[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Linux 3.18 and Linux 4.1 reproducible OOM crashes



osstest service owner writes ("[linux-3.18 test] 97278: regressions - FAIL"):
> flight 97278 linux-3.18 real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/97278/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
...
>  test-amd64-amd64-xl           6 xen-boot    fail REGR. vs. 96188

Loads of these.  It seems it can't boot at all.  Seems to affect amd64
dom0 kernels but not i386 or armhf.  (See flight 97279 for 4.1
results.)

It seems that the oom killer is tripping.  Obviously something is
seriously wrong.

It seems likely that this is a real bug, but whether it's in Xen or
Linux is not clear right now.  If it's in Linux it's because a similar
patch has been backported to both.

Below is serial log extract (from the job mentioned above).

Ian.


Jul 14 00:27:33.715871 [   22.813518] rc.local invoked oom-killer: 
gfp_mask=0x84d0, order=0, oom_score_adj=0

Jul 14 00:27:33.899843 [   22.813541] rc.local cpuset=/ mems_allowed=0

Jul 14 00:27:33.907821 [   22.813550] CPU: 0 PID: 2676 Comm: rc.local Not 
tainted 3.18.37 #1

Jul 14 00:27:33.907861 [   22.813556] Hardware name: Intel Corporation 
SandyBridge Platform/To be filled by O.E.M., BIOS 
S1200BT.86B.02.00.0042.050820141549 05/08/2014

Jul 14 00:27:33.923914 [   22.813565]  0000000000000000 ffff8800024b7968 
ffffffff817dcab5 00000000000084d0

Jul 14 00:27:33.931995 [   22.813573]  0000000000000000 ffff8800024b79c8 
ffffffff8118082f ffff8800024b7988

Jul 14 00:27:33.940021 [   22.813582]  ffffffff8114c315 ffffffff817e5ee9 
0000000000000001 ffff8800024b79c8

Jul 14 00:27:33.948034 [   22.813590] Call Trace:

Jul 14 00:27:33.948065 [   22.813600]  [<ffffffff817dcab5>] dump_stack+0x7c/0x98

Jul 14 00:27:33.955838 [   22.813609]  [<ffffffff8118082f>] 
dump_header.isra.11+0x8f/0x1e0

Jul 14 00:27:33.963831 [   22.813616]  [<ffffffff8114c315>] ? 
__delayacct_freepages_end+0x45/0x50

Jul 14 00:27:33.963872 [   22.813677]  [<ffffffff817e5ee9>] ? 
_raw_spin_unlock_irqrestore+0x29/0x90

Jul 14 00:27:33.971868 [   22.813685]  [<ffffffff8131f93e>] ? 
___ratelimit+0xae/0x160

Jul 14 00:27:33.979838 [   22.813691]  [<ffffffff81180ddc>] 
oom_kill_process+0x20c/0x370

Jul 14 00:27:33.987847 [   22.813699]  [<ffffffff810bbc69>] ? 
has_capability_noaudit+0x19/0x20

Jul 14 00:27:33.987888 [   22.813706]  [<ffffffff811812d1>] 
out_of_memory+0x211/0x330

Jul 14 00:27:33.995847 [   22.813714]  [<ffffffff81186331>] 
__alloc_pages_nodemask+0xb11/0xb50

Jul 14 00:27:34.003844 [   22.813721]  [<ffffffff81186382>] 
__get_free_pages+0x12/0x70

Jul 14 00:27:34.011845 [   22.813728]  [<ffffffff811863f1>] 
get_zeroed_page+0x11/0x20

Jul 14 00:27:34.011882 [   22.813735]  [<ffffffff811a9c31>] 
__pud_alloc+0x21/0x120

Jul 14 00:27:34.019841 [   22.813742]  [<ffffffff811ab312>] 
handle_mm_fault+0x352/0xd50

Jul 14 00:27:34.027863 [   22.813749]  [<ffffffff811a550f>] ? 
follow_page_mask+0x2f/0x4f0

Jul 14 00:27:34.035848 [   22.813756]  [<ffffffff811af392>] ? find_vma+0x62/0x70

Jul 14 00:27:34.035885 [   22.813763]  [<ffffffff811a5b48>] 
__get_user_pages+0x178/0x660

Jul 14 00:27:34.043839 [   22.813770]  [<ffffffff811a607d>] 
get_user_pages+0x4d/0x50

Jul 14 00:27:34.051847 [   22.813776]  [<ffffffff811db546>] 
copy_strings.isra.26+0x176/0x310

Jul 14 00:27:34.051886 [   22.813782]  [<ffffffff811db71a>] 
copy_strings_kernel+0x3a/0x60

Jul 14 00:27:34.059844 [   22.813789]  [<ffffffff811dcad8>] 
do_execve_common.isra.33+0x418/0x680

Jul 14 00:27:34.067846 [   22.813795]  [<ffffffff811dd014>] SyS_execve+0x24/0x30

Jul 14 00:27:34.075839 [   22.813801]  [<ffffffff817e6b19>] 
stub_execve+0x69/0xa0

Jul 14 00:27:34.075876 [   22.813807] Mem-Info:

Jul 14 00:27:34.083841 [   22.813810] DMA per-cpu:

Jul 14 00:27:34.083872 [   22.813813] CPU    0: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.083905 [   22.813817] CPU    1: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.091848 [   22.813821] CPU    2: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.099849 [   22.813825] CPU    3: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.099885 [   22.813829] DMA32 per-cpu:

Jul 14 00:27:34.107837 [   22.813833] CPU    0: hi:  186, btch:  31 usd:  88

Jul 14 00:27:34.107872 [   22.813837] CPU    1: hi:  186, btch:  31 usd:  72

Jul 14 00:27:34.115855 [   22.813841] CPU    2: hi:  186, btch:  31 usd:  47

Jul 14 00:27:34.115892 [   22.813845] CPU    3: hi:  186, btch:  31 usd: 171

Jul 14 00:27:34.123844 [   22.813852] active_anon:0 inactive_anon:0 
isolated_anon:0

Jul 14 00:27:34.131837 [   22.813852]  active_file:0 inactive_file:0 
isolated_file:0

Jul 14 00:27:34.131873 [   22.813852]  unevictable:69 dirty:9 writeback:0 
unstable:0

Jul 14 00:27:34.139847 [   22.813852]  free:1036 slab_reclaimable:1887 
slab_unreclaimable:4091

Jul 14 00:27:34.147839 [   22.813852]  mapped:4099 shmem:154 pagetables:706 
bounce:0

Jul 14 00:27:34.155836 [   22.813852]  free_cma:0

Jul 14 00:27:34.155867 [   22.813875] DMA free:1680kB min:96kB low:120kB 
high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:4kB isolated(anon):0kB isolated(file):0kB present:15976kB 
managed:15892kB mlocked:1048kB dirty:0kB writeback:0kB mapped:408kB shmem:8kB 
slab_reclaimable:92kB slab_unreclaimable:328kB kernel_stack:64kB 
pagetables:148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:34.195860 [   22.813896] lowmem_reserve[]: 0 396 396 396

Jul 14 00:27:34.195898 [   22.813908] DMA32 free:2464kB min:2496kB low:3120kB 
high:3744kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:272kB isolated(anon):0kB isolated(file):0kB present:508308kB 
managed:408952kB mlocked:32180kB dirty:36kB writeback:0kB mapped:15988kB 
shmem:608kB slab_reclaimable:7456kB slab_unreclaimable:16036kB 
kernel_stack:2720kB pagetables:2676kB unstable:0kB bounce:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:34.243847 [   22.813929] lowmem_reserve[]: 0 0 0 0

Jul 14 00:27:34.243882 [   22.813936] DMA: 0*4kB 0*8kB 1*16kB (R) 0*32kB 0*64kB 
1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1680kB

Jul 14 00:27:34.251860 [   22.813956] DMA32: 0*4kB 0*8kB 1*16kB (R) 1*32kB (R) 
0*64kB 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 2480kB

Jul 14 00:27:34.267804 [   22.813976] 15037 total pagecache pages

Jul 14 00:27:34.267825 [   22.813980] 0 pages in swap cache

Jul 14 00:27:34.275851 [   22.813984] Swap cache stats: add 0, delete 0, find 
0/0

Jul 14 00:27:34.283845 [   22.813988] Free swap  = 1949692kB

Jul 14 00:27:34.283879 [   22.813991] Total swap = 1949692kB

Jul 14 00:27:34.283908 [   22.813994] 131071 pages RAM

Jul 14 00:27:34.291912 [   22.813997] 0 pages HighMem/MovableOnly

Jul 14 00:27:34.291946 [   22.814000] 24860 pages reserved

Jul 14 00:27:34.299797 [   22.814003] [ pid ]   uid  tgid total_vm      rss 
nr_ptes swapents oom_score_adj name

Jul 14 00:27:34.307798 [   22.814013] [ 1316]     0  1316     9825      819     
 21        0         -1000 udevd

Jul 14 00:27:34.315792 [   22.814021] [ 2038]     0  2038     1084       25     
  7        0             0 ntpdate

Jul 14 00:27:34.323800 [   22.814027] [ 2040]     0  2040     1569      172     
  9        0             0 lockfile-create

Jul 14 00:27:34.331829 [   22.814034] [ 2047]     0  2047     1060      331     
  7        0             0 startpar

Jul 14 00:27:34.339797 [   22.814040] [ 2061]     0  2061     9270      668     
 24        0             0 rpcbind

Jul 14 00:27:34.347796 [   22.814047] [ 2076]   106  2076     9320      700     
 22        0             0 rpc.statd

Jul 14 00:27:34.355808 [   22.814054] [ 2088]     0  2088     6892       58     
 18        0             0 rpc.idmapd

Jul 14 00:27:34.363800 [   22.814060] [ 2131]     0  2131     1084      410     
  7        0             0 rc

Jul 14 00:27:34.371807 [   22.814066] [ 2137]     0  2137     3202      648     
 13        0             0 startpar

Jul 14 00:27:34.379805 [   22.814073] [ 2160]     0  2160    65721      788     
 33        0             0 rsyslogd

Jul 14 00:27:34.387805 [   22.814079] [ 2175]     0  2175     1064      394     
  8        0             0 acpid

Jul 14 00:27:34.395808 [   22.814085] [ 2211]     0  2211    19431     1167     
 41        0             0 apache2

Jul 14 00:27:34.403799 [   22.814091] [ 2214]    33  2214    91722     1032     
 69        0             0 apache2

Jul 14 00:27:34.411802 [   22.814098] [ 2216]    33  2216    91722     1031     
 69        0             0 apache2

Jul 14 00:27:34.419800 [   22.814104] [ 2422]     0  2422     4756       42     
 13        0             0 atd

Jul 14 00:27:34.427806 [   22.814110] [ 2423]     0  2423     6869      590     
 18        0             0 cron

Jul 14 00:27:34.435804 [   22.814117] [ 2606]     0  2606     8780      883     
 21        0             0 oxenstored

Jul 14 00:27:34.443846 [   22.814123] [ 2619]   105  2619    10560      551     
 24        0             0 dbus-daemon

Jul 14 00:27:34.451802 [   22.814130] [ 2624]     0  2624    13796      700     
 28        0         -1000 sshd

Jul 14 00:27:34.459799 [   22.814136] [ 2631]     0  2631    22682       57     
 19        0             0 xenconsoled

Jul 14 00:27:34.467815 [   22.814143] [ 2643]   104  2643    13313      828     
 26        0             0 exim4

Jul 14 00:27:34.475801 [   22.814149] [ 2652]     0  2652    60164     1974     
 78        0             0 qemu-system-i38

Jul 14 00:27:34.483868 [   22.814156] [ 2656]     0  2656    20648     1510     
 42        0             0 sshd

Jul 14 00:27:34.491859 [   22.814162] [ 2660]   107  2660     8345     1093     
 21        0             0 ntpd

Jul 14 00:27:34.499864 [   22.814169] [ 2673]     0  2673     1084      411     
  7        0             0 rc.local

Jul 14 00:27:34.507852 [   22.814175] [ 2676]     0  2676     1084       44     
  6        0             0 rc.local

Jul 14 00:27:34.515859 [   22.814181] [ 2677]  1000  2677    20648      946     
 41        0             0 sshd

Jul 14 00:27:34.523849 [   22.814188] Out of memory: Kill process 2652 
(qemu-system-i38) score 3 or sacrifice child

Jul 14 00:27:34.531862 [   22.814194] Killed process 2652 (qemu-system-i38) 
total-vm:240656kB, anon-rss:2440kB, file-rss:5456kB

Jul 14 00:27:34.539871 [   22.822755] sshd invoked oom-killer: 
gfp_mask=0x2000d0, order=0, oom_score_adj=0

Jul 14 00:27:34.547800 [   22.822760] sshd cpuset=/ mems_allowed=0

Jul 14 00:27:34.555868 [   22.822764] CPU: 1 PID: 2677 Comm: sshd Not tainted 
3.18.37 #1

Jul 14 00:27:34.563847 [   22.822767] Hardware name: Intel Corporation 
SandyBridge Platform/To be filled by O.E.M., BIOS 
S1200BT.86B.02.00.0042.050820141549 05/08/2014

Jul 14 00:27:34.571852 [   22.822771]  0000000000000000 ffff8800024fb990 
ffffffff817dcab5 00000000002000d0

Jul 14 00:27:34.579860 [   22.822775]  0000000000000000 ffff8800024fb9f0 
ffffffff8118082f ffff8800024fb9b0

Jul 14 00:27:34.587803 [   22.822779]  ffffffff8114c315 ffffffff817e5ee9 
0000000000000001 ffff8800024fb9f0

Jul 14 00:27:34.595862 [   22.822783] Call Trace:

Jul 14 00:27:34.595892 [   22.822787]  [<ffffffff817dcab5>] dump_stack+0x7c/0x98

Jul 14 00:27:34.603859 [   22.822791]  [<ffffffff8118082f>] 
dump_header.isra.11+0x8f/0x1e0

Jul 14 00:27:34.611843 [   22.822794]  [<ffffffff8114c315>] ? 
__delayacct_freepages_end+0x45/0x50

Jul 14 00:27:34.619891 [   22.822798]  [<ffffffff817e5ee9>] ? 
_raw_spin_unlock_irqrestore+0x29/0x90

Jul 14 00:27:34.619933 [   22.822801]  [<ffffffff8131f93e>] ? 
___ratelimit+0xae/0x160

Jul 14 00:27:34.627853 [   22.822804]  [<ffffffff81180ddc>] 
oom_kill_process+0x20c/0x370

Jul 14 00:27:34.635900 [   22.822808]  [<ffffffff810bbc69>] ? 
has_capability_noaudit+0x19/0x20

Jul 14 00:27:34.643851 [   22.822812]  [<ffffffff811812d1>] 
out_of_memory+0x211/0x330

Jul 14 00:27:34.643888 [   22.822815]  [<ffffffff81186331>] 
__alloc_pages_nodemask+0xb11/0xb50

Jul 14 00:27:34.651845 [   22.822820]  [<ffffffff811cb40c>] new_slab+0x2bc/0x320

Jul 14 00:27:34.660009 [   22.822823]  [<ffffffff811cc770>] 
__slab_alloc.constprop.69+0x440/0x570

Jul 14 00:27:34.667913 [   22.822828]  [<ffffffff81054f3e>] ? 
xen_leave_lazy_mmu+0xe/0x10

Jul 14 00:27:34.667952 [   22.822831]  [<ffffffff811b6721>] ? 
anon_vma_fork+0x71/0x140

Jul 14 00:27:34.675855 [   22.822835]  [<ffffffff811b651b>] ? 
anon_vma_clone+0x6b/0x200

Jul 14 00:27:34.683863 [   22.822838]  [<ffffffff811b6721>] ? 
anon_vma_fork+0x71/0x140

Jul 14 00:27:34.691849 [   22.822841]  [<ffffffff811cdad6>] 
kmem_cache_alloc+0x116/0x150

Jul 14 00:27:34.691887 [   22.822844]  [<ffffffff811b6721>] 
anon_vma_fork+0x71/0x140

Jul 14 00:27:34.699848 [   22.822848]  [<ffffffff810b2017>] 
copy_process.part.45+0x1687/0x1a50

Jul 14 00:27:34.707853 [   22.822852]  [<ffffffff810b2577>] do_fork+0xb7/0x3b0

Jul 14 00:27:34.707888 [   22.822856]  [<ffffffff811f3a34>] ? 
get_unused_fd_flags+0x34/0x40

Jul 14 00:27:34.715857 [   22.822860]  [<ffffffff811f3a88>] ? 
__fd_install+0x48/0x60

Jul 14 00:27:34.723872 [   22.822862]  [<ffffffff810b28f1>] SyS_clone+0x11/0x20

Jul 14 00:27:34.723910 [   22.822866]  [<ffffffff817e68c9>] stub_clone+0x69/0x90

Jul 14 00:27:34.731848 [   22.822868]  [<ffffffff817e6589>] ? 
system_call_fastpath+0x12/0x17

Jul 14 00:27:34.739823 [   22.822871] Mem-Info:

Jul 14 00:27:34.739841 [   22.822872] DMA per-cpu:

Jul 14 00:27:34.747843 [   22.822874] CPU    0: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.747879 [   22.822876] CPU    1: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.755836 [   22.822878] CPU    2: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.755870 [   22.822880] CPU    3: hi:    0, btch:   1 usd:   0

Jul 14 00:27:34.763850 [   22.822882] DMA32 per-cpu:

Jul 14 00:27:34.763881 [   22.822884] CPU    0: hi:  186, btch:  31 usd:  88

Jul 14 00:27:34.771838 [   22.822886] CPU    1: hi:  186, btch:  31 usd: 157

Jul 14 00:27:34.779837 [   22.822888] CPU    2: hi:  186, btch:  31 usd:  47

Jul 14 00:27:34.779872 [   22.822890] CPU    3: hi:  186, btch:  31 usd: 173

Jul 14 00:27:34.787884 [   22.822893] active_anon:0 inactive_anon:0 
isolated_anon:0

Jul 14 00:27:34.787920 [   22.822893]  active_file:0 inactive_file:0 
isolated_file:0

Jul 14 00:27:34.795844 [   22.822893]  unevictable:69 dirty:9 writeback:0 
unstable:0

Jul 14 00:27:34.803834 [   22.822893]  free:1036 slab_reclaimable:1887 
slab_unreclaimable:4091

Jul 14 00:27:34.811846 [   22.822893]  mapped:3101 shmem:154 pagetables:650 
bounce:0

Jul 14 00:27:34.811884 [   22.822893]  free_cma:0

Jul 14 00:27:34.819842 [   22.822904] DMA free:1680kB min:96kB low:120kB 
high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:4kB isolated(anon):0kB isolated(file):0kB present:15976kB 
managed:15892kB mlocked:1048kB dirty:0kB writeback:0kB mapped:336kB shmem:8kB 
slab_reclaimable:92kB slab_unreclaimable:328kB kernel_stack:64kB 
pagetables:148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:34.859831 [   22.822914] lowmem_reserve[]: 0 396 396 396

Jul 14 00:27:34.859866 [   22.822920] DMA32 free:2464kB min:2496kB low:3120kB 
high:3744kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:272kB isolated(anon):0kB isolated(file):0kB present:508308kB 
managed:408952kB mlocked:32180kB dirty:36kB writeback:0kB mapped:12068kB 
shmem:608kB slab_reclaimable:7456kB slab_unreclaimable:16036kB 
kernel_stack:2720kB pagetables:2452kB unstable:0kB bounce:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:34.899849 [   22.822930] lowmem_reserve[]: 0 0 0 0

Jul 14 00:27:34.907838 [   22.822934] DMA: 0*4kB 0*8kB 1*16kB (R) 0*32kB 0*64kB 
1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1680kB

Jul 14 00:27:34.915876 [   22.822943] DMA32: 0*4kB 0*8kB 1*16kB (R) 1*32kB (R) 
0*64kB 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 2480kB

Jul 14 00:27:34.931842 [   22.822953] 15037 total pagecache pages

Jul 14 00:27:34.931878 [   22.822955] 0 pages in swap cache

Jul 14 00:27:34.939850 [   22.822957] Swap cache stats: add 0, delete 0, find 
0/0

Jul 14 00:27:34.939888 [   22.822959] Free swap  = 1949692kB

Jul 14 00:27:34.947842 [   22.822960] Total swap = 1949692kB

Jul 14 00:27:34.947874 [   22.822962] 131071 pages RAM

Jul 14 00:27:34.947903 [   22.822963] 0 pages HighMem/MovableOnly

Jul 14 00:27:34.955868 [   22.822965] 24860 pages reserved

Jul 14 00:27:34.955900 [   22.822966] [ pid ]   uid  tgid total_vm      rss 
nr_ptes swapents oom_score_adj name

Jul 14 00:27:34.963860 [   22.822971] [ 1316]     0  1316     9825      819     
 21        0         -1000 udevd

Jul 14 00:27:34.971874 [   22.822976] [ 2038]     0  2038     1084       25     
  7        0             0 ntpdate

Jul 14 00:27:34.979864 [   22.822978] [ 2040]     0  2040     1569      172     
  9        0             0 lockfile-create

Jul 14 00:27:34.987862 [   22.822981] [ 2047]     0  2047     1060      331     
  7        0             0 startpar

Jul 14 00:27:34.995869 [   22.822984] [ 2061]     0  2061     9270      668     
 24        0             0 rpcbind

Jul 14 00:27:35.003863 [   22.822987] [ 2076]   106  2076     9320      700     
 22        0             0 rpc.statd

Jul 14 00:27:35.019848 [   22.822991] [ 2088]     0  2088     6892       58     
 18        0             0 rpc.idmapd

Jul 14 00:27:35.027853 [   22.822994] [ 2131]     0  2131     1084      410     
  7        0             0 rc

Jul 14 00:27:35.035843 [   22.822997] [ 2137]     0  2137     3202      648     
 13        0             0 startpar

Jul 14 00:27:35.043857 [   22.823000] [ 2160]     0  2160    65721      788     
 33        0             0 rsyslogd

Jul 14 00:27:35.051846 [   22.823003] [ 2175]     0  2175     1064      394     
  8        0             0 acpid

Jul 14 00:27:35.059852 [   22.823006] [ 2211]     0  2211    19431     1167     
 41        0             0 apache2

Jul 14 00:27:35.067848 [   22.823009] [ 2214]    33  2214    91722     1032     
 69        0             0 apache2

Jul 14 00:27:35.075852 [   22.823012] [ 2216]    33  2216    91722     1031     
 69        0             0 apache2

Jul 14 00:27:35.083847 [   22.823015] [ 2422]     0  2422     4756       42     
 13        0             0 atd

Jul 14 00:27:35.091857 [   22.823018] [ 2423]     0  2423     6869      590     
 18        0             0 cron

Jul 14 00:27:35.099857 [   22.823021] [ 2606]     0  2606     8780      883     
 21        0             0 oxenstored

Jul 14 00:27:35.107854 [   22.823024] [ 2619]   105  2619    10560      551     
 24        0             0 dbus-daemon

Jul 14 00:27:35.115866 [   22.823027] [ 2624]     0  2624    13796      700     
 28        0         -1000 sshd

Jul 14 00:27:35.123797 [   22.823030] [ 2631]     0  2631    22682       57     
 19        0             0 xenconsoled

Jul 14 00:27:35.131849 [   22.823034] [ 2643]   104  2643    13313      828     
 26        0             0 exim4

Jul 14 00:27:35.139844 [   22.823037] [ 2656]     0  2656    20648     1510     
 42        0             0 sshd

Jul 14 00:27:35.147846 [   22.823040] [ 2660]   107  2660     8345     1093     
 21        0             0 ntpd

Jul 14 00:27:35.155856 [   22.823043] [ 2673]     0  2673     1084      411     
  7        0             0 rc.local

Jul 14 00:27:35.163847 [   22.823046] [ 2676]     0  2676     1084       44     
  6        0             0 rc.local

Jul 14 00:27:35.171852 [   22.823049] [ 2677]  1000  2677    20648      946     
 41        0             0 sshd

Jul 14 00:27:35.179883 [   22.823052] Out of memory: Kill process 2656 (sshd) 
score 2 or sacrifice child

Jul 14 00:27:35.187860 [   22.823055] Killed process 2677 (sshd) 
total-vm:82592kB, anon-rss:848kB, file-rss:2936kB

Jul 14 00:27:35.195838 [   22.825444] oxenstored invoked oom-killer: 
gfp_mask=0x280da, order=0, oom_score_adj=0

Jul 14 00:27:35.203861 [   22.825449] oxenstored cpuset=/ mems_allowed=0

Jul 14 00:27:35.203897 [   22.825453] CPU: 2 PID: 2606 Comm: oxenstored Not 
tainted 3.18.37 #1

Jul 14 00:27:35.211845 [   22.825457] Hardware name: Intel Corporation 
SandyBridge Platform/To be filled by O.E.M., BIOS 
S1200BT.86B.02.00.0042.050820141549 05/08/2014

Jul 14 00:27:35.227847 [   22.825461]  0000000000000000 ffff880008dc3ad8 
ffffffff817dcab5 00000000000280da

Jul 14 00:27:35.235847 [   22.825466]  0000000000000000 ffff880008dc3b38 
ffffffff8118082f ffff880008dc3af8

Jul 14 00:27:35.243891 [   22.825469]  ffffffff8114c315 ffffffff817e5ee9 
0000000000000001 ffff880008dc3b38

Jul 14 00:27:35.243937 [   22.825473] Call Trace:

Jul 14 00:27:35.251845 [   22.825477]  [<ffffffff817dcab5>] dump_stack+0x7c/0x98

Jul 14 00:27:35.251881 [   22.825481]  [<ffffffff8118082f>] 
dump_header.isra.11+0x8f/0x1e0

Jul 14 00:27:35.259846 [   22.825484]  [<ffffffff8114c315>] ? 
__delayacct_freepages_end+0x45/0x50

Jul 14 00:27:35.267864 [   22.825488]  [<ffffffff817e5ee9>] ? 
_raw_spin_unlock_irqrestore+0x29/0x90

Jul 14 00:27:35.275843 [   22.825492]  [<ffffffff8131f93e>] ? 
___ratelimit+0xae/0x160

Jul 14 00:27:35.283904 [   22.825494]  [<ffffffff81180ddc>] 
oom_kill_process+0x20c/0x370

Jul 14 00:27:35.283942 [   22.825498]  [<ffffffff810bbc69>] ? 
has_capability_noaudit+0x19/0x20

Jul 14 00:27:35.291845 [   22.825501]  [<ffffffff811812d1>] 
out_of_memory+0x211/0x330

Jul 14 00:27:35.299881 [   22.825505]  [<ffffffff81186331>] 
__alloc_pages_nodemask+0xb11/0xb50

Jul 14 00:27:35.307834 [   22.825509]  [<ffffffff811ab988>] 
handle_mm_fault+0x9c8/0xd50

Jul 14 00:27:35.307871 [   22.825513]  [<ffffffff811ea170>] ? 
poll_select_copy_remaining+0x130/0x130

Jul 14 00:27:35.315942 [   22.825517]  [<ffffffff810a68dd>] 
__do_page_fault+0x1bd/0x590

Jul 14 00:27:35.323846 [   22.825521]  [<ffffffff[   23.782240] getty invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0

Jul 14 00:27:35.331856 [   23.782247] getty cpuset=/ mems_allowed=0

Jul 14 00:27:35.339842 [   23.782252] CPU: 3 PID: 2685 Comm: getty Not tainted 
3.18.37 #1

Jul 14 00:27:35.339884 [   23.782255] Hardware name: Intel Corporation 
SandyBridge Platform/To be filled by O.E.M., BIOS 
S1200BT.86B.02.00.0042.050820141549 05/08/2014

Jul 14 00:27:35.355849 [   23.782259]  0000000000000000 ffff8800158db7f8 
ffffffff817dcab5 00000000000280da

Jul 14 00:27:35.363925 [   23.782263]  0000000000000000 ffff8800158db858 
ffffffff8118082f ffff8800158db818

Jul 14 00:27:35.371864 [   23.782268]  ffffffff8114c315 ffffffff817e5ee9 
0000000000000001 ffff8800158db858

Jul 14 00:27:35.379855 [   23.782272] Call Trace:

Jul 14 00:27:35.379886 [   23.782278]  [<ffffffff817dcab5>] dump_stack+0x7c/0x98

Jul 14 00:27:35.387850 [   23.782283]  [<ffffffff8118082f>] 
dump_header.isra.11+0x8f/0x1e0

Jul 14 00:27:35.387888 [   23.782287]  [<ffffffff8114c315>] ? 
__delayacct_freepages_end+0x45/0x50

Jul 14 00:27:35.395877 [   23.782291]  [<ffffffff817e5ee9>] ? 
_raw_spin_unlock_irqrestore+0x29/0x90

Jul 14 00:27:35.403858 [   23.782295]  [<ffffffff8131f93e>] ? 
___ratelimit+0xae/0x160

Jul 14 00:27:35.411839 [   23.782298]  [<ffffffff81180ddc>] 
oom_kill_process+0x20c/0x370

Jul 14 00:27:35.419845 [   23.782301]  [<ffffffff811812d1>] 
out_of_memory+0x211/0x330

Jul 14 00:27:35.419883 [   23.782305]  [<ffffffff81186331>] 
__alloc_pages_nodemask+0xb11/0xb50

Jul 14 00:27:35.427868 [   23.782309]  [<ffffffff811ab988>] 
handle_mm_fault+0x9c8/0xd50

Jul 14 00:27:35.435856 [   23.782320]  [<ffffffffa009d912>] ? 
search_dir+0xc2/0x120 [ext4]

Jul 14 00:27:35.443840 [   23.782324]  [<ffffffff810a68dd>] 
__do_page_fault+0x1bd/0x590

Jul 14 00:27:35.443877 [   23.782329]  [<ffffffff81233736>] ? 
dquot_file_open+0x16/0x50

Jul 14 00:27:35.451860 [   23.782334]  [<ffffffffa008c953>] ? 
ext4_file_open+0x43/0x180 [ext4]

Jul 14 00:27:35.459849 [   23.782339]  [<ffffffffa008c910>] ? 
ext4_check_all_de+0xb0/0xb0 [ext4]

Jul 14 00:27:35.467884 [   23.782343]  [<ffffffff810a6cdc>] 
do_page_fault+0x2c/0x40

Jul 14 00:27:35.467920 [   23.782346]  [<ffffffff817e81b8>] page_fault+0x28/0x30

Jul 14 00:27:35.475841 [   23.782350]  [<ffffffff811a47a8>] ? 
copy_page_to_iter+0x68/0x3c0

Jul 14 00:27:35.483845 [   23.782353]  [<ffffffff8117ed88>] 
generic_file_read_iter+0x278/0x5d0

Jul 14 00:27:35.491842 [   23.782357]  [<ffffffff811d4ad9>] 
new_sync_read+0x79/0xb0

Jul 14 00:27:35.491880 [   23.782360]  [<ffffffff811d5303>] vfs_read+0x93/0x180

Jul 14 00:27:35.499846 [   23.782364]  [<ffffffff811d5e61>] SyS_read+0x41/0xb0

Jul 14 00:27:35.507832 [   23.782367]  [<ffffffff817e6589>] 
system_call_fastpath+0x12/0x17

Jul 14 00:27:35.507869 [   23.782370] Mem-Info:

Jul 14 00:27:35.515838 [   23.782371] DMA per-cpu:

Jul 14 00:27:35.515869 [   23.782373] CPU    0: hi:    0, btch:   1 usd:   0

Jul 14 00:27:35.523838 [   23.782375] CPU    1: hi:    0, btch:   1 usd:   0

Jul 14 00:27:35.523873 [   23.782377] CPU    2: hi:    0, btch:   1 usd:   0

Jul 14 00:27:35.531843 [   23.782379] CPU    3: hi:    0, btch:   1 usd:   0

Jul 14 00:27:35.531878 [   23.782381] DMA32 per-cpu:

Jul 14 00:27:35.539836 [   23.782382] CPU    0: hi:  186, btch:  31 usd:   0

Jul 14 00:27:35.539873 [   23.782384] CPU    1: hi:  186, btch:  31 usd:   0

Jul 14 00:27:35.547841 [   23.782386] CPU    2: hi:  186, btch:  31 usd:   0

Jul 14 00:27:35.555833 [   23.782388] CPU    3: hi:  186, btch:  31 usd:   0

Jul 14 00:27:35.555868 [   23.782392] active_anon:0 inactive_anon:0 
isolated_anon:0

Jul 14 00:27:35.563857 [   23.782392]  active_file:0 inactive_file:0 
isolated_file:0

Jul 14 00:27:35.563895 [   23.782392]  unevictable:1 dirty:0 writeback:18 
unstable:0

Jul 14 00:27:35.571859 [   23.782392]  free:1060 slab_reclaimable:1888 
slab_unreclaimable:4074

Jul 14 00:27:35.579860 [   23.782392]  mapped:2923 shmem:153 pagetables:588 
bounce:0

Jul 14 00:27:35.587840 [   23.782392]  free_cma:0

Jul 14 00:27:35.587872 [   23.782403] DMA free:1668kB min:96kB low:120kB 
high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB 
managed:15892kB mlocked:1044kB dirty:0kB writeback:4kB mapped:340kB shmem:8kB 
slab_reclaimable:92kB slab_unreclaimable:332kB kernel_stack:48kB 
pagetables:116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:35.627846 [   23.782413] lowmem_reserve[]: 0 396 396 396

Jul 14 00:27:35.627882 [   23.782419] DMA32 free:2572kB min:2496kB low:3120kB 
high:3744kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:4kB isolated(anon):0kB isolated(file):0kB present:508308kB 
managed:408952kB mlocked:31912kB dirty:0kB writeback:68kB mapped:11352kB 
shmem:604kB slab_reclaimable:7460kB slab_unreclaimable:15964kB 
kernel_stack:2400kB pagetables:2236kB unstable:0kB bounce:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:36 all_unreclaimable? yes

Jul 14 00:27:35.675846 [   23.782431] lowmem_reserve[]: 0 0 0 0

Jul 14 00:27:35.675882 [   23.782433] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 
1*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1664kB

Jul 14 00:27:35.683866 [   23.782442] DMA32: 29*4kB (UR) 0*8kB 3*16kB (R) 
0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 1*2048kB (R) 
0*4096kB = 2660kB

Jul 14 00:27:35.699851 [   23.782452] 15076 total pagecache pages

Jul 14 00:27:35.699885 [   23.782455] 0 pages in swap cache

Jul 14 00:27:35.707849 [   23.782465] Swap cache stats: add 0, delete 0, find 
0/0

Jul 14 00:27:35.715836 [   23.782468] Free swap  = 1949692kB

Jul 14 00:27:35.715869 [   23.782472] Total swap = 1949692kB

Jul 14 00:27:35.715899 [   23.782476] 131071 pages RAM

Jul 14 00:27:35.723847 [   23.782480] 0 pages HighMem/MovableOnly

Jul 14 00:27:35.723882 [   23.782484] 24860 pages reserved

Jul 14 00:27:35.731838 [   23.782487] [ pid ]   uid  tgid total_vm      rss 
nr_ptes swapents oom_score_adj name

Jul 14 00:27:35.739850 [   23.782495] [ 1316]     0  1316     9825      819     
 21        0         -1000 udevd

Jul 14 00:27:35.747851 [   23.782501] [ 2038]     0  2038     1084       25     
  7        0             0 ntpdate

Jul 14 00:27:35.755844 [   23.782506] [ 2040]     0  2040     1569      172     
  9        0             0 lockfile-create

Jul 14 00:27:35.763856 [   23.782509] [ 2047]     0  2047     1060      331     
  7        0             0 startpar

Jul 14 00:27:35.771890 [   23.782512] [ 2061]     0  2061     9270      668     
 24        0             0 rpcbind

Jul 14 00:27:35.779829 [   23.782516] [ 2076]   106  2076     9320      700     
 22        0             0 rpc.statd

Jul 14 00:27:35.787856 [   23.782519] [ 2088]     0  2088     6892       58     
 18        0             0 rpc.idmapd

Jul 14 00:27:35.795799 [   23.782522] [ 2160]     0  2160    65721      785     
 33        0             0 rsyslogd

Jul 14 00:27:35.803857 [   23.782525] [ 2175]     0  2175     1064      394     
  8        0             0 acpid

Jul 14 00:27:35.811854 [   23.782528] [ 2211]     0  2211    19431     1167     
 41        0             0 apache2

Jul 14 00:27:35.820250 [   23.782532] [ 2216]    33  2216    91722     1031     
 69        0             0 apache2

Jul 14 00:27:35.827884 [   23.782534] [ 2422]     0  2422     4756       42     
 13        0             0 atd

Jul 14 00:27:35.835897 [   23.782537] [ 2423]     0  2423     6869      590     
 18        0             0 cron

Jul 14 00:27:35.843893 [   23.782541] [ 2606]     0  2606     8780      883     
 21        0             0 oxenstored

Jul 14 00:27:35.851883 [   23.782544] [ 2619]   105  2619    10560      551     
 24        0             0 dbus-daemon

Jul 14 00:27:35.859858 [   23.782547] [ 2624]     0  2624    13796      700     
 28        0         -1000 sshd

Jul 14 00:27:35.867866 [   23.782550] [ 2631]     0  2631    22682       57     
 19        0             0 xenconsoled

Jul 14 00:27:35.875902 [   23.782554] [ 2643]   104  2643    13313      828     
 26        0             0 exim4

Jul 14 00:27:35.883887 [   23.782557] [ 2660]   107  2660     8346     1093     
 21        0             0 ntpd

Jul 14 00:27:35.891912 [   23.782560] [ 2681]     0  2681     2650      399     
 10        0             0 getty

Jul 14 00:27:35.899869 [   23.782563] [ 2682]     0  2682     3166      417     
 12        0             0 getty

Jul 14 00:27:35.907853 [   23.782566] [ 2683]     0  2683     1063      173     
  7        0             0 getty

Jul 14 00:27:35.915859 [   23.782569] [ 2684]     0  2684     3166      416     
 12        0             0 getty

Jul 14 00:27:35.923807 [   23.782572] [ 2685]     0  2685     3560      425     
 12        0             0 getty

Jul 14 00:27:35.931919 [   23.782575] [ 2686]     0  2686     3166      425     
 11        0             0 getty

Jul 14 00:27:35.939914 [   23.782578] [ 2687]     0  2687     1591      404     
  8        0             0 getty

Jul 14 00:27:35.947888 [   23.782581] [ 2688]     0  2688     3166      437     
 12        0             0 getty

Jul 14 00:27:35.956047 [   23.782587] [ 2690]     0  2690     3166       35     
  9        0             0 getty

Jul 14 00:27:35.964079 [   23.782592] [ 2691]     0  2691     3166       34     
  9        0             0 getty

Jul 14 00:27:35.972103 [   23.782597] [ 2693]     0  2693     9825      195     
 19        0         -1000 udevd

Jul 14 00:27:35.980089 [   23.782601] [ 2694]     0  2694     9825      195     
 18        0         -1000 udevd

Jul 14 00:27:35.987933 [   23.782604] Out of memory: Kill process 2211 
(apache2) score 1 or sacrifice child

Jul 14 00:27:35.996043 [   23.782607] Killed process 2216 (apache2) 
total-vm:366888kB, anon-rss:1556kB, file-rss:2568kB

Jul 14 00:27:36.004061 [   23.790140] getty invoked oom-killer: 
gfp_mask=0x200da, order=0, oom_score_adj=0

Jul 14 00:27:36.011954 [   23.790145] getty cpuset=/ mems_allowed=0

Jul 14 00:27:36.019911 [   23.790150] CPU: 0 PID: 2681 Comm: getty Not tainted 
3.18.37 #1

Jul 14 00:27:36.019951 [   23.790152] Hardware name: Intel Corporation 
SandyBridge Platform/To be filled by O.E.M., BIOS 
S1200BT.86B.02.00.0042.050820141549 05/08/2014

Jul 14 00:27:36.035934 [   23.790156]  0000000000000000 ffff88001220ba78 
ffffffff817dcab5 00000000000200da

Jul 14 00:27:36.043951 [   23.790161]  0000000000000000 ffff88001220bad8 
ffffffff8118082f ffff88001220ba98

Jul 14 00:27:36.051973 [   23.790166]  ffffffff8114c315 ffffffff817e5ee9 
0000000000000001 ffff88001220bad8

Jul 14 00:27:36.059965 [   23.790170] Call Trace:

Jul 14 00:27:36.059997 [   23.790174]  [<ffffffff817dcab5>] dump_stack+0x7c/0x98

Jul 14 00:27:36.067938 [   23.790178]  [<ffffffff8118082f>] 
dump_header.isra.11+0x8f/0x1e0

Jul 14 00:27:36.067977 [   23.790181]  [<ffffffff8114c315>] ? 
__delayacct_freepages_end+0x45/0x50

Jul 14 00:27:36.076024 [   23.790185]  [<ffffffff817e5ee9>] ? 
_raw_spin_unlock_irqrestore+0x29/0x90

Jul 14 00:27:36.083947 [   23.790188]  [<ffffffff8131f93e>] ? 
___ratelimit+0xae/0x160

Jul 14 00:27:36.092010 [   23.790191]  [<ffffffff81180ddc>] 
oom_kill_process+0x20c/0x370

Jul 14 00:27:36.099947 [   23.790195]  [<ffffffff811812d1>] 
out_of_memory+0x211/0x330

Jul 14 00:27:36.099991 [   23.790198]  [<ffffffff81186331>] 
__alloc_pages_nodemask+0xb11/0xb50

Jul 14 00:27:36.107888 [   23.790202]  [<ffffffff811a9757>] 
do_cow_fault.isra.96+0x47/0x2a0

Jul 14 00:27:36.116087 [   23.790207]  [<ffffffff81054bf9>] ? 
__raw_callee_save_xen_pmd_val+0x11/0x1e

Jul 14 00:27:36.123936 [   23.790210]  [<ffffffff811ab675>] 
handle_mm_fault+0x6b5/0xd50

Jul 14 00:27:36.124004 [   23.790214]  [<ffffffff810a68dd>] 
__do_page_fault+0x1bd/0x590

Jul 14 00:27:36.132061 [   23.790217]  [<ffffffff81198939>] ? 
vm_mmap_pgoff+0x99/0xc0

Jul 14 00:27:36.140090 [   23.790220]  [<ffffffff810a6cdc>] 
do_page_fault+0x2c/0x40

Jul 14 00:27:36.148022 [   23.790224]  [<ffffffff817e81b8>] page_fault+0x28/0x30

Jul 14 00:27:36.148057 [   23.790226] Mem-Info:

Jul 14 00:27:36.148083 [   23.790227] DMA per-cpu:

Jul 14 00:27:36.155974 [   23.790229] CPU    0: hi:    0, btch:   1 usd:   0

Jul 14 00:27:36.155995 [   23.790231] CPU    1: hi:    0, btch:   1 usd:   0

Jul 14 00:27:36.164132 [   23.790233] CPU    2: hi:    0, btch:   1 usd:   0

Jul 14 00:27:36.172145 [   23.790235] CPU    3: hi:    0, btch:   1 usd:   0

Jul 14 00:27:36.172208 [   23.790237] DMA32 per-cpu:

Jul 14 00:27:36.180000 [   23.790238] CPU    0: hi:  186, btch:  31 usd:   0

Jul 14 00:27:36.180049 [   23.790241] CPU    1: hi:  186, btch:  31 usd: 128

Jul 14 00:27:36.187954 [   23.790243] CPU    2: hi:  186, btch:  31 usd:   0

Jul 14 00:27:36.187996 [   23.790245] CPU    3: hi:  186, btch:  31 usd:   0

Jul 14 00:27:36.195965 [   23.790248] active_anon:0 inactive_anon:0 
isolated_anon:0

Jul 14 00:27:36.203964 [   23.790248]  active_file:0 inactive_file:0 
isolated_file:0

Jul 14 00:27:36.204010 [   23.790248]  unevictable:1 dirty:4 writeback:0 
unstable:0

Jul 14 00:27:36.211969 [   23.790248]  free:1012 slab_reclaimable:1888 
slab_unreclaimable:4067

Jul 14 00:27:36.219948 [   23.790248]  mapped:2877 shmem:154 pagetables:521 
bounce:0

Jul 14 00:27:36.227943 [   23.790248]  free_cma:0

Jul 14 00:27:36.227982 [   23.790259] DMA free:1680kB min:96kB low:120kB 
high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB 
managed:15892kB mlocked:1044kB dirty:0kB writeback:0kB mapped:340kB shmem:8kB 
slab_reclaimable:92kB slab_unreclaimable:332kB kernel_stack:48kB 
pagetables:84kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes

Jul 14 00:27:36.267995 [   23.790269] lowmem_reserve[]: 0 396 396 396

Jul 14 00:27:36.268052 [   23.790275] DMA32 free:2368kB min:2496kB low:3120kB 
high:3744kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:4kB isolated(anon):0kB isolated(file):0kB present:508308kB 
managed:408952kB mlocked:319
Jul 14 00:28:18.168603 <client 0x1c9b400 connected - now 1 clients>


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.