[Xen-devel] SLUB allocation error on 3.0.3 / 4.1.1

[Nathan March <nathan@xxxxxx>]

Hi All,

Running into temporary pauses in our VMs which correspond to these 
errors in dmesg on the dom0:
[1721485.352560] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[1721485.352563]   cache: kmalloc-2048, object size: 2048, buffer size: 
2048, default order: 3, min order: 0
[1721485.352566]   node 0: slabs: 81, objs: 1296, free: 0
[1721485.352576] swapper: page allocation failure: order:0, mode:0x4020
[1721485.352579] Pid: 0, comm: swapper Not tainted 3.0.3 #1
[1721485.352582] Call Trace:
[1721485.352584] <IRQ>  [<ffffffff810bea48>] warn_alloc_failed+0x12b/0x142
[1721485.352595]  [<ffffffff810be8f2>] ? get_page_from_freelist+0x51c/0x547
[1721485.352601]  [<ffffffff810068a5>] ? xen_force_evtchn_callback+0xd/0xf
[1721485.352605]  [<ffffffff810bf30d>] __alloc_pages_nodemask+0x606/0x67d
[1721485.352610]  [<ffffffff81006eef>] ? xen_restore_fl_direct_reloc+0x4/0x4
[1721485.352614]  [<ffffffff810ed11e>] new_slab+0x7e/0x1f6
[1721485.352617]  [<ffffffff810ed430>] __slab_alloc+0x19a/0x33c
[1721485.352623]  [<ffffffff815b1655>] ? __netdev_alloc_skb+0x1d/0x3c
[1721485.352627]  [<ffffffff810ed84e>] __kmalloc_track_caller+0x106/0x145
[1721485.352631]  [<ffffffff815b1655>] ? __netdev_alloc_skb+0x1d/0x3c
[1721485.352634]  [<ffffffff815b066d>] __alloc_skb+0x69/0x129
[1721485.352638]  [<ffffffff815b1655>] __netdev_alloc_skb+0x1d/0x3c
[1721485.352643]  [<ffffffff81469c03>] e1000_alloc_rx_buffers+0x7f/0x14c
[1721485.352647]  [<ffffffff81469f84>] e1000_clean_rx_irq+0x265/0x28c
[1721485.352651]  [<ffffffff810068a5>] ? xen_force_evtchn_callback+0xd/0xf
[1721485.352655]  [<ffffffff8146b44a>] e1000_clean+0x75/0x24e
[1721485.352658]  [<ffffffff81006eef>] ? xen_restore_fl_direct_reloc+0x4/0x4
[1721485.352663]  [<ffffffff815b888e>] net_rx_action+0xdd/0x20f
[1721485.352668]  [<ffffffff810492d7>] __do_softirq+0xd3/0x1bb
[1721485.352673]  [<ffffffff81094be4>] ? handle_edge_irq+0x9d/0xbc
[1721485.352678]  [<ffffffff81731b1c>] call_softirq+0x1c/0x30
[1721485.352682]  [<ffffffff8100bd89>] do_softirq+0x61/0xbf
[1721485.352686]  [<ffffffff81049082>] irq_exit+0x43/0xb2
[1721485.352691]  [<ffffffff813712b1>] xen_evtchn_do_upcall+0x2f/0x3c
[1721485.352695]  [<ffffffff81731b6e>] xen_do_hypervisor_callback+0x1e/0x30
[1721485.352697] <EOI>  [<ffffffff810013aa>] ? hypercall_page+0x3aa/0x1000
[1721485.352705]  [<ffffffff810013aa>] ? hypercall_page+0x3aa/0x1000
[1721485.352709]  [<ffffffff8100693c>] ? xen_safe_halt+0x10/0x1a
[1721485.352714]  [<ffffffff81010fc3>] ? default_idle+0x5e/0xa6
[1721485.352718]  [<ffffffff81009f81>] ? cpu_idle+0x6d/0xa3
[1721485.352723]  [<ffffffff81701664>] ? rest_init+0x68/0x6a
[1721485.352729]  [<ffffffff81cb2d18>] ? start_kernel+0x412/0x41d
[1721485.352733]  [<ffffffff81cb22cb>] ? x86_64_start_reservations+0xb6/0xba
[1721485.352737]  [<ffffffff81cb5f55>] ? xen_start_kernel+0x59b/0x5a2
[1721485.352739] Mem-Info:
[1721485.352741] DMA per-cpu:
[1721485.352744] CPU    0: hi:    0, btch:   1 usd:   0
[1721485.352746] DMA32 per-cpu:
[1721485.352748] CPU    0: hi:  186, btch:  31 usd: 176
[1721485.352750] Normal per-cpu:
[1721485.352752] CPU    0: hi:  186, btch:  31 usd:   0
[1721485.352757] active_anon:2403 inactive_anon:13164 isolated_anon:0
[1721485.352758]  active_file:66256 inactive_file:75740 isolated_file:0
[1721485.352759]  unevictable:507 dirty:3175 writeback:40742 unstable:0
[1721485.352760]  free:13180 slab_reclaimable:5805 slab_unreclaimable:9005
[1721485.352761]  mapped:1983 shmem:4 pagetables:1147 bounce:0
[1721485.352768] DMA free:15904kB min:16kB low:20kB high:24kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15680kB 
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB 
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[1721485.352774] lowmem_reserve[]: 0 952 10042 10042
[1721485.352784] DMA32 free:36816kB min:1212kB low:1512kB high:1816kB 
active_anon:9612kB inactive_anon:52656kB active_file:265024kB 
inactive_file:302960kB unevictable:2028kB isolated(anon):0kB 
isolated(file):0kB present:975072kB mlocked:2028kB dirty:12700kB 
writeback:162968kB mapped:7932kB shmem:16kB slab_reclaimable:23220kB 
slab_unreclaimable:36020kB kernel_stack:2360kB pagetables:4588kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? no
[1721485.352790] lowmem_reserve[]: 0 0 9090 9090
[1721485.352799] Normal free:0kB min:11600kB low:14500kB high:17400kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9308160kB 
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB 
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[1721485.352805] lowmem_reserve[]: 0 0 0 0
[1721485.352810] DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 
0*512kB 1*1024kB 1*2048kB 3*4096kB = 15904kB
[1721485.352822] DMA32: 588*4kB 256*8kB 150*16kB 168*32kB 285*64kB 
34*128kB 6*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 36816kB
[1721485.352834] Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[1721485.352845] 153277 total pagecache pages
[1721485.352848] 10888 pages in swap cache
[1721485.352850] Swap cache stats: add 3069760, delete 3058872, find 
3791763/4300237
[1721485.352852] Free swap  = 1816844kB
[1721485.352854] Total swap = 1943856kB
[1721485.377309] 2621424 pages RAM
[1721485.377312] 2427446 pages reserved
[1721485.377313] 108302 pages shared
[1721485.377315] 126166 pages non-shared
[1721485.377319] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[1721485.377323]   cache: kmalloc-2048, object size: 2048, buffer size: 
2048, default order: 3, min order: 0
[1721485.377326]   node 0: slabs: 81, objs: 1296, free: 0
[1721485.377560] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[1721485.377564]   cache: kmalloc-2048, object size: 2048, buffer size: 
2048, default order: 3, min order: 0
[1721485.377567]   node 0: slabs: 81, objs: 1296, free: 0

xen7 ~ # xm info
host                   : xen7
release                : 3.0.3
version                : #1 SMP Mon Aug 22 14:25:38 PDT 2011
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 36580
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Mon Aug 22 11:28:50 PDT 2011
xend_config_format     : 4

Seeing this on multiple dom0's which are all running identical hardware 
(Supermicro X8DTT w/ Intel 82574L gige). Dom0's are limited to 1gb 
(dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true) although they 
don't go above 250mb used.
Not sure if this is a xen bug, network driver issue or something else?

- Nathan

--
Nathan March<nathan@xxxxxx>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel