[Xen-bugs] [Bug 1097] New: dom0 networking hangs with "page allocation failure"

[bugzilla-daemon@xxxxxxxxxxxxxxxxxxx]

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1097

           Summary: dom0 networking hangs with "page allocation failure"
           Product: Xen
           Version: 3.0.3
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: Unspecified
        AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx
        ReportedBy: andy@xxxxxxxxxxxxxx


Hi, I have an x86 supermicro dual core Xeon-based server with 8G of RAM which
after ~70 days of uptime is suffering from occasional networking hangs.

At the time of the hangs the following is printed to console:

swapper: page allocation failure. order:0, mode:0x20
 [<c014078d>] __alloc_pages+0x261/0x275
 [<c015621a>] cache_alloc_refill+0x297/0x493
 [<c0104a51>] hypervisor_callback+0x3d/0x48
 [<c020007b>] k_shift+0xc0/0xc8
 [<c0155f6a>] kmem_cache_alloc+0x3b/0x54
 [<c022e9f5>] alloc_skb_from_cache+0x48/0x110
 [<c020d768>] __alloc_skb+0x6c/0x70
 [<c0215dbb>] netif_be_start_xmit+0x118/0x3d5
 [<c02326fe>] dev_hard_start_xmit+0x19a/0x1f0
 [<c0234080>] dev_queue_xmit+0x247/0x2e3
 [<ee4b3dfe>] br_dev_queue_push_xmit+0x155/0x178 [bridge]
 [<ee4b3e64>] br_forward_finish+0x43/0x45 [bridge]
 [<ee4b7ae4>] br_nf_forward_finish+0xc6/0xcc [bridge]
 [<ee4b834a>] br_nf_forward_arp+0x116/0x128 [bridge]
 [<c0246e88>] nf_iterate+0x30/0x61
 [<ee4b3e21>] br_forward_finish+0x0/0x45 [bridge]
 [<c0246fae>] nf_hook_slow+0x3a/0x90
 [<ee4b3e21>] br_forward_finish+0x0/0x45 [bridge]
 [<ee4b3eac>] __br_forward+0x46/0x57 [bridge]
 [<ee4b3e21>] br_forward_finish+0x0/0x45 [bridge]
 [<ee4b3c59>] br_flood+0x65/0x9d [bridge]
 [<ee4b3e66>] __br_forward+0x0/0x57 [bridge]
 [<ee4b3c9b>] br_flood_forward+0xa/0xc [bridge]
 [<ee4b3e66>] __br_forward+0x0/0x57 [bridge]
 [<ee4b4868>] br_handle_frame_finish+0x80/0xcf [bridge]
 [<ee4b4a16>] br_handle_frame+0x15f/0x179 [bridge]
 [<c0232291>] netif_receive_skb+0x25e/0x357
 [<ee0d1130>] e1000_clean_rx_irq_ps+0x4a6/0x569 [e1000]
 [<ee0cfc4c>] e1000_clean+0x69/0x136 [e1000]
 [<c0233d40>] net_rx_action+0x96/0x18f
 [<c011f40a>] __do_softirq+0x5e/0xc3
 [<c011f4a9>] do_softirq+0x3a/0x4a
 [<c0106131>] do_IRQ+0x48/0x53
 [<c020c22c>] evtchn_do_upcall+0x64/0x9b
 [<c0104a51>] hypervisor_callback+0x3d/0x48
 [<c010732e>] raw_safe_halt+0x8c/0xaf
 [<c0102c5f>] xen_idle+0x22/0x2e
 [<c0102d7e>] cpu_idle+0x91/0xab
 [<c03236fc>] start_kernel+0x378/0x37f
Mem-info:
DMA per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
DMA32 per-cpu: empty
Normal per-cpu: empty
HighMem per-cpu:
cpu 0 hot: high 90, batch 15 used:82
cpu 0 cold: high 30, batch 7 used:3
Free pages:       36860kB (35660kB HighMem)
Active:172543 inactive:33283 dirty:25 writeback:0 unstable:0 free:9215
slab:24052 mapped:3492 pagetables:290
DMA free:1200kB min:3452kB low:4312kB high:5176kB active:523552kB
inactive:33676kB present:745464kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 304
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 304                               
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 2432
HighMem free:35660kB min:304kB low:664kB high:1024kB active:166620kB
inactive:99456kB present:311304kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB
0*4096kB = 1200kB
DMA32: empty
Normal: empty
HighMem: 1701*4kB 1429*8kB 1013*16kB 18*32kB 0*64kB 3*128kB 1*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 35660kB
Swap cache: add 969

Sometimes many times over.  This happens for a few seconds or minutes before
normal service is resumed.

I am using Debian etch with etch's stock dom0 kernel and hypervisor:

host                   : curacao
release                : 2.6.18-5-xen-686
version                : #1 SMP Mon Aug 13 01:44:42 UTC 2007
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2333
hw_caps                :
bfebfbff:20100000:00000000:00000140:0004e3bd:00000000:00000001
total_memory           : 8190
free_memory            : 200
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-1
xen_caps               : xen-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : Tue Oct 17 22:09:52 2006 +0100
cc_compiler            : gcc version 4.1.2 20061028 (prerelease) (Debian
4.1.1-19)
cc_compile_by          : ultrotter
cc_compile_domain      : debian.org
cc_compile_date        : Thu Nov  2 20:32:28 CET 2006
xend_config_format     : 2

dom0_mem is set to 1G in the grub config, and as you can see 200M of RAM is
left unallocated.

"top" does not suggest any memory issue while this is happening.

If there is any other information that would be useful please let me know.

Cheers,
Andy


-- 
Configure bugmail: 
http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs