WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 		  
Home Products Support Community News
 
   
 

xen-users

[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

Re: [Xen-users] Re; System Crash ...

from [Gareth Bult] [Permanent Link][Original]
To: Gareth Bult <gareth@xxxxxxxxxxxxx>
Subject: Re: [Xen-users] Re; System Crash ...
From: Gareth Bult <gareth@xxxxxxxxxxxxx>
Date: Sat, 19 Jan 2008 13:26:56 +0000 (GMT)
Cc: Xen-Users <xen-users@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Sat, 19 Jan 2008 05:27:45 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <25940858.121200672206767.JavaMail.root@zimbra>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Ok,

Problem solved, just don't use IPTABLES in Dom0.
(esp. if you are using two bridges)

Something to do with loading Dom0 and installing IPTABLES (conntrack?) breaks things under pressure ...

This seems like a massive flaw  ?!
(I've seen lots of people posting problems that seem likely to come down to the same issue)

???

----- Original Message -----
step 3.: "Gareth Bult" <gareth@xxxxxxxxxxxxx>
To: "Xen-Users" <xen-users@xxxxxxxxxxxxxxxxxxx>
Sent: 18 January 2008 16:03:26 o'clock (GMT) Europe/London
Subject: [Xen-users] Re; System Crash ...

Hi,

I've just has a rather nasty system crash but it's generated lots of nice logging ..

Anyone know what may have caused it and whether there's anything I can do to stop it happening again ?

(Xen3.1/Stock Ubuntu)

Jan 18 14:00:31 nodec kernel: [  362.689029] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.689044]
Jan 18 14:00:31 nodec kernel: [  362.689045] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.689048]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.689092]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.689108]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689126]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689139]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689171]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.689190]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689211]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.689235]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689259]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689271]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689285]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689300]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689327]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.689352]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689375]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.689391]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689405]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689427]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689440]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689456]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689488]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.689504]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.689516]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.689534]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.689562]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.689587]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689622]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689638]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689672]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.689688]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.689706]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689727]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689741]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689755]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689787]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.689802]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.689832]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.689864]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.689874]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.689881]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.689891]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.689900]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.689906]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.689913]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.689922]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.689934]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.689938]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.689958]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.689971]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.689979]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689991]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.690004]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.690012]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.690020]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.690033]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.690073]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.690089]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.690125]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.690139]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.690150]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.690163]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.690170]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.690178]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.690185]  [system_call+0/139] system_call+0x0/0x8b
                                                                                                                                                                                                69,1          Top
Jan 18 14:00:31 nodec kernel: [  362.690192]
Jan 18 14:00:31 nodec kernel: [  362.690194] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.690196] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.690199] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.690203] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690208] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.690212] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690218] Active:58006 inactive:40628 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.690220]  free:272 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.690230] DMA free:1088kB min:2896kB low:3620kB high:4344kB active:232024kB inactive:162512kB present:525200kB pages_scanned:0 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.690237] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.690242] DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1108kB
Jan 18 14:00:31 nodec kernel: [  362.690265] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.690269] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690272] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690277] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.693502] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.693507] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.693509] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.693511] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.693565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.693570]
Jan 18 14:00:31 nodec kernel: [  362.693571] Call Trace:

Jan 18 14:00:31 nodec kernel: [  362.693576]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.693630]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.693641]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693659]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.693677]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693705]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.693723]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693744]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.693766]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693787]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693801]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693816]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693831]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693858]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.693879]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693898]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.693913]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693923]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693942]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693953]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693966]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693988]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.694000]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.694011]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.694026]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.694054]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.694080]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694101]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694115]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694131]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694165]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.694182]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.694201]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.694224]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694234]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694242]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694257]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.694264]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.694280]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.694300]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.694309]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.694314]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.694323]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.694331]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.694337]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.694343]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.694351]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.694363]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.694367]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.694388]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.694401]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.694408]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.694420]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.694434]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.694442]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.694451]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.694462]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.694477]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.694484]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.694500]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.694522]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.694545]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.694573]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.694586]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.694607]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694623]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694644]
Jan 18 14:00:31 nodec kernel: [  362.694648] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.694652] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.694659] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.694665] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.694674] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.694680] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.694688] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.694691]  free:297 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.694700] DMA free:1188kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.694708] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.694713] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1288kB
Jan 18 14:00:31 nodec kernel: [  362.694737] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.694742] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694752] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.697874] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.697881] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.697883] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.697887] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.698059] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.698064]
Jan 18 14:00:31 nodec kernel: [  362.698065] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.698068]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.698119]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.698144]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.698158]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698177]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698191]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.698209]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.698227]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698251]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.698294]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.698332]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.698348]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.698355]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.698369]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.698383]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.698392]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.698403]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.698414]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.698432]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.698437]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.698486]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.698512]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.698526]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.698551]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.698582]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.698603]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.698621]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.698652]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.698686]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.698705]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.698745]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.698777]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.698797]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.698820]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.698838]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.698857]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698874]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698893]
Jan 18 14:00:31 nodec kernel: [  362.698896] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.698900] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.698905] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.698911] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.698920] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.698927] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.698935] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.698937]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.698948] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.698954] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.698959] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.698983] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.698989] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698992] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698995] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.701567] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.701573] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.701575] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.701577] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.701614] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.701617]
Jan 18 14:00:31 nodec kernel: [  362.701618] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.701620]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.701656]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.701668]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.701676]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701684]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701692]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.701701]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.701709]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701724]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.701743]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.701754]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.701770]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.701781]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.701786]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.701796]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.701804]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.701810]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.701818]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.701826]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.701839]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.701843]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.701863]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.701877]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.701884]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.701896]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.701910]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.701919]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.701927]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.701938]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.701954]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.701962]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.701978]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.701992]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.702002]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.702014]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.702023]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.702030]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702038]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702047]
Jan 18 14:00:31 nodec kernel: [  362.702049] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.702050] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.702055] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.702059] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.702064] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.702067] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.702073] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.702075]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.702080] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.702084] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.702088] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.702106] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.702109] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702112] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702114] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.704303] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.704308] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.704310] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.704312] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.704565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.704569]
Jan 18 14:00:31 nodec kernel: [  362.704570] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.704571]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.704607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.704619]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.704628]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704636]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704642]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.704652]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.704660]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704674]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.704692]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.704712]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.704723]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.704734]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.704744]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.704750]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.704758]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.704767]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.704779]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.704784]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.704805]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.704820]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.704828]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.704840]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.704854]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.704863]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.704871]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.704882]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.704898]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.704906]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.704924]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.704938]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.704949]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.704963]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.704971]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.704980]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704987]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704996]
Jan 18 14:00:31 nodec kernel: [  362.704998] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.705001] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.705004] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.705010] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.705013] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.705018] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.705023] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.705024]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.705030] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.705035] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.705039] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.705053] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.705057] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705059] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705061] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707207] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.707211] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.707212] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.707214] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.707262] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.707265]
Jan 18 14:00:31 nodec kernel: [  362.707266] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.707268]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.707305]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.707313]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707321]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707327]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.707335]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.707343]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707359]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.707377]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.707390]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.707408]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.707417]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.707427]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.707436]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.707442]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.707450]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.707459]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.707471]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.707476]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.707497]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.707510]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.707519]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.707531]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.707544]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.707552]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.707561]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.707572]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.707586]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.707595]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.707612]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.707627]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.707638]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.707650]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.707658]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.707667]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707675]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707685] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.707687] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.707690] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.707696] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.707699] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.707704] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.707709] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.707710]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.707716] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.707721] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.707725] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.707741] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.707744] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707749] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.709971] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.709977] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.709978] 31578 pages shared
Jan 18 14:00:31 nodec kernel: [  362.709982] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.881155] swapper: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.881164]
Jan 18 14:00:31 nodec kernel: [  362.881165] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.881168]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.881228]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.881251]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.881266]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881280]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881295]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.881312]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.881328]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881355]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.881392]  [_end+128027760/2130180472] :e1000:e1000_clean_rx_irq+0x368/0x580
Jan 18 14:00:31 nodec kernel: [  362.881441]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.881463]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.881488]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.881504]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.881519]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.881532]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.881549]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.881578]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.881585]  <EOI>  [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0
Jan 18 14:00:31 nodec kernel: [&nbs
_______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  Re: [Xen-users] Creating Virtual Network-Interface under XEN 3.2.0 for Domain0, Maik Brauer
Next by Date:  [Xen-users] Kernel Ooops when shutting down DomU, Gareth Bult
Previous by Thread:  [Xen-users] Re; System Crash ..., Gareth Bult
Next by Thread:  Re: [Xen-users] wine in a domU, Ryan E Roth
Indexes:  [Date] [Thread] [Top] [All Lists]
 
   

Copyright , Citrix Systems Inc. All rights reserved. Legal and Privacy