Re: [Xen-devel] 2.6.39 domU - page allocation failure

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Mon, May 23, 2011 at 10:13:21PM -0400, Christopher S. Aker wrote:
> Moments after deploying 2.6.39 32 bit this evening, got a report of the 
> following.  Recent pv_ops kernels seem to suffer from a too-low 
> vm/min_free_kbytes value.  Raising it from the default of ~2700 to 16k (on a 
> 512M instance) seems to relieve some of these symptoms, but it's not a 
> scientific analysis.
> 
> Noteworthy is the "HighMem free:0kB min:128kB low:136kB high:148kB" line from 
> the spew.  HighMem ratio calculations seem to be insufficient ... maybe I'll 
> try just setting it to 0.
> 
> Any thoughts?  Thanks!
> 
> index.scgi: page allocation failure. order:5, mode:0x20
> Pid: 1997, comm: index.scgi Not tainted 2.6.39-linode33 #1
> Call Trace:
>  [<c0189576>] ? __alloc_pages_nodemask+0x516/0x6d0
>  [<c01af4a0>] ? T.819+0x30/0xc0
>  [<c01af5c5>] ? T.816+0x95/0x250
>  [<c01af9c6>] ? cache_alloc_refill+0x246/0x290
>  [<c01afac4>] ? __kmalloc+0xb4/0xe0
>  [<c0509ffe>] ? pskb_expand_head+0x12e/0x200
>  [<c050a54d>] ? __pskb_pull_tail+0x4d/0x2b0
>  [<c05cfd73>] ? ipv4_confirm+0xd3/0x180
>  [<c0512ecd>] ? dev_hard_start_xmit+0x1dd/0x3e0
>  [<c0591270>] ? ip_finish_output2+0x250/0x250
>  [<c0591270>] ? ip_finish_output2+0x250/0x250
>  [<c0526bf2>] ? sch_direct_xmit+0xb2/0x170
>  [<c05131c9>] ? dev_queue_xmit+0xf9/0x320
>  [<c05913a7>] ? ip_finish_output+0x137/0x300
>  [<c059161a>] ? ip_output+0xaa/0xe0
>  [<c0591270>] ? ip_finish_output2+0x250/0x250
>  [<c05907f8>] ? ip_local_out+0x18/0x20
>  [<c0590bd7>] ? ip_queue_xmit+0x117/0x3d0
>  [<c0188d89>] ? get_page_from_freelist+0x119/0x340
>  [<c01c2d60>] ? __pollwait+0xf0/0xf0
>  [<c01062eb>] ? xen_restore_fl_direct_reloc+0x4/0x4
>  [<c05a420f>] ? tcp_transmit_skb+0x37f/0x660
>  [<c05a6c0b>] ? tcp_write_xmit+0x18b/0x4a0
>  [<c01062f4>] ? check_events+0x8/0xc
>  [<c05a6f4d>] ? tcp_push_one+0x2d/0x40
>  [<c0599dc4>] ? tcp_sendmsg+0x9d4/0xab0
>  [<c05b7f27>] ? inet_sendmsg+0x47/0xb0
>  [<c05025e1>] ? sock_sendmsg+0xf1/0x110
>  [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180
>  [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180
>  [<c0106258>] ? xen_vcpuop_set_next_event+0x48/0x80
>  [<c015a4a0>] ? clockevents_program_event+0xa0/0x160
>  [<c0502ece>] ? sys_sendto+0x11e/0x170
>  [<c010614a>] ? do_stolen_accounting+0x21a/0x250
>  [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180
>  [<c0502f57>] ? sys_send+0x37/0x40
>  [<c05043d4>] ? sys_socketcall+0x1f4/0x2c0
>  [<c0683931>] ? syscall_call+0x7/0xb
>  [<c0680000>] ? migrate_timers+0x80/0xbe
> Mem-Info:
> DMA per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> CPU    1: hi:    0, btch:   1 usd:   0
> CPU    2: hi:    0, btch:   1 usd:   0
> CPU    3: hi:    0, btch:   1 usd:   0
> Normal per-cpu:
> CPU    0: hi:  186, btch:  31 usd: 130
> CPU    1: hi:  186, btch:  31 usd: 168
> CPU    2: hi:  186, btch:  31 usd: 159
> CPU    3: hi:  186, btch:  31 usd: 193
> HighMem per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> CPU    1: hi:    0, btch:   1 usd:   0
> CPU    2: hi:    0, btch:   1 usd:   0
> CPU    3: hi:    0, btch:   1 usd:   0
> active_anon:32680 inactive_anon:32950 isolated_anon:43
>  active_file:22048 inactive_file:23990 isolated_file:19
>  unevictable:0 dirty:122 writeback:8 unstable:0
>  free:2486 slab_reclaimable:1069 slab_unreclaimable:1605
>  mapped:5588 shmem:33178 pagetables:255 bounce:0
> DMA free:2092kB min:84kB low:104kB high:124kB active_anon:588kB 
> inactive_anon:632kB active_file:32kB inactive_file:2996kB unevictable:0kB 
> isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB 
> writeback:0kB mapped:20kB shmem:1220kB slab_reclaimable:0kB 
> slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB 
> bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 490 498 498
> Normal free:7852kB min:2788kB low:3484kB high:4180kB active_anon:130132kB 
> inactive_anon:131168kB active_file:88160kB inactive_file:92964kB 
> unevictable:0kB isolated(anon):172kB isolated(file):76kB present:502240kB 
> mlocked:0kB dirty:488kB writeback:32kB mapped:22332kB shmem:131492kB 
> slab_reclaimable:4276kB slab_unreclaimable:6416kB kernel_stack:568kB 
> pagetables:1020kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 
> all_unreclaimable? no
> lowmem_reserve[]: 0 0 64 64
> HighMem free:0kB min:128kB low:136kB high:148kB active_anon:0kB 
> inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB 
> isolated(anon):0kB isolated(file):0kB present:8192kB mlocked:0kB dirty:0kB 
> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB 
> bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> lowmem_reserve[]: 0 0 0 0
> DMA: 3*4kB 4*8kB 4*16kB 2*32kB 4*64kB 3*128kB 3*256kB 1*512kB 0*1024kB 
> 0*2048kB 0*4096kB = 2092kB
> Normal: 507*4kB 197*8kB 110*16kB 31*32kB 23*64kB 0*128kB 0*256kB 0*512kB 
> 0*1024kB 0*2048kB 0*4096kB = 7828kB
> HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 
> 0*2048kB 0*4096kB = 0kB
> 79628 total pagecache pages
> 460 pages in swap cache
> Swap cache stats: add 1791, delete 1330, find 1257/1321
> Free swap  = 259640kB
> Total swap = 262140kB
> 1050608 pages RAM

Ok, so 4G
> 865282 pages HighMem
> 930420 pages reserved

That is a large amount of reserved.. Did the previous guest (2.6.38) have less 
of 
a reserve amount? 

Can you for fun try to merge #stable/e820.bugfixes ?

> 18088 pages shared
> 104397 pages non-shared
> 
> -Chris
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel