[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [Xen-users] unexpected Out Of Memory (OOM)




Le mercredi 07 aoÃt 2013 Ã 11:35 +0200, Olivier Bonvalet a Ãcrit :
> I have a DomU which is currently running with 8GB of memory (confirmed
> in /proc/meminfo), and only 6GB of memory is used, without process using
> it :
> 
> root! to:~# free -m
>              total       used       free     shared    buffers     cached
> Mem:          8024       6642       1381          0         32        252
> -/+ buffers/cache:       6357       1667
> Swap:          255          0        255
> 
> 
> 
> root! to:~# ps auxw
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> root         1  0.0  0.0   8408   800 ?        Ss   Aug06   0:01 init [2]     
>   
> root         2  0.0  0.0      0     0 ?        S    Aug06   0:00 [kthreadd]
> root         3  0.0  0.0      0     0 ?        S    Aug06   0:00 [ksoftirqd/0]
> root         4  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/0:0]
> root         5  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/0:0H]
> root         7  0.0  0.0      0     0 ?        S    Aug06   0:00 [migration/0]
> root         8  0.0  0.0      0     0 ?        S    Aug06   0:00 [rcu_bh]
> root         9  0.0  0.0      0     0 ?        S    Aug06   0:00 [rcu_sched]
> root        10  0.0  0.0      0     0 ?        S    Aug06   0:00 [watchdog/0]
> root        11  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/1]
> root        12  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/1]
> root        13  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/1]
> root        14  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/1:0]
> root        15  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/1:0H]
> root        16  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/2]
> root        17  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/2]
> root        18  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/2]
> root        19  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/2:0]
> root        20  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/2:0H]
> root        21  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/3]
> root        22  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/3]
> root        23  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/3]
> root        24  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/3:0]
> root        25  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/3:0H]
> root        26  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/4]
> root        27  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/4]
> root        28  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/4]
> root        29  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/4:0]
> root        30  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/4:0H]
> root        31  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/5]
> root        32  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/5]
> root        33  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/5]
> root        34  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/5:0]
> root        35  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/5:0H]
> root        36  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/6]
> root        37  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/6]
> root        38  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/6]
> root        39  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/6:0]
> root        40  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/6:0H]
> root        41  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/7]
> root        42  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/7]
> root        43  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/7]
> root        44  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/7:0]
> root        45  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/7:0H]
> root        46  0.0  0.0      0     0 ?        S<   Aug06   0:00 [khelper]
> root        47  0.0  0.0      0     0 ?        S    Aug06   0:00 [kdevtmpfs]
> root        48  0.0  0.0      0     0 ?        S    Aug06   0:00 [xenwatch]
> root        49  0.0  0.0      0     0 ?        S    Aug06   0:00 [xenbus]
> root        50  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/1:1H]
> root        51  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/2:1H]
> root        52  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/3:1H]
> root        53  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/4:1H]
> root        54  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/5:1H]
> root        55  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/6:1H]
> root        56  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [kworker/7:1H]
> root        57  0.0  0.0      0     0 ?        S<   Aug06   0:00 [writeback]
> root        58  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kintegrityd]
> root        59  0.0  0.0      0     0 ?        S<   Aug06   0:00 [bioset]
> root        60  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kblockd]
> root        61  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [xenbus_frontend]
> root        62  0.0  0.0      0     0 ?        S<   Aug06   0:00 [ata_sff]
> root        63  0.0  0.0      0     0 ?        S<   Aug06   0:00 [edac-poller]
> root        64  0.0  0.0      0     0 ?        S    Aug06   0:02 [kworker/0:1]
> root       106  0.0  0.0      0     0 ?        S    Aug06   0:00 [kswapd0]
> root       107  0.0  0.0      0     0 ?        SN   Aug06   0:00 [ksmd]
> root       108  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [fsnotify_mark]
> root       109  0.0  0.0      0     0 ?        S<   Aug06   0:00 [crypto]
> root       126  0.0  0.0      0     0 ?        S    Aug06   0:00 [khvcd]
> root       127  0.0  0.0      0     0 ?        S<   Aug06   0:00 [iscsi_eh]
> root       128  0.0  0.0      0     0 ?        S<   Aug06   0:00 [deferwq]
> root       129  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [kworker/u16:1]
> root       253  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvda-8]
> root       254  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       257  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       259  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       520  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kdmflush]
> root       704  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvdc-8]
> root       706  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       707  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvdb-8]
> root       708  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       709  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/dm-0-8]
> root       710  0.0  0.0      0     0 ?        S<   Aug06   0:00 
> [ext4-dio-unwrit]
> root       713  0.0  0.0      0     0 ?        S<   Aug06   0:00 [bioset]
> root       714  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-worker-1]
> root       715  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-genwork-1]
> root       716  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-submit-1]
> root       717  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-delalloc-]
> root       718  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-fixup-1]
> root       719  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-endio-1]
> root       720  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-endio-met]
> root       721  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-rmw-1]
> root       722  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-endio-rai]
> root       723  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-endio-met]
> root       724  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-endio-wri]
> root       725  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-freespace]
> root       726  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-delayed-m]
> root       727  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-cache-1]
> root       728  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-readahead]
> root       729  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-flush_del]
> root       730  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-qgroup-re]
> root       731  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-cleaner]
> root       732  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [btrfs-transacti]
> root      1845  0.0  0.0   5984   620 tty1     Ss+  Aug06   0:00 /sbin/getty 
> 38400 tty1
> root      1846  0.0  0.0   5984   624 hvc0     Ss+  Aug06   0:00 /sbin/getty 
> 38400 hvc0
> root      1908  0.0  0.0      0     0 ?        S    Aug06   0:00 
> [kworker/u16:2]
> root      6457  0.0  0.0  76740  3348 ?        Ss   11:23   0:00 sshd: daevel 
> [priv]
> daevel    6460  0.0  0.0  76740  1928 ?        S    11:23   0:00 sshd: 
> daevel@pts/0,pts/1
> daevel    6461  0.0  0.0  25040  7312 pts/0    Ss   11:23   0:00 -bash
> daevel    6640  0.0  0.0  25036  7296 pts/1    Ss   11:23   0:00 -bash
> root      6668  0.0  0.0  25128  1264 pts/1    S    11:23   0:00 sudo su
> root      6669  0.0  0.0  45804  1348 pts/1    S    11:23   0:00 su
> root      6671  0.0  0.0  19944  2168 pts/1    S    11:23   0:00 bash
> daevel    8314  0.0  0.0   5924   560 pts/0    S+   11:32   0:00 sleep 30
> root      8315  0.0  0.0  16900  1176 pts/1    R+   11:33   0:00 ps auxw
> root     11919  0.0  0.0      0     0 ?        S<   02:00   0:00 
> [kworker/0:1H]
> root     11971  0.0  0.0      0     0 ?        S<   02:00   0:00 [bioset]
> root! to:~# 
> 
> root! to:~# cat /proc/meminfo 
> MemTotal:        8217004 kB
> MemFree:         1415044 kB
> Buffers:           33128 kB
> Cached:           258884 kB
> SwapCached:            0 kB
> Active:           123328 kB
> Inactive:         182692 kB
> Active(anon):      14032 kB
> Inactive(anon):      344 kB
> Active(file):     109296 kB
> Inactive(file):   182348 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> SwapTotal:        262140 kB
> SwapFree:         262140 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:         14024 kB
> Mapped:             4300 kB
> Shmem:               396 kB
> Slab:            5105244 kB
> SReclaimable:    1563800 kB
> SUnreclaim:      3541444 kB
> KernelStack:     1360232 kB
> PageTables:          860 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     4370640 kB
> Committed_AS:      20612 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:       16248 kB
> VmallocChunk:   34359721447 kB
> AnonHugePages:         0 kB
> DirectMap4k:     8388608 kB
> DirectMap2M:           0 kB
> root! to:~# 
> 
> 
> root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
>  Active / Total Objects (% used)    : 17298069 / 17301644 (100.0%)
>  Active / Total Slabs (% used)      : 597110 / 597110 (100.0%)
>  Active / Total Caches (% used)     : 72 / 93 (77.4%)
>  Active / Total Size (% used)       : 5045185.95K / 5045752.48K (100.0%)
>  Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
> 10290816 10290816 100%    0.25K 321588       32   2572704K kmalloc-256        
>     
> 1997499 1997499 100%    0.19K  95119       21    380476K dentry               
>   
> 1904890 1904890 100%    0.59K  73265       26   1172240K proc_inode_cache     
>   
> 1305654 1305654 100%    0.19K  62174       21    248696K kmalloc-192          
>   
> 863488 861305  99%    0.06K  13492       64     53968K kmalloc-64             
> 180540 180540 100%    0.11K   5015       36     20060K sysfs_dir_cache        
> 171808 170841  99%    0.12K   5369       32     21476K kmalloc-128            
> 170079 170079 100%    0.81K   4361       39    139552K task_xstate            
> 170050 170050 100%    1.66K   8950       19    286400K task_struct            
> 169860 169860 100%    1.06K   5662       30    181184K signal_cache           
>  19925  19925 100%    0.62K    797       25     12752K shmem_inode_cache      
>  19383  19383 100%    0.10K    497       39      1988K buffer_head            
>   4608   4608 100%    0.01K      9      512        36K kmalloc-8              
>   4592   4592 100%    0.55K    164       28      2624K radix_tree_node        
>   4480   4480 100%    0.91K    128       35      4096K ext4_inode_cache       
>   3660   3660 100%    0.53K    122       30      1952K inode_cache            
>   3200   3200 100%    0.03K     25      128       100K kmalloc-32             
>   2816   2816 100%    0.02K     11      256        44K kmalloc-16             
>   2448   2448 100%    0.04K     24      102        96K ext4_extent_status     
>   2112   2112 100%    0.06K     33       64       132K anon_vma               
>   1632   1632 100%    0.04K     16      102        64K ext4_system_zone       
>   1298   1186  91%    0.18K     59       22       236K vm_area_struct         
>   1024    910  88%    0.50K     32       32       512K kmalloc-512            
>    546    546 100%    0.09K     13       42        52K kmalloc-96             
>    345    345 100%    0.34K     15       23       120K blkdev_requests        
>    340    340 100%    0.05K      4       85        16K shared_policy_node     
>    336    336 100%    0.19K     16       21        64K tw_sock_TCP            
>    324    324 100%    0.11K      9       36        36K jbd2_journal_head      
>    320    320 100%    1.00K     10       32       320K kmalloc-1024           
>    312    312 100%    0.10K      8       39        32K flow_cache             
>    256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
>    232     89  38%    0.27K      8       29        64K 
> nf_conntrack_ffffffff8165cc40
>    195    195 100%    2.06K     13       15       416K idr_layer_cache        
>    195    179  91%    2.06K     13       15       416K sighand_cache          
>    192    192 100%    0.06K      3       64        12K ext4_io_end            
>    170    170 100%    0.02K      1      170         4K numa_policy            
>    128    128 100%    0.06K      2       64         8K kmem_cache_node        
>    128    128 100%    0.03K      1      128         4K jbd2_revoke_record_s   
>    125    125 100%    0.62K      5       25        80K files_cache            
>    125    125 100%    0.16K      5       25        20K sigqueue               
>    120     80  66%    4.00K     15        8       480K kmalloc-4096           
>    112    112 100%    2.00K      7       16       224K kmalloc-2048           
>    100    100 100%    0.62K      4       25        64K sock_inode_cache       
>     96     96 100%    0.25K      3       32        24K kmem_cache             
>     96     96 100%    0.98K      3       32        96K btrfs_inode            
> 
>  Active / Total Objects (% used)    : 17298196 / 17301771 (100.0%)
>  Active / Total Slabs (% used)      : 597115 / 597115 (100.0%)
>  Active / Total Caches (% used)     : 72 / 93 (77.4%)
>  Active / Total Size (% used)       : 5045213.76K / 5045780.30K (100.0%)
>  Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
> 10290880 10290880 100%    0.25K 321590       32   2572720K kmalloc-256        
>     
> 1997520 1997520 100%    0.19K  95120       21    380480K dentry               
>   
> 1904890 1904890 100%    0.59K  73265       26   1172240K proc_inode_cache     
>   
> 1305696 1305696 100%    0.19K  62176       21    248704K kmalloc-192          
>   
> 863488 861305  99%    0.06K  13492       64     53968K kmalloc-64             
> 180540 180540 100%    0.11K   5015       36     20060K sysfs_dir_cache        
> 171808 170841  99%    0.12K   5369       32     21476K kmalloc-128            
> 170079 170079 100%    0.81K   4361       39    139552K task_xstate            
> 170050 170050 100%    1.66K   8950       19    286400K task_struct            
> 169860 169860 100%    1.06K   5662       30    181184K signal_cache           
>  19925  19925 100%    0.62K    797       25     12752K shmem_inode_cache      
>  19383  19383 100%    0.10K    497       39      1988K buffer_head            
>   4608   4608 100%    0.01K      9      512        36K kmalloc-8              
>   4592   4592 100%    0.55K    164       28      2624K radix_tree_node        
>   4480   4480 100%    0.91K    128       35      4096K ext4_inode_cache       
>   3660   3660 100%    0.53K    122       30      1952K inode_cache            
>   3200   3200 100%    0.03K     25      128       100K kmalloc-32             
>   2816   2816 100%    0.02K     11      256        44K kmalloc-16             
>   2448   2448 100%    0.04K     24      102        96K ext4_extent_status     
>   2112   2112 100%    0.06K     33       64       132K anon_vma               
>   1632   1632 100%    0.04K     16      102        64K ext4_system_zone       
>   1298   1186  91%    0.18K     59       22       236K vm_area_struct         
>   1024    910  88%    0.50K     32       32       512K kmalloc-512            
>    546    546 100%    0.09K     13       42        52K kmalloc-96             
>    345    345 100%    0.34K     15       23       120K blkdev_requests        
>    340    340 100%    0.05K      4       85        16K shared_policy_node     
>    336    336 100%    0.19K     16       21        64K tw_sock_TCP            
>    324    324 100%    0.11K      9       36        36K jbd2_journal_head      
>    320    320 100%    1.00K     10       32       320K kmalloc-1024           
>    312    312 100%    0.10K      8       39        32K flow_cache             
>    256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
>    232     89  38%    0.27K      8       29        64K 
> nf_conntrack_ffffffff8165cc40
>    195    195 100%    2.06K     13       15       416K idr_layer_cache        
>    195    179  91%    2.06K     13       15       416K sighand_cache          
>    192    192 100%    0.06K      3       64        12K ext4_io_end            
>    170    170 100%    0.02K      1      170         4K numa_policy            
>    128    128 100%    0.06K      2       64         8K kmem_cache_node        
>    128    128 100%    0.03K      1      128         4K jbd2_revoke_record_s   
>    125    125 100%    0.62K      5       25        80K files_cache            
>    125    125 100%    0.16K      5       25        20K sigqueue               
>    120     80  66%    4.00K     15        8       480K kmalloc-4096           
>    112    112 100%    2.00K      7       16       224K kmalloc-2048           
>    100    100 100%    0.62K      4       25        64K sock_inode_cache       
>     96     96 100%    0.25K      3       32        24K kmem_cache             
>     96     96 100%    0.98K      3       32        96K btrfs_inode            
> root! to:~# 
> 
> 
> 
> 

and the diff between the two slaptop :

- Active / Total Objects (% used)    : 17298069 / 17301644 (100.0%)
- Active / Total Slabs (% used)      : 597110 / 597110 (100.0%)
+ Active / Total Objects (% used)    : 17298196 / 17301771 (100.0%)
+ Active / Total Slabs (% used)      : 597115 / 597115 (100.0%)

- Active / Total Size (% used)       : 5045185.95K / 5045752.48K (100.0%)
+ Active / Total Size (% used)       : 5045213.76K / 5045780.30K (100.0%)

-10290816 10290816 100%    0.25K 321588       32   2572704K kmalloc-256         
   
-1997499 1997499 100%    0.19K  95119       21    380476K dentry                
 
+10290880 10290880 100%    0.25K 321590       32   2572720K kmalloc-256         
   
+1997520 1997520 100%    0.19K  95120       21    380480K dentry                
 

-1305654 1305654 100%    0.19K  62174       21    248696K kmalloc-192           
 
+1305696 1305696 100%    0.19K  62176       21    248704K kmalloc-192           
 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.