Re: [Xen-users] Xen system hang or freeze

[Nick Anderson <nick@xxxxxxxxxxxx>]

On Tue, Apr 21, 2009 at 08:30:32AM -0400, Peter Booth wrote:
> It would be interesting to know whether sar data was captured during  
> this time. From this you could track whether there was any process  
> creation or destruction occurring.
I just had another lockup this weekend.

Sar (from the host)
12:35:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
12:45:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
12:55:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
01:05:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
01:15:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
Average:          all      0.00      0.00      0.00      0.00
0.01     99.98

01:25:53 PM       LINUX RESTART

01:35:02 PM       CPU     %user     %nice   %system   %iowait
%steal     %idle
01:45:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
01:55:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99
02:05:01 PM       all      0.00      0.00      0.00      0.00
0.01     99.99


sar -b
11:55:01 AM     12.22      0.90     11.32     12.90    257.89
12:05:01 PM     13.97      0.49     13.48      7.68    331.48
12:15:01 PM     18.88      7.30     11.59    161.74    260.17
12:25:01 PM     14.34      1.10     13.23     16.53    438.73
12:35:01 PM      9.01      0.43      8.58      6.96    208.50
12:45:01 PM      8.47      0.35      8.12      5.23    186.03
12:55:01 PM     10.00      1.09      8.91     19.22    245.17
01:05:01 PM     11.89      1.82     10.06     27.76    279.90
01:15:01 PM     10.06      0.34      9.72      5.23    214.62
Average:        17.55      6.12     11.43    385.87    369.74

01:25:53 PM       LINUX RESTART

01:35:02 PM       tps      rtps      wtps   bread/s   bwrtn/s
01:45:01 PM     19.01      7.19     11.83    113.49    273.91
01:55:01 PM     12.23      2.44      9.79     37.42    239.82
02:05:01 PM     16.89      2.79     14.10     47.93    422.02
02:15:01 PM     17.09      1.92     15.17     26.93    495.01
02:25:01 PM     13.91      3.42     10.49    164.83    282.82
02:35:01 PM     12.47      2.05     10.42     28.45    256.32
02:45:01 PM     13.67      1.81     11.87     31.78    340.39


sar -c
12:45:01 PM      0.02
12:55:01 PM      0.02
01:05:01 PM      0.02
01:15:01 PM      0.02
Average:         0.03

01:25:53 PM       LINUX RESTART

01:35:02 PM    proc/s
01:45:01 PM      0.02
01:55:01 PM      0.02

sar -q 
12:55:01 PM         0       147      0.00      0.00      0.00
01:05:01 PM         0       147      0.07      0.03      0.01
01:15:01 PM         0       147      0.00      0.00      0.00
Average:            0       147      0.00      0.00      0.00

01:25:53 PM       LINUX RESTART

01:35:02 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
01:45:01 PM         0       147      0.00      0.00      0.00
01:55:01 PM         0       147      0.00      0.00      0.00

sar -r
01:05:01 PM   7312568   1878856     20.44    175416     66532
1044184         0      0.00         0
01:15:01 PM   7311948   1879476     20.45    175416     66544
1044184         0      0.00         0
Average:      7328126   1863298     20.27    175403     67011
1044184         0      0.00         0

01:25:53 PM       LINUX RESTART

01:35:02 PM kbmemfree kbmemused  %memused kbbuffers  kbcached
kbswpfree kbswpused  %swpused  kbswpcad
01:45:01 PM   8620940    570484      6.21     64136     36012
1044184         0      0.00         0
01:55:01 PM   8619824    571600      6.22     64972     36028
1044184         0      0.00         0
02:05:01 PM   8618204    573220      6.24     65800     36040
1044184         0      0.00         0
===============================================================



Now perhaps I have missed something but to me that all looks just
fine. I should setup something to log ps. But in my guests I see steal
pushed through the roof. And its like that for days ahead time. Ive
noticed the steal during the lockups before but either I neglected to
look back several days or forgot what I saw. I didnt recall steal
being at 100% as far back as my logs go.

12:55:01 PM       CPU     %user     %nice   %system   %iowait
%steal     %idle
01:05:01 PM       all      0.00      0.00      0.00      0.00
100.00      0.00
01:15:01 PM       all      0.00      0.00      0.00      0.00
100.00      0.00
Average:          all      0.00      0.00      0.00      0.00
100.00      0.00

01:27:49 PM       LINUX RESTART

01:35:01 PM       CPU     %user     %nice   %system   %iowait
%steal     %idle
01:45:01 PM       all      4.04      0.00      1.80      0.64
0.02     93.50
01:55:01 PM       all      4.10      0.00      1.76      0.31
0.02     93.80
02:05:01 PM       all      5.45      0.00      2.47      0.23
0.02     91.83
02:15:01 PM       all      7.03      0.00      3.22      0.22
0.02     89.51
02:25:01 PM       all      4.82      0.00      2.31      0.18
0.01     92.6




> Might also be worth adding a cron entry to append the output of lsof to a 
> file every N minutes (perhaps with logrotate enabled) to see if you can 
> capture what changed in the running system when this "lockup" occurred?
> Also worth collecting ps output every minute

> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

-- 
Nick Anderson <nick@xxxxxxxxxxxx>
http://www.cmdln.org


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users