[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Bug: xm commands hanging due to poor threading in xend

Ewan Mellor wrote:

Bug url: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=465

In the /var/log/xend-debug.log for both your bugs #465 and #486 you can see
the message "error: can't start new thread".  That's going to be fatal --
there's no way that Xend can proceed if it cannot create new threads.

This points to a resource leak on the machine -- either you are leaking
threads or processes locally to Xend or globally to your machine, which would
show up on ps ax, or you are out of memory, which would show up in free or top
(press m to sort by memory usage).  Possibly, this could be a manifestation of
a file descriptor leak, which would show up in lsof.

Could you try and track down the leak?  This would give us a much better clue
as where to look.

I went ahead and did find some problems. On a server up with 10 days some processes (mysql/httpd) in dom0 were stressed. Swap was 50% in use. I have put in memory minimizing config files for both of these apps. File descriptors is still high even after restart most all services on the server with the higher uptime. I can also try increasing dom0 memory to 512MB or so.

I did 128MB for dom0 with 2.0 and increased this to 256MB with 3.0 because all my hosts can now access their full 8GB.

10 day uptime host:
# lsof -n | wc -l
# free
             total       used       free     shared    buffers     cached
Mem:        262544     218040      44504          0      21300      55592
-/+ buffers/cache:     141148     121396
Swap:       522104      35944     486160

2 day uptime host:
# lsof -n | wc -l
# free
             total       used       free     shared    buffers     cached
Mem:        262544     252076      10468          0      28432      85264
-/+ buffers/cache:     138380     124164
Swap:       522104       3928     518176

File limit is 14343 so fd's shouldn't be a problem.

I do not have any OOM errors in my logs though.

I've also run into this once:

Message from syslogd@vm20 at Fri Jan 20 23:16:52 2006 ...
vm20 xenstored: xenstored corruption: connection id -1: err No such file or directory: No child '(null)' found

If you get this, all bets are off.  There is no way that the system as it
stands will recover gracefully if the store is corrupted.  At best, you'll
just lose configuration data regarding the running VMs -- at worst, the
corruption could persist indefinitely, and you'll be unable to do anything
through Xend.

Do you have xen-unstable changeset 8269:ac3ceb2d37d1 aka xen-3.0-testing
changeset 8250:1e3d31952015?  This fixes the only xenstore corruption bug that
I know of, and if you've got that fix, then it's definitely a new bug.  In
that case, we would appreciate it if you could either find a test case that
takes less than a few days to trigger this bug, or get your hands dirty
yourself and put some tracing and assertions into Xenstored around the TDB
manipulations to try and catch the corruption.

I am running -unstable from the 16th. If that change exists in there then yes I have the fix.

Maybe the corrupted TDB file itself might be useful to someone.  Could you
save that, too?

Yes, normally in a case like this I get a few tdb.xxxxxx where the x's represent a 6 character length hex string.

As far as I'm aware, you are the only person who's ever seen this message, so
tracking it down without your help is going to be impossible.  Is there
anything strange about your setup?  Any network block devices or NFS involved,
any quotas on your filesystems or SELinux?  Any patches that you've applied,
non-standard kernel options, anything like that?

My setup is fairly standard. -unstable, PAE, LVM, routed networking. Just tracking Xen using mercuial.

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.