[Xen-devel] intermittent problems with legacy xmlrpc server in 3.0.4

[John Levon <levon@xxxxxxxxxxxxxxxxx>]

I've been having intermittent problems with xm talking to xend over the legacy
xmlrpc server. In theory it should be reproducable under Linux with an
xm list loop, though you might need a heavy load. DTrace says:

  0  24244                       recv:entry 12954192948474 xend recv(8192)
  0  24245                      recv:return 12954192971001 xend recv() ret -1 
errno 11
  0  24250                       send:entry 12954196092353 xm send(132, POST 
/RPC2 HTTP/1.0
Host:
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 268

)
  0  24251                      send:return 12954196113363 xm send() ret -1 
errno 32

11 = EAGAIN:

     EWOULDBLOCK     The socket is marked  non-blocking  and  the
                     requested operation would block.
32 = EPIPE:

     EPIPE           The socket is shut down for writing, or  the
                     socket  is  connection-mode and is no longer
                     connected. In the latter case, if the socket
                     is  of  type SOCK_STREAM, the SIGPIPE signal
                     is generated to the calling thread.

So for some reason the server is trying to process a request before xm has sent 
it, and the
EWOULDBLOCK is causing the EPIPE it seems.

changeset 12062:5fe8e9ebcf5c made this change:

+        try:
+            self.server.socket.settimeout(1.0)
+            while self.running:
+                self.server.handle_request()

which places xmlrpc.sock in non-blocking mode. SocketServer.py actually
does this on init:

    def __init__(self, request, client_address, server):
        self.request = request
        self.client_address = client_address
        self.server = server
        try:
            self.setup()
            self.handle()
            self.finish()

This self.handle() ends up as the recv() that craps itself when it gets
EAGAIN. This doesn't always happen, presumably the race is between
creating the request thread in SocketServer and xm writing the data.

I've hacked up SocketServer a bit to handle EAGAIN, but this obviously
isn't a good fix. Suggestions welcome, I'm not really familiar with all
this server code.

regards
john

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel