[Xen-devel] RE: [PATCH] Netchannel2 optimizations [2/4]

["Santos, Jose Renato G" <joserenato.santos@xxxxxx>]

> -----Original Message-----
> From: Steven Smith [mailto:steven.smith@xxxxxxxxxx] 
> Sent: Friday, February 20, 2009 1:58 AM
> To: Santos, Jose Renato G
> Cc: Steven Smith; xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [PATCH] Netchannel2 optimizations [2/4]
> 
> > > I've also added a (very stupid) adaptation scheme which tries to 
> > > adjust the max_count_frags_no_event parameter to avoid 
> hitting the 
> > > deadlock too often in the first place.  It seems to do 
> broadly the 
> > > right thing for both UDP floods and TCP stream tests, but it 
> > > probably wouldn't be very hard to come up with some workload for 
> > > which it falls over.
> >   OK, I will test how this work on 10 gig NICs when I have some
> >   time.  I am currently doing some tests on Intel 10gig ixgbe NICs
> >   and I am seeing some behaviour that I cannot explain (without this
> >   adaptation patch).  Netperf is not able to saturate the link and
> >   at the same time both the guest and dom0 cannot not saturate the
> >   CPU either ( I made sure the client is not the bottleneck
> >   either). So some other factor is limiting throughput. (I disabled
> >   the netchannel2 rate limiter but this did not seem to have any
> >   effect either). I will spend some time looking into that.
> Is it possible that we're seeing some kind of 
> semi-synchronous bouncing between the domU and dom0?  
> Something like this:
> 
> -- DomU sends some messages to dom0, wakes it up, and then goes to
>    sleep.
> -- Dom0 wakes up, processes the messages, sends the responses, wakes
>    the domU, and then goes to sleep.
> -- Repeat.
> 
> So that both domains are spending significant time just 
> waiting for the other one to do something, and neither can 
> saturate their CPU.

   Yes, that is what I thought as well. I still need to do a careful xentrace 
analysis though.

> That should be fairly obvious in a xentrace trace if you run 
> it while you're observing the bad behaviour.
> 
> If that is the problem, there are a couple of easy-ish things 
> we could do which might help a bit:
> 
> -- Re-arrange the tasklet a bit so that it sends outgoing messages
>    before checking for incoming ones.  The risk is that processing an
>    incoming message is likely to generate further outgoing ones, so we
>    risk splitting the messages into two flights.
> 
> -- Arrange to kick after N messages, even if we still have more
>    messages to send, so that the domain which is receiving the
>    messages runs in parallel with the sending one.
>

   I tried limiting the messages in each run and some other things but it did 
not help. I can try doing TX before RX but I think there is something else 
going on and I will need to spend more time analysing.  I will postpone this 
until after the Xen summit as I will be busy in the next days writing my slides.

Thanks for the suggestions

Renato 

> Both approaches would risk sending more batches of messages, 
> and hence more event channel notifications, trips through the 
> Xen scheduler, etc., and hence would only ever increase the 
> number of cycles per packet, but if they stop CPUs going idle 
> then they might increase the actual throughput.
> 
> Ideally, we'd only do this kind of thing if the receiving 
> domain is idle, but figuring that out from the transmitting 
> domain in an efficient way sounds tricky.  You could imagine 
> some kind of scoreboard showing which domains are running, 
> maintained by Xen and readable by all domains, but I really 
> don't think we want to go down that route.
> 
> Steven.
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel