> -----Original Message-----
> From: Steven Smith [mailto:steven.smith@xxxxxxxxxx]
> Sent: Friday, February 20, 2009 1:58 AM
> To: Santos, Jose Renato G
> Cc: Steven Smith; xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [PATCH] Netchannel2 optimizations [2/4]
>
> > > I've also added a (very stupid) adaptation scheme which tries to
> > > adjust the max_count_frags_no_event parameter to avoid
> hitting the
> > > deadlock too often in the first place. It seems to do
> broadly the
> > > right thing for both UDP floods and TCP stream tests, but it
> > > probably wouldn't be very hard to come up with some workload for
> > > which it falls over.
> > OK, I will test how this work on 10 gig NICs when I have some
> > time. I am currently doing some tests on Intel 10gig ixgbe NICs
> > and I am seeing some behaviour that I cannot explain (without this
> > adaptation patch). Netperf is not able to saturate the link and
> > at the same time both the guest and dom0 cannot not saturate the
> > CPU either ( I made sure the client is not the bottleneck
> > either). So some other factor is limiting throughput. (I disabled
> > the netchannel2 rate limiter but this did not seem to have any
> > effect either). I will spend some time looking into that.
> Is it possible that we're seeing some kind of
> semi-synchronous bouncing between the domU and dom0?
> Something like this:
>
> -- DomU sends some messages to dom0, wakes it up, and then goes to
> sleep.
> -- Dom0 wakes up, processes the messages, sends the responses, wakes
> the domU, and then goes to sleep.
> -- Repeat.
>
> So that both domains are spending significant time just
> waiting for the other one to do something, and neither can
> saturate their CPU.
Yes, that is what I thought as well. I still need to do a careful xentrace
analysis though.
> That should be fairly obvious in a xentrace trace if you run
> it while you're observing the bad behaviour.
>
> If that is the problem, there are a couple of easy-ish things
> we could do which might help a bit:
>
> -- Re-arrange the tasklet a bit so that it sends outgoing messages
> before checking for incoming ones. The risk is that processing an
> incoming message is likely to generate further outgoing ones, so we
> risk splitting the messages into two flights.
>
> -- Arrange to kick after N messages, even if we still have more
> messages to send, so that the domain which is receiving the
> messages runs in parallel with the sending one.
>
I tried limiting the messages in each run and some other things but it did
not help. I can try doing TX before RX but I think there is something else
going on and I will need to spend more time analysing. I will postpone this
until after the Xen summit as I will be busy in the next days writing my slides.
Thanks for the suggestions
Renato
> Both approaches would risk sending more batches of messages,
> and hence more event channel notifications, trips through the
> Xen scheduler, etc., and hence would only ever increase the
> number of cycles per packet, but if they stop CPUs going idle
> then they might increase the actual throughput.
>
> Ideally, we'd only do this kind of thing if the receiving
> domain is idle, but figuring that out from the transmitting
> domain in an efficient way sounds tricky. You could imagine
> some kind of scoreboard showing which domains are running,
> maintained by Xen and readable by all domains, but I really
> don't think we want to go down that route.
>
> Steven.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home