Re: [Xen-devel] Re: performance of credit2 on hybrid workload

[David Xu <davidxu06@xxxxxxxxx>]

Hi George,

Thanks for your reply. I have similar ideas to you, adding another parameter that indicates the required latency and then letting scheduler determine latency characteristics of a VM automatically. Firstly, adding another parameter and let users set its value in advance sounds similar to SEDF. But sometimes the configuration process is hard and inflexible when workloads in VM is complex. So in my opinion, a task-aware scheduler is better. However, manually configuration can help us to check out the effectiveness of the new parameter. For another hand, as you described, it is also not easy and accurate to make scheduler  determine the latency characteristics of a VM automatically with some information we can get from hypervisor, for instance the delay interrupt. Therefore, the key point for me is to find and implement a scheduling helper to indicate which VM should be scheduled soon. For example, for TCP network, we can implement a tool similar to a packet sniffer to capture the packet and analyze its head information to infer the type of workload. Then the analysis result can help scheduler to make a decision. In fact, not all I/O-intensive workloads require low latency, some of them only require high-throughput. Of course, scheduling latency impact significantly the throughput (You handled this problem with boost mechanism to some extension). What I want to is to only reduce the latency of a VM which require low latency while postpone other VMs, and use other technology such as packet offloading to compensate their lost and improve their throughput.

This is just my course idea and there are many problems as well. I hope I can often discuss with you and share our results. Thanks very much.

Regards,

Cong

2011/6/8 George Dunlap <George.Dunlap@xxxxxxxxxxxxx>

On Tue, Jun 7, 2011 at 8:28 PM, David Xu <davidxu06@xxxxxxxxx> wrote:
> Hi George,
> Could you share some ideas about how to addressed the  "mixed workload"
> problem,  where a single VM does both
> cpu-intensive and latency-sensitive workloads, even though you haven't
> implemented it yet?  I am also working on it, maybe I can try some methods
> and give you feedback. Thanks.

Well the main thing to remember is that you can't give the VM any
*more* time.  The amount of time it's allowed is defined by the
scheduler parameters (and the other VMs running).  So all you can do
is change *when* the VM gets the time.  So what you want the scheduler
to do is give the VM shorter timeslices *so that* it can get time more
frequently.

For example, the credit1 scheduler will let a VM burn through 30ms of
credit.  That means if its "fair share" is (say) 50%, then it has to
wait at least 30ms before being allowed to run again in order to
maintain fairness.  If its "fair share" is 33%, then the VM has to
wait at least 60ms.  If the scheduler were to preempt it after 5ms,
then the VM would only have to be delayed for 5ms or 10ms,
respectively; and if it were preempted after 1ms, it would only have
to be delayed 1s or 2s.

So the real key to giving a VM with a mixed workload better latency
characteristics is not to wake it up sooner, but to preempt it sooner.

The problem is, of course, that preempting workloads which are *not*
latency sensitive too soon adds scheduling overhead, and reduces cache
effectiveness.  So the question becomes, how do I know how long to let
a VM run for?

One solution would be to introduce a scheduling parameter that will
tell the scheduler how long to set the preemption timer for.  Then if
an administrator knows he's running a mixed-workload VM, he can
shorten it down; or if he knows he's running a cpu-cruncher, he can
make it longer.  This would also be useful in verifying the logic of
"shorter timeslices -> less latency for mixed workloads"; i.e,. we
could vary this number and see the effects.

One issue with adding this to the credit1 scheduler is that the
credit1 scheduler is that there are only 3 priorities (BOOST, UNDER,
and OVER), and scheduling is round-robin within each priority.  It's a
known issue with round-robin scheduling that tasks which yield (or are
preempted soon) are discriminated against compared to tasks which use
up their full timeslice (or are preempted less soon).  So there
results may not be representative.

The next step would be to try to get the scheduler to determine the
latency characteristics of a VM automatically.  The key observation
here is that most of the time, latency-sensitive operations are
initiated with an interrupt; or to put it the other way, a pending
interrupt generally means that there is a latency sensitive operation
waiting to happen.  My idea was to have the scheduler look at the
historical rate of interrupts and determine a preemption timeslice
based on those, such that on average, the VM's credit would be enough
to run just when the next interrupt arrived for it to handle.

It occurs to me now that after a certain point, interrupts themselves
become inefficient and drivers sometimes go into "polling" mode, which
would look to the scheduler the same as cpu-bound.  Hmm... bears
thinking about. :-)

Anyway, that's where I got in my thinking on this. Let me know what
you think. :-)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel