[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [RFC PATCH v1 00/16] xen: sched: implement core-scheduling


  • To: Dario Faggioli <dfaggioli@xxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxxx
  • From: Juergen Gross <jgross@xxxxxxxx>
  • Date: Fri, 12 Oct 2018 07:15:11 +0200
  • Autocrypt: addr=jgross@xxxxxxxx; prefer-encrypt=mutual; keydata= xsBNBFOMcBYBCACgGjqjoGvbEouQZw/ToiBg9W98AlM2QHV+iNHsEs7kxWhKMjrioyspZKOB ycWxw3ie3j9uvg9EOB3aN4xiTv4qbnGiTr3oJhkB1gsb6ToJQZ8uxGq2kaV2KL9650I1SJve dYm8Of8Zd621lSmoKOwlNClALZNew72NjJLEzTalU1OdT7/i1TXkH09XSSI8mEQ/ouNcMvIJ NwQpd369y9bfIhWUiVXEK7MlRgUG6MvIj6Y3Am/BBLUVbDa4+gmzDC9ezlZkTZG2t14zWPvx XP3FAp2pkW0xqG7/377qptDmrk42GlSKN4z76ELnLxussxc7I2hx18NUcbP8+uty4bMxABEB AAHNHkp1ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmRlPsLAeQQTAQIAIwUCU4xw6wIbAwcL CQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJELDendYovxMvi4UH/Ri+OXlObzqMANruTd4N zmVBAZgx1VW6jLc8JZjQuJPSsd/a+bNr3BZeLV6lu4Pf1Yl2Log129EX1KWYiFFvPbIiq5M5 kOXTO8Eas4CaScCvAZ9jCMQCgK3pFqYgirwTgfwnPtxFxO/F3ZcS8jovza5khkSKL9JGq8Nk czDTruQ/oy0WUHdUr9uwEfiD9yPFOGqp4S6cISuzBMvaAiC5YGdUGXuPZKXLpnGSjkZswUzY d9BVSitRL5ldsQCg6GhDoEAeIhUC4SQnT9SOWkoDOSFRXZ+7+WIBGLiWMd+yKDdRG5RyP/8f 3tgGiB6cyuYfPDRGsELGjUaTUq3H2xZgIPfOwE0EU4xwFgEIAMsx+gDjgzAY4H1hPVXgoLK8 B93sTQFN9oC6tsb46VpxyLPfJ3T1A6Z6MVkLoCejKTJ3K9MUsBZhxIJ0hIyvzwI6aYJsnOew cCiCN7FeKJ/oA1RSUemPGUcIJwQuZlTOiY0OcQ5PFkV5YxMUX1F/aTYXROXgTmSaw0aC1Jpo w7Ss1mg4SIP/tR88/d1+HwkJDVW1RSxC1PWzGizwRv8eauImGdpNnseneO2BNWRXTJumAWDD pYxpGSsGHXuZXTPZqOOZpsHtInFyi5KRHSFyk2Xigzvh3b9WqhbgHHHE4PUVw0I5sIQt8hJq 5nH5dPqz4ITtCL9zjiJsExHuHKN3NZsAEQEAAcLAXwQYAQIACQUCU4xwFgIbDAAKCRCw3p3W KL8TL0P4B/9YWver5uD/y/m0KScK2f3Z3mXJhME23vGBbMNlfwbr+meDMrJZ950CuWWnQ+d+ Ahe0w1X7e3wuLVODzjcReQ/v7b4JD3wwHxe+88tgB9byc0NXzlPJWBaWV01yB2/uefVKryAf AHYEd0gCRhx7eESgNBe3+YqWAQawunMlycsqKa09dBDL1PFRosF708ic9346GLHRc6Vj5SRA UTHnQqLetIOXZm3a2eQ1gpQK9MmruO86Vo93p39bS1mqnLLspVrL4rhoyhsOyh0Hd28QCzpJ wKeHTd0MAWAirmewHXWPco8p1Wg+V+5xfZzuQY0f4tQxvOpXpt4gQ1817GQ5/Ed/wsDtBBgB CAAgFiEEhRJncuj2BJSl0Jf3sN6d1ii/Ey8FAlrd8NACGwIAgQkQsN6d1ii/Ey92IAQZFggA HRYhBFMtsHpB9jjzHji4HoBcYbtP2GO+BQJa3fDQAAoJEIBcYbtP2GO+TYsA/30H/0V6cr/W V+J/FCayg6uNtm3MJLo4rE+o4sdpjjsGAQCooqffpgA+luTT13YZNV62hAnCLKXH9n3+ZAgJ RtAyDWk1B/0SMDVs1wxufMkKC3Q/1D3BYIvBlrTVKdBYXPxngcRoqV2J77lscEvkLNUGsu/z W2pf7+P3mWWlrPMJdlbax00vevyBeqtqNKjHstHatgMZ2W0CFC4hJ3YEetuRBURYPiGzuJXU pAd7a7BdsqWC4o+GTm5tnGrCyD+4gfDSpkOT53S/GNO07YkPkm/8J4OBoFfgSaCnQ1izwgJQ jIpcG2fPCI2/hxf2oqXPYbKr1v4Z1wthmoyUgGN0LPTIm+B5vdY82wI5qe9uN6UOGyTH2B3p hRQUWqCwu2sqkI3LLbTdrnyDZaixT2T0f4tyF5Lfs+Ha8xVMhIyzNb1byDI5FKCb
  • Cc: Wei Liu <wei.liu2@xxxxxxxxxx>, George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Ian Jackson <ian.jackson@xxxxxxxxxxxxx>, Bhavesh Davda <bhavesh.davda@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>
  • Delivery-date: Fri, 12 Oct 2018 05:15:26 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

On 11/10/2018 19:37, Dario Faggioli wrote:
> Hey,
> 
> Sorry if replying took some time. :-P
> 
> On Fri, 2018-09-07 at 18:00 +0200, Juergen Gross wrote:
>> On 25/08/18 01:35, Dario Faggioli wrote:
>>>
>>> There are git branches here:
>>>  https://gitlab.com/dfaggioli/xen.git rel/sched/core-scheduling-
>>> RFCv1
>>>  https://github.com/fdario/xen.git rel/sched/core-scheduling-RFCv1
>>>
>>> Any comment is more than welcome.
>>
>> Have you thought about a more generic approach?
>>
> I had. And I have thought about it more since this email. :-)
> 
>> Instead of trying to schedule only vcpus of the same domain on a core
>> I'd rather switch form vcpu scheduling to real core scheduling. The
>> scheduler would see guest cores to be scheduled on physical cores. A
>> guest core consists of "guest threads" being vcpus (vcpus are bound
>> to their guest cores, so that part of the topology could even be used
>> by the guest for performance tuning). 
>>
> Right, so I think I got the big picture. And it was something that, as
> I've said above, I've been thinking too, and we've also talked about
> something similar with Andrew in Nanjing.
> 
> I'm still missing how something like this would work in details,
> perhaps because I'm really used to reason within the boundaries of the
> model we currently have.
> 
> So, for example:
> - domain A has vCore0 and vCore1
> - each vCore has 2 threads ({vCore0.0, vCore0.1} and
>   {vCore1.0, vCore1.1})
> - we're on a 2-way SMT host
> - vCore1 is running on physical core 3 on the host
> - more specifically, vCore1.0 is currently executing on thread 0 of
>   physical core 3 of the host, and vCore1.1 is currently executing on
>   thread 1 of core 3 of the host
> - say that both vCore1.0 and vCore1.1 are in guest context
> 
> Now:
> * vCore1.0 blocks. What happens?

It is going to vBlocked (the physical thread is sitting in the
hypervisor waiting for either a (core-)scheduling event or for
unblocking vCore1.0). vCore1.1 keeps running. Or, if vCore1.1
is already vIdle/vBlocked, vCore1 is switching to blocked and the
scheduler is looking for another vCore to schedule on the physical
core.

> * vCore1.0 makes an hypercall. What happens?

Same as today. The hypercall is being executed.

> * vCore1.0 VMEXITs. What happens?

Same as today. The VMEXIT is handled.

In case you referring to a potential rendezvous for e.g. L1TF
mitigation: this would be handled scheduler agnostic.

>> The state machine determining the core state from its vcpus would be
>> scheduler agnostic (schedule.c), same for switching guest cores on a
>> physical core.
>>
> What do you mean with "same for switching guest cores on a physical
> core"?

No per-scheduler handling, but a common scheduler.c function (maybe with
new per-scheduler hooks if needed). So schedule() modified to work on
scheduling entities (threads/cores/sockets).

> All in all, I like the idea, because it is about introducing nice
> abstractions, it is general, etc., but it looks like a major rework of
> the scheduler.

Correct. Finally something to do :-p

> And it's not that I am not up for major reworks, but I'd like to
> understand properly what that is buying us.

I would hope so!

> Note that, while this series which tries to implement core-scheduling
> for Credit1 is rather long and messy, doing the same (and with a
> similar approach) for Credit2 is a lot easier and nicer. I have it
> almost ready, and will send it soon.

Okay, but would it keep vThreads of the same vCore let always running
together on the same physical core?

>> This scheme could even be expanded for socket scheduling.
>>
> Right. But again, in Credit2, I've been able to implement socket-wise
> coscheduling with this approach (I mean, an approach similar to the one
> in this series, but adapted to Credit2).

And then there still is sched_rt.c


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.