Re: [Xen-devel] Notes on stubdoms and latency on ARM

[Volodymyr Babchuk <vlad.babchuk@xxxxxxxxx>]

Hi George,

On 19 June 2017 at 02:37, George Dunlap <george.dunlap@xxxxxxxxxx> wrote:
>>>>> There is no way out: if the stubdom needs events, then we'll have to
>>>>> expose and context switch the vGIC. If it doesn't, then we can skip the
>>>>> vGIC. However, we would have a similar problem with EL0 apps: I am
>>>>> assuming that EL0 apps don't need to handle interrupts, but if they do,
>>>>> then they might need something like a vGIC.
>>>> Hm. Correct me, but if we want make stubdom to handle some requests
>>>> (e.g. emulate MMIO access), then it needs events, and thus it needs
>>>> interrupts. At least, I'm not aware about any other mechanism, that
>>>> allows hypervisor to signal to a domain.
>>>> On other hand, EL0 app (as I see them) does not need such events.
>>>> Basically, you just call function `handle_mmio()` right in the app.
>>>> So, apps can live without interrupts and they still be able to handle
>>>> request.
>>>
>>> So remember that "interrupt" and "event" are basically the same as
>>> "structured callback".  When anything happens that Xen wants to tell the
>>> EL0 app about, it has to have a way of telling it.  If the EL0 app is
>>> handling a device, it has to have some way of getting interrupts from
>>> that device; if it needs to emulate devices sent to the guest, it needs
>>> some way to tell Xen to deliver an interrupt to the guest.
>> Basically yes. There should be mechanism to request something from
>> native application. Question is how this mechanism can be implemented.
>> Classical approach is a even-driven loop:
>>
>> while(1) {
>>     wait_for_event();
>>     handle_event_event();
>>     return_back_results();
>> }
>>
>> wait_for_event() can by anything from WFI instruction to read() on
>> socket. This is how stubdoms are working. I agree with you: there are
>> no sense to repeat this in native apps.
>>
>>> Now, we could make the EL0 app interface "interruptless".  Xen could
>>> write information about pending events in a shared memory region, and
>>> the EL0 app could check that before calling some sort of block()
>>> hypercall, and check it again when it returns from the block() call.
>>
>>> But the shared event information starts to look an awful lot like events
>>> and/or pending bits on an interrupt controller -- the only difference
>>> being that you aren't interrupted if you're already running.
>>
>> Actually there are third way, which I have used. I described it in
>> original email (check out [1]).
>> Basically, native application is dead until it is needed by
>> hypervisor. When hypervisor wants some services from app, it setups
>> parameters, switches mode to EL0 and jumps at app entry point.
>
> What's the difference between "jumps to an app entry point" and "jumps
> to an interrupt handling routine"?
"Jumps to an app entry point" and "Unblocks vcpu that waits for an
interrupt". That would be more precise. There are two differences:
first approach is synchronous, no need to wait scheduler to schedule
vcpu. Also vGIC code can be omitted, which decreases switch latency.


>  And what's the difference between
> "Tells Xen about the location of the app entry point" and "tells Xen
> about the location of the interrupt handling routine"?
There are no difference at all.

> If you want this "EL0 app" thing to be able to provide extra security
> over just running the code inside of Xen, then the code must not be able
> to DoS the host by spinning forever instead of returning.
Right. This is a problem. Fortunately, it is running with interrupts
enabled, so next timer tick will switch back to XEN. There you can
terminate app which is running too long.

> What happens if two different pcpus in Xen decide they want to activate
> some "app" functionality?
There are two possibilities: we can make app single threaded, then
second pcpu can be assigned with another vcpu until app is busy. But I
don't like this approach.
I think that all apps should be multi threaded. They can use simple
spinlocks to control access to shared resources.

>>> I'm pretty sure you could run in this mode using the existing interfaces
>>> if you didn't want the hassle of dealing with asynchrony.  If that's the
>>> case, then why bother inventing an entirely new interface, with its own
>>> bugs and duplication of functionality?  Why not just use what we already
>>> have?
>> Because we are concerned about latency. In my benchmark, my native app
>> PoC is 1.6 times faster than stubdom.
>
> But given the conversation so far, it seems likely that that is mainly
> due to the fact that context switching on ARM has not been optimized.
Yes. Question is: can context switching in ARM be optimized more? I don't know.

> Just to be clear -- I'm not adamantly opposed to a new interface similar
> to what you're describing above.  But I would be opposed to introducing
> a new interface that doesn't achieve the stated goals (more secure, &c),
> or a new interface that is the same as the old one but rewritten a bit.
>
> The point of having this design discussion up front is to prevent a
> situation where you spend months coding up something which is ultimately
> rejected.  There are a lot of things that are hard to predict until
> there's actually code to review, but at the moment the "jumps to an
> interrupt handling routine" approach looks unpromising.
Yes, I'm agree with you. This is why I started those mail threads in
the first place. Actually, after all that discussions I stick more to
some sort of lightweight domain-bound stubdoms (without vGICs, for
example). But I want to discuss all possibilities, including native
apps.
Actually, what we really need right now is a hard numbers. I did one
benchmark, but that was ideal use case. I'm going to do more
experiments: with 1 or 1.5 active vcpu per pcpu, with p2m context
switch stripped off, etc.


-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@xxxxxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel