Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hello Jan,

Thanks for reviewing the design.

> On 23 Mar 2022, at 4:07 pm, Jan Beulich <jbeulich@xxxxxxxx> wrote:
> 
> On 23.03.2022 16:43, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@xxxxxxx>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> 
>> diff --git a/docs/designs/dom0less-evtchn.md 
>> b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen 
>> guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used 
>> in
>> +normal systems with dynamic VMs to communicate between domains by providing 
>> a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is 
>> based
>> +on event channels.
>> +
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event 
>> notifications.
>> +An event channel is a logical connection between 2 domains (more 
>> specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit 
>> from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
> 
> Nit: Since you're talking of channels here, not ports, strictly speaking
> it's two bits - one on each side. Alternatively instead of "They ..."
> you could say "Each port ...".
> 

Ack. I will correct it in next version.

>> +Notifications are received by a guest via an interrupt from Xen to the 
>> guest,
>> +indicating when an event arrives (setting the bit). Further notifications 
>> are
>> +masked until the bit is cleared again. When a domain wants to wait for data 
>> it
>> +will block until an event arrives, and then send an event to signal that 
>> data
>> +has been consumed. Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> +    "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> +    The property is four numbers of tuples of
>> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
> 
> Nit: I think you mean "tuples of four numbers"?

Yes, you are right I will fix this.
> 
>> +    local-port-domU1 is an integer value that will be used to allocte local
>> +    port for domU1 to send an event notification to the remote domain.
>> +
>> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
>> +    will be allocated.
>> +
>> +    local-port-domU2 is an integer value that will be used to allocte local
>> +    port for domU2 to send an event notification to the remote domain.
>> +
>> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
>> +    will be allocated.
>> +
>> +Example:
>> +
>> +    chosen {
>> +        ....
>> +
>> +        domU1: domU1 {
>> +            ......
>> +        };
>> +
>> +        domU2: domU2 {
>> +            ......
>> +        };
>> +
>> +        evtchn@1 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
>> +        };
>> +
>> +        evtchn@2 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
>> +        };
>> +    };
>> +
>> +In above example two event channel comunication will be established between
>> +domU1 and domU2.
>> +
>> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
>> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
>> +
>> +domU1 and domU2 can send the signal to remote domain via hypercall
>> +EVTCHNOP_send(.) on local port.
> 
> How do the DomU-s learn of these ports? I guess information is to be
> extracted into their individual DT representation, but this could do
> with spelling out, including how those nodes (or however the data
> items are called in DT) would be named and what data they would
> contain.

There are two options how the guest learn the local ports.

1. Document the event channel connection information for the end-user in
    the end user documentation. The integrator will make use of the local port 
information
    to write a userspace application to send events to the remote domain. For 
Linux, in this
    case, we need to modify the current  "drivers/xen/evtchn.c” to include a 
new IOCTL to
    bind the local port to the user and set up the interrupt handler. The 
application can use the ioctl
    IOCTL_STATIC_EVTCHN_BIND and IOCTL_EVTCHN_NOTIFY to notify the remote 
domain.

        case IOCTL_STATIC_EVTCHN_BIND: {
                struct ioctl_evtchn_bind bind;

                rc = -EFAULT;
                if (copy_from_user(&bind, uarg, sizeof(bind)))
                        break;

                rc = evtchn_bind_to_user(u, bind.port);
                evtchn_bind_interdom_next_vcpu(bind.port);
                break;                                                          
        
    }

   We can also implement the new hypercall for error checking if the port 
requested from user application is static port
   and already created in XEN.

2. Create the DT node for guest and pass the local port information to guest. 
Guest can
    make use this DT node to create device node “ /dev/xen/eventchn-localport" 
or something else in Linux.
    In this case we may need to write the new driver in linux to support this.
 
> 
> Further I assume this being encoded in the DT passed to Xen means
> implicit consent of the admin security-wise. If so, this could do
> with making explicit as well. If not, the security of this would
> need discussing in even broader a scope.

Yes, it is responsibility of the admin to take care of security when 
defining the event channel in DT.
> 
> Finally I assume such channels are established fully bound, i.e.
> there's nothing to do by guests in order to establish communication.
> This is another aspect that would imo better be said explicitly.
> 

Yes, you are right event channel will be created and established beforehand in 
XEN before
guests started. The guest doesn’t need to do any operation to establish a 
connection. Guest 
only needs hypercall EVTCHNOP_send(local port) to send notifications to the 
remote guest.

Regards,
Rahul
> Jan