[PATCH] design: design doc for shared memory on a dom0less system

[Penny Zheng <penny.zheng@xxxxxxx>]

This commit provides a design doc for static shared memory
on a dom0less system.

Signed-off-by: Penny Zheng <penny.zheng@xxxxxxx>
---
 design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 design/shm-dom0less.md

diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
new file mode 100644
index 0000000..b46199d
--- /dev/null
+++ b/design/shm-dom0less.md
@@ -0,0 +1,182 @@
+# Static Shared Memory between domains on a dom0less system
+
+This design aims to provide an overview of the new feature: setting up static
+shared memory between domains on a dom0less system, through device tree
+configuration.
+
+The new feature is driven by the need of finding a way to build up
+communication channels on dom0less system, since the legacy ways including
+grant table, etc are all absent there.
+
+It was inspired by the patch serie of "xl/libxl-based shared memory", see
+[1] for more details.
+
+# Static Shared Memory Device Tree Configuration
+
+The static shared memory device tree nodes allow users to statically set up
+shared memory among a group of dom0less DomUs and Dom0, enabling domains
+to do shm-based communication.
+
+- compatible
+
+    "xen,domain-shared-memory-v1"
+
+- xen,shm-id
+
+    An u32 value represents the unique identifier of the shared memory region.
+    User valuing per shared memory region shall follow the ascending order,
+    starting from xen,shm-id = <0x0>, to the maximum identifier
+    xen,shm-id = <0x126>. The special xen,shm-id = <0x127> is reserved for
+    INVALID_SHMID.
+
+- xen,shared-mem
+
+    An array takes a physical address, which is the base address of the
+    shared memory region in host physical address space, a size, and a guest
+    physical address, as the target address of the mapping.
+
+- role(Optional)
+
+    A string property specifying the ownership of a shared memory region,
+    the value must be one of the following: "owner", or "borrower"
+    A shared memory region could be explicitly backed by one domain, which is
+    called "owner domain", and all the other domains who are also sharing
+    this region are called "borrower domain".
+    If not specified, the default value is "borrower" and owner is
+    "dom_shared", a system domain.
+
+## Example
+
+chosen {
+    #address-cells = <0x1>;
+    #size-cells = <0x1>;
+    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
+
+    ......
+
+    /* this is for Dom0 */
+    dom0-shared-mem@10000000 {
+        compatible = "xen,domain-shared-memory-v1";
+        xen,shm-id = <0x0>;
+        role = "owner";
+        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
+    }
+
+    domU1 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 131072>;
+        cpus = <2>;
+        vpl011;
+
+        /*
+         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
+         * shared between dom0.
+         */
+        domU1-shared-mem@10000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x0>;
+            role = "borrower";
+            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
+        }
+
+        domU1-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
+        }
+
+        ......
+
+    };
+
+    domU2 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 65536>;
+        cpus = <1>;
+
+        /*
+         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+         * shared between domU1.
+         */
+        domU2-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
+        }
+
+        ......
+    };
+};
+
+It is the example of two static shared memory regions.
+
+In terms of shared memory region identified as 0x0, host physical address
+starting at 0x10000000 of 256MB will be reserved to be shared between Dom0
+and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address
+space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is the
+owner domain, and domU1 is the borrower domain.
+
+And in terms of shared memory region identified as 0x1, host physical address
+starting at 0x50000000 of 512MB will be reserved to be shared between DomU1
+and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address
+space, and at 0x70000000 in DomU2 guest physical address space. Since no owner
+domain is explicitly defined, the default "dom_shared" is the owner domain,
+and both domU1 and domU2 are the borrower domains.
+
+# Overview of Static Shared Memory Flow
+
+Static Shared Memory working flow could be classified into the following
+steps:
+ - Carve out a range of memory in host physical address space to be used
+for sharing. Define it in device tree configuration, then parse and reserve
+it to avoid other use.
+ - Create a special domain "dom_shared". It will be the owner domain which
+is owning the statically shared pages, if "role" property is not specified.
+ - Per shared memory region could be shared with multiple domains. For
+owner domain, it acquires statically shared pages and assign them to itself,
+in the same way with static memory. And other than owner domain, the others
+who are also sharing are called "borrower domain", for which foreign memory
+map of statically shared pages is required.
+ - Expose the shared memory to the domU using the "xen,shared-memory-v1"
+reserved-memory binding. See
+Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
+in Linux for the corresponding device tree binding.
+
+# Memory management of Shared Memory Region
+
+Each memory page needs to have an "owner" and it is likely that in many cases
+the user don't care who the owner is, so it makes sense that users don't
+need to specify the "role" in device tree if they don't want to, in which
+scenario, a default domain shall be the owner domain.
+
+We propose a new system domain "dom_shared" to be the default domain owning all
+statically unowned shared pages, assigning it dom_id 0x7FF5(DOMID_SHARED).
+
+"dom_shared" domain shall get constructed before domain construction and after
+"setup_virt_paging", during system boot-time, so it could successfully do
+p2m initialization.
+
+Owner domain acquires statically shared pages and assign them to itself,
+while borrower domains get and take reference of them, then do foreign memory
+map of these statically shared pages.
+
+When destroying or rebooting a domain, if it is a borrower domain, other than
+removing foreign memory map of statically shared pages in P2M table, we also
+need to drop according gained reference. And if it is an owner domain, since
+statically shared pages are allocated as guest normal ram, it is not needed to
+do extra removing.
+
+However if owner domain is not the default "dom_shared" domain, but specified
+explicitly in device tree, stopping itself will make shared memory region
+unaccessible to all borrower domains, so we need to remove foreign memory map
+for all borrower domains. Notice that all borrowers domains should be stopped
+before stopping the owner domain.
+
+"dom_shared" domain is destroyed when the whole system shuts down, so its
+owning statically shared pages are only freed at system shutdown.
+
+[1] https://marc.info/?l=xen-devel&m=154404821731186
-- 
2.25.1