[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

Hi Andrei,

On 17/12/2019 17:20, Andrei Cherechesu wrote:
On Mon, 16 Dec 2019, Julien Grall wrote:
On 16/12/2019 23:05, Stefano Stabellini wrote:
On Mon, 16 Dec 2019, Julien Grall wrote:
On 16/12/2019 18:02, Andrei Cherechesu wrote:
But even with this patch, RAM in DomU is not direct mapped (i.e Guest
Address == Host Physical Address). This means that DMA-capable device
not work properly in DomU.

We could theoritically map DomU direct mapped, but this would break the
isolation provided by the hypervisor.

Yes, being able to map the DomU memory 1:1 can be pretty useful for some
very embedded dom0less configurations, in fact I was surprised that a
couple of Xilinx users asked me for that recently. Typically, the users
are aware of the consequences but they still find them better than the
alternative (i.e. the lack of isolation is bad but is tolerable in their
This does not make much sense... The whole point of a hypervisor is to isolate
guest between each other... So if you are happy with the lack of isolation,
then why are you using an hypervisor at the first place?

There are a number of reasons, although they are all variation of the
same theme. In all these cases the IOMMU cannot be used for one reason
or the other (a device is not behind the IOMMU, or due to an errata,

- multiple baremetal apps
The user wants to run two or more baremetal (unikernel-like)
applications. The user owns both applications and she is not much
concerned about isolation (although it is always desirable when

- multiple OSes
This is similar to the one before, however, instead of multiple
baremetal apps, we are talking about multiple full OSes. For instance,
Linux and Android or Linux and VxWorks. Again, they are both maintained
by the same user (no multi-tenancy) so isolation is desirable but it is
not the top concern.

- real-time / no real-time
The user wants to run a real-time OS or real-time baremetal app and a
non real-time OS. For instance a tiny baremetal app controlling one
specific device and Linux. Again, the user is responsible for both
systems so isolation is not a concern.

In all these cases the users has to run multiple OSes or baremetal apps
so she needs a hypervisor. However, it is tolerable that the apps are
not actually fully isolated from each others because they are both
developed and deployed together by the same "owner".

Basically, since we do not have an IOMMU, we would be able
to ensure memory isolation via a NXP IP named xRDC (Extended
Resource Domain Controller) that our boards have, which supervises
the access to memory buses.

Ok, so you have some sort of MPU. I assume this will be between the devices and the memory, am I right?

But before we get to think about isolation, we need to enable
basic passthrough functionality (via 1:1 mapping, since no IOMMU).

So you are in better place than what Stefano described. Your use case is probably the only place where a 1:1 mapping would be warrant as isolation is still provided by the HW.

Firstly, a good step forward would be to get any non-DMA-capable
device passed-through and working.
I rebased onto upstream/staging branch and applied the hack
that skips the setting of XEN_DOMCTL_CDF_iommu flag,
that Julien specified.

Then I tried to passthrough the eMMC, but I got the following
(XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request region 
for resource [mem 0x4005d000-0x4005dfff]
(XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc: sdhci_pltfm_init 
failed -16
(XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed with 
error -16

Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the DT.
It seems that the DomU1 kernel does not have access to that memory zone.

Could you paste your partial Device-Tree and domain node?

I'm trying to passthrough the eMMC in order to mount DomU1's root
on a SDCard partition, because I couldn't get to DomU1's Linux prompt
when I tried to boot with a ramdisk module. I always get this error:
(XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image starting 
at 0.

How did you pass the ramdisk to dom1?

Could this be because the ramdisk is too big? The smallest I've tried with
Is approximately 60MB in size. What size are the ramdisks that you
are using in your dom0less booting demos?

How much memory did you give to your guest?


I'll gladly write the patch if you give me some basic
instructions regarding it, because I'm not that familiar with
all the Xen internal mechanisms, and I wouldn't know where
to look in order to ensure everything is properly done.

I am going to suggest a quick and dirty way but it should get you to the point where 1:1 mapping will work in basic use case:

1) Update the guest memory map in xen/include/public/arch-arm.h (see GUEST_*) so all the regions don't overlap your RAM. The best way would be to re-use the same address for the GIC and UART as the host.

2) Allocate the memory bank 1:1. The patch below should do the job (not compiled nor tested):

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index dd9c3b73ba..9155728640 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -374,6 +374,7 @@ static void __init allocate_memory_11(struct domain *d,

+#if 0
 static bool __init allocate_bank_memory(struct domain *d,
                                         struct kernel_info *kinfo,
                                         gfn_t sgfn,
@@ -472,6 +473,7 @@ fail:
           " %ldKB unallocated. Fix the VMs configurations.\n",
           (unsigned long)kinfo->unassigned_mem >> 10);

static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
                                    const struct dt_device_node *node)
@@ -2434,7 +2436,7 @@ static int __init construct_domU(struct domain *d,
     /* type must be set before allocate memory */
     d->arch.type = kinfo.type;
-    allocate_memory(d, &kinfo);
+    allocate_memory_11(d, &kinfo);

     rc = prepare_dtb_domU(d, &kinfo);
     if ( rc < 0 )

If you want to use Xen hypercalls such as for ballooning, you can look at the macro is_domain_direct_mapped(). This will most likely require change in your guest OS because any hypercall modifying memory will not keep the 1:1 assumption. Therefore the OS has to keep track of it. This is a bit tedious but already exists in Linux as we use it for Dom0.

For a more generic approach, you would need to replace the static version suggested 1) with a more dynamic layout that could be based on the host memory layout.

Best regards,

Julien Grall

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.