Re: [PATCH v4 00/11] direct-map memory map

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Hi Penny,

Thanks for the update. I tested the series in a couple of different
configurations and it works great!

You can add my Tested-by to all patches


On Mon, 20 Dec 2021, Penny Zheng wrote:
> Cases where domU needs direct-map memory map:
>   * IOMMU not present in the system.
>   * IOMMU disabled if it doesn't cover a specific device and all the guests
> are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
> a few without, then guest DMA security still could not be totally guaranteed.
> So users may want to disable the IOMMU, to at least gain some performance
> improvement from IOMMU disabled.
>   * IOMMU disabled as a workaround when it doesn't have enough bandwidth.
> To be specific, in a few extreme situation, when multiple devices do DMA
> concurrently, these requests may exceed IOMMU's transmission capacity.
>   * IOMMU disabled when it adds too much latency on DMA. For example,
> TLB may be missing in some IOMMU hardware, which may bring latency in DMA
> progress, so users may want to disable it in some realtime scenario.
>   * Guest OS relies on the host memory layout
> 
> "direct-map" property shall be added under the appropriate domain node,
> when users requesting direct-map memory mapping for the domain.
> 
> Right now, direct-map is only supported when domain on Static Allocation,
> that is, "xen,static-mem" is also necessary in the domain configuration.
> 
> Looking into related [design link](
> https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html)
> for more details.
> 
> The whole design is about Static Allocation and direct-map, and this
> Patch Serie only covers parts of it, which are direct-map memory map.
> Other features will be delievered through different patch series.
> 
> See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html
> for Domain on Static Allocation.
> 
> This patch serie is based on
> https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html\
> ---
> v4 changes:
> - introduce internal const CDF_xxx flags for domain creation
> - introduce internal flag CDF_privileged
> - introduce new internal flag CDF_directmap
> - add a directmap flag under struct arch_domain and use it to
> reimplement is_domain_direct_mapped.
> - expand arch_domain_create/domain_create to include internal-only parameter
> "const unsigned int flags"
> - use mfn_eq() instead, because it is the only value used to indicate
> there is an error and this is more lightweight than mfn_valid()
> - rename function allocate_static_memory_11() to assign_static_memory_11()
> to make clear there is actually no allocation done. Instead we are only
> mapping pre-defined host regions to pre-defined guest regions.
> - remove tot_size to directly substract psize from kinfo->unassigned_mem
> - check kinfo->unassigned_mem doesn't underflow or overflow
> - remove nested if/else
> - remove ASSERT_UNREACHABLE() to avoid breaking compilation on prod build with
> CONFIG_GICV3=n
> - comment and commit message refinement
> ---
> v3 changes:
> - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to 
> xen/include/xen/domain.h,
> to let it be only available for domain created by XEN.
> - name it with extra "INTERNAL" and add comments to warn developers not
> to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
> - reject this flag in x86'es arch_sanitise_domain_config()
> - add ASSERT_UNREACHABLE to catch any misuse in allocate_static_memory()
> and allocate_static_memory_11()
> - add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only
> when CONFIG_STATIC_MEMORY is set.
> - simply map the CPU interface at the GPA vgic_v2_hw.cbase
> - drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)'
> - rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()'
> ---
> v2 changes:
> - remove the introduce of internal flag
> - Refine is_domain_direct_mapped to check whether the flag
> XEN_DOMCTL_CDF_directmap is set
> - reword "1:1 direct-map" to just "direct-map"
> - split the common codes into two helpers: parse_static_mem_prop and
> acquire_static_memory_bank to deduce complexity.
> - introduce a new helper allocate_static_memory_11 for allocating static
> memory for direct-map guests
> - remove panic action since it is fine to assign a non-DMA capable device when
> IOMMU and direct-map both off
> - remove redistributor accessor
> - introduce new helper "is_domain_use_host_layout()"
> - explain why vpl011 initialization before creating its device tree node
> - error out if the domain is direct-mapped and the IRQ is not found
> - harden the code and add a check/comment when the hardware UART region
> is smaller than CUEST_VPL011_SIZE.
> Penny Zheng (4):
>   xen/arm: introduce new helper parse_static_mem_prop and
>     acquire_static_memory_bank
>   xen/arm: introduce direct-map for domUs
>   xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory
>   xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3
> 
> Stefano Stabellini (7):
>   xen: introduce internal CDF_xxx flags for domain creation
>   xen: introduce CDF_directmap
>   xen/arm: avoid setting XEN_DOMCTL_CDF_iommu when IOMMU off
>   xen/arm: if direct-map domain use native addresses for GICv2
>   xen/arm: if direct-map domain use native addresses for GICv3
>   xen/arm: if direct-map domain use native UART address and IRQ number
>     for vPL011
>   xen/docs: Document how to do passthrough without IOMMU
> 
>  docs/misc/arm/device-tree/booting.txt |   6 +
>  docs/misc/arm/passthrough-noiommu.txt |  52 +++++
>  xen/arch/arm/domain.c                 |   5 +-
>  xen/arch/arm/domain_build.c           | 308 +++++++++++++++++++++-----
>  xen/arch/arm/include/asm/domain.h     |  19 +-
>  xen/arch/arm/include/asm/new_vgic.h   |  10 +
>  xen/arch/arm/include/asm/vgic.h       |  11 +
>  xen/arch/arm/include/asm/vpl011.h     |   2 +
>  xen/arch/arm/vgic-v2.c                |  34 ++-
>  xen/arch/arm/vgic-v3.c                |  26 ++-
>  xen/arch/arm/vgic/vgic-v2.c           |  34 ++-
>  xen/arch/arm/vpl011.c                 |  60 ++++-
>  xen/arch/x86/domain.c                 |   3 +-
>  xen/arch/x86/setup.c                  |   2 +-
>  xen/common/domain.c                   |  12 +-
>  xen/common/sched/core.c               |   2 +-
>  xen/include/xen/domain.h              |   9 +-
>  xen/include/xen/sched.h               |   2 +-
>  18 files changed, 490 insertions(+), 107 deletions(-)
>  create mode 100644 docs/misc/arm/passthrough-noiommu.txt
> 
> -- 
> 2.25.1
>