[Xen-devel] [PATCH RFC v1 00/74] Run PV guest in PVH container

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Hi all

This is a patch series to run PV guest inside a PVH container. The series is
still in a very RFC state. We're aware that some code is not very clean yet and
in the process of cleaning things up.

The series can be found at:

    https://xenbits.xen.org/git-http/people/liuw/xen.git wip.pvshim-rfc-v1

The basic idea can be found at page 15 of the slides at [0].

This is a mitigation against one of the CPU vulnerabilities disclosed recently.
This series makes it possible to continue running untrusted PV guests.  Please
refer to XSA-254 [1] for more information.

Given the embargo lifted and vulnerabilities disclosed we opt to develop openly
on xen-devel. Feedback and testing is very welcome.

The series is split into three parts: The first part is for the host that runs
the shim, the second part is for the shim itself, the third part is for
toolstack patches (not yet fully working). See the markers in the list of
patches.

Instructions on using the PV shim:

1. Git clone the branch and configure as one normally would.
2. A xen-shim binary would be built and installed into Xen's firmware
   directory, along side hvmloader and co.
3. Use the hacky way currently provided in the first part of the series to
   boot a PV guest inside a PVH container:
   a. Append type='pvh' in your PV guest config file;
   b. Export two environment variables so that libxl knows where to find
      the shim and what to add to the shim's command line option.
      # export LIBXL_PVSHIM_PATH=$PATH_TO_XEN_SHIM
      # export LIBXL_PVSHIM_CMDLINE="pv-shim console=xen,pv loglvl=all 
guest_loglvl=all apic_verbosity=debug e820-verbose sched=null"
4. xl create -c guest.cfg

You should be able to see some Xen messages first and then guest kernel
messages (the console= shim paramter is required).

Known issues:

1. ARM build and some Clang build are broken by this series.
2. The host will see a lot over-allocation messages, nothing too harmful and
   will be fixed once toolstack is ready.

Wei.

[0] 
https://www.slideshare.net/xen_com_mgr/xpdds17-keynote-towards-a-configurable-and-slimmer-x86-hypervisor-wei-liu-citrix
[1] https://xenbits.xen.org/xsa/advisory-254.html

# Patches for the host:

448f56a363 x86/svm: Offer CPUID Faulting to AMD HVM guests as well
6a78c9ae33 x86: Common cpuid faulting support
05844fec44 x86/upcall: inject a spurious event after setting upcall vector
fc7a48dd74 tools/libxc: initialise hvm loader elf log fd to get more logging
522c9cbaf0 tools/libxc: remove extraneous newline in xc_dom_load_acpi
bd6b572b32 tools/libelf: fix elf notes check for PVH guest
449b932b0c tools/libxc: Multi modules support
cc6dbdc0c1 libxl: Introduce hack to allow PVH mode to add a shim

# Patches for the shim:

8ffbad2060 xen/common: Widen the guest logging buffer slightly
7fc883a17e x86/time: Print a more helpful error when a platform timer can't be 
found
0575f913c1 x86/link: Introduce and use SECTION_ALIGN
62d8196e3b xen/acpi: mark the PM timer FADT field as optional
789db028db xen/domctl: Return arch_config via getdomaininfo
97f38ec4b4 tools/ocaml: Expose arch_config in domaininfo
6da8e1993c tools/ocaml: Extend domain_create() to take arch_domainconfig
0b1f990b03 x86/fixmap: Modify fix_to_virt() to return a void pointer
014520b9d0 ---- x86/Kconfig: Options for Xen and PVH support
62a4c07bfe x86/link: Relocate program headers
3b5a699018 x86: introduce ELFNOTE macro
802fbc7aff x86: produce a binary that can be booted as PVH
782cebfce5 x86/entry: Early PVH boot code
cdc718e5e0 x86/boot: Map more than the first 16MB
025e19e9a5 x86/entry: Probe for Xen early during boot
53aadd1db4 x86/guest: Hypercall support
86aeb49671 x86/shutdown: Support for using SCHEDOP_{shutdown,reboot}
145d616bed x86/pvh: Retrieve memory map from Xen
40bf7f9323 xen/console: Introduce console=xen
c7ad734481 x86: initialise shared_info page
a6940a5516 x86: xen pv clock time source
b69ce8f6eb x86: APIC timer calibration when running as a guest
e2e3665ea3 x86: read wallclock from Xen running in pvh mode
b01d7d338e x86: don't swallow the first command line item in pvh mode
aa1937c569 x86/guest: enable event channels upcalls
9e9c06e556 x86/guest: add PV console code
a87b4fd510 x86/guest: use PV console for Xen/Dom0 I/O
b67edf6cdb --- x86/shim: Kconfig and command line options
ce622d9384 tools/firmware: Build and install xen-shim
c1b1c473b8 x86/pv-shim: Force CPUID faulting in pv-shim mode
576f4be4b9 xen/x86: make VGA support selectable
7dbc3f25f6 xen/x86: report domain id on cpuid
c95b5e63eb xen/pvh: do not mark the low 1MB as IO mem
cc7023e59b sched/null: skip vCPUs on the waitqueue that are blocked
4fcc995c14 xen: introduce rangeset_reserve_hole
22d931454f xen/pvshim: keep track of unused pages
f32f82c439 x86/guest: use unpopulated memory to map the shared_info page
638d3ae921 xen/guest: fetch vCPU ID from Xen
2fb52effec x86/guest: fix upcall vector setup
2ec939a35e x86/guest: unmask console event channel
e57a7c3173 x86/guest: map per-cpu vcpu_info area.
d2a1878ac6 xen/pvshim: remove Dom0 kernel support check
4e898f8c00 xen/pvshim: don't allow access to iomem or ioports
4dff8efebd xen: mark xenstore/console pages as RAM and add them to dom_io
5b6a4b069b xen/pvshim: modify Dom0 builder in order to build a DomU
b865a91f1d xen/pvshim: set correct domid value
f31e0cd535 xen/pvshim: forward evtchn ops between L0 Xen and L2 DomU
97bda4f904 xen/pvshim: add grant table operations
e37cf1367a x86/pv-shim: shadow PV console's page for L2 DomU
8cc21ecc09 xen/pvshim: add migration support
a7ff975bb3 xen/pvshim: add shim_mem cmdline parameter
bcdc208ad2 xen/pvshim: set max_pages to the value of tot_pages
a5b1f98a15 xen/pvshim: support vCPU hotplug
6be1b4e645 xen/pvshim: memory hotplug
7ce9abb399 xen/shim: modify shim_mem parameter behaviour
7952196d0c xen/pvshim: use default position for the m2p mappings
6cbe2150d3 xen/shim: crash instead of reboot in shim mode
8667344fa4 xen/shim: allow DomU to have as many vcpus as available

# Patches for toolstack (not yet fully working):

8faff727c4 libxl: libxl__build_hvm: Introduce separate b_info parameter
4b5a346b96 libxl__domain_build_info_setdefault_pvhhvm: introduce
dfcaf56b00 libxl_bitmap_copy_alloc: copy 0,NULL as 0,NULL
a9b73202d1 libxl: pvshim: Check state->shim_path before domain type
48a6a2217a libxl: pvshim: Provide first-class config settings to enable shim 
mode
e40e08c4fd libxl: pvshim: Introduce pvhshim_extra
c6bd9ca574 xl: pvshim: Provide and document xl config
d3e5c3a0d6 libxl: pvshim: Set video_memkb to ~0

 .gitignore                            |   5 +
 docs/man/xl.cfg.pod.5.in              |  28 +
 docs/misc/xen-command-line.markdown   |  36 +-
 stubdom/grub/kexec.c                  |   7 +-
 tools/firmware/Makefile               |  11 +
 tools/firmware/xen-dir/Makefile       |  59 +++
 tools/firmware/xen-dir/shim.config    |  86 +++
 tools/helpers/init-xenstore-domain.c  |   4 +-
 tools/libxc/include/xc_dom.h          |  48 +-
 tools/libxc/include/xenctrl.h         |   1 +
 tools/libxc/xc_dom_compat_linux.c     |   2 +-
 tools/libxc/xc_dom_core.c             | 154 ++++--
 tools/libxc/xc_dom_hvmloader.c        |   1 +
 tools/libxc/xc_dom_x86.c              |  65 +--
 tools/libxc/xc_domain.c               |   1 +
 tools/libxl/libxl.h                   |   8 +
 tools/libxl/libxl_create.c            |  65 ++-
 tools/libxl/libxl_dom.c               |  73 ++-
 tools/libxl/libxl_internal.h          |   7 +
 tools/libxl/libxl_types.idl           |   4 +
 tools/libxl/libxl_utils.c             |   8 +-
 tools/ocaml/libs/xc/xenctrl.ml        |  31 +-
 tools/ocaml/libs/xc/xenctrl.mli       |  30 +-
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  48 +-
 tools/xl/xl_parse.c                   |  11 +
 xen/Makefile                          |   8 +-
 xen/arch/x86/Kconfig                  |  40 +-
 xen/arch/x86/Makefile                 |  11 +-
 xen/arch/x86/acpi/lib.c               |   2 +-
 xen/arch/x86/apic.c                   |  38 +-
 xen/arch/x86/boot/build32.mk          |   1 +
 xen/arch/x86/boot/cmdline.c           |   5 +-
 xen/arch/x86/boot/head.S              |  48 ++
 xen/arch/x86/boot/trampoline.S        |   7 +
 xen/arch/x86/boot/x86_64.S            |   5 +-
 xen/arch/x86/cpu/amd.c                |  16 +-
 xen/arch/x86/cpu/common.c             |  78 ++-
 xen/arch/x86/cpu/intel.c              |  81 +--
 xen/arch/x86/dom0_build.c             |  50 +-
 xen/arch/x86/domctl.c                 |   2 +
 xen/arch/x86/e820.c                   |   7 +-
 xen/arch/x86/efi/efi-boot.h           |   4 +
 xen/arch/x86/guest/Makefile           |   4 +
 xen/arch/x86/guest/hypercall_page.S   |  79 +++
 xen/arch/x86/guest/pvh-boot.c         | 140 +++++
 xen/arch/x86/guest/xen.c              | 401 ++++++++++++++
 xen/arch/x86/hvm/hvm.c                |   1 +
 xen/arch/x86/hvm/irq.c                |   4 +
 xen/arch/x86/hvm/svm/svm.c            |   6 +
 xen/arch/x86/mm.c                     |  19 +-
 xen/arch/x86/mpparse.c                |   2 +-
 xen/arch/x86/msi.c                    |   3 +-
 xen/arch/x86/msr.c                    |   3 +-
 xen/arch/x86/platform_hypercall.c     |   2 +
 xen/arch/x86/pv/Makefile              |   1 +
 xen/arch/x86/pv/dom0_build.c          |  49 +-
 xen/arch/x86/pv/shim.c                | 955 ++++++++++++++++++++++++++++++++++
 xen/arch/x86/setup.c                  |  86 ++-
 xen/arch/x86/shutdown.c               |  41 +-
 xen/arch/x86/smpboot.c                |   4 +
 xen/arch/x86/tboot.c                  |   4 +-
 xen/arch/x86/time.c                   | 132 ++++-
 xen/arch/x86/traps.c                  |   1 +
 xen/arch/x86/xen.lds.S                |  82 ++-
 xen/common/compat/grant_table.c       |   5 +
 xen/common/domain.c                   |  54 +-
 xen/common/event_channel.c            | 100 ++--
 xen/common/grant_table.c              |  10 +
 xen/common/libelf/libelf-dominfo.c    |   9 +-
 xen/common/memory.c                   |  14 +
 xen/common/page_alloc.c               |  15 +
 xen/common/rangeset.c                 |  51 ++
 xen/common/sched_null.c               |  11 +-
 xen/common/schedule.c                 |   3 +-
 xen/drivers/acpi/apei/apei-io.c       |   2 +-
 xen/drivers/acpi/tables/tbfadt.c      |   2 +-
 xen/drivers/char/Makefile             |   2 +
 xen/drivers/char/console.c            |  49 +-
 xen/drivers/char/consoled.c           | 145 ++++++
 xen/drivers/char/ehci-dbgp.c          |   2 +-
 xen/drivers/char/ns16550.c            |   2 +-
 xen/drivers/char/xen_pv_console.c     | 205 ++++++++
 xen/drivers/video/Kconfig             |   8 +-
 xen/include/asm-x86/apicdef.h         |   2 +-
 xen/include/asm-x86/asm_defns.h       |  12 +
 xen/include/asm-x86/cpuid.h           |   3 -
 xen/include/asm-x86/dom0_build.h      |   4 +
 xen/include/asm-x86/e820.h            |   1 +
 xen/include/asm-x86/fixmap.h          |   6 +-
 xen/include/asm-x86/guest.h           |  37 ++
 xen/include/asm-x86/guest/hypercall.h | 202 +++++++
 xen/include/asm-x86/guest/pvh-boot.h  |  57 ++
 xen/include/asm-x86/guest/xen.h       | 111 ++++
 xen/include/asm-x86/processor.h       |   4 +-
 xen/include/asm-x86/pv/shim.h         | 124 +++++
 xen/include/asm-x86/setup.h           |   6 +
 xen/include/public/arch-x86/cpuid.h   |   3 +-
 xen/include/public/domctl.h           |   1 +
 xen/include/xen/consoled.h            |  27 +
 xen/include/xen/domain.h              |   1 +
 xen/include/xen/event.h               |  15 +
 xen/include/xen/pv_console.h          |  38 ++
 xen/include/xen/rangeset.h            |   4 +
 xen/include/xen/sched.h               |   6 +-
 104 files changed, 4153 insertions(+), 415 deletions(-)
 create mode 100644 tools/firmware/xen-dir/Makefile
 create mode 100644 tools/firmware/xen-dir/shim.config
 create mode 100644 xen/arch/x86/guest/Makefile
 create mode 100644 xen/arch/x86/guest/hypercall_page.S
 create mode 100644 xen/arch/x86/guest/pvh-boot.c
 create mode 100644 xen/arch/x86/guest/xen.c
 create mode 100644 xen/arch/x86/pv/shim.c
 create mode 100644 xen/drivers/char/consoled.c
 create mode 100644 xen/drivers/char/xen_pv_console.c
 create mode 100644 xen/include/asm-x86/guest.h
 create mode 100644 xen/include/asm-x86/guest/hypercall.h
 create mode 100644 xen/include/asm-x86/guest/pvh-boot.h
 create mode 100644 xen/include/asm-x86/guest/xen.h
 create mode 100644 xen/include/asm-x86/pv/shim.h
 create mode 100644 xen/include/xen/consoled.h
 create mode 100644 xen/include/xen/pv_console.h

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel