Re: Virtio on Xen with Rust

[Viresh Kumar <viresh.kumar@xxxxxxxxxx>]

+rust-vmm@xxxxxxxxxxxxxxxxx

On Thu, 14 Apr 2022 at 14:54, Viresh Kumar <viresh.kumar@xxxxxxxxxx> wrote:
>
> +xen-devel
>
> On 14-04-22, 14:45, Viresh Kumar wrote:
> > Hello,
> >
> > We verified our hypervisor-agnostic Rust based vhost-user backends with Qemu
> > based setup earlier, and there was growing concern if they were truly
> > hypervisor-agnostic.
> >
> > In order to prove that, we decided to give it a try with Xen, a type-1
> > bare-metal hypervisor.
> >
> > We are happy to announce that we were able to make progress on that front 
> > and
> > have a working setup where we can test our existing Rust based backends, 
> > like
> > I2C, GPIO, RNG (though only I2C is tested as of now) over Xen.
> >
> > Key components:
> > --------------
> >
> > - Xen: https://github.com/vireshk/xen
> >
> >   Xen requires MMIO and device specific support in order to populate the
> >   required devices at the guest. This tree contains four patches on the top 
> > of
> >   mainline Xen, two from Oleksandr (mmio/disk) and two from me (I2C).
> >
> > - libxen-sys: https://github.com/vireshk/libxen-sys
> >
> >   We currently depend on the userspace tools/libraries provided by Xen, like
> >   xendevicemodel, xenevtchn, xenforeignmemory, etc. This crates provides 
> > Rust
> >   wrappers over those calls, generated automatically with help of bindgen
> >   utility in Rust, that allow us to use the installed Xen libraries. Though 
> > we
> >   plan to replace this with Rust based "oxerun" (find below) in longer run.
> >
> > - oxerun (WIP): https://gitlab.com/mathieupoirier/oxerun/-/tree/xen-ioctls
> >
> >   This is Rust based implementations for Ioctl and hypercalls to Xen. This 
> > is WIP
> >   and should eventually replace "libxen-sys" crate entirely (which are C 
> > based
> >   implementation of the same).
> >
> > - vhost-device: https://github.com/vireshk/vhost-device
> >
> >   These are Rust based vhost-user backends, maintained inside the rust-vmm
> >   project. This already contain support for I2C and RNG, while GPIO is under
> >   review. These are not required to be modified based on hypervisor and are
> >   truly hypervisor-agnostic.
> >
> >   Ideally the backends are hypervisor agnostic, as explained earlier, but
> >   because of the way Xen maps the guest memory currently, we need a minor 
> > update
> >   for the backends to work. Xen maps the memory via a kernel file
> >   /dev/xen/privcmd, which needs calls to mmap() followed by an ioctl() to 
> > make
> >   it work. For this a hack has been added to one of the rust-vmm crates,
> >   vm-virtio, which is used by vhost-user.
> >
> >   
> > https://github.com/vireshk/vm-memory/commit/54b56c4dd7293428edbd7731c4dbe5739a288abd
> >
> >   The update to vm-memory is responsible to do ioctl() after the already 
> > present
> >   mmap().
> >
> > - vhost-user-master (WIP): https://github.com/vireshk/vhost-user-master
> >
> >   This implements the master side interface of the vhost protocol, and is 
> > like
> >   the vhost-user-backend (https://github.com/rust-vmm/vhost-user-backend) 
> > crate
> >   maintained inside the rust-vmm project, which provides similar 
> > infrastructure
> >   for the backends to use. This shall be hypervisor independent and provide 
> > APIs
> >   for the hypervisor specific implementations. This will eventually be
> >   maintained inside the rust-vmm project and used by all Rust based 
> > hypervisors.
> >
> > - xen-vhost-master (WIP): https://github.com/vireshk/xen-vhost-master
> >
> >   This is the Xen specific implementation and uses the APIs provided by
> >   "vhost-user-master", "oxerun" and "libxen-sys" crates for its functioning.
> >
> >   This is designed based on the EPAM's "virtio-disk" repository
> >   (https://github.com/xen-troops/virtio-disk/) and is pretty much similar 
> > to it.
> >
> >   One can see the analogy as:
> >
> >   Virtio-disk == "Xen-vhost-master" + "vhost-user-master" + "oxerun" + 
> > "libxen-sys" + "vhost-device".
> >
> >
> >
> > Test setup:
> > ----------
> >
> > 1. Build Xen:
> >
> >   $ ./configure --libdir=/usr/lib --build=x86_64-unknown-linux-gnu 
> > --host=aarch64-linux-gnu --disable-docs --disable-golang 
> > --disable-ocamltools 
> > --with-system-qemu=/root/qemu/build/i386-softmmu/qemu-system-i386;
> >   $ make -j9 debball CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64
> >
> > 2. Run Xen via Qemu on X86 machine:
> >
> >   $ qemu-system-aarch64 -machine virt,virtualization=on -cpu cortex-a57 
> > -serial mon:stdio \
> >         -device virtio-net-pci,netdev=net0 -netdev 
> > user,id=net0,hostfwd=tcp::8022-:22 \
> >         -device virtio-scsi-pci -drive 
> > file=/home/vireshk/virtio/debian-bullseye-arm64.qcow2,index=0,id=hd0,if=none,format=qcow2
> >  -device scsi-hd,drive=hd0 \
> >         -display none -m 8192 -smp 8 -kernel /home/vireshk/virtio/xen/xen \
> >         -append "dom0_mem=5G,max:5G dom0_max_vcpus=7 loglvl=all 
> > guest_loglvl=all" \
> >         -device 
> > guest-loader,addr=0x46000000,kernel=/home/vireshk/kernel/barm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2
> >  console=hvc0 earlyprintk=xen" \
> >         -device ds1338,address=0x20     # This is required to create a 
> > virtual I2C based RTC device on Dom0.
> >
> >   This should get Dom0 up and running.
> >
> > 3. Build rust crates:
> >
> >   $ cd /root/
> >   $ git clone https://github.com/vireshk/xen-vhost-master
> >   $ cd xen-vhost-master
> >   $ cargo build
> >
> >   $ cd ../
> >   $ git clone https://github.com/vireshk/vhost-device
> >   $ cd vhost-device
> >   $ cargo build
> >
> > 4. Setup I2C based RTC device
> >
> >   $ echo ds1338 0x20 > /sys/bus/i2c/devices/i2c-0/new_device; echo 0-0020 > 
> > /sys/bus/i2c/devices/0-0020/driver/unbind
> >
> > 5. Lets run everything now
> >
> >   # Start the I2C backend in one terminal (open new terminal with "ssh
> >   # root@localhost -p8022"). This tells the I2C backend to hook up to
> >   # "/root/vi2c.sock0" socket and wait for the master to start transacting.
> >   $ /root/vhost-device/target/debug/vhost-device-i2c -s /root/vi2c.sock -c 
> > 1 -l 0:32
> >
> >   # Start the xen-vhost-master in another terminal. This provides the path 
> > of
> >   # the socket to the master side and the device to look from Xen, which is 
> > I2C
> >   # here.
> >   $ /root/xen-vhost-master/target/debug/xen-vhost-master --socket-path 
> > /root/vi2c.sock0 --name i2c
> >
> >   # Start guest in another terminal, i2c_domu.conf is attached. The guest 
> > kernel
> >   # should have Virtio related config options enabled, along with i2c-virtio
> >   # driver.
> >   $ xl create -c  i2c_domu.conf
> >
> >   # The guest should boot fine now. Once the guest is up, you can create 
> > the I2C
> >   # RTC device and use it. Following will create /dev/rtc0 in the guest, 
> > which
> >   # you can configure with 'hwclock' utility.
> >
> >   $ echo ds1338 0x20 > /sys/bus/i2c/devices/i2c-0/new_device
> >
> >
> > Hope this helps.
> >
> > --
> > viresh
>
> i2c_domu.conf
>
> > kernel="/root/Image"
> > memory=512
> > vcpus=2
> > command="console=hvc0 earlycon=xenboot"
> > name="domu"
> > i2c = [ "virtio=true, irq=1, base=1" ]
>
> --
> viresh