Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski <brchuckz@xxxxxxx>]

On 1/3/2023 4:50 PM, Chuck Zmudzinski wrote:
> On 1/3/2023 10:14 AM, Alex Williamson wrote:
> > On Mon, 2 Jan 2023 18:10:24 -0500
> > Chuck Zmudzinski <brchuckz@xxxxxxxxxxxx> wrote:
> >
> > > On 1/2/23 12:46 PM, Michael S. Tsirkin wrote:
> > > > On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:  
> > > > > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > > > > as noted in docs/igd-assign.txt in the Qemu source code.
> > > > > ... 
> > > > > Signed-off-by: Chuck Zmudzinski <brchuckz@xxxxxxx>  
> > > >
> > > > I'm not sure why is the issue xen specific. Can you explain?
> > > > Doesn't it affect kvm too?  
> > > 
> > > Recall from docs/igd-assign.txt that there are two modes for
> > > igd passthrough: legacy and upt, and the igd needs to be
> > > at slot 2 only when using legacy mode which gives one
> > > single guest exclusive access to the Intel igd.
> > > 
> > > It's only xen specific insofar as xen does not have support
> > > for the upt mode so xen must use legacy mode which
> > > requires the igd to be at slot 2. I am not an expert with
> >
> > UPT mode never fully materialized for direct assignment, the folks at
> > Intel championing this scenario left.
>
> Thanks for clarifying that for me.
>
> >
> > > kvm, but if I understand correctly, with kvm one can use
> > > the upt mode with the Intel i915 kvmgt kernel module
> > > and in that case the guest will see a virtual Intel gpu
> > > that can be at any arbitrary slot when using kvmgt, and
> > > also, in that case, more than one guest can access the
> > > igd through the kvmgt kernel module.
> >
> > This is true, IIRC an Intel vGPU does not need to be in slot 2.
> >
> > > Again, I am not an expert and do not have as much
> > > experience with kvm, but if I understand correctly it is
> > > possible to use the legacy mode with kvm and I think you
> > > are correct that if one uses kvm in legacy mode and without
> > > using the Intel i915 kvmgt kernel module, then it would be
> > > necessary to reserve slot 2 for the igd on kvm.
> >
> > It's necessary to configure the assigned IGD at slot 2 to make it
> > functional, yes, but I don't really understand this notion of
> > "reserving" slot 2.  If something occupies address 00:02.0 in the
> > config, it's the user's or management tool's responsibility to move it
> > to make this configuration functional.  Why does QEMU need to play a
> > part in reserving this bus address.  IGD devices are not generally
> > hot-pluggable either, so it doesn't seem we need to reserve an address
> > in case an IGD device is added dynamically later.
>
> As I said in earlier message in this thread, the xenlight toolstack (libxl) 
> that is
> provided as the default toolstack for building xen guests with pci passthrough
> is not the most flexible management tool, and that is why, in the case of xen,
> it is simpler to reserve slot 2 while qemu assigns the slot addresses of the
> qemu emulated pci devices so that the igd can use slot 2. IIRC, In 
> hw/pci/pci.c,
> once the slot value is assigned, it is constant and cannot be changed later on
> by a management tool.
>
> >  
> > > Your question makes me curious, and I have not been able
> > > to determine if anyone has tried igd passthrough using
> > > legacy mode on kvm with recent versions of linux and qemu.
> >
> > Yes, it works.
> >
> > > I will try reproducing the problem on kvm in legacy mode with
> > > current versions of linux and qemu and report my findings.
> > > With kvm, there might be enough flexibility to specify the
> > > slot number for every pci device in the guest. Such a
> >
> > I think this is always the recommendation, libvirt will do this by
> > default in order to make sure the configuration is reproducible.  This
> > is what we generally rely on for kvm/vfio IGD assignment to place the
> > GPU at the correct address.
> >
> > > capability is not available using the xenlight toolstack
> > > for managing xen guests, so I have been using this patch
> > > to ensure that the Intel igd is at slot 2 with xen guests
> > > created by the xenlight toolstack.
> >
> > Seems like a deficiency in xenlight.  I'm not sure why QEMU should take
> > on this burden to support support tool stacks that lack such basic
> > features.
>
> So you would prefer to patch xenlight (libxl) to make it flexible enough to 
> properly
> handle this case of legacy igd passthrough.
>
> >  
> > > The patch as is will only fix the problem on xen, so if the
> > > problem exists on kvm also, I agree that the patch should
> > > be modified to also fix it on kvm.
> >
> > AFAICT, it's not a problem on kvm/vfio because we generally make use of
> > invocations that specify bus addresses for each device by default,
> > making this a configuration requirement for the user or management tool
> > stack.  Thanks,
>
> Unfortunately, and as I mentioned in an earlier message on this thread,
> the xenlight management tool stack (libxl) is not so flexible and does not
> make it so easy for the administrator to specify the bus address of each
> device, and that is why either this patch is needed for igd legacy passtrhough
> on xen, or the libxl management tool needs to be patched so it is flexible
> enough to enable the slot addresses to be assigned correctly using
> that tool instead of relying on qemu to reserve slot 2 for the igd.
>
> If there is a consensus that this should be fixed in libxl instead of in qemu,
> I will work on a patch to libxl, but I will wait a while for some feedback 
> from
> the xen people (Anthony, Paul) before I do that.

Hello Anthony and Paul,

I am requesting your feedback to Alex Williamson's suggestion that this
problem with assigning the correct slot address to the igd on xen should
be fixed in libxl instead of in qemu.

It seems to me that the xen folks and the kvm folks have two different
philosophies regarding how a tool stack should be designed. kvm/libvirt
provides much greater flexibility in configuring the guest which puts
the burden on the administrator to set all the options correctly for
a given feature set, while xen/xenlight does not provide so much
flexibility and tries to automatically configure the guest based on
a high-level feature option such as the igd-passthrough=on option that
is available for xen guests using qemu but not for kvm guests using
qemu.

What do you think? Should libxl be patched instead of fixing the problem
with this patch to qemu, which is contrary to Alex's suggestion?

Thanks in advance,

Chuck