Re: RMRRs and Phantom Functions

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Apr 27, 2022 at 10:05:54AM +0000, Andrew Cooper wrote:
> On 27/04/2022 07:59, Jan Beulich wrote:
> > On 26.04.2022 19:51, Andrew Cooper wrote:
> >> Hello,
> >>
> >> Edvin has found a machine with some very weird properties.  It is an HP
> >> ProLiant BL460c Gen8 with:
> >>
> >>  \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
> >>              +-01.0-[11]--
> >>              +-01.1-[02]--
> >>              +-02.0-[04]--+-00.0  Emulex Corporation OneConnect 10Gb NIC
> >> (be3)
> >>              |            +-00.1  Emulex Corporation OneConnect 10Gb NIC
> >> (be3)
> >>              |            +-00.2  Emulex Corporation OneConnect 10Gb
> >> iSCSI Initiator (be3)
> >>              |            \-00.3  Emulex Corporation OneConnect 10Gb
> >> iSCSI Initiator (be3)
> >>
> >> yet all 4 other functions on the device periodically hit IOMMU faults
> >> (~once every 5 mins, so definitely stats).
> >>
> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
> >> bdf80000
> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
> >> bdf80000
> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
> >> bdf80000
> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
> >> bdf80000
> >>
> >> There are several RMRRs covering the these devices, with:
> >>
> >> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> >> (XEN) [VT-D] endpoint: 0000:03:00.0
> >> (XEN) [VT-D] endpoint: 0000:01:00.0
> >> (XEN) [VT-D] endpoint: 0000:01:00.2
> >> (XEN) [VT-D] endpoint: 0000:04:00.0
> >> (XEN) [VT-D] endpoint: 0000:04:00.1
> >> (XEN) [VT-D] endpoint: 0000:04:00.2
> >> (XEN) [VT-D] endpoint: 0000:04:00.3
> >> (XEN) [VT-D]dmar.c:608:   RMRR region: base_addr bdf8f000 end_addr bdf92fff
> >>
> >> being the one relevant to these faults.  I've not manually decoded the
> >> DMAR table because device paths are horrible to follow but there are at
> >> least the correct number of endpoints.  The functions all have SR-IOV
> >> (disabled) and ARI (enabled).  None have any Phantom functions described.
> >>
> >> Specifying pci-phantom=04:00,1 does appear to work around the faults,
> >> but it's not right, because functions 1 thru 3 aren't actually phantom.
> > Indeed, and I think you really mean "pci-phantom=04:00,4".
> 
> As a quick tangent, the cmdline docs for pci-phantom= are in desperate
> need of an example and a description of how stride works.  I've got some
> ideas and notes jotted down.
> 
> Do we really mean ,4 here?  What happens for function 1?
> 
> > I guess we
> > should actually refuse "pci-phantom=04:00,1" in a case like this one.
> > The problem is that at the point we set pdev->phantom_stride we may
> > not know of the other devices, yet. But I guess we could attempt a
> > config space read of the supposed phantom function's device/vendor
> > and do <whatever> if these aren't both 0xffff.
> 
> At a minimum, we ought to warn when it looks like something is wonky,
> but I wouldn't go as far as rejecting.
> 
> All of these options to work around firmware/system screwups are applied
> to an already-non-working system, and there is absolutely no guarantee
> that necessary fixes make any kind of logical sense.

AFAICT with stride = 1 Xen will treat functions 1-7 as phantom
functions depending from function 0, which means the pdev struct won't
get updated when those phantom functions are assigned to a domain as
part of assigning function 0.  That would imply that functions 1 to 3
will be considered phantom but would also have a matching pdev that
allows them to be independently assigned to a domain, nothing good
will came out of it.

I agree with Jan that we need to explicitly reject strides that cover
functions that would otherwise be considered devices (ie: have valid
config space entries).  Or alternatively we need to remove the pdevs
for those functions now considered phantom.

Thanks, Roger.