Re: RMRRs and Phantom Functions

[Jan Beulich <jbeulich@xxxxxxxx>]

On 27.04.2022 12:05, Andrew Cooper wrote:
> On 27/04/2022 07:59, Jan Beulich wrote:
>> On 26.04.2022 19:51, Andrew Cooper wrote:
>>> Hello,
>>>
>>> Edvin has found a machine with some very weird properties.  It is an HP
>>> ProLiant BL460c Gen8 with:
>>>
>>>  \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
>>>              +-01.0-[11]--
>>>              +-01.1-[02]--
>>>              +-02.0-[04]--+-00.0  Emulex Corporation OneConnect 10Gb NIC
>>> (be3)
>>>              |            +-00.1  Emulex Corporation OneConnect 10Gb NIC
>>> (be3)
>>>              |            +-00.2  Emulex Corporation OneConnect 10Gb
>>> iSCSI Initiator (be3)
>>>              |            \-00.3  Emulex Corporation OneConnect 10Gb
>>> iSCSI Initiator (be3)
>>>
>>> yet all 4 other functions on the device periodically hit IOMMU faults
>>> (~once every 5 mins, so definitely stats).
>>>
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.4] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.5] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.6] fault addr
>>> bdf80000
>>> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:04:00.7] fault addr
>>> bdf80000
>>>
>>> There are several RMRRs covering the these devices, with:
>>>
>>> (XEN) [VT-D]found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D] endpoint: 0000:03:00.0
>>> (XEN) [VT-D] endpoint: 0000:01:00.0
>>> (XEN) [VT-D] endpoint: 0000:01:00.2
>>> (XEN) [VT-D] endpoint: 0000:04:00.0
>>> (XEN) [VT-D] endpoint: 0000:04:00.1
>>> (XEN) [VT-D] endpoint: 0000:04:00.2
>>> (XEN) [VT-D] endpoint: 0000:04:00.3
>>> (XEN) [VT-D]dmar.c:608:   RMRR region: base_addr bdf8f000 end_addr bdf92fff
>>>
>>> being the one relevant to these faults.  I've not manually decoded the
>>> DMAR table because device paths are horrible to follow but there are at
>>> least the correct number of endpoints.  The functions all have SR-IOV
>>> (disabled) and ARI (enabled).  None have any Phantom functions described.
>>>
>>> Specifying pci-phantom=04:00,1 does appear to work around the faults,
>>> but it's not right, because functions 1 thru 3 aren't actually phantom.
>> Indeed, and I think you really mean "pci-phantom=04:00,4".
> 
> As a quick tangent, the cmdline docs for pci-phantom= are in desperate
> need of an example and a description of how stride works.  I've got some
> ideas and notes jotted down.
> 
> Do we really mean ,4 here?  What happens for function 1?

With stride 4 function 1's single phantom function is function 5. With
stride 1, as you had it before, functions 1...7 would all be considered
phantom functions of function 0.

Jan