[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

  • To: Jan Beulich <jbeulich@xxxxxxxx>
  • From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>
  • Date: Fri, 11 Jun 2021 06:53:25 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kTOpmOIDaR981wQQoovjRFppx1CKxP5ikGvnwSG0umU=; b=iEitRxcS8ONyrO+81aBfJYS9PGc4RFlG7p4qnD30mAgUs5Je9sIgBRFqTN7Xqek5R8rzIJ1a08aSJKDwtr4AaC4DlGZ2jhoRH1OXy/gWMrqDvY4Ta7yLk4XmFyj78MZIKnH6UCi0CjaPq5/Evb4Ho9xuxKWUV1CeTeaDvHxLxwGmnQfWI78s4jympXIZXR2PZlDX2hNjm7UnQL1L2OJ2r/OryKAgRK+u+3xXeJNX2P85+B/XruIOCA+v/LbeFyfgUaqJR4GGca4wNh7VkYw6bIDunz1rsSnr387u9BcWhd00HPKZdK0H9IvunFFFLqGxxSvU7m/xOxs1C6tnRjVjug==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OeipfMNk9Io0KE34PwOxvslfYi0KJ4pNxUWXh+vpODlR0CsTEoNEVgtEgbmGufjhXvyGmG1Nk7eldUhGvWUTQ1ZFfDVyXWRW8XorFvICyln7GhhgOShl32mXdqZ3YT6mBjfITifbw38JJ9EESQlRLRR3LaxdptVsisp6roojS8VslkknFSdZmx+u7ztQ7n2BgpAUszU1f2uk89nGdG8J327kjWrlPVHzfGBTaNdduuLepF4R0O9db/QrQW4Z1SD/0rGbyyiyuo/iLTrvakdArxuq5fadTrna84MkQIB3iWSzeLeIi++tOYlI8vcs3YoaGYZgwJyU0Mk+ePxPYHLVUg==
  • Authentication-results: suse.com; dkim=none (message not signed) header.d=none;suse.com; dmarc=none action=none header.from=epam.com;
  • Cc: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Delivery-date: Fri, 11 Jun 2021 06:53:39 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHXWQwW/O3obtZgqkmpGRvt7ZNDkasM6aMAgAAjXoCAAEW+AIAAFw0AgAD+3oCAAAIxgA==
  • Thread-topic: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

On 11.06.21 09:45, Jan Beulich wrote:
> On 10.06.2021 17:33, Oleksandr Andrushchenko wrote:
>> On 10.06.21 17:10, Roger Pau Monné wrote:
>>> On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
>>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>>> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>>>>>> Hi, all!
>>>>>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>>>>>> of current PCI on ARM support [2]. The question I have for this series
>>>>>> is if we really need emulating SR-IOV code in Xen?
>>>>>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>>>>>> patches)
>>>>>> and it "works for me": MSI support is still WIP, but I was able to see 
>>>>>> that
>>>>>> VFs are properly seen in the guest and BARs are properly programmed in 
>>>>>> p2m.
>>>>>> What I can't fully understand is if we can live with this approach or 
>>>>>> there
>>>>>> are use-cases I can't see.
>>>>>> Previously I've been told that this approach might not work on FreeBSD
>>>>>> running
>>>>>> as Domain-0, but is seems that "PCI Passthrough is not supported
>>>>>> (Xen/FreeBSD)"
>>>>>> anyways [4].
>>>>> PCI passthorgh is not supported on FreeBSD dom0 because PCI
>>>>> passthrough is not supported by Xen itself when using a PVH dom0, and
>>>>> that's the only mode FreeBSD dom0 can use.
>>>> So, it is still not clear to me: how and if PCI passthrough is supported
>>>> on FreeBSD, what are the scenarios and requirements for that?
>>>>> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
>>>>> to work. I however think this is not the proper way to implement
>>>>> SR-IOV support.
>>>> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
>>>> could you please point me to where are these used?
>>> Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
>>> implement them anymore. They are implemented on Linux for x86 PV dom0,
>>> AFAIK Arm doesn't use them either.
>> Well, ARM didn't until we started implementing PCI passthrough [1].
>> It was previously discussed [2], "# Discovering PCI devices:" and proposed
>> to use PHYSDEVOP_pci_device_add.
>> Long story short, it is not easy for ARM to enumerate PCI devices in Xen as 
>> there is
>> no unified way of doing so: different platforms implement different PCI host 
>> bridges
>> which require complex initialization including clocks, power domains etc.
> Just for my own understanding: If this isn't done by firmware, doesn't
> this mean you can't boot an Arm system from e.g. a disk connected through
> a PCI-based controller? Host bridge setup is definitely firmware's job on
> x86 ...

On the platforms I work with: no, you can't. Well, it is possible to add PCI

support to the firmware, but we normally boot out of eMMC, SD, network and

all those are typically NOT PCI devices.

Even more. In my everyday work I don't use (need) any PCI device to run

the system at all.

> Jan
Thank you,




Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.