[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV


  • To: David Woodhouse <dwmw2@xxxxxxxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Mon, 25 Oct 2021 16:12:34 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8Qb7MN/jnEZwBzzlZMq5VTbnICJ88x13Shn0e0Jmuto=; b=IAP3IVarYuEVv5TUnXD+sgGgBTfMum3ARH2TNTtywfkXix4/VhsVx6l+zLga2ypMlC5csJKjQWEBD0oqQbTjpCwl/KpBkSfYPe9kzetDNv1BCU5GCOUXLWT4xvnmGOa79CO16YKnkc9A5d3IYRS9evq2wQM/dz/QASi/mNtPEB9LPWKYqmMArmTJO37PzXRV2IjB9rrVUCZf2cTRGdmbqXqOFIUM/FBVUS18hYgu9sYTTNP30jWCg8a0xKSNBlxxVlc1fUX3M0mbqjwjM8vJGX4QP66ZFxnEZaffghgATNQjtk8CTG2K2+KrjF5sMRStIbOcowGRYuNH/75BL799iw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CUPv6nHF07c/FoIRpmjqLN4Lb5FUIvD+vZHi9oiwanaJL0Klu8o569jggpZYzdHYHrRerJBIg5kz511hMuGw/bloE38XXycDuM1lRYtcmCa0VbO8mkDVuGYLL+xMcYycZ4uPi1hu78PWAduzFYkMyAY3rs0QnHSsUoYGeRoE0l0iMUti+aYRtc8HaoW/BgCff53oOfZ7wh6TPeDL447XQvhOrE2mAK5xpGK69gwxCx94FoEVxFMcnUPuO/hadz1w0ekpOVFw9q+XkWyObaBs2wjVE89jHOrOTfZN9fPnplCSGu44HiZdhu45p1AeLNmTgcu3EA6/uKHrYUgAm3X+Zg==
  • Authentication-results: esa5.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: Jason Andryuk <jandryuk@xxxxxxxxx>, <josef@xxxxxxxxxxx>, <boris.ostrovsky@xxxxxxxxxx>, <helgaas@xxxxxxxxxx>, <jgross@xxxxxxxx>, <linux-pci@xxxxxxxxxxxxxxx>, <maz@xxxxxxxxxx>, <tglx@xxxxxxxxxxxxx>, <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Mon, 25 Oct 2021 14:12:59 +0000
  • Ironport-data: A9a23:1Sb1FKPwyzpbSLnvrR3WkcFynXyQoLVcMsEvi/4bfWQNrUpx0WAAz TMWUG3SP6zbZGXzeN8ibY3j/UlUvJfRx4A1TQto+SlhQUwRpJueD7x1DKtR0wB+jCHnZBg6h ynLQoCYdKjYdpJYz/uUGuCJQUNUjMlkfZKhTr6bUsxNbVU8En540Ug+w7dRbrNA2rBVPSvc4 bsenOWHULOV82Yc3rU8sv/rRLtH5ZweiRtA1rAMTakjUGz2zhH5OKk3N6CpR0YUd6EPdgKMq 0Qv+5nilo/R109F5tpICd8XeGVSKlLZFVDmZna7x8FOK/WNz8A/+v9TCRYSVatYozqjhI177 /9yj5+baz4XIbeVurUkajANRkmSPYUekFPGCX22sMjVxEzaaXr8hf5pCSnaP6VBpLwxWzsXs 6VFdnZdNXhvhMrvqF6/YvNrick5atHiIasUu216zCGfBvEjKXzGa/iXuIMEh2xr7ixINc37a dIoeTNKUDTnczdIOQcoOqk8tuj90xETdBUH8QnI9MLb+VP7yA13zaioKNHUUsKFSN8TnUuCo G/CuWPjDXkyNtOFziGe2mmxneKJliT+MKoXHa+Q9fNwhlCXgGsJB3U+RVa95PW0lEO6c9ZeM FAPvDojq7Ao806mRcW7WAe3yFaYsRoMc95RFfAm8gaLy7qS7wvxLmoNSDJbecElnMAzTD0uk FSOmrvBCSNqr+e9SHST7L6YoDq+fy8PIgcqZygfQAUt5NXuuoU6iB6JVdsLOK64itL8ECrYw D2QqiU6wbIJgqYj3aKh/EuBhCmwvJ/XVQ0kzgLNV2mh40VyY4vNWmCzwQGFt7Aadt/fFwTf+ ihf8ySD0AwQJZ6nj3GHZ88yJrKCxc+7LyeFow9WJrB0olxB5EWfVYxX5Th/ImJgPcAFZSLlb SfvhO9B2HNAFCD1NfEvMupdH+xvlPK6TY20CZg4e/IXOsApHDJr6h2CcqJ5M4rFq0Mrjb0kc amSdcKhHB724ow2kWLoGY/xPVItrx3SJF8/p7iml3xLMpLEPRZ5rIvp13PVNogEAFus+lm9z jqmH5LiJ+9jeOP/eDLL1oUYMEoHK3M2bbiv9ZcKL7bbeFs+QDtxYxM0/V/HU9Y190iyvryQl kxRp2cCkAav7ZE5AVzSApycVF8fdckm9i9qVcDdFV2px2Iice6SAFQ3LPMKkU0c3LU7l5ZcF qBdE+3ZW6gnYmmXqlw1MMimxKQ/JUvDuO57F3f8CNTJV8U7HFKhFx6NVlaHyRTi+QLu5Jtg+ OX7jV2KKXfBLiw7ZPvrhDuU5wrZlVAWmf5oXluOJd9WeU7295NtJTC3hfgyS/zg4z2YrtdD/ wrJUxoeu8fXpIo5rIvAiaye9t/7GOpiBEtKWWLc6O/uZyXd+2Oix65GUfqJIm+BBD+lpv36a LUH1ez4Pd0GgE1O79h2HYF0wP9s/NDovbJbkFhpRS2Zc1SxB7p8CXCaxs0T5LZVz7pUtFLuC EKC89VXI5uTP8bhHAJDLQYpdL3bh/oVhiPT/bI+J0CjvH17+7+OUENzORiQiXMCcOspYd19m ep44ZwY8Q2yjBYuI+2qtCEM+jTeNGEEXoUmqooeXN3hhD00xwwQepfbECL3vs2CMo0eLkkwL zaIr6PenLAAlFHaen8+GHWRj+pQgZMC5EJDwFMYfgnbn9PEgrk83QFL8CRxRQNQl00V3+V2M 2ltFkt0OaTRoGs42JkdBzihS1NbGRmU2k3t0F9YxmTWQn6hWnHJMGBga/2G+1oU8j4EczVWl F1CJL0Jjdo+kBnN4xYP
  • Ironport-hdrordr: A9a23:tP8i7ampb189e0cWhTCBw8wR1FvpDfOwimdD5ihNYBxZY6Wkfp +V8sjzhCWatN9OYh0dcLC7WJVpQRvnhPhICK0qTMqftW7dyReVxeBZnPHfKljbehEWmdQtsJ uIH5IObOEYSGIK8voSgzPIY+rIouP3iJxA7N22pxwGIHAIGsMQnDuRSDzraXGeLDM2dKbRf6 Dsn/avyQDQHkj+Oa+Adwc4tqX41pD2vaOjRSRDKw8s6QGIgz/twLnmEyKA1hNbdz9U278t/U XMjgS8v8yYwr6G4y6Z81WWw4VdmdPnxNcGLMuQivINIjGprgqzfoxuV5CLoThwiuCy71QBls XKvn4bTohOwkKUWlvwjQrm2gHm3jprw3j+yWWAiX+mmsD9TCJSMbsIuatpNj/ir2YwttB116 xGm0iDsYBMMB/GlCPho/DVShBDjCOP0DofuN9Wq0YafZoVabdXo4Ba1lhSCo08ECXz751iOP VyDfvb+O1dfTqhHjHkV1FUsZ6Rt0kIb1K7qhBogL3Q79EWpgE286Ig/r1dop9an6hNDKWt5I z/Q+1Vff91P4krhJlGdZI8qP2MexrwqCL3QRCvyGvcZdU60lL22tXKCeYOlauXkKJh9upEpH 2GaiIAiVIP
  • Ironport-sdr: raXoSIRjazv7HnrKmFEveVUWdjK8sQOxaA52f7fd4U1luQwWrZ9BaV8LAQeVSe/G+fEIeYjzNg wMX/zFFJypds0rZgqoRadcEF5cM25/ul3+HR5xMmwYEy/vxQBvV6oJzT+3A1dVQb6hEM9l/Xxe LHdNdRldPjO+2B6bP4faEvknX1XiXFdOL03fPDmLp2ceH5xkXuYNoNhdNfuzyorqgsXypCavod Ml+BebBk8gppIGFBbeSyqhIjpKbn0tYim+lTUo6A4UP6QX4jw9uOXdZCR/3T1VBLz1vbbVhBvI q0hxxX/KydOjAWeOy17Zaz8q
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Mon, Oct 25, 2021 at 02:02:38PM +0100, David Woodhouse wrote:
> On Mon, 2021-10-25 at 14:58 +0200, Roger Pau Monné wrote:
> > On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> > > On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > > > It's kind of optional for HVM guests, as it depends on
> > > > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > > > guests, thus dropping any benefits from having hardware assisted APIC
> > > > virtualization or posted interrupts support.
> > > 
> > > Indeed. After implementing PIRQ support for Xen guests running under
> > > KVM, I spent a "happy" couple of days swearing at it because it
> > > actually *worked* if something would just *unmask* the f***ing MSI, but
> > > the guest inexplicably (to me) didn't do that.
> > > 
> > > Took me a while to work out that Xen itself is *snooping* on the MSI
> > > table writes even while they are *masked*, to capture the magic MSI
> > > message (with vector==0) which means it's actually a PIRQ# in the
> > > destination ID bits, and then magically unmask the MSI when the guest
> > > binds that PIRQ to an event channel.
> > > 
> > > I did not enjoy implementing that part.
> > 
> > I can see that. It's even better because none of this is actually
> > documented.
> 
> Indeed. I still haven't worked out if/how Xen actually *masks* the
> corresponding MSI-X again. It can't do so when the evtchn is masked,
> since that's just a bit in the shinfo page. So while the evtchn is
> masked, the MSI can still be screaming into the void?

I think so, it's quite weird because as a side effect of this mangling
Xen is transforming an edge triggered interrupt to a level triggered
one, as masked event channels belonging to MSI vectors will get set to
pending.

So it's not entirely screaming into the void because it will get
(wrongly) set as pending when masked.

> Perhaps it does so when the PIRQ is unbound from the evtchn?
> 
> > > FWIW the *guest* could potentlaly be smarter here and elect not to use
> > > PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> > > in the CPUID that Xen advertises, which indicate that? 
> > 
> > Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
> > XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
> > specifications gives us much benefit. We avoid a couple of vm exits
> > for sure, but the cost of having all those modifications in guest
> > OSes is not worth it.
> 
> These days with posted interrupts, it doesn't even save us any vmexits;
> it's all that additional guest complexity just to give us *more*
> vmexits than we would have had :)

Oh indeed. I was thinking about hardware without APIC hw
virtualization or posted interrupts.

Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.