[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [EXT] Re: DOMU: Virtual Function FLR in PCI passthrough is crashing


  • To: Jan Beulich <jbeulich@xxxxxxxx>
  • From: Naresh Bhat <nareshb@xxxxxxxxxxx>
  • Date: Mon, 25 Apr 2022 11:01:37 +0000
  • Accept-language: en-GB, en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vMrGmSzcMKOyH3oAW9hQTtRmv7evMx7ZBE6+j+OOV6Q=; b=W8qXz2t48tIuoSsU5Pmv8zi2L7xQ+tWSmbXatoyTR7ZUiELW2h9Fone9fn7LvTxGhN35SZnzt43RSVJ+jBuWJuST1MKYXLexYaetpe5FwNYsfAN6iEuk9P/9zjvu8onhd7LUjfr6pHrCeihr812kS8FhXREsFeU091ZeKybcdovOJyOeff/VrDpuNKJL525ScjSUrFcWpDy+jpb2XvsOEj6r9c3/L3q4mKuNBzuURtTFQ6J0EODeeUU5q4L6V4kgl/h68FerfN0KjQtfK57aWGbsqxRMMyJ8WX4sQpWbpcWebTL8fVu4b7AIe7dd/1+ZaF7gVPTi0ibwwLubJT51uw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZAf0mLHle1qrOmPX/juPLNzwLeaOYoRjoR1I6l056agEx51IaFW7R3TosSUsFGBQEXwGLyhp/Q30SOOP+tMrN6rzI3hG+hz8dZli4ET+ITD1n2p7UtT0hOmS1tFK8QKR062fmWD1E0qlYUmL7ynYzBoDu8nfiEKhad3QtL8z+kS0pCCdIawmpQBByvpkbygS+uvU6SlbDZukUMTAdNN1tJY6zaqFcfPNjrIfmAOn06wB2h3LR1cPxGDN5VlqnXDxsT5kNJ9kgHWhzWf6AoYF46pZcCLOGzy1hUh2zRr2F/YBISacNyWc1janJvAPrWRJFRyAvkmR4yzSwpASztpskg==
  • Cc: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Mon, 25 Apr 2022 11:01:51 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Suggested_attachment_session_id: 9f1f0a53-5fa2-a218-d889-a5bb3569783d
  • Thread-index: AQHYVLOBoc8zYi1khUW56WNBox6p/Kz4xa6AgAe2FfE=
  • Thread-topic: [EXT] Re: DOMU: Virtual Function FLR in PCI passthrough is crashing

Hi Jan Beulich, 

Thank you very much. Please see my inline comments below.

From: Jan Beulich <jbeulich@xxxxxxxx>
Sent: 20 April 2022 18:37
To: Naresh Bhat <nareshb@xxxxxxxxxxx>
Cc: julien@xxxxxxx <julien@xxxxxxx>; sstabellini@xxxxxxxxxx 
<sstabellini@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx 
<xen-devel@xxxxxxxxxxxxxxxxxxxx>
Subject: [EXT] Re: DOMU: Virtual Function FLR in PCI passthrough is crashing 
 
External Email

----------------------------------------------------------------------
On 20.04.2022 14:48, Naresh Bhat wrote:
> I have the following setup and try to test the Function Level Reset feature.  
> Any suggestions or pointers will be very much helpful.
> 
> DOM0
> Distribution: Ubuntu-20.04.3 (kernel 5.8.0-43)
> Xen version : 4.11.4-pre
> 
> DOMU
> Distribution: Ubuntu-18.04.6 LTS (kernel 5.8.0)
> PCIe device with SRIOV support, VF (Virtual Function) interface connected to 
> DOMU via PCI pass-through
> 
> Issue on DOMU: 
> 1. Enable MSIX on DOMU (We have used the following kernel APIs 
> pci_enable_msix_range, pci_alloc_irq_vectors)
> 2. Execute FLR (Function Level Reset) via sysfs interface on the PCIe 
> passthrough device in DOMU
>    # echo "1" > /sys/bus/pci/devices/<ID>/reset
> 
> The following crash observed 
> 
> [ 4126.391455] BUG: unable to handle page fault for address: ffffc90040029000
> [ 4126.391489] #PF: supervisor write access in kernel mode
> [ 4126.391503] #PF: error_code(0x0003) - permissions violation
> [ 4126.391516] PGD 94980067 P4D 94980067 PUD 16a155067 PMD 16a156067 PTE 
> 80100000a000c075
> [ 4126.391537] Oops: 0003 [#1] SMP NOPTI
> [ 4126.391550] CPU: 0 PID: 971 Comm: bash Tainted: G           OE     5.8.0 #1
> [ 4126.391570] RIP: e030:__pci_write_msi_msg+0x59/0x150
> [ 4126.391580] Code: 8b 50 d8 85 d2 75 31 83 78 fc 03 74 2b f6 47 54 01 74 6e 
> f6 47 55 02 75 1f 0f b7 47 56 c1 e0 04 48 98 48 03 47 60 74 10 8b 16 <89> 10 
> 8b 56 04 89 50 04 8b 56 08 89 50 08 48 8b 03 49 89 44 24 20
> [ 4126.391606] RSP: e02b:ffffc90040407cc0 EFLAGS: 00010286

The RSP related selector value suggests you're talking about a PV DomU.
Such a DomU cannot write the MSI-X table directly, yet at a guess (from
the PTE displayed) that's what the insn does where the crash occurred. I
would guess you've hit yet another place in the kernel where proper PV
abstraction is missing. You may want to check with newer kernels.

[Naresh]: We have tested with latest kernel i.e. 5.17.0 kernel, issue persists.

As to FLR - I guess this operation as a whole needs passing through
pcifront to pciback, such that the operation can be carried out safely
(e.g. to save and restore active MSIs, which is what I infer is being
attempted here, as per the stack trace).

[Naresh]: Any idea when the support will be added to Xen ?

Jan



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.