Xen project Mailing List

DOMU: Virtual Function FLR in PCI passthrough is crashing

To: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>

From: Naresh Bhat <nareshb@xxxxxxxxxxx>

Date: Wed, 20 Apr 2022 12:48:09 +0000

Accept-language: en-GB, en-US

Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none

Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tZ4XQfscFW4XNzWpat06473mKPO2/jW7rq0jSMlmmFA=; b=B9eTMSrnTo2tbA0vGQjmjBjy2mjMqJvhTaoeqSmJOvG/MHUP0GViF5pzmSaVPE2fKYQqfM2zpumw58bki6Gu8A1hNU+a72BWw3/oWCX1VDWGFcKLBDK772Z6WLAk6NOVvRnxdadVuEG8nq5aVv6QyRe66JgYECdkDALwU5KUj1bH3RmEuSMYjRR3q3pZ58430PekbhRYuGaemuCF0fVxohaLNFMfb+Ym7XsekPJ1QroCyhGsIYFzM212y1Z018JraotQwqpU+l7BiZpQJmYEhpcG3TjPqYC0WpWwmuvX01iHqwxmaXiovwop5sxLtRmrouhs87r0mZaSBe3a3Nl19g==

Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n8ZOIPNiWx5JW5GFsNa+m8hNtSz1o9ouA4C/xXk6mB+kccEQpZzmaZA2NXTwYg0KxjGEQ/04xFGdPD/3P2sJ7tzGQN1LTfPx+RA5DAf3fhxhbT9JSRksx676qMsEOn0C0NheX7DGrc100RqzKm4pMzzN9XE5uJsAYkfcgd9ELAhlwE9r5wYPcWP7vXHvYlVTwYcCTPThPe5i/YakVckmbDdIywueYF/06n7VvasBnFxaFZNX7MWHokyqaV1f3+x9RMrZI81B9OWJORjN4kaNcMCikLi1avIh4MNxlGiQTWoPZ4kkG7S8Ijxbu3bl6ubgObvxRzUcB+H11oFXMGzT/A==

Cc: "julien@xxxxxxx" <julien@xxxxxxx>, "sstabellini@xxxxxxxxxx" <sstabellini@xxxxxxxxxx>

Delivery-date: Wed, 20 Apr 2022 12:48:26 +0000

List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Suggested_attachment_session_id: 3881f8c8-5b13-69d0-d21d-4a02acccd4cb

Thread-index: AQHYVLOBoc8zYi1khUW56WNBox6p/A==

Thread-topic: DOMU: Virtual Function FLR in PCI passthrough is crashing

Hi, I have the following setup and try to test the Function Level Reset feature. Any suggestions or pointers will be very much helpful. DOM0 Distribution: Ubuntu-20.04.3 (kernel 5.8.0-43) Xen version : 4.11.4-pre DOMU Distribution: Ubuntu-18.04.6 LTS (kernel 5.8.0) PCIe device with SRIOV support, VF (Virtual Function) interface connected to DOMU via PCI pass-through Issue on DOMU: 1. Enable MSIX on DOMU (We have used the following kernel APIs pci_enable_msix_range, pci_alloc_irq_vectors) 2. Execute FLR (Function Level Reset) via sysfs interface on the PCIe passthrough device in DOMU # echo "1" > /sys/bus/pci/devices/<ID>/reset The following crash observed [ 4126.391455] BUG: unable to handle page fault for address: ffffc90040029000 [ 4126.391489] #PF: supervisor write access in kernel mode [ 4126.391503] #PF: error_code(0x0003) - permissions violation [ 4126.391516] PGD 94980067 P4D 94980067 PUD 16a155067 PMD 16a156067 PTE 80100000a000c075 [ 4126.391537] Oops: 0003 [#1] SMP NOPTI [ 4126.391550] CPU: 0 PID: 971 Comm: bash Tainted: G OE 5.8.0 #1 [ 4126.391570] RIP: e030:__pci_write_msi_msg+0x59/0x150 [ 4126.391580] Code: 8b 50 d8 85 d2 75 31 83 78 fc 03 74 2b f6 47 54 01 74 6e f6 47 55 02 75 1f 0f b7 47 56 c1 e0 04 48 98 48 03 47 60 74 10 8b 16 <89> 10 8b 56 04 89 50 04 8b 56 08 89 50 08 48 8b 03 49 89 44 24 20 [ 4126.391606] RSP: e02b:ffffc90040407cc0 EFLAGS: 00010286 [ 4126.391623] RAX: ffffc90040029000 RBX: ffff888164cfb120 RCX: 0000000000000000 [ 4126.391639] RDX: 0000000000000000 RSI: ffff888164cfb120 RDI: ffff888164cfb100 [ 4126.391653] RBP: ffffc90040407cf8 R08: 000053f2d6975617 R09: ffff888169c4e238 [ 4126.391672] R10: 0000000000000000 R11: ffffffff8266b248 R12: ffff888164cfb100 [ 4126.391688] R13: ffff88815e81c2e0 R14: ffff88815e81c130 R15: ffff8881648394a0 [ 4126.391723] FS: 00007f72b4b9b740(0000) GS:ffff88816ac00000(0000) knlGS:0000000000000000 [ 4126.391742] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4126.391756] CR2: ffffc90040029000 CR3: 0000000167b10000 CR4: 0000000000040660 [ 4126.391781] Call Trace: [ 4126.391799] default_restore_msi_irq+0x38/0x70 [ 4126.391818] default_restore_msi_irqs+0x2f/0x80 [ 4126.391836] arch_restore_msi_irqs+0x15/0x20 [ 4126.391851] pci_restore_msi_state+0xa1/0x230 [ 4126.391870] pci_restore_state.part.0+0x319/0x440 [ 4126.391888] pci_dev_restore+0x4a/0x60 [ 4126.391901] pci_reset_function+0x4b/0x70 [ 4126.391915] reset_store+0x5d/0xa0 [ 4126.391931] dev_attr_store+0x17/0x30 [ 4126.391944] sysfs_kf_write+0x3e/0x50 [ 4126.391958] kernfs_fop_write+0xda/0x1b0 [ 4126.391973] vfs_write+0xc9/0x200 [ 4126.391986] ksys_write+0x67/0xe0 [ 4126.392002] __x64_sys_write+0x1a/0x20 [ 4126.392018] do_syscall_64+0x52/0xc0 [ 4126.392033] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 4126.392049] RIP: 0033:0x7f72b4277224 [ 4126.392064] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 c1 07 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5 [ 4126.392093] RSP: 002b:00007ffc5236f578 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4126.392114] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f72b4277224 [ 4126.392128] RDX: 0000000000000002 RSI: 0000562bde52f440 RDI: 0000000000000001 [ 4126.392140] RBP: 0000562bde52f440 R08: 000000000000000a R09: 0000000000000001 [ 4126.392155] R10: 000000000000000a R11: 0000000000000246 R12: 00007f72b4553760 [ 4126.392171] R13: 0000000000000002 R14: 00007f72b454f2a0 R15: 00007f72b454e760 [ 4126.392185] Modules linked in: <driver function> intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl xen_pcifront sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 [ 4126.392228] CR2: ffffc90040029000 [ 4126.392238] ---[ end trace 23e8ad345e1ef956 ]--- Thanks and Regards -Naresh Bhat

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.