Linux PV/PVH domU crash on (guest) resume from suspend

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I'm observing Linux PV/PVH guest crash when I resume it from sleep. I do
this with:

    virsh -c xen dompmsuspend <vmname> mem
    virsh -c xen dompmwakeup <vmname> 

But it's possible to trigger it with plain xl too:

    xl save -c <vmname> <some-file>

The same on HVM works fine.

This is on Xen 4.14.1, and with guest kernel 5.4.90, the same happens
with 5.4.98. Dom0 kernel is the same, but I'm not sure if that's
relevant here. I can reliably reproduce it.

The crash message:

[  219.844995] Freezing user space processes ... (elapsed 0.011 seconds) done.
[  219.856564] OOM killer disabled.
[  219.856566] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) 
done.
[  277.562118] register_vcpu_info failed: cpu=0 err=-22
[  219.858384] xen:grant_table: Grant tables using version 1 layout
[  219.858442] ------------[ cut here ]------------
[  219.858446] kernel BUG at drivers/xen/events/events_fifo.c:369!
[  219.858503] invalid opcode: 0000 [#1] SMP NOPTI
[  219.858511] CPU: 0 PID: 11 Comm: migration/0 Not tainted 
5.4.90-1.qubes.x86_64 #1
[  219.858527] RIP: e030:evtchn_fifo_resume+0x58/0x90
[  219.858532] Code: eb 48 8b 04 dd 80 29 3e 82 4e 8b 04 20 4d 85 c0 74 d5 48 
0f a3 1d b8 40 20 01 73 10 4c 89 c6 89 ef e8 5c fb ff ff 85 c0 79 bd <0f> 0b 31 
f6 4c 89 c7 e8 7c 8a c8 ff 48 8b 04 dd 80 29 3e 82 4a c7
[  219.858538] RSP: e02b:ffffc9000025be10 EFLAGS: 00010082
[  219.858542] RAX: ffffffffffffffea RBX: 0000000000000000 RCX: 0000000000000000
[  219.858545] RDX: ffff888018400000 RSI: ffffc9000025bde0 RDI: 000000000000000b
[  219.858548] RBP: 0000000000000000 R08: ffff888018143000 R09: 00000000000001e0
[  219.858552] R10: ffff88800e50f440 R11: ffffc9000025bcbd R12: 0000000000026ea0
[  219.858555] R13: 0000000000000000 R14: ffffc9000029bdf8 R15: 0000000000000003
[  219.858567] FS:  0000000000000000(0000) GS:ffff888018400000(0000) 
knlGS:0000000000000000
[  219.858571] CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[  219.858574] CR2: 0000581c2753e030 CR3: 000000000260a000 CR4: 0000000000000660
[  219.858578] Call Trace:
[  219.858615]  xen_irq_resume+0x1b/0xe0
[  219.858620]  xen_suspend+0x13e/0x190
[  219.858626]  multi_cpu_stop+0x6c/0x100
[  219.858630]  ? stop_machine_yield+0x10/0x10
[  219.858633]  cpu_stopper_thread+0xb0/0x110
[  219.858638]  smpboot_thread_fn+0xc5/0x160
[  219.858641]  ? smpboot_register_percpu_thread+0xf0/0xf0
[  219.858645]  kthread+0x115/0x140
[  219.858648]  ? __kthread_bind_mask+0x60/0x60
[  219.858653]  ret_from_fork+0x22/0x40
[  219.858657] Modules linked in: nf_conntrack_netlink nft_reject_ipv4 
nft_reject xt_nat nf_tables_set nft_ct nf_tables nfnetlink e1000e rfkill 
xt_REDIRECT ip6table_filter ip6table_mangle ip6table_raw ip6_tables 
edac_mce_amd pcspkr ipt_REJECT nf_reject_ipv4 xen_pcifront xt_state 
xt_conntrack iptable_filter iptable_mangle iptable_raw xt_MASQUERADE 
iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c 
xen_scsiback target_core_mod xen_netback xen_privcmd xen_gntdev xen_gntalloc 
xen_blkback xen_evtchn drm fuse ip_tables overlay xen_blkfront
[  219.858754] ---[ end trace 54d868ea756768db ]---
[  219.858758] RIP: e030:evtchn_fifo_resume+0x58/0x90
[  219.858762] Code: eb 48 8b 04 dd 80 29 3e 82 4e 8b 04 20 4d 85 c0 74 d5 48 
0f a3 1d b8 40 20 01 73 10 4c 89 c6 89 ef e8 5c fb ff ff 85 c0 79 bd <0f> 0b 31 
f6 4c 89 c7 e8 7c 8a c8 ff 48 8b 04 dd 80 29 3e 82 4a c7
[  219.858768] RSP: e02b:ffffc9000025be10 EFLAGS: 00010082
[  219.858770] RAX: ffffffffffffffea RBX: 0000000000000000 RCX: 0000000000000000
[  219.858774] RDX: ffff888018400000 RSI: ffffc9000025bde0 RDI: 000000000000000b
[  219.858777] RBP: 0000000000000000 R08: ffff888018143000 R09: 00000000000001e0
[  219.858780] R10: ffff88800e50f440 R11: ffffc9000025bcbd R12: 0000000000026ea0
[  219.858783] R13: 0000000000000000 R14: ffffc9000029bdf8 R15: 0000000000000003
[  219.858790] FS:  0000000000000000(0000) GS:ffff888018400000(0000) 
knlGS:0000000000000000
[  219.858793] CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[  219.858796] CR2: 0000581c2753e030 CR3: 000000000260a000 CR4: 0000000000000660
[  219.858801] Kernel panic - not syncing: Fatal exception
[  219.858819] Kernel Offset: disabled

Note the time besides "register_vcpu_info failed" - it is in the future.
I think the offset depends on the uptime, with shorter uptime I get much
smaller difference (like 49 vs 51).

Any ideas?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature