[Xen-devel] Repeated problems with network receive path in Linux

[Sarah Newman <srn@xxxxxxxxx>]

Hi,

We have experienced 3 full crashes over the last 3 months in the Linux network 
receive path, and one additional crash of a vif queue that killed
networking for a single guest but did not bring down the entire host. I am not 
100% sure if it is Xen related or not, but it seems possible.

Because we have only seen it on one specific host, and the problem has 
persisted across different pieces of hardware, I believe the cause is likely
data dependent or maybe can only be triggered in a reasonable amount of time if 
there is large amounts of traffic exchanged between guests on the same
physical server. We've observed some guests that are exchanging a large amount 
of traffic on-host, which is unusual for our systems.

So far the Linux versions we've had problems with are 4.9.32 and 4.9.58.

We are using bridged networking.

The dom0 has up until today had two vcpus allocated to it.

In our latest crash we had gro disabled on our physical devices but not on any 
vifs. Since then we have disabled gro, gso, tso on our physical devices
and all of our vifs.

Suggestions for anything else to try would be quite welcome.

Here is the latest backtrace (module lines removed to try to avoid erroneous 
search results:)

[5172268.346847] general protection fault: 0000 [#2] SMP
[5172268.347810] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G      D         
4.9.58-29.el6.x86_64 #1
[5172268.347904] Hardware name: Supermicro 
X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015
[5172268.348010] task: ffff8802fbf780c0 task.stack: ffffc900400ac000
[5172268.348072] RIP: e030:[<ffffffff81799a57>]  [<ffffffff81799a57>] 
skb_release_data+0x67/0xf0
[5172268.348171] RSP: e02b:ffffc900400af348  EFLAGS: 00010206
[5172268.348233] RAX: 0000000000000030 RBX: ffff880327f8fc40 RCX: 
0000000000000000
[5172268.348313] RDX: 0000000000000010 RSI: 000000000000006b RDI: 
ffff88021f76b000
[5172268.348392] RBP: ffffc900400af388 R08: 0000000000000000 R09: 
ffff880260b0db80
[5172268.348470] R10: 0000000000000011 R11: 0000000000000000 R12: 
ffff88021f76b000
[5172268.348548] R13: c26009c5e4500110 R14: 0000000000000000 R15: 
ffffc900400af348
[5172268.348630] FS:  0000000000000000(0000) GS:ffff880301c00000(0000) 
knlGS:000000000017f980
[5172268.348720] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[5172268.348784] CR2: 00007ff440dd6000 CR3: 00000002bd993000 CR4: 
0000000000042660
[5172268.348863] Stack:
[5172268.348894]  ffff88024723b000 ffff88021f76b000 ffff88021f76b000 
ffff88021f76b000
[5172268.348980]  ffff88021f76b000 ffffffff817eb756 0000000000000000 
ffff8802cea048c0
[5172268.349077]  ffffc900400af3a8 ffffffff81799b08 ffffc900400af450 
ffff88021f76b000
[5172268.349161] Call Trace:
[5172268.349197]  [<ffffffff817eb756>] ? nf_hook_slow+0x96/0xb0
[5172268.349259]  [<ffffffff81799b08>] skb_release_all+0x28/0x30
[5172268.349329]  [<ffffffff81799b26>] __kfree_skb+0x16/0x80
[5172268.349384]  [<ffffffffc035f0cf>] ? ebt_in_hook+0x1f/0x24 [ebtable_filter]
[5172268.349460]  [<ffffffff81799cb7>] kfree_skb+0x47/0xb0
[5172268.349520]  [<ffffffff818ddbb6>] ? _raw_spin_unlock_irqrestore+0x16/0x20
[5172268.349596]  [<ffffffff817eb756>] nf_hook_slow+0x96/0xb0
[5172268.349662]  [<ffffffffc02b6277>] __br_forward+0xd7/0x1f0 [bridge]
[5172268.349765]  [<ffffffffc02b6830>] ? br_flood+0x160/0x160 [bridge]
[5172268.349835]  [<ffffffffc02b64b3>] br_forward+0xb3/0xc0 [bridge]
[5172268.349904]  [<ffffffffc02b7e4a>] br_handle_frame_finish+0x1aa/0x340 
[bridge]
[5172268.349985]  [<ffffffffc02b7ca0>] ? br_handle_local_finish+0x50/0x50 
[bridge]
[5172268.350067]  [<ffffffffc02df215>] br_nf_hook_thresh+0x65/0xd0 
[br_netfilter]
[5172268.350148]  [<ffffffff8123c173>] ? poison_obj+0x33/0x50
[5172268.350215]  [<ffffffff817eb588>] ? nf_iterate+0x68/0xa0
[5172268.350269]  [<ffffffffc02e10dc>] 
br_nf_pre_routing_finish_ipv6+0x18c/0x210 [br_netfilter]
[5172268.350362]  [<ffffffffc02b7ca0>] ? br_handle_local_finish+0x50/0x50 
[bridge]
[5172268.350442]  [<ffffffff817eb6fb>] ? nf_hook_slow+0x3b/0xb0
[5172268.350506]  [<ffffffffc02b60db>] ? br_dev_queue_push_xmit+0xbb/0x180 
[bridge]
[5172268.350586]  [<ffffffffc02e15b4>] br_nf_pre_routing_ipv6+0xd4/0x164 
[br_netfilter]
[5172268.350670]  [<ffffffffc02e0f50>] ? br_nf_pre_routing_finish+0x320/0x320 
[br_netfilter]
[5172268.350758]  [<ffffffffc02e0147>] br_nf_pre_routing+0x1a7/0x370 
[br_netfilter]
[5172268.350846]  [<ffffffffc02b6830>] ? br_flood+0x160/0x160 [bridge]
[5172268.350909]  [<ffffffff817eb588>] nf_iterate+0x68/0xa0
[5172268.350968]  [<ffffffff817eb6fb>] nf_hook_slow+0x3b/0xb0
[5172268.351032]  [<ffffffffc02b819c>] br_handle_frame+0x1bc/0x370 [bridge]
[5172268.351104]  [<ffffffffc02b7ca0>] ? br_handle_local_finish+0x50/0x50 
[bridge]
[5172268.351186]  [<ffffffffc02b7fe0>] ? br_handle_frame_finish+0x340/0x340 
[bridge]
[5172268.351268]  [<ffffffff817afc37>] __netif_receive_skb_core+0x227/0x9e0
[5172268.351343]  [<ffffffff81024d7d>] ? xen_force_evtchn_callback+0xd/0x10
[5172268.351416]  [<ffffffff810254d6>] ? check_events+0x16/0x30
[5172268.351478]  [<ffffffff8123c173>] ? poison_obj+0x33/0x50
[5172268.351539]  [<ffffffff8123c990>] ? 
cache_alloc_debugcheck_after+0x130/0x190
[5172268.351617]  [<ffffffff8179744e>] ? __build_skb+0x2e/0xe0
[5172268.351680]  [<ffffffffc02b7fe0>] ? br_handle_frame_finish+0x340/0x340 
[bridge]
[5172268.351761]  [<ffffffff81240756>] ? kmem_cache_alloc+0xb6/0x240
[5172268.351829]  [<ffffffff81123f25>] ? ktime_get_with_offset+0x75/0x100
[5172268.351900]  [<ffffffff817b0412>] __netif_receive_skb+0x22/0x70
[5172268.351967]  [<ffffffff817b0633>] netif_receive_skb_internal+0x33/0x90
[5172268.352040]  [<ffffffff817b16c2>] napi_gro_receive+0xd2/0xf0
[5172268.352108]  [<ffffffffc0202275>] igb_clean_rx_irq+0x1e5/0x370 [igb]
[5172268.352180]  [<ffffffff816354fe>] ? scsi_run_queue+0x7e/0x90
[5172268.352245]  [<ffffffffc0202466>] igb_poll+0x66/0xb0 [igb]
[5172268.352308]  [<ffffffff817b10fb>] napi_poll+0xdb/0x1f0
[5172268.352373]  [<ffffffff8101e1ea>] ? xen_load_sp0+0x9a/0x1a0
[5172268.352432]  [<ffffffff817b12d5>] net_rx_action+0xc5/0x240
[5172268.352496]  [<ffffffff8102eb7b>] ? __switch_to+0x20b/0x690
[5172268.352560]  [<ffffffff810e6425>] ? put_prev_entity+0xe5/0x140
[5172268.352626]  [<ffffffff818dde50>] ? _raw_spin_lock_irq+0x10/0x30
[5172268.352694]  [<ffffffff8111b53c>] ? run_timer_softirq+0x27c/0x2c0
[5172268.352763]  [<ffffffff81025225>] ? xen_clocksource_read+0x15/0x20
[5172268.352833]  [<ffffffff810389c9>] ? sched_clock+0x9/0x10
[5172268.352893]  [<ffffffff810d8647>] ? sched_clock_cpu+0x87/0xb0
[5172268.352959]  [<ffffffff818e0e0e>] __do_softirq+0xce/0x2bf
[5172268.353022]  [<ffffffff810ab199>] run_ksoftirqd+0x29/0x50
[5172268.353084]  [<ffffffff810ca7ac>] smpboot_thread_fn+0x10c/0x160
[5172268.353151]  [<ffffffff810ca6a0>] ? smpboot_create_threads+0x80/0x80
[5172268.353223]  [<ffffffff810c63f5>] kthread+0xe5/0x100
[5172268.353281]  [<ffffffff810c6310>] ? __kthread_init_worker+0x40/0x40
[5172268.353351]  [<ffffffff818de2d5>] ret_from_fork+0x25/0x30
[5172268.353411] Code: fe ff 89 d0 f0 0f c1 43 20 8d 04 02 85 c0 75 65 80 3b 00 
74 44 4c 8d 7d c0 45 31 f6 49 63 c6 48 83 c0 03 48 c1 e0 04 4c 8b 2c
18 <49> 8b 45 20 a8 01 49 89 07 75 77 f0 41 ff 4d 1c 74 47 49 8b 45
[5172268.353756] RIP  [<ffffffff81799a57>] skb_release_data+0x67/0xf0
[5172268.353825]  RSP <ffffc900400af348>
[5172268.357084] ---[ end trace 592a6fbe8ea728a7 ]---
[5172268.403008] general protection fault: 0000 [#3] SMP
[5172268.403948] CPU: 1 PID: 24031 Comm: v66154b-q1-gues Tainted: G      D      
   4.9.58-29.el6.x86_64 #1
[5172268.429059] Hardware name: Supermicro 
X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015
[5172268.429161] task: ffff88028f190f00 task.stack: ffffc90048acc000
[5172268.429227] RIP: e030:[<ffffffff81799a57>]  [<ffffffff81799a57>] 
skb_release_data+0x67/0xf0
[5172268.429325] RSP: e02b:ffffc90048acfd38  EFLAGS: 00010206
[5172268.429385] RAX: 0000000000000030 RBX: ffff880327f8fec0 RCX: 
ffff880292c19a20
[5172268.429463] RDX: ffff88021f76b6e4 RSI: ffffc90048acfe08 RDI: 
ffff88021f76b600
[5172268.429541] RBP: ffffc90048acfd78 R08: 0000000000000000 R09: 
ffffc90048ab45c0
[5172268.429619] R10: ffff88021f76b600 R11: ffff88028f191948 R12: 
ffff88021f76b600
[5172268.429697] R13: 32310805f6923532 R14: 0000000000000000 R15: 
ffffc90048acfd38
[5172268.429778] FS:  0000000000000000(0000) GS:ffff880301c40000(0000) 
knlGS:000000000017f980
[5172268.429866] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[5172268.429930] CR2: 00007ff440dd6000 CR3: 00000002644e1000 CR4: 
0000000000042660
[5172268.430008] Stack:
[5172268.452945]  ffffc90048acfda8 ffffffff81427252 ffffc90048acfd98 
ffff88021f76b600
[5172268.454972]  ffff88021f76b600 ffffffffc03e0fce ffffc90048abef98 
ffff88028f190f00
[5172268.455062]  ffffc90048acfd98 ffffffff81799b08 ffffc90048ab45c0 
ffff88021f76b600
[5172268.455151] Call Trace:
[5172268.455188]  [<ffffffff81427252>] ? __radix_tree_lookup+0x92/0x110
[5172268.455261]  [<ffffffffc03e0fce>] ? xenvif_rx_copy_flush+0xbe/0x100 
[xen_netback]
[5172268.455343]  [<ffffffff81799b08>] skb_release_all+0x28/0x30
[5172268.455406]  [<ffffffff81799b26>] __kfree_skb+0x16/0x80
[5172268.455465]  [<ffffffff8142742d>] ? radix_tree_lookup+0xd/0x10
[5172268.455531]  [<ffffffff81799cb7>] kfree_skb+0x47/0xb0
[5172268.455590]  [<ffffffff8151190e>] ? info_for_irq+0xe/0x30
[5172268.455652]  [<ffffffffc03e0fce>] xenvif_rx_copy_flush+0xbe/0x100 
[xen_netback]
[5172268.455733]  [<ffffffffc03e16b5>] xenvif_rx_action+0x55/0x60 [xen_netback]
[5172268.455810]  [<ffffffffc03e183a>] xenvif_kthread_guest_rx+0x5a/0x16c 
[xen_netback]
[5172268.478007]  [<ffffffffc03e17e0>] ? xenvif_queue_carrier_on+0x70/0x70 
[xen_netback]
[5172268.478095]  [<ffffffff810c63f5>] kthread+0xe5/0x100
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.

Here is the network trace from when only a specific vif crashed:

[1465674.033243] ------------[ cut here ]------------
[1465674.033321] kernel BUG at drivers/net/xen-netback/rx.c:325!
[1465674.033385] invalid opcode: 0000 [#1] SMP
[1465674.034295] CPU: 1 PID: 26198 Comm: va7bdd6-q0-gues Not tainted 
4.9.58-29.el6.x86_64 #1
[1465674.034382] Hardware name: Supermicro 
X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015
[1465674.034484] task: ffff88028f18cec0 task.stack: ffffc90045e08000
[1465674.034550] RIP: e030:[<ffffffffc03e1436>] [<ffffffffc03e1436>] 
xenvif_rx_data_slot+0x1e6/0x1f0 [xen_netback]
[1465674.034666] RSP: e02b:ffffc90045e0bd78 EFLAGS: 00010246
[1465674.034725] RAX: 00000000000004b8 RBX: ffffc90045e0bdd8 RCX: 
ffffc90045df3df8
[1465674.034803] RDX: 0000000000000000 RSI: 0000000000000021 RDI: 
ffffc90045de93e8
[1465674.034881] RBP: ffffc90045e0bdc8 R08: 0000000000000056 R09: 
ffffc90045de93e8
[1465674.034959] R10: 0000000000327fc7 R11: 0017ffffc0000000 R12: 
0000000000000056
[1465674.035037] R13: 0000000000000056 R14: 0000000000000056 R15: 
0000000000001000
[1465674.035119] FS: 0000000000000000(0000) GS:ffff880301c40000(0000) 
knlGS:ffff880301c40000
[1465674.035208] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[1465674.035271] CR2: 00007ff440e29000 CR3: 00000002aa906000 CR4: 
0000000000042660
[1465674.035350] Stack:
[1465674.035376] ffff88028f18cf40 ffffc90042b31298 ffffc90045de9000 
ffffc90042b31298
[1465674.035466] ffffc90000000000 ffffc90045de9000 ffffc90045e0bdd8 
ffff8802ac0e98c0
[1465674.035555] ffff880166f4bb00 ffff88028f18cec0 ffffc90045e0be38 
ffffffffc03e15ac
[1465674.035645] Call Trace:
[1465674.035679] [<ffffffffc03e15ac>] xenvif_rx_skb+0x16c/0x220 [xen_netback]
[1465674.035755] [<ffffffffc03e16a1>] xenvif_rx_action+0x41/0x60 [xen_netback]
[1465674.035832] [<ffffffffc03e183a>] xenvif_kthread_guest_rx+0x5a/0x16c 
[xen_netback]
[1465674.035916] [<ffffffffc03e17e0>] ? xenvif_queue_carrier_on+0x70/0x70 
[xen_netback]
[1465674.036005] [<ffffffff810c63f5>] kthread+0xe5/0x100
[1465674.036062] [<ffffffff810c6310>] ? __kthread_init_worker+0x40/0x40
[1465674.036162] [<ffffffff818de2d5>] ret_from_fork+0x25/0x30
[1465674.036223] Code: c8 03 eb c1 48 85 c0 74 96 b8 04 00 00 00 eb 91 8b 82 cc 
00 00 00 48 8b 92 d0 00 00 00 48 8b 44 02 08 48 89 43 10 e9 68 ff ff
ff <0f> 0b eb fe 66 0f 1f 44 00 00 55 48 89 e5 41 54 53 48 83 ec 50
[1465674.036565] RIP [<ffffffffc03e1436>] xenvif_rx_data_slot+0x1e6/0x1f0 
[xen_netback]
[1465674.036653] RSP <ffffc90045e0bd78>
[1465674.040869] ---[ end trace 592a6fbe8ea728a6 ]---

Here is a third backtrace, from before gro was disabled:

[1838269.012349] general protection fault: 0000 [#1] SMP
[1838269.013521] CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.9.39-32
[1838269.013637] Hardware name: Supermicro Super Server/X10DRi-LN4+, BIOS 2.0a 
09/16/2016
[1838269.013743] task: ffff88030008c4c0 task.stack: ffffc90041978000
[1838269.013826] RIP: e030:[<ffffffff81403b66>]  [<ffffffff81403b66>] 
memcpy_erms+0x6/0x10
[1838269.013952] RSP: e02b:ffffc9004197bac0  EFLAGS: 00010202
[1838269.014026] RAX: ffff88032fcafe16 RBX: 0000000000000004 RCX: 
0000000000000004
[1838269.014124] RDX: 0000000000000004 RSI: 62a16ddedc6dbcb3 RDI: 
ffff88032fcafe16
[1838269.014222] RBP: ffffc9004197bb20 R08: 0000000000000004 R09: 
0000000000000004
[1838269.014320] R10: ffff88026ae89500 R11: 0000000044639632 R12: 
0000000000000048
[1838269.014417] R13: 0000000000000000 R14: 0000000044639632 R15: 
0000000000000048
[1838269.014519] FS:  0000000000000000(0000) GS:ffff880306640000(0000) 
knlGS:ffff880306640000
[1838269.014629] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[1838269.014709] CR2: ffffffffff600400 CR3: 0000000051939000 CR4: 
0000000000042660
[1838269.014808] Stack:
[1838269.014840]  ffffffff81744c17 ffff88026ae89500 0000000044639632 
ffff88030008c4c0
[1838269.014952]  ffffffff00000004 0000000000000004 ffff88032fcafe16 
ffff88026ae89500
[1838269.015064]  0000000000000004 0000000000000004 000000000000004c 
0000000000000028
[1838269.015176] Call Trace:
[1838269.015217]  [<ffffffff81744c17>] ? skb_copy_bits+0x137/0x2c0
[1838269.015299]  [<ffffffff8174794f>] __pskb_pull_tail+0x7f/0x3b0
[1838269.015382]  [<ffffffff817cad15>] tcp_gro_receive+0x2c5/0x300
[1838269.015465]  [<ffffffff8185495a>] tcp6_gro_receive+0x13a/0x1a0
[1838269.015547]  [<ffffffff818542d6>] ipv6_gro_receive+0x1c6/0x380
[1838269.015630]  [<ffffffff8175b649>] dev_gro_receive+0x269/0x3b0
[1838269.015712]  [<ffffffff8175b7c8>] napi_gro_receive+0x38/0xf0
[1838269.015796]  [<ffffffffc017eb4e>] igb_clean_rx_irq+0x38e/0x690 [igb]
[1838269.015886]  [<ffffffffc017f1b2>] igb_poll+0x362/0x720 [igb]
[1838269.015968]  [<ffffffff810df33e>] ? dequeue_entity+0x26e/0xa90
[1838269.016051]  [<ffffffff8101c9fb>] ? xen_mc_flush+0x17b/0x1b0
[1838269.016131]  [<ffffffff8175b1d8>] net_rx_action+0x158/0x360
[1838269.016212]  [<ffffffff818806b1>] __do_softirq+0xd1/0x283
[1838269.016290]  [<ffffffff810caae0>] ? sort_range+0x30/0x30
[1838269.016366]  [<ffffffff810adc99>] run_ksoftirqd+0x29/0x50
[1838269.016443]  [<ffffffff810cabf0>] smpboot_thread_fn+0x110/0x160
[1838269.016525]  [<ffffffff810c6cd7>] kthread+0xd7/0xf0
[1838269.016595]  [<ffffffff810c6c00>] ? kthread_park+0x60/0x60
[1838269.016673]  [<ffffffff8187db95>] ret_from_fork+0x25/0x30
[1838269.016758] Code: ff 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 
03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89
d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38
[1838269.017183] RIP  [<ffffffff81403b66>] memcpy_erms+0x6/0x10
[1838269.017264]  RSP <ffffc9004197bac0>
[1838269.020618] ---[ end trace 3506ce1d7200529a ]---
[1838269.079891] Kernel panic - not syncing: Fatal exception in interrupt
[1838269.080014] Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.

Here is a fourth backtrace, also from before gro was disabled:

[4083386.299221] ------------[ cut here ]------------
[4083386.299358] WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:1473 
inet_gro_complete+0xbb/0xd0
[4083386.300888] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.39-32
[4083386.301002] Hardware name: Supermicro Super Server/X10DRi-LN4+, BIOS 2.0a 
09/16/2016
[4083386.301109]  ffff880306603d90 ffffffff813f5935 0000000000000000 
0000000000000000
[4083386.301221]  ffff880306603dd0 ffffffff810a7e01 000005c18174578a 
ffff8802f94a9a00
[4083386.301333]  ffff8802f0824450 0000000000000000 0000000000000040 
0000000000000040
[4083386.301445] Call Trace:
[4083386.301483]  <IRQ> [4083386.301519]  [<ffffffff813f5935>] 
dump_stack+0x63/0x8e
[4083386.301596]  [<ffffffff810a7e01>] __warn+0xd1/0xf0
[4083386.301665]  [<ffffffff810a7eed>] warn_slowpath_null+0x1d/0x20
[4083386.301747]  [<ffffffff817d9b2b>] inet_gro_complete+0xbb/0xd0
[4083386.301830]  [<ffffffff8175ac53>] napi_gro_complete+0x73/0xa0
[4083386.301911]  [<ffffffff8175acdf>] napi_gro_flush+0x5f/0x80
[4083386.301988]  [<ffffffff8175ad6a>] napi_complete_done+0x6a/0xb0
[4083386.302075]  [<ffffffffc01881dd>] igb_poll+0x38d/0x720 [igb]
[4083386.302156]  [<ffffffffc017ec4e>] ? igb_msix_ring+0x2e/0x40 [igb]
[4083386.302255]  [<ffffffff8110544b>] ? __handle_irq_event_percpu+0x4b/0x1a0
[4083386.302349]  [<ffffffff8175b1d8>] net_rx_action+0x158/0x360
[4083386.302430]  [<ffffffff818806b1>] __do_softirq+0xd1/0x283
[4083386.302507]  [<ffffffff810adf09>] irq_exit+0xe9/0x100
[4083386.302580]  [<ffffffff814dd655>] xen_evtchn_do_upcall+0x35/0x50
[4083386.302665]  [<ffffffff8187e9de>] xen_do_hypervisor_callback+0x1e/0x40
[4083386.302754]  <EOI> [4083386.302787]  [<ffffffff810013aa>] ? 
xen_hypercall_sched_op+0xa/0x20
[4083386.302876]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[4083386.302965]  [<ffffffff81022560>] ? xen_safe_halt+0x10/0x20
[4083386.303043]  [<ffffffff8187ce5e>] ? default_idle+0x1e/0xd0
[4083386.303122]  [<ffffffff81036a6f>] ? arch_cpu_idle+0xf/0x20
[4083386.303200]  [<ffffffff8187d27c>] ? default_idle_call+0x2c/0x40
[4083386.303284]  [<ffffffff810ed51c>] ? cpu_startup_entry+0x1ac/0x240
[4083386.303370]  [<ffffffff81870767>] ? rest_init+0x77/0x80
[4083386.303462]  [<ffffffff81faa143>] ? start_kernel+0x4a7/0x4b4
[4083386.303568]  [<ffffffff81fa9a8a>] ? set_init_arg+0x55/0x55
[4083386.303670]  [<ffffffff81fa95d7>] ? x86_64_start_reservations+0x24/0x26
[4083386.303776]  [<ffffffff81fb0cd8>] ? xen_start_kernel+0x555/0x561
[4083386.303873] ---[ end trace 8294f59ced689507 ]---
[4083386.303958] general protection fault: 0000 [#1] SMP
[4083386.305179] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       
4.9.39-32
[4083386.305307] Hardware name: Supermicro Super Server/X10DRi-LN4+, BIOS 2.0a 
09/16/2016
[4083386.305414] task: ffffffff81e0e540 task.stack: ffffffff81e00000
[4083386.305498] RIP: e030:[<ffffffff817456c3>]  [<ffffffff817456c3>] 
skb_release_data+0x73/0xf0
[4083386.305617] RSP: e02b:ffff880306603d90  EFLAGS: 00010206
[4083386.305692] RAX: 0000000000000030 RBX: f5b36db76bd162c7 RCX: 
ffffffff81e60048
[4083386.305790] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 
ffff8802f94a9a00
[4083386.305887] RBP: ffff880306603db0 R08: 0000000000004277 R09: 
0000000000000000
[4083386.305985] R10: 0000000000000005 R11: 0000000000000002 R12: 
0000000000000000
[4083386.306083] R13: ffff8802f94a9a00 R14: ffff88032f527740 R15: 
0000000000000040
[4083386.306186] FS:  0000000000000000(0000) GS:ffff880306600000(0000) 
knlGS:0000000000000000
[4083386.306296] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[4083386.306407] CR2: 0000000001692ed8 CR3: 000000022b3c9000 CR4: 
0000000000042660
[4083386.306505] Stack:
[4083386.306537]  ffff8802f94a9a00 ffff8802f94a9a00 ffffffff8175ac3e 
0000000000000040
[4083386.306649]  ffff880306603dc8 ffffffff81745764 ffff8802f94a9a00 
ffff880306603df0
[4083386.306762]  ffffffff817457c2 ffff8802f94a9a00 ffff8802f0824450 
0000000000000000
[4083386.306874] Call Trace:
[4083386.306911]  <IRQ> [4083386.306944]  [<ffffffff8175ac3e>] ? 
napi_gro_complete+0x5e/0xa0
[4083386.307038]  [<ffffffff81745764>] skb_release_all+0x24/0x30
[4083386.307133]  [<ffffffff817457c2>] kfree_skb+0x32/0x90
[4083386.307206]  [<ffffffff8175ac3e>] napi_gro_complete+0x5e/0xa0
[4083386.307287]  [<ffffffff8175acdf>] napi_gro_flush+0x5f/0x80
[4083386.307365]  [<ffffffff8175ad6a>] napi_complete_done+0x6a/0xb0
[4083386.307449]  [<ffffffffc01881dd>] igb_poll+0x38d/0x720 [igb]
[4083386.307530]  [<ffffffffc017ec4e>] ? igb_msix_ring+0x2e/0x40 [igb]
[4083386.307617]  [<ffffffff8110544b>] ? __handle_irq_event_percpu+0x4b/0x1a0
[4083386.307720]  [<ffffffff8175b1d8>] net_rx_action+0x158/0x360
[4083386.307800]  [<ffffffff818806b1>] __do_softirq+0xd1/0x283
[4083386.307877]  [<ffffffff810adf09>] irq_exit+0xe9/0x100
[4083386.307949]  [<ffffffff814dd655>] xen_evtchn_do_upcall+0x35/0x50
[4083386.308034]  [<ffffffff8187e9de>] xen_do_hypervisor_callback+0x1e/0x40
[4083386.308124]  <EOI> [4083386.308156]  [<ffffffff810013aa>] ? 
xen_hypercall_sched_op+0xa/0x20
[4083386.308246]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[4083386.308334]  [<ffffffff81022560>] ? xen_safe_halt+0x10/0x20
[4083386.308413]  [<ffffffff8187ce5e>] ? default_idle+0x1e/0xd0
[4083386.308491]  [<ffffffff81036a6f>] ? arch_cpu_idle+0xf/0x20
[4083386.308568]  [<ffffffff8187d27c>] ? default_idle_call+0x2c/0x40
[4083386.308651]  [<ffffffff810ed51c>] ? cpu_startup_entry+0x1ac/0x240
[4083386.308737]  [<ffffffff81870767>] ? rest_init+0x77/0x80
[4083386.308811]  [<ffffffff81faa143>] ? start_kernel+0x4a7/0x4b4
[4083386.308890]  [<ffffffff81fa9a8a>] ? set_init_arg+0x55/0x55
[4083386.308968]  [<ffffffff81fa95d7>] ? x86_64_start_reservations+0x24/0x26
[4083386.309060]  [<ffffffff81fb0cd8>] ? xen_start_kernel+0x555/0x561
[4083386.309144] Code: f0 41 0f c1 46 20 39 c2 74 09 5b 41 5c 41 5d 41 5e 5d c3 
45 31 e4 41 80 3e 00 74 39 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 1c
06 <48> 8b 43 20 a8 01 75 6f f0 ff 4b 1c 74 55 48 8b 03 48 c1 e8 33
[4083386.309571] RIP  [<ffffffff817456c3>] skb_release_data+0x73/0xf0
[4083386.309658]  RSP <ffff880306603d90>
[4083386.313000] ---[ end trace 8294f59ced689508 ]---
[4083386.389667] Kernel panic - not syncing: Fatal exception in interrupt
[4083386.389791] Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel