|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: Xen 4.18rc/ARM64 on Raspberry Pi 4B: Traffic in DomU crashing Dom0 when using VLANs
Elliot, Bertrand, George: Thanks for your replies. Am 23.01.2024 um 09:29 schrieb Bertrand Marquis: Hi, will try to explain some of the messages here after but I am not sure of the reason of the crash so I will just set some pointers...On 22 Jan 2024, at 11:46, George Dunlap <george.dunlap@xxxxxxxxx> wrote: On Fri, Jan 19, 2024 at 8:32 PM Elliott Mitchell <ehem+xen@xxxxxxx> wrote:On Sun, Jan 14, 2024 at 10:54:24PM +0100, Paul Leiber wrote:Am 22.10.2023 um 07:42 schrieb Paul Leiber:Am 13.10.2023 um 20:56 schrieb Paul Leiber:Hi Xen developers list, TL;DR: ------ Causing certain web server traffic on a secondary VLAN on Raspberry Pi under vanilla Debian/UEFI in combination with Xen leads to complete system reboot (watchdog triggering for Dom0). Other strange things are happening. Description: ---------- I recently set up Xen (self compiled, Version 4.18-rc) on a Raspberry Pi 4B (on vanilla Debian Bookworm, UEFI boot mode). Until some time ago, everything worked well with Dom0, one DomU and one bridge. Then I wanted to actually make use of the virtualization and started to set up a second Debian Bookworm DomU (using xen-create-image) for monitoring my systems with zabbix (a webserver based system monitoring solution). The bridge used for this setup was the device bridging the hardware NIC. I installed zabbix, set it up, and everything went well, I could access the web interface without any problem. Then I set up VLANs (initally using VLAN numbers 1 and 2) to separate network traffic between the DomUs. I made the existing device bridge VLAN 1 (bridge 1) and created a secondary device for bridging VLAN 2 (bridge 2). Using only bridge 1 / VLAN 1 everything works well, I can access the zabbix web interface without any noticeable issue. After switching the zabbix DomU to VLAN 2 / bridge 2, everything seemingly keeps on working well, I can ping different devices in my network from the zabbix DomU and vice versa, I can ssh into the machine. However, as soon as I remotely access the zabbix web interface, the complete system (DomUs and Dom0) becomes unresponsive and reboots after some time (usually seconds, sometimes 1-2 minutes). The reboot is reliably reproducable. I didn't see any error message in any log (zabbix, DomU syslog, Dom0 syslog) except for the following lines immediately before the system reboots on the Xen serial console: (XEN) Watchdog timer fired for domain 0 (XEN) Hardware Dom0 shutdown: watchdog rebooting machine As soon as I change the bridge to bridge 1 (with or without VLAN setup), the web interface is accessible again after booting the zabbix DomU, no reboots. So I assume that causing specific traffic on the virtual NIC when using a VLAN setup with more than one VLAN under Xen makes the Dom0 system hard crash. Of course, there might be other causes that I'm not aware of, but to me, this seems to be the most likely explanation right now. What I tried: ------------- 1. I changed the VLAN numbers. First to 101, 102, 103 etc. This was when I noticed another strange thing: VLANs with numbers >99 simply don't work on my Raspberry Pi under Debian, with or without Xen. VLAN 99 works, VLAN 100 (or everything else >99 that I tried) doesn't work. If I choose a number >99, the VLAN is not configured, "ip a" doesn't list it. Other Debian systems on x64 architecture don't show this behavior, there, it was no problem to set up VLANs > 99. Therefore, I've changed the VLANs to 10, 20, 30 etc., which worked. But it didn't solve the initial problem of the crashing Dom0 and DomUs. 2. Different bridge options, without noticable effect: bridge_stp off # dont use STP (spanning tree proto) bridge_waitport 0 # dont wait for port to be available bridge_fd 0 # no forward delay 3. Removing IPv6: No noticable effect. 4. Network traffic analysis: Now, here it becomes _really_ strange. I started tcpdumps on Dom0, and depending on on which interface/bridge traffic was logged, the problem went away, meaning, the DomU was running smoothly for hours, even when accessing the zabbix web interface. Stopping the log makes the system crash (as above, after seconds up to 1-2 minutes) reproducably if I access the zabbix web interface. Logging enabcm6e4ei0 (NIC): no crashes Logging enabcm6e4ei0.10 (VLAN 10): instant crash Logging enabcm6e4ei0.20 (VLAN 20): no crashes Logging xenbr0 (on VLAN 10): instant crash Logging xenbr1 (on VLAN 20): no crashes I am clinging to the thought that there must be a rational explanation for why logging the traffic on certain interfaces/bridges should avoid the crash of the complete system, while logging other interfaces/bridges doesn't. I myself can't think of one. I checked the dumps of enabcm6e4ei0.10 and xenbr0 (where the system crashes) with wireshark, nothing sticks out to me (but I am really no expert in analyzing network traffic). Dumps can be provided. 5. Watchdog: I tried to dig deeper into the cause for the watchdog triggering. However, I didn't find any useful documentation on the web on how the watchdog works or how to enable logging. 6. Eliminating Xen as cause: I booted the Debian system (which in Xen setup would be Dom0) without Xen and set it up to use the VLAN 20 bridge (the same that leads to a reboot when using it in the DomU) as primary network interface. Everything seemed to be working, I could download large files from the internet without any problem. Setting up Zabbix on the base Debian system showed that the same setup (VLANs 10 and 20, bridges 1 and 2, using bridge 2 as interface for Zabbix) without Xen is working reliably, no reboots. This points to some Xen related component being the root cause, I think. 7. Eliminating Apache as root cause: Reloading the Apache starting page hosted on DomU several times per second didn't lead to a reboot. 8. Recompiling Xen: Independent of which Xen master branch version I was using (all 4.18), the behavior was the same. I didn't get Xen working on ARM64/UEFI in version 4.17. Current situation:(XEN) d3v0 Unhandled SMC/HVC: 0x84000050(XEN) d3v0 Unhandled SMC/HVC: 0x8600ff01 (XEN) d3v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0 (XEN) common/grant_table.c:1909:d3v0 Expanding d3 grant table from 1 to 2 frames (XEN) common/grant_table.c:1909:d3v0 Expanding d3 grant table from 2 to 3 frames (XEN) common/grant_table.c:1909:d3v0 Expanding d3 grant table from 3 to 4 frames------------------ I am out of ideas what to do next. Everything that was recommended to me on xen-users didn't lead to significant insight or solve the problem. I'd appreciate any hints how to troubleshoot this and/or how to proceed otherwise.O.k., let's try to break that issue down. Firstly, how can I get more information on why the Xen watchdog triggers? Is there documentation? Are there any logs? I couldn't find anything useful with my search skills. Is this also relevant or not?I couldn't identify this yet. There is a mailing list exchange that could be related: https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00382.htmlAlthough I'm a little bit puzzled because when comparing with the other messages, this log meassage seems to be missing some parts "vGICD: unhandled word write 0x000000" instead of "vGICD: unhandled word write 0x000000xxxxxxx to y" ffffffff to ICACTIVER0 (XEN) common/grant_table.c:1909:d2v0 Expanding d2 grant table from 1 to 2 frames (XEN) common/grant_table.c:1909:d2v0 Expanding d2 grant table from 2 to 3 frames (XEN) common/grant_table.c:1909:d2v0 Expanding d2 grant table from 3 to 4 frames (XEN) Watchdog timer fired for domain 0 (XEN) Hardware Dom0 shutdown: watchdog rebooting machineI'm unsure whose attention to draw to this report. This might be a scheduler issue since the watchdog timer is triggering. This might be an ACPI issue as ACPI is in use here. This might be an ARM Linux kernel issue. In the end this is someone running into trouble with Xen on an ARM device. Yet despite bringing up the issue hasn't gotten any help...Hey Elliot, Thanks for raising the visibility of this. I'm not familiar with ARM, but if I were investigating this I'd try to figure out what the "unhandled" error messages are. "gnttab_mark_dirty not implemented yet" looks pretty sus too, and also sounds like it might be something ARM-specific.I tried to explain those and they are not the reason of the problem.I don't see anything suspicious WRT the scheduler, but a simple way to test that would be to set the scheduler to credit1 and see if that changes things.I would definitely suggest to investigate the 2 unhandled SMC/HVC calls as if one of them is to enable some clock it could explain why the system is getting stuck at some point (maybe waiting for something to be started). Other than that i did not see anything that could point to an obvious issue. I'll try to use credit1 scheduler and report results when I've got some more time, just to make sure. From what I could find out in ARM documentation, the unhandled calls seem to be the following: 0x84000050 = TRNG_VERSION, returns the implemented TRNG (True Random Number Generator) ABI version [1] 0x8600ff01 = Call UID Query for Vendor Specific Hypervisor Service, Returns a unique identifier of the service provider [2] Does this make sense? What could be next steps? Thanks, Paul [1] https://developer.arm.com/documentation/den0098/latest/[2] https://documentation-service.arm.com/static/628b755ce3c4322a76af56de?token=
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |