[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0


  • To: G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Mon, 20 Dec 2021 14:51:59 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mv0+3kzokXdmEQ/Fdh+QFho2JUgAqqg1nHlzm3hkPWg=; b=b9aUdgQ3MDe1OmOdjCRYEEyD4wd6oJy+keEhQrEvSielTIXbj6ERAPfy/k2FHlZdS0hebi+4rhJB8QeEHmFcIJGgp+uarlc2fJFU5D74zjZ/N76+RLcrAmDodJcsrdIaWEbjB7jLsotCssAFufVcYRDiDCtXypI2lWuWnnZ6nzYj82/gnas6xRi6fHst82mLfTeFuN9xR/UHt2CXUoUr5Z9AoreaEwSBzlRH/Kqx5rodWoGtV279Q4bmFH8j9PWtewaxrJVxzk4e4OvaUF4bOYIkoOh6xnFw7lJcUN+h6RLaSIFJafzaDCDgofgzuSaxZVDQApBsXVA1i3HvoeCZ5w==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I1O817cJsLtGb2t9hvCnTh+e0EBKpeE361UN+H7ROPuEm1cuCq5XSp/1wgvKDFxW4mnJhycYHVJ3oqhbNcsIl05IQiB4BJCAcQ+VGtiMAEU1QNeJQSfdxW4pOzTUheePBtFcnakAXdugEcd/fxuF+x3Gu7icZVFbQDe/0/ky7p7pe2L/pBImJaNDgkjom2M9ettMiAn83gAQ4giVG1NQRD906KwgG629tijZ+iYRBHh/xoHpRoJTpbuFtPuYWIjlrt3dGNsqF/XEPvWGYaDhYiWMMMXgh6lutUymi/aX0qCAL6frKa+qfUf1BcEBzbQZB802KRgJxgZRUlyGK84p6g==
  • Authentication-results: esa4.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: xen-devel <xen-devel@xxxxxxxxxxxxx>
  • Delivery-date: Mon, 20 Dec 2021 13:52:52 +0000
  • Ironport-data: A9a23:VPGmlaBJ5RgWYBVW/+fkw5YqxClBgxIJ4kV8jS/XYbTApDIj0zUCx mtMWjyBaauCZ2qjeN1xYY61oRsFscSHndFiQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA xA2M4GYRCwMo/u1Si6FatANl1ElvU2zbue6WL6s1hxZH1c+EX5700o7wobVv6Yz6TSHK1LV0 T/Ni5W31G+Ng1aY5UpNtspvADs21BjDkGtwUm4WPJinj3eH/5UhN7oNJLnZEpfNatI88thW5 Qr05OrREmvxp3/BAz4++1rxWhVirrX6ZWBihpfKMkQLb9crSiEai84G2PQghUh/jBWMkO5w5 s53iYWTagdyMvHWuf0gekwNe81+FfUuFL7vJHG+tYqYzlHccmuqyPJrZK00FdRGoKAtWzgIr KFGbmBWBvyAr7veLLaTUO5ji95lNMD2FIgepmth3XfSCvNOrZXrHf+XvYICh21YasZmLM/aR ZMAMWVVUBGcfiBRGFMSV70CpbL97pX4W2IB8w/EzUYt2EDDwQo03LXzPd79ft2RWd4Tjkue4 GXc8AzRDBAAOdmS1TeC6FqxneLVhmXgX58IH7C28eRljRuVy3B7NfENfQLl+7/j0Bf4Ao8Bb RxPksYzkUQs3EuLS9bDXjOjmkKdnT0cANYMCcwj1SjYn8I4/D2lLmQDSzdAbvkvu8k3WSEm2 ze1oj/5OdB8mObLECzAr994uRv3YHFIdjFaOUfoWCNcu4G7yLzfmC4jWTqK/ESdqtTuUQ/9z DmRxMTVr+VC1JVbv0lXEL2uvt5NmnQrZlNsjuk0djj8hu+cWGJCT9b3gWU3Fd4acO6koqCp5 RDoYfS24uEUFo2qnyeQWugLF7zBz6/ba2yN2g8zQcd7pmvFF5ufkWd4um4WGauUGpxcJW+Bj LH75Gu9G6O/zFP1NPQqMupd+uwhzLT6FMSNaxwnRoEmX3SFTyfepHsGTRfJhwjFyRFw+Ylia cbzWZv9Vh4yVPU4pAdass9AiNfHMAhlnjiNLX06pjz6uYejiIm9Fe1YbQDQN79ht8tpYmz9q r5iCidD8D0GOMXWaSjL648Da1cMKHkwH5ftrMJLMOWEJ2Jb9KsJUae5LWoJd9M3kqJLuP3P+ 33hCEZUxECm3S/MKBmQa2AlY7TqBM4toXU+NC0qHFCpx3l8Ptr/sPZBL8M6Les96ehu7f9oV P1ZKc+ONetCF2bc8DMHYJij8IE7LEa3hRiDNjaOaSQke8IyXBTA/9LpJ1O99CQHAietm9E5p rmsilHSTZYZHlwwB8fKcvO/iVi2uCFFyu51WkLJJPhVeVntr9c2e3Cg0KdvLphVexvZxzac2 wKHOjsipLHA890v7d3EpaGYtIP1QeFwKVVXQjvA5rGsOCiEomf6md1cUPyFdCz2XX/v/Pnwf v1cyvzxPaFVnFtOtIYgQb9nwbhnuonqrr5eiA9lAG/KfxKgDbY5eiuK2sxGt6tswL5FuFTpB hLTq4cCYbjZatn4FFMxJRY+arXR3P4ZrTDe8PApLRio/yRw5reGDR1fMhTkZPax91ep3FfJG dschfM=
  • Ironport-hdrordr: A9a23:r25Yga2RWN8KBpNRNOI5EQqjBVByeYIsimQD101hICG9Lfb2qy n+ppgmPEHP5Qr5OEtApTiBUJPwJk800aQFm7X5XI3SJzUO3VHHEGgM1/qB/9SNIVyaygcZ79 YcT0EcMqyPMbEZt7eC3ODQKb9Jq7PmgcOVbKXlvg9QpGlRGt5dBmxCe2Cm+yNNNW177c1TLu vh2iMLnUvpRV0nKuCAQlUVVenKoNPG0LrgfB49HhYirC2Dlymh5rLWGwWRmk52aUIE/Z4StU z+1yDp7KSqtP+2jjfaym/o9pxT3P/s0MFKCsCggtUcbh/slgGrToJ8XKDqhkF/nMifrHIR1P XcqRYpOMp+r1vXY2GOuBPonzLt1T4/gkWSvmOwsD/Gm4jUVTg6A81OicZyaR3C8Xctu9l6ze Ziw3+Zn4A/N2KOoA3No/zzEz16nEu9pnQv1cQJiWZEbIcYYLhN6aQC4UJuFosaFi6S0vFqLA BXNrCc2B9qSyLbU5iA1VMfg+BEH05DUytue3Jy9PB8iFNt7TJEJ0hx/r1qop5PzuN5d3B+3Z W1Dk1frsA6ciYnV9MNOA4/e7rFNoXse2O7DIvAGyWvKEk4U0i92aIfpo9FoN2XRA==
  • Ironport-sdr: v6J7V0aeymLCUi+aXGJM6GGy6qJXtzPIRDygLsUFLu0V+MXaHR3+gSkm5JxXPbqt8BBFWYpS2I VL6D4pL9bO2p1al7nTIPYNdR0qSDdW5oky2rqdkctSx+kYVW5o+u6XXD+hzS9QDwOz9IDpSJ3o gCIYdd8XYpyko2PyviWuoWUXW4WNsrYRZN45SIFULa7P+v7xy/9t+epJKm0iGUp8NUvYv0s4Eo 1wxxa+by1a2f2zXbm4ejfEJUvZ5Nw2fpgUfl8O/BRv71pwNnG8h8CNgzjmqPvP6FptbpeVGb1W OI9GU7GCHGIJFEhv5zsESV7h
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Sun, Dec 19, 2021 at 02:35:56AM +0800, G.R. wrote:
> Hi all,
> 
> I ran into the following error report in the DOM0 kernel after a recent 
> upgrade:
> [  501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset:
> 2872, size: 1460
> [  501.840828] vif vif-1-0 vif1.0: fatal error; disabling device
> [  501.841076] xenbr0: port 2(vif1.0) entered disabled state
> Once this error happens, the DOM-U behind this vif is no-longer
> accessible. And recreating the same DOM-U does not fix the problem.
> Only a reboot on the physical host machine helps.
> 
> The problem showed up after a recent upgrade on the DOM-U OS from
> FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while
> leaving other services like NFS intact.
> The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively.
> So far I have tried the following combos:
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed
> 
> I plan to try out the XEN 4.14 version which is the latest I can get
> from the distro (Debian).
> If that still does not fix the problem, I would build the 4.16 version
> from source as my last resort.
> 
> I have to admit that this trial process is blind as I have no idea
> which component in the combo is to be blamed. Is it a bug in the
> backend-driver, frontend-driver or the hypervisor itself? Or due to
> incompatible versions? Any suggestion on other diagnose ideas (e.g.
> debug logs) will be welcome, while I work on the planned experiments.

This is a bug in FreeBSD netfront, so no matter which Linux or Xen
version you use.

Does it make a difference if you disable TSO and LRO from netfront?

$ ifconfig xn0 -tso -lro

Do you have instructions I can follow in order to try to reproduce the
issue?

Thanks, Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.