Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote:
> > > > > > But seems like this patch is not stable enough yet and has its own
> > > > > > issue -- memory is not properly released?
> > > > >
> > > > > I know. I've been working on improving it this morning and I'm
> > > > > attaching an updated version below.
> > > > >
> > > > Good news.
> > > > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > > > panic, at least for a little while.
> > > > I'm going to keep it up and running for a while to see if it's stable 
> > > > over time.
> > >
> > > Thanks again for all the testing. Do you see any difference
> > > performance wise?
> > I'm still on a *debug* kernel build to capture any potential panic --
> > none so far -- no performance testing yet.
> > Since I'm a home user with a relatively lightweight workload, so far I
> > didn't observe any difference in daily usage.
> >
> > I did some quick iperf3 testing just now.
>
> Thanks for doing this.
>
> > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
> > The peak is roughly 12 Gbits/s when domU is the server.
> > But I do see regression down to ~8.5 Gbits/s when I repeat the test in
> > a short burst.
> > The regression can recover when I leave the system idle for a while.
> >
> > When dom0 is the iperf3 server, the transfer rate is much lower, down
> > all the way to 1.x Gbits/s.
> > Sometimes, I can see the following kernel log repeats during the
> > testing, likely contributing to the slowdown.
> >              interrupt storm detected on "irq2328:"; throttling interrupt 
> > source
>
> I assume the message is in the domU, not the dom0?
Yes, in the TrueNAS domU.
BTW, I rebooted back to the stock kernel and the message is no longer observed.

With the stock kernel, the transfer rate from dom0 to nas domU can be
as high as 30Gbps.
The variation is still observed, sometimes down to ~19Gbps. There is
no retransmission in this direction.

For the reverse direction, the observed low transfer rate still exists.
It's still within the range of 1.x Gbps, but should still be better
than the previous test.
The huge number of re-transmission is still observed.
The same behavior can be observed on a stock FreeBSD 12.2 image, so
this is not specific to TrueNAS.

According to the packet capture, the re-transmission appears to be
caused by packet reorder.
Here is one example incident:
1. dom0 sees a sequence jump in the incoming stream and begins to send out SACKs
2. When SACK shows up at domU, it begins to re-transmit lost frames
   (the re-transmit looks weird since it show up as a mixed stream of
1448 bytes and 12 bytes packets, instead of always 1448 bytes)
3. Suddenly the packets that are believed to have lost show up, dom0
accept them as if they are re-transmission
4. The actual re-transmission finally shows up in dom0...
Should we expect packet reorder on a direct virtual link? Sounds fishy to me.
Any chance we can get this re-transmission fixed?

So looks like at least the imbalance between two directions are not
related to your patch.
Likely the debug build is a bigger contributor to the perf difference
in both directions.

I also tried your patch on a release build, and didn't observe any
major difference in iperf3 numbers.
Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.

>
> > Another thing that looks alarming is the retransmission is high:
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec   212 MBytes  1.78 Gbits/sec  110    231 KBytes
> > [  5]   1.00-2.00   sec   230 MBytes  1.92 Gbits/sec    1    439 KBytes
> > [  5]   2.00-3.00   sec   228 MBytes  1.92 Gbits/sec    3    335 KBytes
> > [  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec    1    486 KBytes
> > [  5]   4.00-5.00   sec   201 MBytes  1.69 Gbits/sec  812    258 KBytes
> > [  5]   5.00-6.00   sec   179 MBytes  1.51 Gbits/sec    1    372 KBytes
> > [  5]   6.00-7.00   sec  50.5 MBytes   423 Mbits/sec    2    154 KBytes
> > [  5]   7.00-8.00   sec   194 MBytes  1.63 Gbits/sec  339    172 KBytes
> > [  5]   8.00-9.00   sec   156 MBytes  1.30 Gbits/sec  854    215 KBytes
> > [  5]   9.00-10.00  sec   143 MBytes  1.20 Gbits/sec  997   93.8 KBytes
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-10.00  sec  1.76 GBytes  1.51 Gbits/sec  3120             
> > sender
> > [  5]   0.00-10.45  sec  1.76 GBytes  1.44 Gbits/sec                  
> > receiver
>
> Do you see the same when running the same tests on a debug kernel
> without my patch applied? (ie: a kernel build yourself from the same
> baseline but just without my patch applied)
>
> I'm mostly interested in knowing whether the patch itself causes any
> regressions from the current state (which might not be great already).
>
> >
> > 2. between a remote box <=> nas domU, through a 1Gbps ethernet cable.
> > Roughly saturate the link when domU is the server, without obvious perf drop
> > When domU running as a client, the achieved BW is ~30Mbps lower than the 
> > peak.
> > Retransmission sometimes also shows up in this scenario, more
> > seriously when domU is the client.
> >
> > I cannot test with the stock kernel nor with your patch in release
> > mode immediately.
> >
> > But according to the observed imbalance between inbounding and
> > outgoing path, non-trivial penalty applies I guess?
>
> We should get a baseline using the same sources without my path
> applied.
>
> Thanks, Roger.