Re: 5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith <andy@xxxxxxxxxxxxxx>]

Hi Jan,

On Wed, Jul 21, 2021 at 04:49:26PM +0200, Jan Beulich wrote:
> On 21.07.2021 16:19, Andy Smith wrote:
> > I understand that below 4GiB memory use of swiotlb is disabled so
> > all the time previously this was not used, and now is. Perhaps the
> > bug is in there?
> > 
> > I was told that the only simple way on a Xen dom0 to disable use of
> > swiotlb would be to set the memory below 4GiB again, so I might try
> > that.
> 
> I have no idea where you take this 4GiB aspect from. What the kernel
> considers "below 4GiB" in its view of the world may be at a much
> higher address in system address space. And the mere amount of
> memory doesn't matter here at all.

Ah, I was taking that from:

    
https://elixir.bootlin.com/linux/v5.10.40/source/arch/x86/kernel/pci-swiotlb.c#L41

…which I had found while looking around how to disable use of
swiotlb. But I think I'm probably confused - should I only be
looking at arch/x86/xen/pci-swiotlb-xen.c in the case of a PV domain
like dom0?

I have not been able to reproduce the problem by giving a test
system with identical hardware more RAM and getting fio in a guest
to do random reads with a blocksize between 4kiB and 4MiB.

Perhaps it is highly workload dependent then. In some ways it's a
pity that I do not get call traces for the later occurrences as then
I could see if it's always the same 62.xvda-0 process (and thus same
guest) triggering it.

It's happened three more times since my previous email, but these
have been up to 46 hours apart. These were all reads, so MD just
satisfied the read from the other device without kicking the nvme
device out.

Hmm, I have the sector offset in the MD device so maybe I can
convert that into a logical volume to know if a particular guest is
provoking it…

If anyone happens to have any suggestions as to what kind of IO might
provoke this at all so I could maybe get fio to reproduce it, please
do let me know.

Thanks,
Andy