Re: Re: Re: Re: AW: Re: [Xen-devel] AW: Load increase after memory upgrade?

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Wed, Jun 29, 2011 at 11:17:53PM +0200, Carsten Schiers wrote:
> > Lets first do the c) experiment as that will likely explain your load 
> average increase.
> ...
> > >c). If you want to see if the fault here lies in the bounce buffer 
> > being used more
> > >often in the DomU b/c you have 8GB of memory now and you end up using 
> 
> > more pages
> > >past 4GB (in DomU), I can cook up a patch to figure this out. But an 
> > easier way is
> > >to just do (on the Xen hypervisor line): mem=4G and that will make 
> > think you only have
> > >4GB of physical RAM.  If the load comes back to the normal "amount" 
> > then the likely
> > >culprit is that and we can think on how to fix this.
> 
> You are on the right track. Load was going down to "normal" 10% when 
> reducing
> Xen to 4GB by the parameter. Load seems to be still a little, little bit 
> lower
> with Xenified Kernel (8-9%), but this is drastically lower than the 20% 
> we had
> before.
> 
> What to do next, Konrad?

Yikes. I was hoping that was not the problem but it seems so. The issue is
here (I think) is that the PCI device you have passed in, are 32-bit.
Meaning they can only address memory up to 4GB. You should see if this
is the case - if the device is a PCI card, it definitly can only do 32-bit.

If it is PCIe, then it might be that the driver has set the DMA mask
to be only up to 32-bit.

Back to the explanation ..So if your guest (or the guest application) is
entirely above the 4GB region any of DMA operations have to be "bounced"
(or copied) to a region below 4GB. When the DMA operation is complete the
result is copied from below the 4GB back to the region above 8Gb.

Some drivers use the 'dma_alloc_coherent' which ends up allocating a
buffer that is below 4GB and use that all the time as to avoid the
bouncing. Mostly the IDE, SCSI and NICs do it.

One way that immediately comes in my mind to "fix" this is to force
the guest to be allocated from memory below 4GB. But not sure what
fancy parameters need to be turned on for this.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel