This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Assigning contiguous memory to a driver domain

To: Jan Beulich <JBeulich@xxxxxxxxxx>
Subject: Re: [Xen-devel] Assigning contiguous memory to a driver domain
From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Date: Mon, 20 Sep 2010 15:48:36 -0400
Cc: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx>, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx>
Delivery-date: Mon, 20 Sep 2010 12:50:34 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4C91027802000078000165E8@xxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4C90B31902000078000163E8@xxxxxxxxxxxxxxxxxx> <4C90A32A.7010803@xxxxxxxxxxxxxxxxxxxxxx> <4C90C3200200007800016442@xxxxxxxxxxxxxxxxxx> <4C90A8E4.9090004@xxxxxxxxxxxxxxxxxxxxxx> <4C90CF39020000780001647E@xxxxxxxxxxxxxxxxxx> <87pqwfgqvt.fsf@xxxxxxxxxxxxxxxxxxxx> <20100915120626.GA2024@email> <4C90EB1102000078000164C6@xxxxxxxxxxxxxxxxxx> <20100915144413.GB2098@email> <4C91027802000078000165E8@xxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.20 (2009-06-14)
On Wed, Sep 15, 2010 at 04:29:28PM +0100, Jan Beulich wrote:
> >>> On 15.09.10 at 16:44, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx> wrote:
> > On Wed, Sep 15, 2010 at 02:49:37PM +0100, Jan Beulich wrote:
> >> Because on suspend the driver frees the memory which on resume
> >> it will allocate back?
> > I am a bit lost.
> > By "frees the memory" you mean "return contiguous memory to Xen free 
> > memory" 
> > ?
> > Does it really work this way ?
> Yes - the "special" memory gets exchanged back to "normal" memory
> upon freeing of it by the driver. The exception is if Xen has no "normal"
> memory left to give back out in exchange - in that case the domain will
> retain the "special" memory indefinitely. Yes, you can call this a leak,
> but no, I don't think there's much you can do about it (without adding
> likely rather complex extra code).

Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
memory under the 4GB limit. Those chunks stay in DomU and are used
during the the runtime of the DomU. They don't go back to Xen unless the
domain has been terminated. Any of the DMA operations that any driver
does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
outside the driver capabilities (say, your ping buffer is allocated above
the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
to "bounce" the memory).

> > If so, it requires nonzero Xen free memory ? And that is why when I do
> > "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
> > fails ?

There are couple of things happening when you do ifconfig eth0 up. The 
PTE used for virtual address for the BARs are updated with the _PAGE_IOMAP
which means that the GMFN->PFN->MFN is shortcircuited to be GMFN->MFN.
Obviously that doesn' use any Xen heap memory. The next thing is
that the the driver might allocate coherent DMA mappings. Those are the
ones I think Jan is referring to. For coherent DMA mappings we just
do page_alloc and then we swap the memory behind those pages with Xen
to be under the 32-bit limit (xen_create_contiguous_region).
Naturally when the driver is unloaded the de-allocation will call
xen_destroy_contiguous_region. Loking at the code I think it swaps with
the highest bit order (so with memory close to the end of physical space).

> Generally the second "up" shouldn't fail as long as the prior "down"
> properly returned all resources. See the restrictions above.

Yeah, it might be worth looking at what it is doing to cause this. The e1000/igb
are pretty good at cleaning everying so you can do ifup;ifdown indefinitly.

In reference to the Xen-SWIOTLB for other versions that upstream, there are a 
of implementations at:

for different Linux versions. Which version of kernel do you guys use?
> Jan
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>