This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Assigning contiguous memory to a driver domain

To: Jan Beulich <JBeulich@xxxxxxxxxx>
Subject: Re: [Xen-devel] Assigning contiguous memory to a driver domain
From: Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx>
Date: Wed, 15 Sep 2010 13:07:16 +0200
Cc: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Delivery-date: Wed, 15 Sep 2010 04:08:28 -0700
Dkim-signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:mime-version:to:cc:subject:references:in-reply-to:content-type; s=smtpout; bh=5S7eOgiLWXJwEegOJSi4BJ+Q0hA=; b=BltIWU75Kzgqn+9nXCcsF7sj8WENWgfDnTKh59orP1gV+PHxLp4iZhF2WXVmHq+bYXi1V0DjKxm2jC5KeIxq+ff49i4zJPzfir/ixxn9mqIlHsmDrOjS+UoaB74v40fdAm2yj7wVoL1HSzEgaYulxyUvq/YbaxI4I3+SLDXwI7c=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4C90C3200200007800016442@xxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20100914092439.GA1000@email> <4C8F5E580200007800015F71@xxxxxxxxxxxxxxxxxx> <20100915090842.GB1583@email> <20100915093910.GC1583@email> <4C90B31902000078000163E8@xxxxxxxxxxxxxxxxxx> <4C90A32A.7010803@xxxxxxxxxxxxxxxxxxxxxx> <4C90C3200200007800016442@xxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20100621 Fedora/3.0.5-1.fc13 Lightning/1.0b2pre Thunderbird/3.0.5
On 09/15/10 12:59, Jan Beulich wrote:
>>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx> 
>>>> wrote:
>> On 09/15/10 11:50, Jan Beulich wrote:
>>>>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx> 
>>>>>> wrote:
>>>> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>>>>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>>>>>>>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx> 
>>>>>>>>> wrote:
>>>>>>> Hello,
>>>>>>> Could someone guide me in the right direction with the topic of 
>>>>>>> assigning
>>>>>>> contiguous memory to a domain.
>>>>>>> I have an issue with a PV domain that is assigned a PCI device. 
>>>>>>> Sometimes, 
>>>>>>> the 
>>>>>>> driver fails to load
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>>>>>>> firmware: requesting iwlwifi-4965-2.ucode
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
>>>> loaded
>>>>>>> firmware version
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
>>>> failed
>>>>>>> to allocate pci memory
>>>>>>> The reason seems to be that the domain does not have enough contiguous
>>>>>>> memory, in mfn terms.
>>>>>> No, how (dis)contiguous the memory of a domain is doesn't matter
>>>>>> here. What matters is whether the domain can *make* the requested
>>>>>> memory contiguous, and that depends on how much contiguous
>>>>>> memory Xen has at the point of the allocation.
>>>>> Ah, so you are saying that regardless of whether a domain has some
>>>>> contiguous memory, the driver will call xen_create_contiguous_region when 
>>>> allocating
>>>>> memory (via dma_alloc_coherent ?).
>>>>> Slightly out-of-the-list-scope: is there a convention when a driver should
>>>>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>>>>> has loaded, it will no longer need to call xen_create_contiguous_region
>>>>> anymore and we can use up all free Xen memory ? 
>>>> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
>>>> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
>>>> eth0 up" 
>>>> failed with "SIOCSIFFLAGS: Cannot allocate memory".
>>> Sure - why would the driver waste resources when the device may
>>> not be used.
>>>> So, it looks like in order to make a PV driver domain work, there must be 
>>>> some Xen free memory 
>>>> all the time ?
>>> Potentially yes, but this really depends on how the respective
>>> driver is written.
>>>> Moreover, this free Xen memory must be contiguous to some
>>>> extent; is there any way to assure this ?
>>> No.
>> Wait! Are you saying there is no *way* to guarantee proper operation of
>> a driver domain in Xen?
> No, that's to strict a statement. The admin can guarantee this. Xen
> on its own can't.
Can they? How? Your answer below suggest it cannot be *guaranteed*...

>> Sure, we can tune our memory balancer to always keep some 100MB (or
>> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
>> contain enough continues pages, in case some driver in some driver
>> domain calls dma_alloc_coherent(), so the call will succeed.
> It likely doesn't need to be that much, because allocation happens
> top-down. That is, Xen will try to avoid the (no longer explicitly
> separate) DMA pool unless explicitly requested (or unless otherwise
> not being able to fulfill a request). Therefore, if you always keep
> just enough memory in Xen so that it'll never touch a reasonably
> small set of pages you want to be available for DMA in guests,
> these regions (the combined set of what Xen has and what was
> given out as contiguous memory to the driver domain) will remain
> contiguous. It'll get more problematic if you have more than one
> driver domain.

Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
it that the drivers in Dom0 never has this problem with allocating
continues memory (even if we keep xen free memory = 0)?

> Further, most drivers will prefer to allocate DMA memory once
> (at startup or when enabling use of the device) rather than
> repeatedly in the I/O path. In that case, you don't even need
> to continuously make guarantees on available memory in Xen.

But it is not feasible to modify/ensure e.g. all the Linux networking
drivers behave like this. That would require review/patching of hundreds
of drivers, unless there is some generic way to ensure that?


>> But this is not a good solution: not only because it's a waste of memory
>> (I'd rather use this memory for Dom0/storage domain page cache instead)
>> but also, and most importantly, because I don't want to build a system
>> based on *hope*!
>> Can we do something about it?
> I can't easily imagine anything beyond how it's working today.
> Jan

Attachment: signature.asc
Description: OpenPGP digital signature

Xen-devel mailing list