WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] Dom0 hypercall for adding and removing PCI devices

To: "Keir Fraser" <keir.fraser@xxxxxxxxxxxxx>, "Tian, Kevin" <kevin.tian@xxxxxxxxx>, "Espen Skoglund" <espen.skoglund@xxxxxxxxxxxxx>
Subject: RE: [Xen-devel] Dom0 hypercall for adding and removing PCI devices
From: "Han, Weidong" <weidong.han@xxxxxxxxx>
Date: Thu, 24 Jul 2008 17:14:36 +0800
Cc: joshua.levasseur@xxxxxxxxxxxxx, xen-devel@xxxxxxxxxxxxxxxxxxx, "Li, Xin B" <xin.b.li@xxxxxxxxx>, "Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx>
Delivery-date: Thu, 24 Jul 2008 02:15:10 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C4AE001B.1B835%keir.fraser@xxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <D470B4E54465E3469E2ABBC5AFAC390F024D95D6@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> <C4AE001B.1B835%keir.fraser@xxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: Acjs70tXQ1Zw51sKQ7WpTwpC8cMMEgAOoQJAAA7bGqAAAFdQCgAAGrNwAABf8OoAACMKsAAAMqSdAADURyA=
Thread-topic: [Xen-devel] Dom0 hypercall for adding and removing PCI devices
VT-d spec says: Software must not modify fields other than the Present (P) 
field of currently present root-entries or context-entries. If modifications to 
other fields are required, software must first make these entries not-present 
(P=0), which requires serial invalidation of context-cache and IOTLB, and then 
transition them to present (P=1) state along with the modifications.

So your suggestion is not feasible. 

Randy (Weidong)

Keir Fraser wrote:
> Exactly my thought.
> 
>  K.
> 
> On 24/7/08 09:43, "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> 
>> Isn't enough to first switch VT-d page table, and then flush IOTLB?
>> As long as RMRRs are kept same in two VT-d tables, and in any
>> time a valid entry (either in IOTLB or by walking) can be found,
>> above sequence seems complete.
>> 
>> Thanks,
>> Kevin
>> 
>>> From: Keir Fraser
>>> Sent: 2008年7月24日 16:38
>>> 
>>> Can the pagetable switch not be made atomic (i.e., so that the RMRR
>>> regions appear continuously available throughout)? I'd have thought
>>> that would just naturally happen.
>>> 
>>> If creating/destroying RMRR mappings is part of the switch
>>> operation, you'd have to destroy RMRR mappings in the dom0 table
>>> after the switch, and create RMRR mappings in the fallback table
>>> before the switch. Or just have RMRR mappings always mapped into
>>> all tables. 
>>> 
>>> -- Keir
>>> 
>>> On 24/7/08 09:32, "Han, Weidong" <weidong.han@xxxxxxxxx> wrote:
>>> 
>>>> We found USB (has RMRR) is firstly assigned to dom0, but
>>>> pci_bus_probe() failed, then it was removed from dom0. The
>>>> removing needs switch to RMRR VT-d page table. At the same time,
>>>> BIOS was using its RMRR. 
>>>> 
>>>> Randy (Weidong)
>>>> 
>>>> Keir Fraser wrote:
>>>>> If a device is assigned to a domain (in this case dom0) then that
>>>>> domain's VT-d pagetables will contain the RMRR mappings for that
>>>>> device. Hence BIOS can perform DMA in those RMRR-indicated regions
>>>>> without swapping to and fro between domain tables and fallback
>>>>> RMRR tables. The new fallback RMRR table would be just that -- a
>>>>> fallback table used for any device not currently assigned to any
>>>>> domain (and hence those devices should only have DMAs initiated
>>>>> by the BIOS, if at all). 
>>>>> 
>>>>>  -- Keir
>>>>> 
>>>>> On 24/7/08 09:20, "Han, Weidong" <weidong.han@xxxxxxxxx> wrote:
>>>>> 
>>>>>> I have another concern, when BIOS is initiating DMA operation
>>>>>> using RMRR, can we use RMRR VT-d page table to replace dom0 VT-d
>>>>>> page table? Does it result in some DMA failures?
>>>>>> 
>>>>>> Randy (weidong)
>>>>>> 
>>>>>> Han, Weidong wrote:
>>>>>>> Espen Skoglund wrote:
>>>>>>>> [Keir Fraser]
>>>>>>>>> On 23/7/08 10:26, "Han, Weidong" <weidong.han@xxxxxxxxx>
>>>>>>>>> wrote: 
>>>>>>>>>>> So this would be one extra VT-d pagetable, for the whole
>>>>>>>>>>> system, which would be the fallback location for RMRR
>>>>>>>>>>> mappings for devices which are currently not assigned to
>>>>>>>>>>> any domain? Thus allowing firmware to successfully initiate
>>>>>>>>>>> DMA operations on those devices? Sounds sensible.
>>>>>>>> 
>>>>>>>> Well, the VT-d page table for RMRR devices need not contain the
>>>>>>>> whole system memory---only those regions specified in the DMAR
>>>>>>>> tables. 
>>>>>>>> 
>>>>>>>>>> Is it possible that idle_domain owns the RMRR VT-d page
>>>>>>>>>> table? 
>>>>>>>> 
>>>>>>>>> If that's a convenient place to stash it then why not? Either
>>>>>>>>> way, seems you're going to have it special-cased in the code
>>>>>>>>> as fallback owner for unassigned devices. It's possible that
>>>>>>>>> having it stashed in the idle domain will simply make the
>>>>>>>>> code more confusing. I'm not sure though.
>>>>>>>> 
>>>>>>>> Right.  I don't see any particular good reason to associate it
>>>>>>>> with the idle domain.  As noted above, the page table need not
>>>>>>>> even cover the whole memory, and it will never change after
>>>>>>>> being set up at boot time.  If special case code is needed
>>>>>>>> anyway, then one might as well install a custom VT-d page
>>>>>>>> table. 
>>>>>>> 
>>>>>>> What does "custom VT-d page table" mean?
>>>>>>> 
>>>>>>> Due to domain id is not the same with DID field in context, we
>>>>>>> can reverse a DID for RMRR VT-d page table, it can avoid to
>>>>>>> associate with idle domain. 
>>>>>>> 
>>>>>>> Currently we reassign the device from dom0 to target domain when
>>>>>>> assign a device, and return the device to dom0 when target
>>>>>>> domain tears down. It's not correct due to some devices may be
>>>>>>> not assigned to any domain. Current device_assigned() also
>>>>>>> needs to be changed. Maybe it needs more changes on VT-d.
>>>>>>> 
>>>>>>> I have some concerns, maybe I missed something. Why did you use
>>>>>>> dom0 hypercall approach to replace original method? What's the
>>>>>>> main benefit? I also wonder it's suitable to wrap
>>>>>>> pci_bus_probe() function. 
>>>>>>> 
>>>>>>> Randy (Weidong)
>>>>>>> 
>>>>>>>> 
>>>>>>>> If supported by hardware, the extra page tables can even be
>>>>>>>> skipped altogether and the device marked as having passthrough
>>>>>>>> access. That would give the RMRR device complete access to
>>>>>>>> system memory, though. 
>>>>>>>> 
>>>>>>>> eSk
>>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel