Re: [Xen-devel] Re: [Patch RFC] ttm: nouveau accelerated on Xen pv-ops kernel

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Wed, Jun 09, 2010 at 01:43:42PM -0400, Konrad Rzeszutek Wilk wrote:
> > > >>>         vma->vm_private_data = bo;
> > > >>> -       vma->vm_flags |= VM_RESERVED | VM_IO | VM_MIXEDMAP |
> > > >>> VM_DONTEXPAND;
> > > >>> +       vma->vm_flags |= VM_RESERVED | VM_MIXEDMAP |
> > > >>> VM_DONTEXPAND;
> > > >>> +       if (!((bo->mem.placement & TTM_PL_MASK_MEM) &
> > > >>> TTM_PL_FLAG_TT))
> > > >>> +               vma->vm_flags |= VM_IO;
> > > >>> +       vma->vm_page_prot = vma_get_vm_prot(vma->vm_flags);
> > > >>>         return 0;
> > > >>>  out_unref:
> > > >>>         ttm_bo_unref(&bo);
> > > >>>
> .. snip.
> > > >>> nVidia GeForce 9400 GT.
> 
> ..snip..
> > 
> > Hmm... I just verified that this patch fixes KMS/Nouveau issues in Xen on 
> > my two primary test boxes (GeForce 6200, GeForce 7300).  However, on my 
> > really old machines (AGP GeForce2 MX200), this causes a new crash.  These 
> > old boxes were actually working fine in Xen prior to this patch, just 
> > w/out 3d acceleration.  Now I get the following messages in dmesg:
> > 
> > [  129.637319] [drm] nouveau 0000:01:00.0: Allocating FIFO number 1
> > [  129.638853] [drm] nouveau 0000:01:00.0: nouveau_channel_alloc: 
> > initialised FIFO 1
> > [  129.643791] X: Corrupted page table at address 40412000
> > [  129.643815] *pdpt = 0000000015216001 *pde = 0000000000000000 
> > [  129.643856] Bad pagetable: 000f [#1] SMP 
> .. snip..
> > [  129.652216] [drm] nouveau 0000:01:00.0: nouveau_channel_free: freeing 
> > fifo 1
> > 
> > 
> > And my X log ends abruptly after this line:
> > (II) NOUVEAU(0): Opened GPU Channel 1
> 
> 
> So, I've spent the last two weeks trying to get this to work.
> 
> What I found out was that the majority of the problems were with the Linux 
> AGP code
> (drivers/char/agp/*). Now I can get Kernel Modesetting 
> working under Radeon HD3200 (PCI-e), NVidia GeForce FX5200 (AGP), and I think
> other ones too (hadn't tested yet). However, I am still failing at the same
> spot as Michael: the dreaded Opened GPU Channel 1...
> 
> Anyhow.. if you want the patches:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/kms-fixes
> 
> (and the "devel/drm.backport" which is a drop of drm/* from 2.6.34-rc7 in
> 2.6.32 universe).
> 
> The explanation:
> 
> 1). Newer boxes with PCI-e cards with more than 4GB allocated to dom0.
> Those cards have their own IOMMU built in on the cards and use the PCI
> DMA. The problem is that some of those cards set the DMA mask to 32-bit,
> meaning they are only comfortable allocating pages under the 4GB mark.
> The Xen-SWIOTLB kicks in and happily gives a bounce buffer for pages
> that are above the 32-bit mark, which are then programmed to the IOMMU.
> Except that the graphic drivers do not sync the bounce buffer so the
> result ends up stored in the bounce buffer and not in the buffer the
> graphic drivers expect (oops).  That however should not have happend
> as any pages allocated from the TTM library use the 'alloc_page' with
>  _GFP_DMA32 flag, which means that all of those pages WILL be under
> 4GB under baremetal. However under Xen that flag is useless and the page
> can be very well allocated above the 4GB mark (and is if you are using
> the debug Xen hypervisor). The patch titled
>  "ttm: When TTM_PAGE_FLAG_DMA32 allocate pages under" fixes that by
> replacing the PFNs with ones under the 4GB.
> 
> It makes my Radeon HD3200 work, and I believe other PCI-e cards too.
> 
> 2). i915 cards:
> The earlier cards (before PCI-e) depend on the AGP to the virtual to
> physical address translations.  The old AGP code is safe for specific 
> controllers: say i915 - and only if you have CONFIG_DMAR enabled. So
> there is a patch if you don't have that enabled:
> 
>       intel-agp: Warn when !USE_PCI_DMA_API and inserting bogus bus addresses.
> 
> 3) i915 and before.
> For older ones, such as the ICH5 (which has an i860 chipset), the Linux
> code blindly uses the virt_to_phys and page_to_phys which are not
> Xen-safe. For this fancy NV30 card (which is AGP) I've gotten to the
> point of the framebuffer displaying properly. The patches that fix this
> are:
>   intel-agp: Use Xen back-door to get page's physical address for i8[1,3]0
>       agp-backend: Use Xen back-door to get bus address for scratch page.
>       agp: If _GFP_DMA_32 flag is set enforce pages to be under 4GB under Xen.
>       agp: Program the GART with the real physical address under Xen.
>       agp: Use Xen back-door to get bus address for legacy code.
> 
> 
> 4). I did check out the AGP code for the other chipsets, so added
> experimental support for AMD64 chipset. Haven't tested it at all.
> 
> Summary:
> Most of those patches are not upstream materials. Right now I am
> just trying to get those bugs fixed and then later on revisit them with
> a more comprehensive patch.
> 

Wow. Impressive work!

Hopefully the "Opened GPU channel 1" -thing gets beaten sooner than later too.. 
unfortunately I don't have Nvidia GPU at the moment so I can't really help with 
that..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel