Re: [Xen-devel] Buffers not reachable by PCI

["Taylor, Neal E" <Neal.Taylor@xxxxxx>]

At Leonid's suggestion (and provision of a patch) I tried it with page aligned 
memory and it works. Is this "the" real solution or just a change that happens 
to make things work for the case I have?  The patch I used is:

--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -162,7 +162,7 @@
        /*
         * Get IO TLB memory from any location.
         */
-       xen_io_tlb_start = alloc_bootmem(bytes);
+       xen_io_tlb_start = alloc_bootmem_pages(bytes);
        if (!xen_io_tlb_start)
                panic("Cannot allocate SWIOTLB buffer");

The test that was failing now passes: finding the "bus" address of 7f3ffff to 
be within a 32-bit mask and the last few MFN entries are:
[    0.000000] MFN 0x20ff->0x2080
[    0.000000] MFN 0x20bf->0x2040
[    0.000000] MFN 0x207f->0x2000
[    0.000000] MFN 0x203f->0x7fc0
[    0.000000] MFN 0x7fff->0x7f80
[    0.000000] MFN 0x7fbf->0x7f40
[    0.000000] MFN 0x7f7f->0x7f00
[    0.000000] Placing 64MB software IO TLB between d832c000 - dc32c000
[    0.000000] software IO TLB at phys 0x1832c000 - 0x1c32c000
[    0.000000] software IO TLB at bus 0x1c0000 - 0x120fdf000

(This last address, 0x120fdf000, hasn't been through the final 
"phys_to_machine" translation. My error. Sorry.)

Neal


-----Original Message-----
From: Kalev, Leonid 
Sent: Wednesday, December 14, 2011 10:42 AM
To: Taylor, Neal E
Cc: Jan Beulich; Konrad Rzeszutek Wilk; Tushar N Dave; xen-devel
Subject: Re: [Xen-devel] Buffers not reachable by PCI

On 12/14/2011 06:42 PM, Taylor, Neal E wrote:
> The last quoted printout is calculated the same way as the test that fails 
> which leads me to question the computation's validity.

The computation validity seems OK to me (the 'phys' address is, as correctly 
stated 
in the comments, not suitable for DMA, but it is first translated via 
xen_phys_to_bus, which gives the machine address - and on x86 that is also the 
bus 
address)

I have a STUPID question, though: why are the start and end addresses of the 
swiotlb 
memory area not page-aligned???

The io_tlb_end address is one byte PAST the valid area, which is why the 
xen_swiotlb_dma_supported function uses (xen_io_tlb_end-1). However, this will 
work 
properly only if the value is page-aligned. If it isn't, then decrementing it 
by one 
will keep the value in the same page (which is one page past the last valid 
one).

The function that allocates the memory uses alloc_bootmem(), which provides 
just 
cache-aligned memory (not page-aligned).

If it is OK for the swiotlb area not to be page-aligned, then the 
xen_swiotlb_dma_supported should use (xen_io_tlb_end - (PAGE_SIZE-1))

If the memory should be in fact aligned, then the allocation must be changed to 
alloc_bootmem_pages() (which is the same as alloc_bootmem, but page-aligned).


>
> The test that fails is (from drivers/xen/swiotlb-xen.c):
> xen_swiotlb_dma_supported(struct device *hwdev, u64 mask)
> {
>          return xen_virt_to_bus(xen_io_tlb_end - 1)<= mask;
> }
>
>
> "xen_virt_to_bus" in turn:
> static dma_addr_t xen_virt_to_bus(void *address)
> {
>          return xen_phys_to_bus(virt_to_phys(address));
> }
>
>
> Now, "virt_to_phys" (out of arch/x86/include/asm/io.h) is defined as follows 
> but carries a comment bothersome to this usage of it as we're, basically, 
> dealing with a dma "transfer" and calling from a device driver:
> /**
>   *      virt_to_phys    -       map virtual addresses to physical
>   *      @address: address to remap
>   *
>   *      The returned physical address is the physical (CPU) mapping for
>   *      the memory address given. It is only valid to use this function on
>   *      addresses directly mapped or allocated via kmalloc.
>   *
>   *      This function does not give bus mappings for DMA transfers. In
>   *      almost all conceivable cases a device driver should not be using
>   *      this function
>   */
> static inline phys_addr_t virt_to_phys(volatile void *address)
> {
>          return __pa(address);
> }
>
>
> Going a couple of steps further, "__pa" is defined (in 
> arch/x86/include/asm/page.h) as:
> #define __pa(x)         __phys_addr((unsigned long)(x))
>
>
> and "__phys_addr" (in arch/x86/mm/physaddr.c) as:
> unsigned long __phys_addr(unsigned long x)
> {
>          if (x>= __START_KERNEL_map) {
>                  x -= __START_KERNEL_map;
>                  VIRTUAL_BUG_ON(x>= KERNEL_IMAGE_SIZE);
>                  x += phys_base;
>          } else {
>                  VIRTUAL_BUG_ON(x<  PAGE_OFFSET);
>                  x -= PAGE_OFFSET;
>                  VIRTUAL_BUG_ON(!phys_addr_valid(x));
>          }
>          return x;
> }
> EXPORT_SYMBOL(__phys_addr);
>
>
> So, if "virt_to_phys" isn't yielding a valid test of whether addresses will 
> be reachable from a PCI device, what's the correct way to test it and should 
> xen_swiotlb_dma_supported be updated to the correct way (I think so) or a new 
> function be created?
>
> Neal
>
>
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
> Sent: Wednesday, December 14, 2011 1:20 AM
> To: Taylor, Neal E
> Cc: Kalev, Leonid; Konrad Rzeszutek Wilk; Tushar N Dave; xen-devel
> Subject: Re: [Xen-devel] Buffers not reachable by PCI
>
>>>> On 14.12.11 at 01:38, "Taylor, Neal E"<Neal.Taylor@xxxxxx>  wrote:
>> [    0.000000] MFN 0x7f7f->0x7f00
>
> This is clearly indicating the last chunk ends well below the 4G boundary.
>
>> [    0.000000] Placing 64MB software IO TLB between d832cf00 - dc32cf00
>> [    0.000000] software IO TLB at phys 0x1832cf00 - 0x1c32cf00
>> [    0.000000] software IO TLB at bus 0x1c0f00 - 0x120fdff00
>
> Consequently, the question is how you got to this value, or what
> changed between the first and last quoted printouts.
>
> Jan
>


-- 
Leonid Kalev
CA Technologies
Principal Software Engineer
Tel:     +972 4 825 3952
Mobile:  +972 54 4631508
Leonid.Kalev@xxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel