Re: [Xen-devel] Loading PCIe Device Driver at Dom0

[Kenneth Wong <kenwong@xxxxxxxxxxx>]

Hi Konrad and others,

Oh, there are Xen/dom0 specific APIs for PCI and DMA?  

May I ask the names and where can I can more info on the APIs?

Many thanks!

Kenneth

________________________________________
From: Konrad Rzeszutek Wilk [konrad.wilk@xxxxxxxxxx]
Sent: Wednesday, May 23, 2012 5:17 PM
To: Kenneth Wong
Cc: xen-devel@xxxxxxxxxxxxx
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> Dear all,
>
> I have a PCIe device driver that I have been using on various Linux 
> distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
>
> I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 
> 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) 
> successfully at Dom0 without any issue.  I can do reads and write to the 
> hardware device.  But once booted to Xen, the driver failed to complete the 
> driver load (via insmod manually) at Dom 0 and the console just hangs.
>
> >From my debug messages, it appears it hangs because the driver doesn't 
> >receive any interrupt after a command is sent to the hardware device by 
> >writing a parameter to the mapped register.  Once that register is written, 
> >the device is expected to DMA the command from the buffer allocated by the 
> >driver.
>
> The things that I can only think of that might have caused the problem are 1) 
> IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.

The 2).
>
>
> What the driver does:

You do need to use the PCI API (or the DMA API).

>
> Set up a command buffer:
> Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> unsigned long buf_addr = __pa(buf);
> unsigned int buf_addr_low = (unsigned int)buf_addr;
>
> Tell device about the buffer:
> iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);

>
> Set up IRQ:
>     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
>         (!pci_enable_msi(dev)))
>     {
>         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, 
> DRIVER_NAME, my_dev))
>         {
>             return  -ENODEV;
>         }
>         my_dev->intr_mode = INTERRUPT_MSI;
>     }
>
> Ask device to fetch command from buffer (Expect interrupt after this after 
> device fetched the command from buf.  But interrupt did not happen.):
> iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
>
>
> >From dmesg, it looks like IRQ initialization is complete.
> [  241.743769] My_driver initialization
> [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> [  241.743801] Already setup the GSI :16
> [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> 
> IRQ 16
> [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
>
> /proc/interrupts:
>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
> CPU6       CPU7
> ......
> ......
> 339:          0          0          0          0          0          0        
>   0          0  xen-pirq-msi       my-driver
> ......
> ......
>
> Any idea what might cause the problem?
>
> Is there anything we have to be enable/disable, use different functions, or 
> do differently in drivers written for Xen Dom0 environment regarding the 
> following?
> 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> 2)       Requesting MSI irq.
>
> Please advise!
>
> Thanks a lot in advance!!
>
> Kenneth

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel