Re: [Xen-devel] [PATCH v11 09/12] vpci/msi: add MSI handlers

[Paul Durrant <Paul.Durrant@xxxxxxxxxx>]

> -----Original Message-----
> From: Roger Pau Monne [mailto:roger.pau@xxxxxxxxxx]
> Sent: 20 March 2018 15:16
> To: xen-devel@xxxxxxxxxxxxxxxxxxxx
> Cc: Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>; Konrad Rzeszutek Wilk
> <konrad.wilk@xxxxxxxxxx>; Roger Pau Monne <roger.pau@xxxxxxxxxx>; Jan
> Beulich <jbeulich@xxxxxxxx>; Andrew Cooper
> <Andrew.Cooper3@xxxxxxxxxx>; George Dunlap
> <George.Dunlap@xxxxxxxxxx>; Ian Jackson <Ian.Jackson@xxxxxxxxxx>; Julien
> Grall <julien.grall@xxxxxxx>; Stefano Stabellini <sstabellini@xxxxxxxxxx>;
> Tim (Xen.org) <tim@xxxxxxx>; Wei Liu <wei.liu2@xxxxxxxxxx>; Paul Durrant
> <Paul.Durrant@xxxxxxxxxx>
> Subject: [PATCH v11 09/12] vpci/msi: add MSI handlers
> 
> Add handlers for the MSI control, address, data and mask fields in
> order to detect accesses to them and setup the interrupts as requested
> by the guest.
> 
> Note that the pending register is not trapped, and the guest can
> freely read/write to it.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> Reviewed-by: Jan Beulich <jbeulich@xxxxxxxx>

io header changes...

Reviewed-by: Paul Durrant <paul.durrant@xxxxxxxxxx>

> ---
> Cc: Jan Beulich <jbeulich@xxxxxxxx>
> Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> Cc: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>
> Cc: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
> Cc: Julien Grall <julien.grall@xxxxxxx>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> Cc: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> Cc: Tim Deegan <tim@xxxxxxx>
> Cc: Wei Liu <wei.liu2@xxxxxxxxxx>
> Cc: Paul Durrant <paul.durrant@xxxxxxxxxx>
> ---
> Changes since v8:
>  - Add a FIXME about the lack of testing and a comment regarding the
>    lack of cleaning done in the init_msi error path.
>  - Free msi struct when cleaning up if an init function failed.
>  - Remove the 'error' label of init_msi, the caller will already
>    perform the cleaning.
> 
> Changes since v7:
>  - Don't store pci segment/bus on local variables.
>  - Add an error label to init_msi.
>  - Don't trap accesses to the PBA.
>  - Fix msi_pending_bits_reg macro so it matches coding style.
>  - Move the position of vectors in the vpci_msi struct.
>  - Add a comment to clarify the expected state of vectors after
>    pt_irq_create_bind and use XEN_DOMCTL_VMSI_X86_UNMASKED.
> 
> Changes since v6:
>  - Use domain_spin_lock_irq_desc instead of open coding it.
>  - Reduce the size of printed debug messages.
>  - Constify domain in vpci_dump_msi.
>  - Lock domlist_read_lock before iterating over the list of domains.
>  - Make max_vectors and vectors uint8_t.
>  - Drop the vpci_ prefix from the static functions in msi.c.
>  - Turn the booleans in vpci_msi into bitfields.
>  - Apply the mask bits to all vectors when enabling msi.
>  - Remove the pos field.
>  - Remove the usage of __msi_set_{enable/disable}.
>  - Update the bindings when the message or data fields are updated.
>  - Make vpci_msi_arch_disable return void, it wasn't returning any
>    error.
>  - Prevent the guest from writing to the pending bits field, it's read
>    only as defined in the spec.
>  - Add the must_check attribute to vpci_msi_arch_enable.
> 
> Changes since v5:
>  - Update to new lock usage.
>  - Change handlers to match the new type.
>  - s/msi_flags/msi_gflags/, remove the local variables and use the new
>    DOMCTL_VMSI_* defines.
>  - Change the MSI arch function to take a vpci_msi instead of a
>    vpci_arch_msi as parameter.
>  - Fix the calculation of the guest vector for MSI injection to take
>    into account the number of bits that can be modified.
>  - Use INVALID_PIRQ everywhere.
>  - Simplify exit path of vpci_msi_disable.
>  - Remove the conditional when setting address64 and masking fields.
>  - Add a process_pending_softirqs to the MSI dump loop.
>  - Place the prototypes for the MSI arch-specific functions in
>    xen/vpci.h.
>  - Add parentheses around the INVALID_PIRQ definition.
> 
> Changes since v4:
>  - Fix commit message.
>  - Change the ASSERTs in vpci_msi_arch_mask into ifs.
>  - Introduce INVALID_PIRQ.
>  - Destroy the partially created bindings in case of failure in
>    vpci_msi_arch_enable.
>  - Just take the pcidevs lock once in vpci_msi_arch_disable.
>  - Print an error message in case of failure of pt_irq_destroy_bind.
>  - Make vpci_msi_arch_init return void.
>  - Constify the arch parameter of vpci_msi_arch_print.
>  - Use fixed instead of cpu for msi redirection.
>  - Separate the header includes in vpci/msi.c between xen and asm.
>  - Store the number of configured vectors even if MSI is not enabled
>    and always return it in vpci_msi_control_read.
>  - Fix/add comments in vpci_msi_control_write to clarify intended
>    behavior.
>  - Simplify usage of masks in vpci_msi_address_{upper_}write.
>  - Add comment to vpci_msi_mask_{read/write}.
>  - Don't use MASK_EXTR in vpci_msi_mask_write.
>  - s/msi_offset/pos/ in vpci_init_msi.
>  - Move control variable setup closer to it's usage.
>  - Use d%d in vpci_dump_msi.
>  - Fix printing of bitfield mask in vpci_dump_msi.
>  - Fix definition of MSI_ADDR_REDIRECTION_MASK.
>  - Shuffle the layout of vpci_msi to minimize gaps.
>  - Remove the error label in vpci_init_msi.
> 
> Changes since v3:
>  - Propagate changes from previous versions: drop xen_ prefix, drop
>    return value from handlers, use the new vpci_val fields.
>  - Use MASK_EXTR.
>  - Remove the usage of GENMASK.
>  - Add GFLAGS_SHIFT_DEST_ID and use it in msi_flags.
>  - Add "arch" to the MSI arch specific functions.
>  - Move the dumping of vPCI MSI information to dump_msi (key 'M').
>  - Remove the guest_vectors field.
>  - Allow the guest to change the number of active vectors without
>    having to disable and enable MSI.
>  - Check the number of active vectors when parsing the disable
>    mask.
>  - Remove the debug messages from vpci_init_msi.
>  - Move the arch-specific part of the dump handler to x86/hvm/vmsi.c.
>  - Use trylock in the dump handler to get the vpci lock.
> 
> Changes since v2:
>  - Add an arch-specific abstraction layer. Note that this is only implemented
>    for x86 currently.
>  - Add a wrapper to detect MSI enabling for vPCI.
> ---
> NB: I've only been able to test this with devices using a single MSI
> interrupt and no mask register. I will try to find hardware that
> supports the mask register and more than one vector, but I cannot make
> any promises.
> 
> If there are doubts about the untested parts we could always force Xen
> to report no per-vector masking support and only 1 available vector,
> but I would rather avoid doing it.
> ---
>  xen/arch/x86/hvm/vmsi.c      | 142 +++++++++++++++++++
>  xen/arch/x86/msi.c           |   3 +
>  xen/drivers/vpci/Makefile    |   2 +-
>  xen/drivers/vpci/msi.c       | 324
> +++++++++++++++++++++++++++++++++++++++++++
>  xen/drivers/vpci/vpci.c      |   1 +
>  xen/include/asm-x86/hvm/io.h |   5 +
>  xen/include/asm-x86/msi.h    |   3 +
>  xen/include/xen/irq.h        |   1 +
>  xen/include/xen/vpci.h       |  38 +++++
>  9 files changed, 518 insertions(+), 1 deletion(-)
>  create mode 100644 xen/drivers/vpci/msi.c
> 
> diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c
> index 7126de7841..be59c56d43 100644
> --- a/xen/arch/x86/hvm/vmsi.c
> +++ b/xen/arch/x86/hvm/vmsi.c
> @@ -31,6 +31,7 @@
>  #include <xen/errno.h>
>  #include <xen/sched.h>
>  #include <xen/irq.h>
> +#include <xen/vpci.h>
>  #include <public/hvm/ioreq.h>
>  #include <asm/hvm/io.h>
>  #include <asm/hvm/vpic.h>
> @@ -621,3 +622,144 @@ void msix_write_completion(struct vcpu *v)
>      if ( msixtbl_write(v, ctrl_address, 4, 0) != X86EMUL_OKAY )
>          gdprintk(XENLOG_WARNING, "MSI-X write completion failure\n");
>  }
> +
> +static unsigned int msi_gflags(uint16_t data, uint64_t addr, bool masked)
> +{
> +    /*
> +     * We need to use the DOMCTL constants here because the output of this
> +     * function is used as input to pt_irq_create_bind, which also takes the
> +     * input from the DOMCTL itself.
> +     */
> +    return MASK_INSR(MASK_EXTR(addr, MSI_ADDR_DEST_ID_MASK),
> +                     XEN_DOMCTL_VMSI_X86_DEST_ID_MASK) |
> +           MASK_INSR(MASK_EXTR(addr, MSI_ADDR_REDIRECTION_MASK),
> +                     XEN_DOMCTL_VMSI_X86_RH_MASK) |
> +           MASK_INSR(MASK_EXTR(addr, MSI_ADDR_DESTMODE_MASK),
> +                     XEN_DOMCTL_VMSI_X86_DM_MASK) |
> +           MASK_INSR(MASK_EXTR(data, MSI_DATA_DELIVERY_MODE_MASK),
> +                     XEN_DOMCTL_VMSI_X86_DELIV_MASK) |
> +           MASK_INSR(MASK_EXTR(data, MSI_DATA_TRIGGER_MASK),
> +                     XEN_DOMCTL_VMSI_X86_TRIG_MASK) |
> +           /* NB: by default MSI vectors are bound masked. */
> +           (masked ? 0 : XEN_DOMCTL_VMSI_X86_UNMASKED);
> +}
> +
> +void vpci_msi_arch_mask(struct vpci_msi *msi, const struct pci_dev *pdev,
> +                        unsigned int entry, bool mask)
> +{
> +    unsigned long flags;
> +    struct irq_desc *desc = domain_spin_lock_irq_desc(pdev->domain,
> +                                                      msi->arch.pirq + entry,
> +                                                      &flags);
> +
> +    if ( !desc )
> +        return;
> +    guest_mask_msi_irq(desc, mask);
> +    spin_unlock_irqrestore(&desc->lock, flags);
> +}
> +
> +int vpci_msi_arch_enable(struct vpci_msi *msi, const struct pci_dev *pdev,
> +                         unsigned int vectors)
> +{
> +    struct msi_info msi_info = {
> +        .seg = pdev->seg,
> +        .bus = pdev->bus,
> +        .devfn = pdev->devfn,
> +        .entry_nr = vectors,
> +    };
> +    unsigned int i;
> +    int rc;
> +
> +    ASSERT(msi->arch.pirq == INVALID_PIRQ);
> +
> +    /* Get a PIRQ. */
> +    rc = allocate_and_map_msi_pirq(pdev->domain, -1, &msi->arch.pirq,
> +                                   MAP_PIRQ_TYPE_MULTI_MSI, &msi_info);
> +    if ( rc )
> +    {
> +        gdprintk(XENLOG_ERR, "%04x:%02x:%02x.%u: failed to map PIRQ:
> %d\n",
> +                 pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                 PCI_FUNC(pdev->devfn), rc);
> +        return rc;
> +    }
> +
> +    for ( i = 0; i < vectors; i++ )
> +    {
> +        uint8_t vector = MASK_EXTR(msi->data, MSI_DATA_VECTOR_MASK);
> +        uint8_t vector_mask = 0xff >> (8 - fls(msi->vectors) + 1);
> +        struct xen_domctl_bind_pt_irq bind = {
> +            .machine_irq = msi->arch.pirq + i,
> +            .irq_type = PT_IRQ_TYPE_MSI,
> +            .u.msi.gvec = (vector & ~vector_mask) |
> +                          ((vector + i) & vector_mask),
> +            .u.msi.gflags = msi_gflags(msi->data, msi->address,
> +                                       (msi->mask >> i) & 1),
> +        };
> +
> +        pcidevs_lock();
> +        rc = pt_irq_create_bind(pdev->domain, &bind);
> +        if ( rc )
> +        {
> +            gdprintk(XENLOG_ERR,
> +                     "%04x:%02x:%02x.%u: failed to bind PIRQ %u: %d\n",
> +                     pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                     PCI_FUNC(pdev->devfn), msi->arch.pirq + i, rc);
> +            while ( bind.machine_irq-- )
> +                pt_irq_destroy_bind(pdev->domain, &bind);
> +            spin_lock(&pdev->domain->event_lock);
> +            unmap_domain_pirq(pdev->domain, msi->arch.pirq);
> +            spin_unlock(&pdev->domain->event_lock);
> +            pcidevs_unlock();
> +            msi->arch.pirq = INVALID_PIRQ;
> +            return rc;
> +        }
> +        pcidevs_unlock();
> +    }
> +
> +    return 0;
> +}
> +
> +void vpci_msi_arch_disable(struct vpci_msi *msi, const struct pci_dev
> *pdev)
> +{
> +    unsigned int i;
> +
> +    ASSERT(msi->arch.pirq != INVALID_PIRQ);
> +
> +    pcidevs_lock();
> +    for ( i = 0; i < msi->vectors; i++ )
> +    {
> +        struct xen_domctl_bind_pt_irq bind = {
> +            .machine_irq = msi->arch.pirq + i,
> +            .irq_type = PT_IRQ_TYPE_MSI,
> +        };
> +        int rc;
> +
> +        rc = pt_irq_destroy_bind(pdev->domain, &bind);
> +        ASSERT(!rc);
> +    }
> +
> +    spin_lock(&pdev->domain->event_lock);
> +    unmap_domain_pirq(pdev->domain, msi->arch.pirq);
> +    spin_unlock(&pdev->domain->event_lock);
> +    pcidevs_unlock();
> +
> +    msi->arch.pirq = INVALID_PIRQ;
> +}
> +
> +void vpci_msi_arch_init(struct vpci_msi *msi)
> +{
> +    msi->arch.pirq = INVALID_PIRQ;
> +}
> +
> +void vpci_msi_arch_print(const struct vpci_msi *msi)
> +{
> +    printk("vec=%#02x%7s%6s%3sassert%5s%7s dest_id=%lu pirq: %d\n",
> +           MASK_EXTR(msi->data, MSI_DATA_VECTOR_MASK),
> +           msi->data & MSI_DATA_DELIVERY_LOWPRI ? "lowest" : "fixed",
> +           msi->data & MSI_DATA_TRIGGER_LEVEL ? "level" : "edge",
> +           msi->data & MSI_DATA_LEVEL_ASSERT ? "" : "de",
> +           msi->address & MSI_ADDR_DESTMODE_LOGIC ? "log" : "phys",
> +           msi->address & MSI_ADDR_REDIRECTION_LOWPRI ? "lowest" :
> "fixed",
> +           MASK_EXTR(msi->address, MSI_ADDR_DEST_ID_MASK),
> +           msi->arch.pirq);
> +}
> diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c
> index 8c89f072a8..5567990fbd 100644
> --- a/xen/arch/x86/msi.c
> +++ b/xen/arch/x86/msi.c
> @@ -30,6 +30,7 @@
>  #include <public/physdev.h>
>  #include <xen/iommu.h>
>  #include <xsm/xsm.h>
> +#include <xen/vpci.h>
> 
>  static s8 __read_mostly use_msi = -1;
>  boolean_param("msi", use_msi);
> @@ -1527,6 +1528,8 @@ static void dump_msi(unsigned char key)
>                 attr.guest_masked ? 'G' : ' ',
>                 mask);
>      }
> +
> +    vpci_dump_msi();
>  }
> 
>  static int __init msi_setup_keyhandler(void)
> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
> index 241467212f..62cec9e82b 100644
> --- a/xen/drivers/vpci/Makefile
> +++ b/xen/drivers/vpci/Makefile
> @@ -1 +1 @@
> -obj-y += vpci.o header.o
> +obj-y += vpci.o header.o msi.o
> diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c
> new file mode 100644
> index 0000000000..c3c69ec453
> --- /dev/null
> +++ b/xen/drivers/vpci/msi.c
> @@ -0,0 +1,324 @@
> +/*
> + * Handlers for accesses to the MSI capability structure.
> + *
> + * Copyright (C) 2017 Citrix Systems R&D
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms and conditions of the GNU General Public
> + * License, version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> GNU
> + * General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public
> + * License along with this program; If not, see
> <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/sched.h>
> +#include <xen/softirq.h>
> +#include <xen/vpci.h>
> +
> +#include <asm/msi.h>
> +
> +static uint32_t control_read(const struct pci_dev *pdev, unsigned int reg,
> +                             void *data)
> +{
> +    const struct vpci_msi *msi = data;
> +
> +    return MASK_INSR(fls(msi->max_vectors) - 1, PCI_MSI_FLAGS_QMASK)
> |
> +           MASK_INSR(fls(msi->vectors) - 1, PCI_MSI_FLAGS_QSIZE) |
> +           (msi->enabled ? PCI_MSI_FLAGS_ENABLE : 0) |
> +           (msi->masking ? PCI_MSI_FLAGS_MASKBIT : 0) |
> +           (msi->address64 ? PCI_MSI_FLAGS_64BIT : 0);
> +}
> +
> +static void control_write(const struct pci_dev *pdev, unsigned int reg,
> +                          uint32_t val, void *data)
> +{
> +    struct vpci_msi *msi = data;
> +    unsigned int vectors = min_t(uint8_t,
> +                                 1u << MASK_EXTR(val, PCI_MSI_FLAGS_QSIZE),
> +                                 msi->max_vectors);
> +    bool new_enabled = val & PCI_MSI_FLAGS_ENABLE;
> +
> +    /*
> +     * No change if the enable field and the number of vectors is
> +     * the same or the device is not enabled, in which case the
> +     * vectors field can be updated directly.
> +     */
> +    if ( new_enabled == msi->enabled &&
> +         (vectors == msi->vectors || !msi->enabled) )
> +    {
> +        msi->vectors = vectors;
> +        return;
> +    }
> +
> +    if ( new_enabled )
> +    {
> +        /*
> +         * If the device is already enabled it means the number of
> +         * enabled messages has changed. Disable and re-enable the
> +         * device in order to apply the change.
> +         */
> +        if ( msi->enabled )
> +        {
> +            vpci_msi_arch_disable(msi, pdev);
> +            msi->enabled = false;
> +        }
> +
> +        if ( vpci_msi_arch_enable(msi, pdev, vectors) )
> +            return;
> +    }
> +    else
> +        vpci_msi_arch_disable(msi, pdev);
> +
> +    msi->vectors = vectors;
> +    msi->enabled = new_enabled;
> +
> +    pci_conf_write16(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                     PCI_FUNC(pdev->devfn), reg,
> +                     control_read(pdev, reg, data));
> +}
> +
> +static void update_msi(const struct pci_dev *pdev, struct vpci_msi *msi)
> +{
> +    if ( !msi->enabled )
> +        return;
> +
> +    vpci_msi_arch_disable(msi, pdev);
> +    if ( vpci_msi_arch_enable(msi, pdev, msi->vectors) )
> +        msi->enabled = false;
> +}
> +
> +/* Handlers for the address field (32bit or low part of a 64bit address). */
> +static uint32_t address_read(const struct pci_dev *pdev, unsigned int reg,
> +                             void *data)
> +{
> +    const struct vpci_msi *msi = data;
> +
> +    return msi->address;
> +}
> +
> +static void address_write(const struct pci_dev *pdev, unsigned int reg,
> +                          uint32_t val, void *data)
> +{
> +    struct vpci_msi *msi = data;
> +
> +    /* Clear low part. */
> +    msi->address &= ~0xffffffffull;
> +    msi->address |= val;
> +
> +    update_msi(pdev, msi);
> +}
> +
> +/* Handlers for the high part of a 64bit address field. */
> +static uint32_t address_hi_read(const struct pci_dev *pdev, unsigned int
> reg,
> +                                void *data)
> +{
> +    const struct vpci_msi *msi = data;
> +
> +    return msi->address >> 32;
> +}
> +
> +static void address_hi_write(const struct pci_dev *pdev, unsigned int reg,
> +                             uint32_t val, void *data)
> +{
> +    struct vpci_msi *msi = data;
> +
> +    /* Clear and update high part. */
> +    msi->address &= 0xffffffff;
> +    msi->address |= (uint64_t)val << 32;
> +
> +    update_msi(pdev, msi);
> +}
> +
> +/* Handlers for the data field. */
> +static uint32_t data_read(const struct pci_dev *pdev, unsigned int reg,
> +                          void *data)
> +{
> +    const struct vpci_msi *msi = data;
> +
> +    return msi->data;
> +}
> +
> +static void data_write(const struct pci_dev *pdev, unsigned int reg,
> +                       uint32_t val, void *data)
> +{
> +    struct vpci_msi *msi = data;
> +
> +    msi->data = val;
> +
> +    update_msi(pdev, msi);
> +}
> +
> +/* Handlers for the MSI mask bits. */
> +static uint32_t mask_read(const struct pci_dev *pdev, unsigned int reg,
> +                          void *data)
> +{
> +    const struct vpci_msi *msi = data;
> +
> +    return msi->mask;
> +}
> +
> +static void mask_write(const struct pci_dev *pdev, unsigned int reg,
> +                       uint32_t val, void *data)
> +{
> +    struct vpci_msi *msi = data;
> +    uint32_t dmask = msi->mask ^ val;
> +
> +    if ( !dmask )
> +        return;
> +
> +    if ( msi->enabled )
> +    {
> +        unsigned int i;
> +
> +        for ( i = ffs(dmask) - 1; dmask && i < msi->vectors;
> +              i = ffs(dmask) - 1 )
> +        {
> +            vpci_msi_arch_mask(msi, pdev, i, (val >> i) & 1);
> +            __clear_bit(i, &dmask);
> +        }
> +    }
> +
> +    msi->mask = val;
> +}
> +
> +static int init_msi(struct pci_dev *pdev)
> +{
> +    uint8_t slot = PCI_SLOT(pdev->devfn), func = PCI_FUNC(pdev->devfn);
> +    unsigned int pos = pci_find_cap_offset(pdev->seg, pdev->bus, slot, func,
> +                                           PCI_CAP_ID_MSI);
> +    uint16_t control;
> +    int ret;
> +
> +    if ( !pos )
> +        return 0;
> +
> +    pdev->vpci->msi = xzalloc(struct vpci_msi);
> +    if ( !pdev->vpci->msi )
> +        return -ENOMEM;
> +
> +    ret = vpci_add_register(pdev->vpci, control_read, control_write,
> +                            msi_control_reg(pos), 2, pdev->vpci->msi);
> +    if ( ret )
> +        /*
> +         * NB: there's no need to free the msi struct or remove the register
> +         * handlers form the config space, the caller will take care of the
> +         * cleanup.
> +         */
> +        return ret;
> +
> +    /* Get the maximum number of vectors the device supports. */
> +    control = pci_conf_read16(pdev->seg, pdev->bus, slot, func,
> +                              msi_control_reg(pos));
> +
> +    /*
> +     * FIXME: I've only been able to test this code with devices using a 
> single
> +     * MSI interrupt and no mask register.
> +     */
> +    pdev->vpci->msi->max_vectors = multi_msi_capable(control);
> +    ASSERT(pdev->vpci->msi->max_vectors <= 32);
> +
> +    /* The multiple message enable is 0 after reset (1 message enabled). */
> +    pdev->vpci->msi->vectors = 1;
> +
> +    /* No PIRQ bound yet. */
> +    vpci_msi_arch_init(pdev->vpci->msi);
> +
> +    pdev->vpci->msi->address64 = is_64bit_address(control);
> +    pdev->vpci->msi->masking = is_mask_bit_support(control);
> +
> +    ret = vpci_add_register(pdev->vpci, address_read, address_write,
> +                            msi_lower_address_reg(pos), 4, pdev->vpci->msi);
> +    if ( ret )
> +        return ret;
> +
> +    ret = vpci_add_register(pdev->vpci, data_read, data_write,
> +                            msi_data_reg(pos, pdev->vpci->msi->address64), 2,
> +                            pdev->vpci->msi);
> +    if ( ret )
> +        return ret;
> +
> +    if ( pdev->vpci->msi->address64 )
> +    {
> +        ret = vpci_add_register(pdev->vpci, address_hi_read,
> address_hi_write,
> +                                msi_upper_address_reg(pos), 4, 
> pdev->vpci->msi);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    if ( pdev->vpci->msi->masking )
> +    {
> +        ret = vpci_add_register(pdev->vpci, mask_read, mask_write,
> +                                msi_mask_bits_reg(pos,
> +                                                  
> pdev->vpci->msi->address64),
> +                                4, pdev->vpci->msi);
> +        if ( ret )
> +            return ret;
> +        /*
> +         * FIXME: do not add any handler for the pending bits for the 
> hardware
> +         * domain, which means direct access. This will be revisited when
> +         * adding unprivileged domain support.
> +         */
> +    }
> +
> +    return 0;
> +}
> +REGISTER_VPCI_INIT(init_msi);
> +
> +void vpci_dump_msi(void)
> +{
> +    const struct domain *d;
> +
> +    rcu_read_lock(&domlist_read_lock);
> +    for_each_domain ( d )
> +    {
> +        const struct pci_dev *pdev;
> +
> +        if ( !has_vpci(d) )
> +            continue;
> +
> +        printk("vPCI MSI d%d\n", d->domain_id);
> +
> +        list_for_each_entry ( pdev, &d->arch.pdev_list, domain_list )
> +        {
> +            const struct vpci_msi *msi;
> +
> +            if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) )
> +                continue;
> +
> +            msi = pdev->vpci->msi;
> +            if ( msi && msi->enabled )
> +            {
> +                printk("%04x:%02x:%02x.%u MSI\n", pdev->seg, pdev->bus,
> +                       PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
> +
> +                printk("  enabled: %d 64-bit: %d",
> +                       msi->enabled, msi->address64);
> +                if ( msi->masking )
> +                    printk(" mask=%08x", msi->mask);
> +                printk(" vectors max: %u enabled: %u\n",
> +                       msi->max_vectors, msi->vectors);
> +
> +                vpci_msi_arch_print(msi);
> +            }
> +
> +            spin_unlock(&pdev->vpci->lock);
> +            process_pending_softirqs();
> +        }
> +    }
> +    rcu_read_unlock(&domlist_read_lock);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index e5b49b9d82..3012b30013 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -47,6 +47,7 @@ void vpci_remove_device(struct pci_dev *pdev)
>          xfree(r);
>      }
>      spin_unlock(&pdev->vpci->lock);
> +    xfree(pdev->vpci->msi);
>      xfree(pdev->vpci);
>      pdev->vpci = NULL;
>  }
> diff --git a/xen/include/asm-x86/hvm/io.h b/xen/include/asm-x86/hvm/io.h
> index 16465ceb30..0fedb3473c 100644
> --- a/xen/include/asm-x86/hvm/io.h
> +++ b/xen/include/asm-x86/hvm/io.h
> @@ -127,6 +127,11 @@ void hvm_dpci_eoi(struct domain *d, unsigned int
> guest_irq,
>  void msix_write_completion(struct vcpu *);
>  void msixtbl_init(struct domain *d);
> 
> +/* Arch-specific MSI data for vPCI. */
> +struct vpci_arch_msi {
> +    int pirq;
> +};
> +
>  enum stdvga_cache_state {
>      STDVGA_CACHE_UNINITIALIZED,
>      STDVGA_CACHE_ENABLED,
> diff --git a/xen/include/asm-x86/msi.h b/xen/include/asm-x86/msi.h
> index 37d37b820e..10387dce2e 100644
> --- a/xen/include/asm-x86/msi.h
> +++ b/xen/include/asm-x86/msi.h
> @@ -48,6 +48,7 @@
>  #define MSI_ADDR_REDIRECTION_SHIFT  3
>  #define MSI_ADDR_REDIRECTION_CPU    (0 <<
> MSI_ADDR_REDIRECTION_SHIFT)
>  #define MSI_ADDR_REDIRECTION_LOWPRI (1 <<
> MSI_ADDR_REDIRECTION_SHIFT)
> +#define MSI_ADDR_REDIRECTION_MASK   (1 <<
> MSI_ADDR_REDIRECTION_SHIFT)
> 
>  #define MSI_ADDR_DEST_ID_SHIFT               12
>  #define       MSI_ADDR_DEST_ID_MASK          0x00ff000
> @@ -152,6 +153,8 @@ int msi_free_irq(struct msi_desc *entry);
>       ( (is64bit == 1) ? base+PCI_MSI_DATA_64 : base+PCI_MSI_DATA_32
> )
>  #define msi_mask_bits_reg(base, is64bit) \
>       ( (is64bit == 1) ? base+PCI_MSI_MASK_BIT :
> base+PCI_MSI_MASK_BIT-4)
> +#define msi_pending_bits_reg(base, is64bit) \
> +     ((base) + PCI_MSI_MASK_BIT + ((is64bit) ? 4 : 0))
>  #define msi_disable(control)         control &= ~PCI_MSI_FLAGS_ENABLE
>  #define multi_msi_capable(control) \
>       (1 << ((control & PCI_MSI_FLAGS_QMASK) >> 1))
> diff --git a/xen/include/xen/irq.h b/xen/include/xen/irq.h
> index 0aa817e266..586b78393a 100644
> --- a/xen/include/xen/irq.h
> +++ b/xen/include/xen/irq.h
> @@ -133,6 +133,7 @@ struct pirq {
>      struct arch_pirq arch;
>  };
> 
> +#define INVALID_PIRQ (-1)
>  #define pirq_info(d, p) ((struct pirq *)radix_tree_lookup(&(d)->pirq_tree,
> p))
> 
>  /* Use this instead of pirq_info() if the structure may need allocating. */
> diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h
> index 6bf8b22b4f..116b93f519 100644
> --- a/xen/include/xen/vpci.h
> +++ b/xen/include/xen/vpci.h
> @@ -87,6 +87,30 @@ struct vpci {
>          /* FIXME: currently there's no support for SR-IOV. */
>      } header;
>  #endif
> +
> +    /* MSI data. */
> +    struct vpci_msi {
> +#ifdef __XEN__
> +      /* Address. */
> +        uint64_t address;
> +        /* Mask bitfield. */
> +        uint32_t mask;
> +        /* Data. */
> +        uint16_t data;
> +        /* Maximum number of vectors supported by the device. */
> +        uint8_t max_vectors : 5;
> +        /* Enabled? */
> +        bool enabled        : 1;
> +        /* Supports per-vector masking? */
> +        bool masking        : 1;
> +        /* 64-bit address capable? */
> +        bool address64      : 1;
> +        /* Number of vectors configured. */
> +        uint8_t vectors     : 5;
> +        /* Arch-specific data. */
> +        struct vpci_arch_msi arch;
> +#endif
> +    } *msi;
>  };
> 
>  struct vpci_vcpu {
> @@ -97,6 +121,20 @@ struct vpci_vcpu {
>      bool rom_only : 1;
>  };
> 
> +#ifdef __XEN__
> +void vpci_dump_msi(void);
> +
> +/* Arch-specific vPCI MSI helpers. */
> +void vpci_msi_arch_mask(struct vpci_msi *msi, const struct pci_dev *pdev,
> +                        unsigned int entry, bool mask);
> +int __must_check vpci_msi_arch_enable(struct vpci_msi *msi,
> +                                      const struct pci_dev *pdev,
> +                                      unsigned int vectors);
> +void vpci_msi_arch_disable(struct vpci_msi *msi, const struct pci_dev
> *pdev);
> +void vpci_msi_arch_init(struct vpci_msi *msi);
> +void vpci_msi_arch_print(const struct vpci_msi *msi);
> +#endif /* __XEN__ */
> +
>  #else /* !CONFIG_HAS_VPCI */
>  struct vpci_vcpu {};
>  #endif
> --
> 2.16.2

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel