[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen optimization


Sorry for the formatting.

On Fri, 12 Oct 2018, 17:36 Milan Boberic, <milanboberic94@xxxxxxxxx> wrote:
Hi Stefano, glad to have you back :D,
this is my setup:
        - dom0 is PetaLinux, has 1 vCPU and it's pinned for pCPU0
        - there is only one domU and this is my bare-metal app that also have one vCPU and it's pinned for pCPU1
so yeah, there is only dom0 and bare-metal app on the board.

Jitter is the same with and without Dario's patch.

I'm still not sure about timer's passthrough because there is no mention of triple timer counter is device tree so I added:

&ttc0 {
   xen,passthrough = <0x1>;

Would you mind to explain what is the triple timer counter?

at the end of the xen-overlay.dtsi file which I included in attachment.

About patch you sent, I can't find this funcion void vgic_inject_irq in /xen/arch/arm/vgic.c file, this is link of git repository from where I build my xen so you can take a look if that printk can be put somewhere else.

There was some vGIC rework in Xen 4.11. There was also a new vGIC added (selectable using NEW_VGIC). It might be worth to look at it.

This is not the official Xen repository and look like patches have been applied on top. I am afraid, I am not going to be able help here. Could you do the same experiment with Xen 4.11?

I ran some more testing and realized that results are the same with or without vwfi=native, which I think again points out that passthrough that I need to provide in device tree isn't valid.

This could also means that wfi is not used by the guest or you never go to the idle vCPU.

 And of course, higher the frequency of interrupts results in higher jitter. I'm still battling with Xilinx SDK and triple timer counter that's why I can't figure out what is the exact frequency set (I'm just rising it and lowering it), I'll give my best to solve that ASAP because we need to know exact value of frequency set. 

Thanks in advance!


On Fri, Oct 12, 2018 at 12:29 AM Stefano Stabellini <stefano.stabellini@xxxxxxxxxx> wrote:
On Thu, 11 Oct 2018, Milan Boberic wrote:
> On Wed, Oct 10, 2018 at 6:41 PM Meng Xu <xumengpanda@xxxxxxxxx> wrote:
> >
> > The jitter may come from Xen or the OS in dom0.
> > It will be useful to know what is the jitter if you run the test on PetaLinux.
> > (It's understandable the jitter is gone without OS. It is also common
> > that OS introduces various interferences.)
> Hi Meng,
> well... I'm using bare-metal application and I need it exclusively to
> be ran on one CPU as domU (guest) without OS (and I'm not sure how
> would I make the same app to be ran on PetaLinux dom0 :D haha).
> Is there a chance that PetaLinux as dom0 is creating this jitter and
> how? Is there a way of decreasing it?
> Yes, there are no prints.
> I'm not sure about this timer interrupt passthrough because I didn't
> find any example of it, in attachment I included xen-overlay.dtsi file
> which I edited to add passthrough, in earlier replies there are
> bare-metal configuration file. It would be helpful to know if those
> setting are correct. If they are not correct it would explain the
> jitter.
> Thanks in advance, Milan Boberic!

Hi Milan,

Sorry for taking so long to go back to this thread. But I am here now :)

First, let me ask a couple of questions to understand the scenario
better: is there any interference from other virtual machines while you
measure the jitter? Or is the baremetal app the only thing actively
running on the board?

Second, it would be worth double-checking that Dario's patch to fix
sched=null is not having unexpected side effects. I don't think so, it
would be worth testing with it and without it to be sure.

I gave a look at your VM configuration. The configuration looks correct.
There is no dtdev settings, but given that none of the devices you are
assigning to the guest does any DMA, it should be OK. You want to make
sure that Dom0 is not trying to use those same devices -- make sure to
add "xen,passthrough;" to each corresponding node on the host device

The error messages "No valid vCPU found" are due to the baremetal
applications trying to configure as target cpu for the interrupt cpu1
(the second cpu in the system), while actually only 1 vcpu is assigned
to the VM. Hence, only cpu0 is allowed. I don't think it should cause
any jitter issues, because the request is simply ignored. Just to be
safe, you might want to double check that the physical interrupt is
delivered to the right physical cpu, which would be cpu1 in your
configuration, the one running the only vcpu of the baremetal app. You
can do that by adding a printk to xen/arch/arm/vgic.c:vgic_inject_irq,
for example:

diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 5a4f082..208fde7 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -591,6 +591,7 @@ void vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
     spin_unlock_irqrestore(&v->arch.vgic.lock, flags);

+    if (v != current) printk("DEBUG irq slow path!\n");
     /* we have a new higher priority irq, inject it into the guest */

You don't want "DEBUG irq slow path!" to get printed.

Finally, I would try to set the timer to generate events less frequently
than every 1us and see what happens, maybe every 5-10us. In my tests,
the IRQ latency overhead caused by Xen is around 1us, so injecting 1
interrupt every 1us, plus 1us of latency caused by Xen, cannot lead to
good results.

I hope this helps, please keep us updated with your results, they are
very interesting!
Xen-devel mailing list
Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.