Re: [Xen-devel] apic-v reduce network performance in my test case

["Liuqiming (John)" <john.liuqiming@xxxxxxxxxx>]

On 2015/2/6 3:26, Konrad Rzeszutek Wilk wrote:
> On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> > On 2015/2/5 10:57, Liuqiming (John) wrote:
> >> sorry for late replay
> >>
> >> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> >>> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> >>>> On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> >>>>> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> >>>>>> Hi Jan,
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> On 2015/2/2 18:12, Jan Beulich wrote:
> >>>>>>>>>> On 31.01.15 at 11:29, <john.liuqiming@xxxxxxxxxx> wrote:
> >>>>>>>>          Recently I met an odd performance problem: when I turn
> >>> on APIC
> >>>>>>>> Virtualization feature (apicv=1), the network performance of a
> >>> windows
> >>>>>>>> guest become worse.
> >>>>>>>>
> >>>>>>>>          My test case like this: host only have one windows 2008
> >>> R2 HVM
> >>>>>>>> guest running,and this guest has a SR-IOV VF network passthrough
> >>> to it.
> >>>>>>>> Guest using this network access a NAS device. No fontend or
> >>> backend of
> >>>>>>>> network and storage, all data transfered through network.
> >>>>>>>>
> >>>>>>>>          The xentrace data shows: the mainly difference between
> >>> apicv and
> >>>>>>>> non-apicv, is the way guest write apic registers, and
> >>>>>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >>>>>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> >>> is much
> >>>>>>>> less than using APIC-v.
> >>>>>>>
> >>>>>>> There being heavier use of the pause VMEXIT doesn't by itself say
> >>>>>>> anything, I'm afraid. It may suggest that you have a C-state exit
> >>>>>>> latency problem - try lowering the maximum C-state allowed, or
> >>>>>>> disabling use of C-states altogether.
> >>>>>> Sorry, I forgot to mention  my test scenario:
> >>>>>> Its a video test suite,I am not sure what the logic inside the
> >>> tools exactly (not opensource tool).
> >>>>>> The basic flow is:
> >>>>>>         1) test suite start several thread to read video file from
> >>> disk (from NAS through network in my case)
> >>>>>>         2) decode these video data as a frame one by one
> >>>>>>         3) if  any frame delay more than 40ms, then mark as lost
> >>>>>>
> >>>>>> test result:
> >>>>>>           apicv=1,  there can be 15 thread running at the same time
> >>> without lost frame
> >>>>>>           apicv=0,  there can be 22 thread running at the same time
> >>> without lost frame
> >>>>>>
> >>>>>> so when I'm saying apicv reduce the performance, I got the
> >>> conclusion from the test result not from what xentrace shows.
> >>>>>>>
> >>>>>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >>>>>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> >>> APIC-V",
> >>>>>>>> msr based apic is disabled when apic-v is on, I wonder can they
> >>> co-exist
> >>>>>>>> in some way? seems for windows guest msr-based apic has better
> >>> performance.
> >>>>>>>
> >>>>>>> The whole purpose is to avoid the costly MSR access exits. Why
> >>>>>>> would you want to reintroduce that overhead?
> >>>>>>>
> >>>>>>> Jan
> >>>>>>>
> >>>>>> I agree to avoid the MSR access vmexit by using apicv, I just do
> >>> not know what's the side effect.
> >>>>>> Because from the test result,  apicv replacing  msr-based access
> >>> brings performance reduction.
> >>>>>
> >>>>> It sounds like it brings latency disruption, not neccessarily
> >>> performance reduction.
> >>>>>
> >>>>> What happens if you run with the cpufreq turned to performance, or
> >>> as Jan suggested -
> >>>>> with disabling C-states?
> >>>>>
> >>>> I have tried disable C-states in BIOS, the test result is still the
> >>> same.
> >>>> There is no frontend-backend device in my test, dom0 is not
> >>> involved, so you mean cpufreq in DomU, right?
> >>>
> >>> I meant on Xen (cpufreq=performance). But if you disable C-states then
> >>> that should
> >>> not matter.
> >>>
> >>>> I have no idea how to change cpufreq of a windows os but I will
> >>> figure out and try.
> >>>
> >>> That should not be needed.
> >>>
> >>> But back to your problem. When you say you have 'frame delay' - is it
> >>> because
> >>> of the NAS storage not getting the I/Os fast enough?
> >>>
> >> The  NAS storage should not be the bottleneck, it's a ramdisk connected
> >> with 10G network card.
> >> Besides, I'am using the same test environment, the only difference is
> >> whether apicv=1 or not.
> >>
> >>> What are you using as network device? Is it an SR-IOV device or an
> >>> emulated
> >>> device (e1000, ne2k, rtl81..?)?
> >> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> >> Function passthrough network and has the corresponding driver installed.
> >>>>>
> >>>>>> _______________________________________________
> >>>>>> Xen-devel mailing list
> >>>>>> Xen-devel@xxxxxxxxxxxxx
> >>>>>> http://lists.xen.org/xen-devel
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >> When  tracing the performance problem, I found whether apicv is on
> >> change the result.
> >> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> >> presents or not is the key factor.
>
> Ah, willing to send out a patch for that?
I have not get the conclusion: why CPUID4A_MSR_BASED_APIC affect performance.
But I will continue to dig in, and happy to send out what I found.
> >>
> >>
> >>
> >>
> > sorry, forget cc the list
> >
> >
for testing purpose, reverting this commit will do:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=7f2e992b824ec62a2818e64390ac2ccfbd74e6b7

"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V"

> .


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel