Re: [Xen-devel] HVM CPU enumeration, mapping to VCPU ID (Was: Re: [Xen-users] FreeBSD PVHVM call for testing)

[Matt Wilson <msw@xxxxxxxxxx>]

On Mon, Jun 03, 2013 at 01:33:51PM -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Jun 03, 2013 at 08:57:26AM -0700, Matt Wilson wrote:

[...]

> > 
> > Right, all of the LAPIC information is provided to the guest OS via
> > the MADT. I believe what I'm observing is that Linux and Windows use
> > the order of entries to enumerate processors in the system.
> > 
> > What we typically see on bare metal Intel systems is something like
> > this (example system has 16 cores with HT):
> > 
> > All of the "cores"...
> > [    0.000000] ACPI: Local APIC address 0xfee00000
> > [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
> > ...
> > 
> > Followed by all of the "threads"...
> > [    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x01] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x03] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x05] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x09] enabled)
> > 
> > Since Xen hard codes the LAPIC ID (and x2APIC ID) to 0, 2, 4, 6, 8,
> > etc. (vCPUID * 2), everything looks like a core.
> 
> OK.
>
> > > So .. assuming this was thought out, why are we starting on vCPUs
> > > that don't match to this? That seems like a bug? (Note, this is 
> > > with maxvcpus=32, vcpus=1 and the starting of a VCPU1 actually
> > > ended up starting at VCPU4?!).
> > 
> > I'm lost. What?
> 
> http://lists.xen.org/archives/html/xen-devel/2013-05/msg00941.html

Aah, gotcha.

> > > I think all of this can be sorted out if the hvmloader sets the
> > > LAPIC CPU == VCPU ID properly.
> > 
> > No, that's not the right answer. Or, at least, not completely. Right
> > now Xen provides the same ID for both the LAPIC and x2APIC. In order
> > for cpu topology discovery to work, the x2APIC needs to follow a
> > particular structure. See the Intel whitepaper on processor topology
> > enumeration:
> >   
> > http://software.intel.com/sites/default/files/m/d/4/1/d/8/Kuo_CpuTopology_rc1.rh1.final.pdf
> 
> Nice explanation. However, I was under impression that right now we
> don't virtualize the x2APIC registers?

Hmm?

http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=21c4d089ba2a236dcf33a06222070bd6c60c6e3d

> > > So perhaps a better question is - why is it not setup properly
> > > nowadays? If the formal is baked in for the PVHVM guests, somewhere
> > > the formula is not being evaluated properly?
> > 
> > The "LAPIC ID = 2 * vCPUID" formula is not baked into any OS that I
> > know of, and it shouldn't be. It should all be discovered via
> > firmware/BIOS tables. The enumeration order in the tables should,
> > under best practices, match the logical processor ID assignment in the
> > OS.
> 
> OK, good. That is my understanding too.
> 
> > 
> > > The new hypercall to figure this out could be used, but that wouldn't
> > > explain why we are failing to start on the correct VCPU?
> > 
> > I didn't follow the jump here. Can you provide an example?
> 
> http://lists.xen.org/archives/html/xen-devel/2013-05/msg00941.html

OK, got it.

[   84.619508] smpboot: Booting Node 0 Processor 1 APIC 0x8

So it seems like, in this case:

int __cpuinit native_cpu_up(unsigned int cpu)
{
        int apicid = apic->cpu_present_to_apicid(cpu);

apic->cpu_present_to_apicid(1) returned 8 instead of 2.

All of that should have been set up correctly ahead of time by
generic_processor_info() for all possible CPUs. Do you have the full
boot log?

--msw

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel