Re: [Xen-devel] xen: arm: beginning the removal of mode_switch.S

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Fri, 2013-08-16 at 16:44 +0100, Julien Grall wrote:
> Unfortunately I haven't a patch for this part. My change was a quick
> hack to try to move Arndale kick cpus in C code. But I had some issue
> with the page table. I don't remember what was the exact problem and I
> lost my hack :/.

Sadly I think removing kick_cpus is going to require at least some
reworking of the boot time pages tables, for at least secondary CPUs.
Note that I still think we should do it ;-)

I think currently we have:

        BOOT CPU                        SECONDARY CPU
        
        startup                         held in firmware
        
        call platform kick cpu.
        
                                        wfe loop 1, checking smp_up_cpu for our 
CPU
        
        
*** At this point all CPUs are running Xen code, at the original 
*** physical load address and with paging disabled
        
        ensure we are in hyp mode
        (mode_switch.S called here, 
         needs to go!)
        
        zero bss
        
        call proc specific cpu_init
        
        setup MMU (HMAIR, HTCR, HSCTLR)
        
        HTTBR = boot_pgtable
        
        build page tables in boot_pgtable,
        contains regular vaddr mapping of
        Xen at 2M and a 1:1 mapping of the
        physical load address. This might
        play badly if the load address was 2M?
        
        enable paging
        
*** At this point the boot CPU is running on the boot_pgtable at the 
*** original physical load address on an identity mapping.
*** Secondary CPUs are still spinning at the start with paging disabled.
        
        jump to Xen virtual mapping at 2M
        
*** At this point the boot CPU is running on the boot_pgtable at the 
*** original physical load address on the normal vaddr (text at 2M).
*** Secondary CPUs are still spinning at the original paddr with paging
*** disabled.
        
        load boot stack pointer
        
        branch to start_xen
        
        setup_pagetables
          copies xen image, including
          boot_pgtable to final physical
          location
        
          remove 1:1 mapping from copy, 
          leave in the original
        
          switch to relocated page tables
        
*** At this point the boot CPU is now running on the relocated 
*** boot_pgtable at the normal vaddr.
*** Secondary CPUs are still spinning at the original paddr with paging
*** disabled.
    (at this point we have something of a bug, which is that we rely on 
     luck to not overwrite the original physical address...)

        ... other setup
        
        make_cpus_ready:
          for each cpu (0..max_cpus):
            set smp_up_cpu at orig paddr.
            sev
            wait for ready_cpus to change
        
                                        exits wfe loop 1
        
                                        ensure in hyp mode (mode_switch.S, must 
die)
        
                                        proc specific cpu init
        
                                        setup MMU (HMAIR, HTCR, HSCTLR)
        
                                        HTTBR = boot_pgtable (original paddr)
        
                                        enable paging
        
*** At this point the boot CPU is now running on the relocated 
*** boot_pgtable at the normal vaddr.
*** Secondary CPUs are now running on the boot_pgtable at the original 
*** load address, on the 1:1 mapping original used by the boot CPU. 

                                        jump to 2MB text mapping
        
*** At this point the boot CPU is now running on the relocated 
*** boot_pgtable at the normal vaddr.
*** Secondary CPUs are now running on the boot_pgtable at the original
*** load address, on the 1:1 mapping original used by the boot CPU. 

                                        switch from boot_pgtable to boot_ttbr
        
*** At this point all CPUs are now running on the relocated 
*** pgtables at the normal vaddr.

                                        increment ready_cpus
        
                                        wfe loop 1, checking smp_up_cpu for our 
CPU
                                        this time smp_up_cpu at relocated 
address
        
        for each secondary:
          __cpu_up:
            write cpu to smp_up_cpu
        
            sev
        
            wait for cpu to come up
        
                                        wakes up
        
                                        calls start_secondary
        
                                        marks cpu online
        
                                        enters idle loop
        
            cpu comes on line
        
        build and start dom0.
        
        idle loop
        
*** we are done

So this is all pretty complex (not to mention hard to describe in ASCII)
and in lockstep, the secondary cpus wait twice once on the original
smp_cpu_up and then again on the relocated version. There is a subtle
reliance on the 1:1 mapping being retained in the original copy of the
page tables.

I think the original wait is actually a workaround for lack of firmware
on the fastmodels, and should be implemented by either the firmware or
bootwrapper.

The second wait is something of a consequence of this and is because the
secondary CPUs start on the original paddr. They should instead be told
to start directly on the relocated copy of Xen. This gets rid of both
the second wait and the make_cpus_ready call I think.

Ultimately it should be possible for the boot cpu to bootstrap itself
all the way to "for each secondary: __cpu_up" while the secondaries are
still spinning in the firmware. __cpu_up should instruct (using the
platforms existing firmware interface, SYSREG or PSCI etc) the
secondaries to start at a paddr within the relocated image (removing the
bug with crossing our fingers and hoping the original is still there).
Secondary CPUs would then come all the way up to idling without needing
to wait for anything.

However there is a slight wrinkle which is that the relocated page
tables do not contain the 1:1 mapping needed to enable paging on the
secondaries.

I think the right solution here is to have a special set of page tables
which are used only during bring up (and eventually for pCPU hotplug
etc) which contain the necessary 1:1 mapping + 2M mapping of Xen.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel