Xen project Mailing List

Re: [Xen-devel] Failed to enable MMU when booting EFI on Seattle

On Tue, Oct 7, 2014 at 6:45 PM, Suravee Suthikulanit <suravee.suthikulpanit@xxxxxxx> wrote: > On 10/7/2014 4:21 AM, Ian Campbell wrote: >> >> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote: >>> >>> Ian/Roy, >>> >>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and >>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to >>> address 0x2005F8, which has all zeros value. This is why we getting the >>> "Synchronous Exception" >>> >>> ...... >>> 1: >>> PRINT("- Turning on paging -\r\n") >>> >>> ldr x1, =paging /* Explicit vaddr, not >>> RIP-relative */ >>> mrs x0, SCTLR_EL2 >>> orr x0, x0, #SCTLR_M /* Enable MMU */ >>> orr x0, x0, #SCTLR_C /* Enable D-cache */ >>> dsb sy /* Flush PTE writes and finish >>> reads */ >>> b . >>> msr SCTLR_EL2, x0 /* now paging is enabled */ >>> isb /* Now, flush the icache */ >>> br x1 /* Get a proper vaddr into PC */ >>> paging: >>> ..... >>> >>> Notes: >>> * This is not the case when booting non-EFI. >>> >>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it >>> seems to branch to "paging:" fine. Although, I don't this this is the >>> right thing to do since it should already be using VA). >>> >>> So, I inspected the page tables and compare the ones from booting with >>> and without EFI. At this point, it seems that the contents of the page >>> tables are set up properly. The only thing different is the phys-offset >>> and the location of the page table: >>> >>> BOOTING WITH EFI: >>> >>> x20 (phys-offset) = 0x83fc2d0000 >>> x19 (start paddr) = 0x83fc4d0000 >>> (.text starts at 2MB = 0x200000 offset) >>> >>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000 >>> boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000 >>> boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000 >>> boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000 >>> >>> BOOTING WITHOUT EFI: >>> >>> x20 (phys-offset) = 0x8008500000 >>> x19 (start paddr) = 0x8008700000 >>> (.text starts at 2MB = 0x200000 offset) >>> >>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000 >>> boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000 >>> boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000 >>> boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000 >>> >>> Here is the memory map available from UEFI: >>> >>> Shell> memmap >>> Type Start End #pages Attributes >>> RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000 >>> 800000000000000F >>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE >>> 000000000000000F >>> BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002 >>> 000000000000000F >>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE >>> 000000000000000F >>> BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002 >>> 000000000000000F >>> Available 0000008020000000-000000802FFFFFFF 0000000000010000 >>> 000000000000000F >>> RT_Code 0000008030000000-0000008030000FFF 0000000000000001 >>> 800000000000000F >>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF >>> 000000000000000F >>> BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020 >>> 000000000000000F >>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3 >>> 000000000000000F >>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF >>> 000000000000000F >>> >>> However, it seems that the location falls in the "Available", which >>> should be ok to use. >> >> >> Right, I don't see anything wrong with the location of the page tables. >> >> What do the contents of these PTs look like? >> >> In particular for Xen's load address of 0x200000 I think entries at >> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are >> of interest. Do they correctly map their next level and/or the physical >> address with the actual Xen bits in? >> >> For completeness it would also be worth knowing the entries >> corresponding to the 1:1 pa mapping. >> >> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42, >> 0x100] (I think, but please do check my maths!). I think this will >> result in boot_pgtable[1] mapping boot_first_id which will contain a >> gigabyte page super mapping in slot [0]. >> >> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd]. >> Which I would again expect to result in boot_pgtable[1] mapping >> boot_first_id with a gigabyte mapping in slot 0xf this time. >> >> I expect the PA mapping to be correct, since as you say we are able to >> run off it, it's worth checking though in case we are somehow creating >> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead >> of boot_first_id when creating the PA mappings -- that sort of thing). >> >> Ian. > > > So, I have experimented with: > > /* > * Preserve x0 (fdt pointer) across call to __flush_dcache_area, > * restore for entry into Xen. > */ > mov x20, x0 > > /* > * Flush dcache covering current runtime addresses > * of xen text/data. Then flush all of icache. > */ > adrp x1, _start > add x1, x1, #:lo12:_start > mov x0, x1 <--- Fixed X0 > adrp x2, _end > add x2, x2, #:lo12:_end > sub x1, x2, x1 > > bl __flush_dcache_area > ic ialluis > tlbi alle2 <---- Added this > > Now, that I added the "tlbi alle2", it works after we enable MMU :) I guess > even though we set up the new page table, but it was still using the stale > info in the TLB. > > On to the next hurdle: > > Shell> FS0:xen -cfg=xen-seattle.cfg > Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty) EFI > loader > Image: 0x00000083fbbca000-0x00000083fc4ca890 > - UART enabled - > - CPU 00000000 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Zero BSS - > - Setting up control registers - > - Setup boot first - > - Setup boot second - > - Setup boot third - > - Turning on paging - > - Ready - > (XEN) Checking for initrd in /chosen > (XEN) RAM: 0000008001000000 - 0000008007ffdfff > (XEN) RAM: 0000008007ffe000 - 0000008007ffffff > (XEN) RAM: 0000008008000000 - 000000801fffdfff > (XEN) RAM: 000000801fffe000 - 000000801fffffff > (XEN) RAM: 0000008020000000 - 000000802fffffff > (XEN) RAM: 0000008030001000 - 00000083f0ffffff > (XEN) RAM: 00000083f1000000 - 00000083f101ffff > (XEN) RAM: 00000083f1020000 - 00000083fbbc7fff > (XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff > (XEN) RAM: 00000083fc6b2000 - 00000083fec25fff > (XEN) RAM: 00000083fec26000 - 00000083fee8bfff > (XEN) RAM: 00000083fee8c000 - 00000083ff225fff > (XEN) RAM: 00000083ff226000 - 00000083ff263fff > (XEN) RAM: 00000083ff265000 - 00000083ff2c4fff > (XEN) RAM: 00000083ffe70000 - 00000083ffffffff > (XEN) > (XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree > (XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel console=hvc0 > console=ttyAMA0,115200 earlycon=pl011,0xe1010000 show_styx_info > root=/dev/sda2 rootwait maxcpus=1 > (XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel > (XEN) > (XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x > dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2 > (XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000 > (XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 => > 000000802fe00000-000000802ff02d81 > (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44 > (XEN) CPU0: Unexpected Trap: Hypervisor > (XEN) ----[ Xen-4.5-unstable arm64 debug=y Not tainted ]---- > (XEN) CPU: 0 > (XEN) PC: 00000000002794b4 bootmem_region_add+0x194/0x1c4 > (XEN) LR: 00000000002794ac > (XEN) SP: 00000000002afd50 > (XEN) CPSR: 800003c9 MODE:64-bit EL2h (Hypervisor, handler) > (XEN) X0: 0000800000000000 X1: 0000800000000000 X2: 0000000000000000 > (XEN) X3: 0000800000000000 X4: ffffffffffffffff X5: 0000000000000000 > (XEN) X6: 0000800000000010 X7: 000000000000000a X8: 0000000000000000 > (XEN) X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038 > (XEN) X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030 > (XEN) X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc > (XEN) X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048 > (XEN) X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000 > (XEN) X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff > (XEN) X27: 0000000000298038 X28: 0000008007ffe000 FP: 00000000002afd50 > (XEN) > (XEN) VTCR_EL2: 80000000 > (XEN) VTTBR_EL2: 0000000000000000 > (XEN) > (XEN) SCTLR_EL2: 30cd183d > (XEN) HCR_EL2: 000000000038643f > (XEN) TTBR0_EL2: 000000802feec000 > (XEN) > (XEN) ESR_EL2: 96000044 > (XEN) HPFAR_EL2: 0000000000000000 > (XEN) FAR_EL2: 0000800000000000 > (XEN) > (XEN) Xen stack trace from sp=00000000002afd50: > (XEN) 00000000002afd80 0000000000279530 0000000000290048 0000000000000000 > (XEN) 0000008001000000 0000008001000000 00000000002afdd0 000000000024b154 > (XEN) 0000000000000000 0000000000000000 0000008001000000 0000008001000000 > (XEN) 0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac > (XEN) 00000000002afe20 0000000000283d50 0000000000298030 0000000000000418 > (XEN) 0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000 > (XEN) 0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690 > (XEN) 00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000 > (XEN) 0000000000400000 0000000000000000 0000000000000001 00000083fc544be0 > (XEN) 00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000 > (XEN) 00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 > (XEN) Xen call trace: > (XEN) [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC) > (XEN) [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR) > (XEN) [<0000000000279530>] init_boot_pages+0x4c/0x174 > (XEN) [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0 > (XEN) [<0000000000283d50>] start_xen+0xc38/0xc58 > (XEN) [<0000000000200690>] paging+0x88/0xc0 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) CPU0: Unexpected Trap: Hypervisor > (XEN) > (XEN) **************************************** > (XEN) > (XEN) Manual reset required ('noreboot' specified) > > Any thoughts?? > > Thanks, > > Suravee > > > > After running a bit with your patch (minus the TLB flushing), I found I was seeing some crashes with a corrupt FDT. From looking at the code, I didn't think we would need to flush the FDT, as I don't think it is accessed before mmu/caching is turned back on. This was an intermittent crash on the FVP model with cache state modelling enabled. The following hacky patch seems (in limited testing) to fix this: diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 2b8998b..1c2346b 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -744,6 +744,8 @@ ENTRY(efi_xen_start) * restore for entry into Xen. */ mov x20, x0 + mov x1, 0x100000 + bl __flush_dcache_area /* * Flush dcache covering current runtime addresses 1 MByte should cover the FDT - if we really do need to flush it I will do a proper flush of the correct size. The code where you are crashing it is examining the FDT to see if there are any mem-reserve regions, so if the FDT is corrupt it could be causing your crashes as well. (There won't be any mem-reserve regions, as the EFI boot code removes them along with the memory nodes from the FDT.) I'll be interested to hear if flushing the FDT fixes your crashes, and I need to further investigate why this flushing seems to be required. Roy _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.