Re: Problem loading linux 5.19 as PV dom0

[Jan Beulich <jbeulich@xxxxxxxx>]

On 23.06.2022 10:06, Juergen Gross wrote:
> On 23.06.22 09:55, Jan Beulich wrote:
>> On 22.06.2022 18:06, Juergen Gross wrote:
>>> A Linux kernel 5.19 can only be loaded as dom0, if it has been
>>> built with CONFIG_AMD_MEM_ENCRYPT enabled. This is due to the
>>> fact that otherwise the (relevant) last section of the built
>>> kernel has the NOLOAD flag set (it is still marked with
>>> SHF_ALLOC).
>>>
>>> I think at least the hypervisor needs to be changed to support
>>> this layout. Otherwise it will put the initial page tables for
>>> dom0 at the same position as this last section, leading to
>>> early crashes.
>>
>> Isn't Xen using the bzImage header there, rather than any ELF
>> one? In which case it would matter how the NOLOAD section is
> 
> For a PV kernel? No, I don't think so.

Actually it's a mix (and the same for PV and PVH) - the bzImage
header is parsed to get at the embedded ELF header. XenoLinux was
what would/could be loaded as plain ELF.

>> actually represented in that header. Can you provide a dump (or
>> binary representation) of both headers?
> 
> Program Header:
>      LOAD off    0x0000000000200000 vaddr 0xffffffff81000000 paddr 
> 0x0000000001000000 align 2**21
>           filesz 0x000000000145e114 memsz 0x000000000145e114 flags r-x
>      LOAD off    0x0000000001800000 vaddr 0xffffffff82600000 paddr 
> 0x0000000002600000 align 2**21
>           filesz 0x00000000006b7000 memsz 0x00000000006b7000 flags rw-
>      LOAD off    0x0000000002000000 vaddr 0x0000000000000000 paddr 
> 0x0000000002cb7000 align 2**21
>           filesz 0x00000000000312a8 memsz 0x00000000000312a8 flags rw-
>      LOAD off    0x00000000020e9000 vaddr 0xffffffff82ce9000 paddr 
> 0x0000000002ce9000 align 2**21
>           filesz 0x00000000001fd000 memsz 0x0000000000317000 flags rwx

20e9000 + 317000 = 240000

>      NOTE off    0x000000000165df10 vaddr 0xffffffff8245df10 paddr 
> 0x000000000245df10 align 2**2
>           filesz 0x0000000000000204 memsz 0x0000000000000204 flags ---
> 
> 
> Sections:
> Idx Name          Size      VMA               LMA               File off  Algn
> ...
>   30 .smp_locks    00009000  ffffffff82edc000  0000000002edc000  022dc000  
> 2**2
>                    CONTENTS, ALLOC, LOAD, READONLY, DATA
>   31 .data_nosave  00001000  ffffffff82ee5000  0000000002ee5000  022e5000  
> 2**2
>                    CONTENTS, ALLOC, LOAD, DATA
>   32 .bss          0011a000  ffffffff82ee6000  0000000002ee6000  022e6000  
> 2**12
>                    ALLOC

2ee6000 + 11a000 = 240000

>   33 .brk          00026000  ffffffff83000000  ffffffff83000000  00000000  
> 2**0
>                    ALLOC

This space isn't covered by any program header. Which in turn may be a
result of its LMA matching its VMA, unlike for all other sections.
Looks like a linker script or linker issue to me: While ...

> And the related linker script part:
> 
>          __end_of_kernel_reserve = .;
> 
>          . = ALIGN(PAGE_SIZE);
>          .brk (NOLOAD) : AT(ADDR(.brk) - LOAD_OFFSET) {

... this AT() looks correct to me, I'm uncertain of the use of NOLOAD.
Note that .bss doesn't have NOLOAD, matching the vast majority of the
linker scripts ld itself has.

Jan

>                  __brk_base = .;
>                  . += 64 * 1024;         /* 64k alignment slop space */
>                  *(.bss..brk)            /* areas brk users have reserved */
>                  __brk_limit = .;
>          }
> 
>          . = ALIGN(PAGE_SIZE);           /* keep VO_INIT_SIZE page aligned */
>          _end = .;
> 
> 
> Juergen