Re: issue with dom0_pvh on Xen 4.20

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 02/09/2025 1:52 pm, Andrew Cooper wrote:
> On 02/09/2025 1:24 pm, Manuel Bouyer wrote:
>> On Tue, Sep 02, 2025 at 01:14:29PM +0100, Andrew Cooper wrote:
>>> On 02/09/2025 12:23 pm, Manuel Bouyer wrote:
>>>> On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
>>>>> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
>>>>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>>>>> Hello,
>>>>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>>>>
>>>>>>>> The boot options are:
>>>>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH 
>>>>>>>> console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M 
>>>>>>>> console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all 
>>>>>>>> gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>>>>
>>>>>>>> and the full log from serial console is attached.
>>>>>>>>
>>>>>>>> With 4.20 the boot fails with:
>>>>>>>>
>>>>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch 
>>>>>>>> input)
>>>>>>>> (XEN) Freed 664kB init memory
>>>>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0  x86_64  debug=y  Tainted:   C  
>>>>>>>>   ]----
>>>>>>>> (XEN) CPU:    7
>>>>>>>> (XEN) RIP:    0008:[<000000000020e268>]
>>>>>>>> (XEN) RFLAGS: 0000000000010006   CONTEXT: hvm guest (d0v0)
>>>>>>>> (XEN) rax: 000000002024c003   rbx: 000000000020e260   rcx: 
>>>>>>>> 00000000000dfeb7
>>>>>>>> (XEN) rdx: 0000000000100000   rsi: 0000000000103000   rdi: 
>>>>>>>> 000000000013e000
>>>>>>>> (XEN) rbp: 0000000080000000   rsp: 00000000014002e4   r8:  
>>>>>>>> 0000000000000000
>>>>>>>> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
>>>>>>>> 0000000000000000
>>>>>>>> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
>>>>>>>> 0000000000000000
>>>>>>>> (XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 
>>>>>>>> 0000000000000000
>>>>>>>> (XEN) cr3: 0000000000000000   cr2: 0000000000000000
>>>>>>>> (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 
>>>>>>>> 0000000000000000
>>>>>>>> (XEN) ds: 0010   es: 0010   fs: 0000   gs: 0000   ss: 0010   cs: 0008
>>>>>>>>
>>>>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>>>>
>>>>>>>> I tracked it down to this code:
>>>>>>>>         cmpl    $0,%ecx                 ;       /* zero-sized? */      
>>>>>>>>  \
>>>>>>>>         je      2f                      ; \
>>>>>>>>         pushl   %ebp                    ; \
>>>>>>>>         movl    RELOC(nox_flag),%ebp    ; \
>>>>>>>> 1:      movl    %ebp,(PDE_SIZE-4)(%ebx) ;       /* upper 32 bits: NX 
>>>>>>>> */ \
>>>>>>>>         movl    %eax,(%ebx)             ;       /* store phys addr */  
>>>>>>>>  \
>>>>>>>>         addl    $PDE_SIZE,%ebx          ;       /* next PTE/PDE */     
>>>>>>>>  \
>>>>>>>>         addl    $PAGE_SIZE,%eax         ;       /* next phys page */   
>>>>>>>>  \
>>>>>>>>         loop    1b                      ; \
>>>>>>>>         popl    %ebp                    ; \
>>>>>>>> 2:                                      ;
>>>>>>>>
>>>>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>>>>> (in fact the exact same fragment is called just before with different
>>>>>>>> inputs and it doesn't fault). So the culprit it probably the write to 
>>>>>>>> (%ebx),
>>>>>>>> which would be 0x20e260
>>>>>>>> This is in the range:
>>>>>>>> (XEN)  [0000000000100000, 0000000040068e77] (usable)
>>>>>>>> so I can't see why this would be a problem.
>>>>>>>>
>>>>>>>> Any idea, including how to debug this further, welcome
>>>>>>> Even though triple fault's are aborts, they're generally accurate under
>>>>>>> virt, so 0x20e268 is most likely where things die.
>>>>>> but that's the RIP of the last fault, not the first one, right ?
>>>>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>>>>> first fault triggers an exception, but the exeption handler isn't set up 
>>>>>> yet
>>>>>> so we end up jumping to some random value.
>>>>> Double and Triple faults occur when trying to deliver an exception
>>>>> generates an exception.  So while multiple faults are involved, only one
>>>>> instruction typically is.
>>>>>
>>>>> Is this an Intel or an AMD system?  One thing virt can do is break apart
>>>>> a triple fault, but the logic to do so is vendor specific.
>>>> it's an old intel system:
>>>> cpu0: "Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz"
>>>> cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
>>>> cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
>>>>
>>> Hmm.  Westmere EP.  Are you running with EPT active, or with Shadow Paging?
>> How do I know ?
> Something like:
>
> HVM: Hardware Assisted Paging (HAP) detected
>
> on boot.

Sorry - i missed the log on the root of the conversation.  You do have
HAP on this system.

~Andrew