Re: [Xen-devel] HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2 <ariel.atom2@xxxxxxxxxx>]

Am 17.11.15 um 00:10 schrieb Atom2:
> Am 17.11.15 um 00:01 schrieb Andrew Cooper:
> > On 16/11/2015 19:16, Atom2 wrote:
> > > Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
> > > > > > > Your analysis was absolutely spot on. After re-thinking this for a
> > > > > > > moment, I thought going down that route first would make a lot of
> > > > > > > sense
> > > > > > > as PV guests still do work and one of the differences to HVM 
> > > > > > > domUs is
> > > > > > > that the former do _not_ require SeaBIOS. Looking at my log 
> > > > > > > files of
> > > > > > > installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
> > > > > > > 1.8.2 in
> > > > > > > the relevant timeframe which obviously had not made it to the
> > > > > > > hvmloader
> > > > > > > of xen-4.5.1 as I did not re-compile xen after the upgrade of
> > > > > > > SeaBIOS.
> > > > > > >
> > > > > > > So I re-compiled xen-4.5.1 (obviously now using the installed 
> > > > > > > SeaBIOS
> > > > > > > 1.8.2) and the same error as with xen-4.5.2 popped up - and that
> > > > > > > seemed
> > > > > > > to strongly indicate that there indeed might be an issue with
> > > > > > > SeaBIOS as
> > > > > > > this probably was the only variable that had changed from the
> > > > > > > original
> > > > > > > install of xen-4.5.1.
> > > > I recall seeing this way back in Fedora 20 days. I narrowed it down 
> > > > the
> > > > SeaBIOS version that was a standalone package to not have CONFIG_XEN.
> > > >
> > > > Having that fixed in the SeaBIOS package fixed it.
> > > Hi Konrad, Doug, Andrew (specifically added to this part of the 
> > > thread)!
> > > Konrad, you might have found an interesting point. I did have a look
> > > at the ebuild for the failing version and in there I found the
> > > following comment:
> > > ====== comment from ebuild =======
> > >      # Upstream hasn't released a new binary.  We snipe ours from
> > > Fedora for now.
> > >      # 
> > > http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
> > > ====== end comment from ebuild =======
> > > which might in fact underline that there might be an issue similar to
> > > what you described above.
> > >
> > > What is also pretty interesting is the fact that the old (working)
> > > SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
> > > is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
> > > 1.8.2 installed in the same location is only half as big: 131.072 
> > > bytes.
> > > I checked at the download site and the 1.8.2 binary version is indeed
> > > not available from http://code.coreboot.org/p/seabios/downloads/. But
> > > both the binary versions for 1.7.5 and 1.8.0 are available and both
> > > are acutually 262.144 bytes in size, so I'd be very surprised if the
> > > 1.8.2 version is really only half that size. By the way, the old
> > > working version (according to the ebuild) was directly downloaded from
> > > the above url and also shows an identical SHA1 digest to that version
> > > available for download there.
> > >
> > > To me this looks as if something is really wrong here. If anybody of
> > > you has access to a 1.8.2 version, could you please confirm whether
> > > there's really that big a size difference between the 1.7.5 and the
> > > 1.8.2 versions? Or is that difference probably attributable to the
> > > missing CONFIG_XEN option?
> > >
> > > Andrew: I havent't gotten around to run the debug version of the
> > > hypervisor again, but if the current suspicion turns out to be true,
> > > there's probably not much value in that anyways. Would you agree?
> > Sadly not.
> Fair enough. I'll try to get things done, hopefully somewhen tomorrow 
> or, in case that doesn't work out, on Wednesday and will send you the 
> requested information.
> Many thanks for your support, Atom2
> > I accept that this issue is possibly fixed in newer SeaBIOS by working
> > around the issue.
> >
> > However, I stand by my original point.  *There is no way the guest
> > should be able to get into this situation in the first place*, and its
> > implication of *there is a genuine hypervisor bug which we should track
> > down*, irrespective of whether the issue has been worked around elsehow
Hi Andrew,
as promised I have again tried with a debug build and the results are 
very mixed. I initially tried to better understand what the debug USE 
flag actually does in gentoo and my understanding (after reading the so 
called ebuilds) is now that the XEN hypervisor will be built by adding a 
gcc option of "debug=y" - and that's what should compile a debug build - 
right? So I went on and again enabled the debug USE flag plus gdb 
symbols and rebuilt the hypervisor in the hope that this created a valid 
and working debug build.
It, however, seems there's another problem lurking somewhere which only 
manifests itself when I boot from the debug build of the hypervisor. The 
system crashes early on with a DOUBLE FAULT in doIRQ - we have had this 
already earlier in that thread. I am however a step further as the 
disass in gdb now seems to provide not just an empty page full of NULL 
values but rather something that might give you a hint why it crashes 
that early on: Please see the attached disass file (doIRQ) together with 
the serial console output (serial.dbg). The old NULL value file was 
probably because I did not include gdb symbols in the debug build at 
that time - my bad.
So I am at loss why a debug build actually results in a crash early on, 
but probably Doug could step in here and explain that ... it initially 
thought that it might have to do with the hardened sources that I am 
using, but discarded that option because linux does not play any role 
that early on in the boot process (albeit all lines in the serial log 
are prefixed with "(XEN)").
Because using a debug build of the hypervisor did not work out, I at 
least used the debug build of the xen-tools (which includes hvmloader) 
and run this on the working hypervisor (i.e. the standard, non-debug 
build) in the hope that this might prove useful.
Please find attached the "xl dmesg" from an attempt to start a HVM domU. 
There's a bit more information included compared to the previous xl 
dmesg output, but I don't know whether that's of any help. Unfortunately 
I could not spot any lines "prefixed with [HVM:$DOMID.$VCPUID]" which 
you had expected. But I assume that's because I was forced to boot from 
a non-debug build of the hypervisor in order to get the system up at all.
So to me it looks as if we are currently stuck - unless you (or Doug) 
are able to find the root cause of the early crash when using the debug 
build of the hypervisor. If that is resolved, I could then move on to 
the next step and hopefully be able to provide the information that you 
are after.
Sorry for not being able to provide better input right now.

Thanks Atom2

Attachment: serial.dbg
Description: Text document

Attachment: doIRQ
Description: Text document

Attachment: xl.dmesg
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel