[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Regression between Xen 4.6.0 and 4.7.0, Direct kernel boot on a qemu-xen and seabios HVM guest doesn't work anymore.

Den 25. aug. 2016 23:18, skrev linux@xxxxxxxxxxxxxx:
On 2016-08-25 22:34, Doug Goldstein wrote:
On 8/25/16 4:21 PM, linux@xxxxxxxxxxxxxx wrote:
Today i tried to switch some of my HVM guests (qemu-xen) from booting of
a kernel *inside* the guest, to a dom0 supplied kernel, which is
described as "Direct Kernel Boot" here:
https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html :

    Direct Kernel Boot

    Direct kernel boot allows booting directly from a kernel and initrd
stored in the host physical
    machine OS, allowing command line arguments to be passed directly.
PV guest direct kernel boot
    is supported. HVM guest direct kernel boot is supported with
limitation (it's supported when
    using qemu-xen and default BIOS 'seabios'; not supported in case of
stubdom-dm and old rombios.)

    kernel="PATHNAME"    Load the specified file as the kernel image.
    ramdisk="PATHNAME"   Load the specified file as the ramdisk.

But qemu fails to start, output appended below.

I tested with:
- current Xen-unstable, which fails.
- xen-stable-4.7.0 release, which fails.
- xen-stable-4.6.0 release, works fine.

Can you include the logs from xl dmesg around that time frame as well?

Ah i thought there wasn't any, but didn't check thoroughly or wasn't there
since the release builds are non-debug by default.

However, back on xen-unstable:
(XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU
(XEN) [2016-08-25 21:09:15.172] HVM19 save: PIC
(XEN) [2016-08-25 21:09:15.172] HVM19 save: IOAPIC
(XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC
(XEN) [2016-08-25 21:09:15.172] HVM19 save: LAPIC_REGS
(XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_IRQ
(XEN) [2016-08-25 21:09:15.172] HVM19 save: ISA_IRQ
(XEN) [2016-08-25 21:09:15.172] HVM19 save: PCI_LINK
(XEN) [2016-08-25 21:09:15.172] HVM19 save: PIT
(XEN) [2016-08-25 21:09:15.172] HVM19 save: RTC
(XEN) [2016-08-25 21:09:15.172] HVM19 save: HPET
(XEN) [2016-08-25 21:09:15.172] HVM19 save: PMTIMER
(XEN) [2016-08-25 21:09:15.172] HVM19 save: MTRR
(XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_DOMAIN
(XEN) [2016-08-25 21:09:15.172] HVM19 save: CPU_XSAVE
(XEN) [2016-08-25 21:09:15.172] HVM19 save: VIRIDIAN_VCPU
(XEN) [2016-08-25 21:09:15.172] HVM19 save: VMCE_VCPU
(XEN) [2016-08-25 21:09:15.172] HVM19 save: TSC_ADJUST
(XEN) [2016-08-25 21:09:15.172] HVM19 restore: CPU 0
(XEN) [2016-08-25 21:09:16.126] d0v1 Over-allocation for domain 19: 262401 > 262400
(XEN) [2016-08-25 21:09:16.126] memory.c:213:d0v1 Could not allocate order=0 extent: id=19 memflags=0 (192 of 512)

Hmm some off by one issue ?


Just wondering how much RAM you're domain is defined with as well?

1024 Mb, there is more than enough unallocated memory for xen to start the guest (and dom0 is fixed with dom0_mem=1536M,max:1536M and ballooning is off)

--
Sander




I've got the same issue, reported it in xen-users som time ago. I never caught on that internal/external kernel would trigger it. I'll just paste my entire message from xen-users below:
------
I have been trying for some time now to upgrade from Xen 4.6.* to 4.7.
Trying several different dom0 kernel versions, and jiggling the xl.cfg
files. All to no avail.

I am unable to launch most of my guests under 4.7, though they run fine
under 4.6 (except for some usb/pci-pass-though -related issues)  . As
seen from the device-model log below, qemu claims it is unable to
allocate ram: "qemu: hardware error: xen: failed to populate ram at
280050000", but I have plenty ram available, and this same VM (and many
more) launch fine under 4.6.*

I admit I am a rank amateur at this, so my config is probably pretty
weird, possibly leading to a set-up that nobody knowledgeable would run.
If somebody can give me a hint on how to work around this issue I'll
happily test patches and provide logs.

Example VM which does not start under 4.7. :

------xl.cfg for media.hvm (i pass pci-pass-through for usb-card on
command-line. Works OK)name = "media.hvm"
builder = "hvm"
xen_platform_pci = '1'
pvh=1
memory = 7168
mmio_hole=3072
vcpus = 6
cap=600
cpus_soft="node:0"
cpu_weight=6144
device_model_version="qemu-xen"
serial = 'pty'
disk = [ 'vdev=xvda, format=raw, target=/dev/system/media-backend'
        ,'vdev=xvdb, format=raw, target=/dev/system/media-backend-swap'
    ,'vdev=xvdd, format=raw, target=/dev/system/apub'
        ,'vdev=xvde, format=raw, target=/dev/system/apub1'
        ,'vdev=xvdf, format=raw, target=/dev/system/apub2'
        ,'vdev=xvdg, format=raw, target=/dev/system/apub3'
        ,'vdev=xvdh, format=raw, target=/dev/system/apub4'
        ,'vdev=xvdi, format=raw, target=/dev/system/apub5'
        ,'vdev=xvdj, format=raw, target=/dev/system/apub6'
        ,'vdev=xvdk, format=raw, target=/dev/system/apub7' ]
kernel = "/etc/xen/media-boot/vmlinuz-4.1.12-gentoo"
extra = "root=/dev/xvda intel_iommu=on console=ttyS0 console=vga
init=/usr/lib/systemd/systemd elevator=deadline xen_blkfront.max=128"
vif = ['mac=02:16:3e:00:00:07,bridge=br0']
_on_poweroff_ = 'destroy'
_on_reboot_ = 'restart'
_on_crash_ = 'restart'
boot = 'd'
acpi = '1'
sdl = '0'
vnc = '1'
--------

This results in the following:

-----VM console log: ----

Parsing config from /etc/xen/media.hvm
libxl: error: libxl_dm.c:2187:device_model_spawn_outcome: domain 3
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1422:domcreate_devmodel_started: device
model did not start: -3
libxl: error: libxl_dm.c:2301:kill_device_model: Device Model already exited
libxl: error: libxl.c:1583:libxl__destroy_domid: non-existant domain 3
libxl: error: libxl.c:1542:domain_destroy_callback: unable to destroy
guest with domid 3
libxl: error: libxl.c:1471:domain_destroy_cb: destruction of domain 3 failed

------ dom0 console: ---

(XEN) [2016-08-10 10:14:09] HVM3 save: CPU
(XEN) [2016-08-10 10:14:09] HVM3 save: PIC
(XEN) [2016-08-10 10:14:09] HVM3 save: IOAPIC
(XEN) [2016-08-10 10:14:09] HVM3 save: LAPIC
(XEN) [2016-08-10 10:14:09] HVM3 save: LAPIC_REGS
(XEN) [2016-08-10 10:14:09] HVM3 save: PCI_IRQ
(XEN) [2016-08-10 10:14:09] HVM3 save: ISA_IRQ
(XEN) [2016-08-10 10:14:09] HVM3 save: PCI_LINK
(XEN) [2016-08-10 10:14:09] HVM3 save: PIT
(XEN) [2016-08-10 10:14:09] HVM3 save: RTC
(XEN) [2016-08-10 10:14:09] HVM3 save: HPET
(XEN) [2016-08-10 10:14:09] HVM3 save: PMTIMER
(XEN) [2016-08-10 10:14:09] HVM3 save: MTRR
(XEN) [2016-08-10 10:14:09] HVM3 save: VIRIDIAN_DOMAIN
(XEN) [2016-08-10 10:14:09] HVM3 save: CPU_XSAVE
(XEN) [2016-08-10 10:14:09] HVM3 save: VIRIDIAN_VCPU
(XEN) [2016-08-10 10:14:09] HVM3 save: VMCE_VCPU
(XEN) [2016-08-10 10:14:09] HVM3 save: TSC_ADJUST
(XEN) [2016-08-10 10:14:09] HVM3 restore: CPU 0
(XEN) [2016-08-10 10:14:11] d0v0 Over-allocation for domain 3: 1835265 >
1835264
(XEN) [2016-08-10 10:14:11] memory.c:209:d0v0 Could not allocate order=0
extent: id=3 memflags=0 (192 of 512)

------------- xl info output: ---------

host                   : gentoo
release                : 4.1.29-gentoo
version                : #1 SMP Wed Aug 10 03:47:43 CEST 2016
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 23
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2394
hw_caps                :
b7ebfbff:77fef3ff:2c100800:00000021:00000001:000037ab:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 65376
free_memory            : 47044
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 7
xen_extra              : .0
xen_version            : 4.7.0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : ssd-xen-4.7-marker console_timestamps=date
loglvl=all guest_loglvl=all sync_console iommu=1,verbose,debug
iommu_inclusive_mapping=1 com1=115200,8n1 console=com1 dom0_max_vcpus=4
dom0_vcpus_pin=1 dom0_mem=7G,max:7G cpufreq=xen,performance,verbose
sched_smt_power_savings=1 apic_verbosity=debug e820-verbose=1
core_parking=power cpuidle=0
cc_compiler            : x86_64-pc-linux-gnu-gcc (Gentoo 5.4.0 p1.0,
pie-0.6.5) 5.4.0
cc_compile_by          :
cc_compile_domain      : alstadheim.priv.no
cc_compile_date        : Tue Aug  9 17:12:07 CEST 2016
build_id               : 124ae07d4d637e3a8dc4150d03008027ce5c4d54
xend_config_format     : 4

-------- device model log: -------

char device redirected to /dev/pts/8 (label serial0)
qemu: hardware error: xen: failed to populate ram at 280050000
CPU #0:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #1:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #2:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #3:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #4:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #5:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
-------

---

Regards, Håkon A.




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.