[Xen-users] XEN live migration: cannot console or ssh to the migrated guest VM (domU)

[Wenda Ni <wonda.ni@xxxxxxxxx>]

I am now testing XEN live migration on two physical hosts with XEN 4.0.1 pvops Ubuntu 10.10 (2.6.35-22). Host A also acts as a NFS server, and Host B acts as a NFS client. 
When I migrate a guest domain from B to A. The ssh connection experiences a downtime of about 1 minute (the terminal does not react to the keyboard input till about 1 minute later). However, the "sudo xm console" does not work unless a reboot is triggered. 
When I migrate the guest domain from A to back B, the situation is even worse -- neither ssh connection nor console works. 
In both migration cases, you can notice that the guest domain is there by using "sudo xm list". 

Could anyone shed some light on the potential problems I am facing? I have been working on this for quite some time, but have no idea what goes wrong. The following is the log file copied from host B, when the guest domain is migrated from A back to B.



[2011-02-06 23:35:51 5208] DEBUG (XendDomainInfo:235) XendDomainInfo.restore(['domain', ['domid', '8'], ['cpu_weight', '256'], ['cpu_cap', '0'], ['bootloader', '/usr/bin/pygrub'], ['on_crash', 'restart'], ['uuid', 'aaf5c3c7-a5a6-d306-dc77-483cea04bff5'], ['bootloader_args', '-q'], ['vcpus', '1'], ['description'], ['name', 'Elite'], ['cpus', [[]]], ['on_reboot', 'restart'], ['on_poweroff', 'destroy'], ['maxmem', '128'], ['memory', '128'], ['shadow_memory', '0'], ['vcpu_avail', '1'], ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', '1297053017.52'], ['cpu_time', '7.409878829'], ['online_vcpus', '1'], ['image', ['linux', ['kernel'], ['args', 'root=/dev/sda2 ro '], ['superpages', '0'], ['videoram', '4'], ['pci', []], ['nomigrate', '0'], ['tsc_mode', '0'], ['notes', ['HV_START_LOW', '4118806528'], ['FEATURES', '!writable_page_tables|pae_pgdir_above_4gb'], ['VIRT_BASE', '3221225472'], ['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '0'], ['GUEST_OS', 'linux'], ['HYPERCALL_PAGE', '3222282240'], ['LOADER', 'generic'], ['SUSPEND_CANCEL', '1'], ['PAE_MODE', 'yes'], ['ENTRY', '3229974528'], ['XEN_VERSION', 'xen-3.0']]]], ['status', '2'], ['state', '-b----'], ['store_mfn', '472195'], ['console_mfn', '472194'], ['device', ['vif', ['ip', '138.15.106.226'], ['mac', '00:16:3E:E9:17:69'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid', 'b3761d83-4167-d29e-2833-98065be549c8'], ['backend', '0']]], ['device', ['console', ['protocol', 'vt100'], ['location', '2'], ['uuid', '99ef56f7-bbd9-0a3c-277e-bbcc8f5cebce']]], ['device', ['vbd', ['protocol', 'x86_32-abi'], ['uuid', '638d020f-b8cd-4100-69a0-c3efc5334b08'], ['bootable', '1'], ['dev', 'sda2:disk'], ['uname', 'file:/home/user1/vm-storage-export/domains/Elite/disk.img'], ['mode', 'w'], ['backend', '0'], ['VDI']]], ['device', ['vbd', ['protocol', 'x86_32-abi'], ['uuid', 'dac29d55-cda5-29c5-b362-dee4b24e60d6'], ['bootable', '0'], ['dev', 'sda1:disk'], ['uname', 'file:/home/user1/vm-storage-export/domains/Elite/swap.img'], ['mode', 'w'], ['backend', '0'], ['VDI']]], ['change_home_server', 'False']])
[2011-02-06 23:35:51 5208] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2011-02-06 23:35:51 5208] DEBUG (balloon:220) Balloon: 263812 KiB free; need 16384; done.
[2011-02-06 23:35:51 5208] DEBUG (XendDomain:464) Adding Domain: 15
[2011-02-06 23:35:51 5208] DEBUG (XendDomainInfo:3400) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'aaf5c3c7-a5a6-d306-dc77-483cea04bff5', 'on_reboot': 'restart', 'start_time': '1297053017.52', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (kernel ) (args 'root=/dev/sda2 ro ') (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES '!writable_page_tables|pae_pgdir_above_4gb') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3229974528) (XEN_VERSION xen-3.0)))", 'name': 'Elite'}
[2011-02-06 23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: console : {'protocol': 'vt100', 'location': '2', 'uuid': '99ef56f7-bbd9-0a3c-277e-bbcc8f5cebce'}
[2011-02-06 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/15/0'} to /local/domain/15/device/console/0.
[2011-02-06 23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain': 'Elite', 'frontend': '/local/domain/15/device/console/0', 'uuid': '99ef56f7-bbd9-0a3c-277e-bbcc8f5cebce', 'frontend-id': '15', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/15/0.
[2011-02-06 23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: vbd : {'protocol': 'x86_32-abi', 'uuid': '638d020f-b8cd-4100-69a0-c3efc5334b08', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda2:disk', 'uname': 'file:/home/user1/vm-storage-export/domains/Elite/disk.img', 'mode': 'w', 'backend': '0'}
[2011-02-06 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'virtual-device': '2050', 'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/15/2050'} to /local/domain/15/device/vbd/2050.
[2011-02-06 23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain': 'Elite', 'frontend': '/local/domain/15/device/vbd/2050', 'uuid': '638d020f-b8cd-4100-69a0-c3efc5334b08', 'bootable': '1', 'dev': 'sda2', 'state': '1', 'params': '/home/user1/vm-storage-export/domains/Elite/disk.img', 'mode': 'w', 'online': '1', 'frontend-id': '15', 'type': 'file'} to /local/domain/0/backend/vbd/15/2050.
[2011-02-06 23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: vbd : {'protocol': 'x86_32-abi', 'uuid': 'dac29d55-cda5-29c5-b362-dee4b24e60d6', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'sda1:disk', 'uname': 'file:/home/user1/vm-storage-export/domains/Elite/swap.img', 'mode': 'w', 'backend': '0'}
[2011-02-06 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'virtual-device': '2049', 'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/15/2049'} to /local/domain/15/device/vbd/2049.
[2011-02-06 23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain': 'Elite', 'frontend': '/local/domain/15/device/vbd/2049', 'uuid': 'dac29d55-cda5-29c5-b362-dee4b24e60d6', 'bootable': '0', 'dev': 'sda1', 'state': '1', 'params': '/home/user1/vm-storage-export/domains/Elite/swap.img', 'mode': 'w', 'online': '1', 'frontend-id': '15', 'type': 'file'} to /local/domain/0/backend/vbd/15/2049.
[2011-02-06 23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: vif : {'ip': '138.15.106.226', 'mac': '00:16:3E:E9:17:69', 'script': '/etc/xen/scripts/vif-bridge', 'uuid': 'b3761d83-4167-d29e-2833-98065be549c8', 'backend': '0'}
[2011-02-06 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3E:E9:17:69', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/15/0'} to /local/domain/15/device/vif/0.
[2011-02-06 23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain': 'Elite', 'handle': '0', 'uuid': 'b3761d83-4167-d29e-2833-98065be549c8', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '138.15.106.226', 'mac': '00:16:3E:E9:17:69', 'frontend-id': '15', 'state': '1', 'online': '1', 'frontend': '/local/domain/15/device/vif/0'} to /local/domain/0/backend/vif/15/0.
[2011-02-06 23:35:51 5208] DEBUG (XendDomainInfo:1804) Storing domain details: {'image/entry': '3229974528', 'console/port': '2', 'image/loader': 'generic', 'vm': '/vm/aaf5c3c7-a5a6-d306-dc77-483cea04bff5', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/virt-base': '3221225472', 'memory/target': '131072', 'image/guest-version': '2.6', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '3222282240', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '0', 'console/type': 'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2011-02-06 23:35:52 5208] DEBUG (XendCheckpoint:286) restore:shadow=0x0, _static_max=0x8000000, _static_min=0x0, 
[2011-02-06 23:35:52 5208] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 4 15 1 2 0 0 0 0
[2011-02-06 23:35:52 5208] INFO (XendCheckpoint:423) xc_domain_restore start: p2m_size = 8000
[2011-02-06 23:35:52 5208] INFO (XendCheckpoint:423) Reloading memory pages:   0%
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) ERROR Internal error: Error when reading batch size
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) ERROR Internal error: error when buffering batch, finishing
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) 
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) 100%
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) Memory reloaded (9067 pages)
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) read VCPU 0
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) Completed checkpoint load
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) Domain ready to be built.
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) Restore exit with rc=0
[2011-02-06 23:36:03 5208] DEBUG (XendCheckpoint:394) store-mfn 163723
[2011-02-06 23:36:03 5208] DEBUG (XendCheckpoint:394) console-mfn 163722
[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:2992) XendDomainInfo.completeRestore
[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref': '163722', 'image/entry': '3229974528', 'console/port': '2', 'store/ring-ref': '163723', 'image/loader': 'generic', 'vm': '/vm/aaf5c3c7-a5a6-d306-dc77-483cea04bff5', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/virt-base': '3221225472', 'memory/target': '131072', 'image/guest-version': '2.6', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '3222282240', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '0', 'console/type': 'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:3005) XendDomainInfo.completeRestore done
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices tap2.
[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vif.
[2011-02-06 23:36:03 5208] DEBUG (DevController:144) Waiting for 0.
[2011-02-06 23:36:03 5208] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/15/0/hotplug-status.
[2011-02-06 23:36:03 5208] DEBUG (DevController:642) hotplugStatusCallback 1.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vkbd.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices ioports.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices tap.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vif2.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices console.
[2011-02-06 23:36:03 5208] DEBUG (DevController:144) Waiting for 0.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vscsi.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vbd.
[2011-02-06 23:36:03 5208] DEBUG (DevController:144) Waiting for 2050.
[2011-02-06 23:36:03 5208] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/15/2050/hotplug-status.
[2011-02-06 23:36:03 5208] DEBUG (DevController:642) hotplugStatusCallback 1.
[2011-02-06 23:36:03 5208] DEBUG (DevController:144) Waiting for 2049.
[2011-02-06 23:36:03 5208] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/15/2049/hotplug-status.
[2011-02-06 23:36:03 5208] DEBUG (DevController:642) hotplugStatusCallback 1.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices irq.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vfb.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices pci.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vusb.
[2011-02-06 23:36:03 5208] DEBUG (DevController:139) Waiting for devices vtpm.



--

Cheers,

 

Wenda Ni, Ph.D.

Dept. of Computer Science and Engineering

State University of New York (SUNY) at Buffalo

T: 716 982 6746

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users