Re: [Xen-devel] [Xen-users] xend (12, 'Cannot allocate memory')

[Keir Fraser <keir.fraser@xxxxxxxxxxxxx>]

Given you are running dom0 with plenty of swap, it's most likely that Xen
has run out of memory to allocate to your new domU, rather than the
allocation failure coming from within dom0. Probably auto-ballooning is
screwing up somehow (that's the bit of xend that gives away memory back to
Xen to make sure it has enough memory to create your new domU).

Sorry I don't have a precise diagnosis for you, but I think looking at Xen's
allocator is at least the right direction for you to take. 'xm info' tells
you how much memory Xen thinks it has.

 -- Keir

On 26/3/08 14:54, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:

> I'm getting the error:
> xend (12, 'Cannot allocate memory')
>  From the command line after the Dom-0 goes through a period of heavy
> memory usage. I don't have any hard numbers on how to duplicate it, but
> I'll give you the steps I've gone through to duplicate the problem.
> 
> First, some information about my setup. It's a new setup, and has been
> in heavy development for the past few months. I'm new to Xen as of this
> project we started in January. As far as I can tell, this problem has
> existed since we started, and may have even happened before but we were
> too new to think it wasn't our fault (which it may still be....)  I've
> found the same error addressed here:
> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
> The 'hack' to fix this problem is already built into my
> XendDomainInfo.py script already, so I'm a little more than lost.
> 
> [root@xen1 ~]# xm info
> host                   : xen1.fiber.net
> release                : 2.6.18-53.1.13.el5xen
> version                : #1 SMP Tue Feb 12 13:33:07 EST 2008
> machine                : x86_64
> nr_cpus                : 8
> nr_nodes               : 1
> sockets_per_node       : 2
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 1866
> hw_caps                :
> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
> total_memory           : 16382
> free_memory            : 2
> xen_major              : 3
> xen_minor              : 1
> xen_extra              : .0-53.1.13.el5
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
> cc_compile_by          : mockbuild
> cc_compile_domain      :
> cc_compile_date        : Tue Feb 12 12:55:35 EST 2008
> xend_config_format     : 2
> 
> 
> Here's the error in action:
> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
> Using config file "./centos-5.1_core_image".
> Error: (12, 'Cannot allocate memory')
> [root@xen1 centos-5.1_core_image]#
> 
> 
> Here is the output from xend.log that occurs when I get this error:
> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
> '/usr/lib/xen/boot/hvmloader'], ['device_model',
> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]])
> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> (XendDomainInfo:306) parseConfig: config is ['vm', ['name',
> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
> '/usr/lib/xen/boot/hvmloader'], ['device_model',
> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]
> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
> 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot':
> 'restart', 'localtime': None, 'image': ['hvm', ['kernel',
> '/usr/lib/xen/boot/hvmloader'], ['device_model',
> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> '/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy',
> 'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image',
> 'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None,
> 'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname',
> 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev',
> 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname',
> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac',
> '00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu':
> None, 'maxmem': 4024}
> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> (XendDomainInfo:1349) XendDomainInfo.construct: None
> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB
> free; need 2048; done.
> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR
> (XendDomainInfo:212) Domain construction failed
> Traceback (most recent call last):
>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> line 204, in create
>    vm.construct()
>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> line 1369, in construct
>    hvm = hvm)
> Error: (12, 'Cannot allocate memory')
> 
> 
> Here's the configuration file being used:
> name = "centos-5.1_core_image"
> maxmem = 4024
> memory = 4024
> vcpus = 6
> builder = "hvm"
> kernel = "/usr/lib/xen/boot/hvmloader"
> boot = "cda"
> pae = 1
> acpi = 0
> apic = 0
> on_poweroff = "destroy"
> on_reboot = "restart"
> on_crash = "restart"
> device_model = "/usr/lib64/xen/bin/qemu-dm"
> sdl = 0
> vnc = 1
> vncunused = 1
> keymap = "en-us"
> disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' ,
> 'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ]
> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
> serial = "pty"
> 
> At the time the error occured, my memory looked like this:
> [root@xen1 centos-5.1_core_image]# free
>             total       used       free     shared    buffers     cached
> Mem:      15845376   15821080      24296          0     113596   15081468
> -/+ buffers/cache:     626016   15219360
> Swap:      4192956          4    4192952
> 
> The only way I've been able to duplicate this error is by making
> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB
> but du shows 2GB) for different virtual machines and booting them up. I
> can get up to about 8 new virtual machines roughly, before I start
> seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl'
> script, I've modified it slightly to meet our needs, and launch our
> systems with that tool. It's basic functionality is to read in a xen
> configuration file you want to use as a template, and output a modified
> one based on arguments you pass to the scripts, and then create a floppy
> disk image and copy the image from the template config file, and then
> boot the new system.
> 
> I've read in previous emails that this error occurs because of a bug in
> the hvmloader, and look for an update. That email response was for a 3.0
> version of xen, and I'm on 3.1.
> 
> What other information can I provide to help diagnose this problem?
> 
> -Kai Meyer
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel