WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] SOLUTION FOUND FOR xend does not start "No space left...

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] SOLUTION FOUND FOR xend does not start "No space left..."
From: Oliver Heil <o.heil@xxxxxxxxxxxxxxxxxx>
Date: Thu, 08 Mar 2007 14:56:26 +0100
Delivery-date: Fri, 09 Mar 2007 01:25:20 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <45EE7F2A.1080408@xxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <45EE7F2A.1080408@xxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 1.5.0.10 (Windows/20070221)
Hi,

I finally found a solution for my problem.
I still do not know, what the exact problem was. The symptoms
are, that the xen dom0, Domain-0 does not start and that e.g.
xm asks, if xend is running, but xend is in the process list.

>> # xm list
>> Error: Unable to connect to xend: No such file or directory. Is xend
>> running?

The xend.log file reports error messages like

> RuntimeError: (28, 'No space left on device, while writing 
/vm/00000000-0000-0000-0000-
> 000000000000/xend/restart_count : 0')

but every device and memory has plenty space available.

The system is a openSUSE 10.2 on 2 processor XEON 64bit machine
with 4Gbyte RAM. XEN is build from source xen-3.0.4_1

To workaround and to get xen back again I just
removed the directories /var/lib/xen, /var/lib/xend
and /var/lib/xenstored.
In /var/lib/xen/save was 1 Gbyte file named like the only
guest virtual machine I am running on the host:
/var/lib/xen/save/vm01
This is propably a memory image of the guest vm and I believe
that the existance of this file was the cause of the trouble.

There are a few other threads unanswered in the net, which maybe
solved the same way. Here are some of the subjects I found with google:
[Fedora-xen] Xen0 not booting into Login console
[Fedora-xen] Exception when starting xend: error while reading 
/vm/00000000-0000-0000-0000-000000000000/ssidref

Sincerelly,

Oli

Oliver Heil wrote:
Hi XEN Users and Developers,

xend does not start into a clean state. The process is running...

#ps -efl | grep xen
1 S root 20 19 0 70 -5 - 0 xenwat Mar06 ? 00:00:00 [xenwatch] 1 S root 21 19 0 70 -5 - 0 xb_rea Mar06 ? 00:00:00 [xenbus] 5 S root 12663 1 0 76 0 - 1274 - 09:56 ? 00:00:00 xenstored --pid-file /var/run/xenstore.pid 5 S root 12667 1 0 84 0 - 3552 stext 09:56 ? 00:00:00 xenconsoled 1 S root 12671 1 0 79 0 - 13265 wait 09:56 ? 00:00:00 python /usr/sbin/xend start 5 S root 12673 12671 0 76 0 - 15412 184466 09:56 ? 00:00:00 python /usr/sbin/xend start

...but no virtual machine can be started and xm answers e.g with this:
# xm list
Error: Unable to connect to xend: No such file or directory. Is xend running?

In the xend.log there is the following error:
[2007-03-07 09:56:13 xend 12659] WARNING (XendAuthSessions:27) python-pam is required for XenAPI support.
[2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:322) Xend Daemon started
[2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:326) Xend changeset: unavailable. [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:333) Xend version: Unknown. [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG (XendDomainInfo:121) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 11 74593321820L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'r unning': 1, 'maxmem_kb': 17179869180, 'shutdown': 0, 'mem_kb': 3985144, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'blocked': 0, 'name': 'Domain-0'})
[2007-03-07 09:56:13 xend.XendDomainInfo 12673] INFO (XendDomainInfo:135) Recreating domain 0, UUID 00000000-0000-0000-0000-00000000
0000. at /local/domain/0
[2007-03-07 09:56:13 xend.XendDomainInfo 12673] WARNING (XendDomainInfo:149) /local/domain/0/vm is missing. recreate is confused, tr
ying our best to recover
[2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG (XendDomainInfo:1714) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_m emory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'name': 'Domain-0', 'on_reboot': 'restart', 'on_poweroff': 'destroy', ' on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'memory': '3892', 'max
mem': '16777216'}
[2007-03-07 09:56:13 xend 12673] ERROR (SrvDaemon:344) Exception starting xend ((28, 'No space left on device, while writing /vm/000
00000-0000-0000-0000-000000000000/xend/restart_count : 0'))
Traceback (most recent call last):
File "//usr/lib64/python/xen/xend/server/SrvDaemon.py", line 336, in run
    servers = SrvServer.create()
File "//usr/lib64/python/xen/xend/server/SrvServer.py", line 226, in create
    root.putChild('xend', SrvRoot())
File "//usr/lib64/python/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "//usr/lib64/python/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "//usr/lib64/python/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
File "//usr/lib64/python/xen/xend/server/SrvDomainDir.py", line 41, in __init__
    self.xd = XendDomain.instance()
File "//usr/lib64/python/xen/xend/XendDomain.py", line 1399, in instance
    inst.init()
  File "//usr/lib64/python/xen/xend/XendDomain.py", line 116, in init
    dom0 = XendDomainInfo.recreate(dom0info, True)
File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 174, in recreate
    vm._storeVmDetails()
File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1716, in _storeVmDetails
    self._writeVm(to_store)
File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 672, in _writeVm
    return xstransact.Write(self.vmpath, *args)
File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, in Write
    complete(path, lambda t: t.write(*args))
File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 354, in complete
    result = f(t)
File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, in <lambda>
    complete(path, lambda t: t.write(*args))
File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 87, in write
    self._write(key, d[key])
File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 75, in _write
    ('%s, while writing %s : %s' %
RuntimeError: (28, 'No space left on device, while writing /vm/00000000-0000-0000-0000-000000000000/xend/restart_count : 0') [2007-03-07 09:56:13 xend 12675] WARNING (XendAuthSessions:27) python-pam is required for XenAPI support.

Does somebody know a cure for this behaviour?

With regards,

Oli


--
/////////////////////////////////
Oliver Heil

DKFZ - Deutsches Krebsforschungszentrum
Im Neuenheimer Feld 515
69120 Heidelberg

Tel    (+49 6221) / 42 4708

Email  o.heil@xxxxxxx
WWW    www.dkfz.de
/////////////////////////////////



-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.0.6 (MingW32)

mQGiBDyzHxERBACFUsv9ZzMAstEUKpV7ZI+//CqMwNWteCdcet8ONYQkE+meq7qp
L2yQgqdGUdlRz/O0Ix9ERuMw+6qnRYxXCGvRjWX6t+wDSALfFjjEIUVecedsMKGv
7bINfUQEJQ+7hZAonOJ/iEL3adh8fvl1Bl0yNYxATbqp+q1YGawsHLu4TwCg10VR
AkS9zZnv1gsGBg2rDm8f2CcD/0siN0AUvjvj38Hzxr7CCG22c5QYHZLhFvM7+jSA
OF2qTNGtFJ0fVkIlqzkd1D21efmM2xKo9Jzlhb5D1Dr1GQ7epxvpcGV/n8vHtwOv
+N8UoOaNfrJIoM9lV4IXKTAnwlTcyy61Bthx4L+dKNTH0ejbPk3/SIUlYAju6V9y
uqOCA/0euPYCj0GQ4YfdULUu/ZTLOAX3X+fUXqSQThBUt8BpTv/wqNqSifURttH9
ERb+Q22E9ck5mYs2zZmd2xvmxU6swFqJJVZMBzs1/iQcc7mC1v84Riddcuj+C4YA
whph8LxHAdFPN+ZHoaQTIu8vC0KdtHJX88KNxvl7HVYL8x9yW7QcT2xpdmVyIEhl
aWwgPG8uaGVpbEBka2Z6LmRlPohXBBMRAgAXBQI8sx8RBQsHCgMEAxUDAgMWAgEC
F4AACgkQ87yOFfCjArRQUgCgqL5irUC6wdZntArL+8n19RWQWrMAn32Dee0njHm1
2qTW27TcBSPAWGwSuQINBDyzH18QCADC8Em8AKrM1QrBR1qpmb9kN+KmF5FZDwXP
4ioO4k0ee2cfcriUiZJXeNo/TpUvGpGRhINMyCKDpiXeU970Gxhy6ksU+q+hEqFZ
nAqYuq00xQQ+SxMb+TgGytgmq3Jz8q8jJFokkd3Mc0mcicPKlKNv4Sq+sGAzHXTk
in/XJRTB7RA7zl7K+iz+u4lnXyyhM+cT82EtduQ2XvPGYjQwymLMcIRhoWlbvB1w
1uGGCaOQSdlH36LEG+P41TwMjB4Eq2dhk9PL+6pUpbRXrtWtnUzfQuEFwHlMwITv
RCit6Pp+5f2USyWeUP0oc/39I0iTOqINZ7S031FD7xa61vtMK01bAAMGCAC2sX9z
vormmdWvHWyH5+d9PiYUS86JNy8ncVqgoLtin6fqpy0UlGMHH2ewfpExfVip8CWj
Cp7vzpmzhsu+wIDyVdAVx3OldKFUH7Al3PDdMe4hNkfB0om+DDAYPO33C7PhpcUL
+Muz14fc6ti5PKs8zNcqENGyL0sv/Uun28Ebb9cOHWfEcgJ8ZbfmLiW8b2vXYFPj
F6T0+tOTCSNDuWGBAWfvumOhZB6sFokPUxs+5CqibGTgGtCUYW1VXVTu+d3bugMN
NcuQBucGKdq7v6rREgfNfqvfodyk77Zy9w9wbGvPROUe77PaYubdRviiCiMYfQQc
501JeSIwwARsSt5viEYEGBECAAYFAjyzH18ACgkQ87yOFfCjArSc9ACdEY4YqlqM
lIuB4uWb6XZ3DTTUs3QAoL+a4UJp9OLLVtjTsijb/hpu3uhO
=I3OA
-----END PGP PUBLIC KEY BLOCK-----

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>