WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] Infinite loop shutting down xendomains

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] Infinite loop shutting down xendomains
From: "John A. Sullivan III" <jsullivan@xxxxxxxxxxxxxxxxxxx>
Date: Sat, 28 Jan 2006 23:44:38 -0500
Delivery-date: Sun, 29 Jan 2006 04:54:08 +0000
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Pardon me for a long detailed post and for pestering the developer list
but this looks pretty serious.  The bottom line is that, when I attempt
to shut down or reboot my simple xen-2.0.7 server running FC3 with two
domUs, shutting down xendomains appears to send python into an infinite
loop.  This may be due to a serious misconfiguration on my part but it
is certainly a big problem.

If we shutdown the domUs manually first, the system reboots perfectly
fine.

I'll describe some history and include lots of log information.  The
configuration is slightly unusual.  Both domUs are VPN gateways running
the 2.4.30 kernel with openswan and openvpn.  Because of the openvpn,
but are privileged domains but the only device driver configured in the
kernel is the openvpn tun/tap adapter.

Since we have three interfaces, we slightly modified the network script.
It is included below.

We first noticed problems during shutdown in that the last domU to
shutdown crashed very nastily when closing openvpn.  We could see this
even when manually stopping the domUs.  We surmised that the problem was
we had tun0 and tun1 defined in both domains and killing them in one
must be killing them in another.  We renamed those tun adapters in the
second domain to tun10 and tun11 and that problem went away.

We still noticed very long reboot times.  The hang seemed to be at
killall but the "logjam" seemed to break after xendomains errored out.
We assumed this was because killall killed something that xendomains
needed to shutdown gracefully so we added explicit K scripts in rc0.d
and rc6.d for xend and xendomains.

At first this seemed to work but it must have been only coincidental
because the problem returned.  We then realized we had brain cramped and
shutdown xend before xendomains.  We thought for sure we had found the
problem and put them in the proper order but, alas, the problem did not
go away.

To gain a little more insight into exactly what was happening, we
manually ran each kill script in rc6.d.  When we ran K78xendomains stop,
we saw the problem, or at least many aspect of it.

>From a xen console, we saw that the first domU had cleanly shutdown but
the second one was still listed howbeit with zero memory.

The hard disks were thrashing madly and top showed python consuming 99.9
percent CPU utilization.  Most of the time, during reboot, we do not see
the madly thrashing disks.  They do so for a moment as one might expect
when shutting down the domains but then just blip occasionally
thereafter.

Here is what we see in the logs:

>From messages (ntpd was the last service stopped before xendomains):
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled

Here is the complete xend-debug.log:
network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7
bridge=xen-br0
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 44763, 'rx_shmem_frame':
44764, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 44767, 'rx_shmem_frame':
44768, 'handle': 1}
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 69339, 'rx_shmem_frame':
69340, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 69343, 'rx_shmem_frame':
69344, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
VirqClient.virqReceived> 4
vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4
bridge=xen-br0
Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/twisted/protocols/basic.py",
line 229, in dataReceived
    return self.rawDataReceived(data)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1015, in rawDataReceived
    self.allContentReceived()
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1006, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 557, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
165, in process
    self.render(resrc)
  File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
172, in render
    body = resrc.render(self)
  File "/usr/lib/python2.3/site-packages/twisted/web/resource.py", line
201, in render
    return m(request)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180, in
render_POST
    return self.perform(req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in
perform
    return self._perform(op, op_method, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in
_perform
    return self._perform_err(err, op, req)
  File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in
_perform
    val = op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47, in
op_shutdown
    val = fn(req.args, {'dom': self.dom.id})
  File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__
    return self.call_with_form_args(self.fn, fargs, xargs=xargs)
  File "/usr/lib/python/xen/xend/Args.py", line 121, in
call_with_form_args
    return fn(*params, **keys)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in
domain_shutdown
    val = xend.domain_shutdown(dominfo.id, reason, key)
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723, in
domain_shutdown
    ctrl = self.domainCF.getController(dom)
  File "/usr/lib/python/xen/xend/server/controller.py", line 323, in
getController
    ctrl = self.createController(dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 20, in
createController
    return DomainController(self, dom)
  File "/usr/lib/python/xen/xend/server/domain.py", line 38, in __init__
    self.registerChannel()
  File "/usr/lib/python/xen/xend/server/controller.py", line 220, in
registerChannel
    self.channel = self.channelFactory.domChannel(self.dom)
  File "/usr/lib/python/xen/xend/server/channel.py", line 83, in
domChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 253, in
__init__
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 137, in
createPort
    remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface
/usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  return '''<div class="instance"><span class="instanceName">%s instance
@ %s</span>
/usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w('<div class="dict"><span class="heading">Dictionary instance @ %
s</span>' % hex(id(d)))
/usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
  w('<div class="list"><span class="heading">List instance @ %s</span>'
% hex(id(l)))

 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 88257, 'rx_shmem_frame':
88258, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 88261, 'rx_shmem_frame':
88262, 'handle': 1}
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c0319394
 Init. ramdisk: c031a000->c031a000
 Phys-Mach map: c031a000->c0332000
 Page tables:   c0332000->c0334000
 Start info:    c0334000->c0335000
 Boot stack:    c0335000->c0336000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

Here is the information from xend.log:
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend.log
Row 6079 Col 1   11:04  Ctrl-K H for help
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown ['vpn1dc1', '2', 'halt']
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown ['niagras1', '1', 'halt']
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=niagras1 id=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=1 reason=poweroff
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 1 poweroff 0
[2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain:
name=niagras1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=0
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=0
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 1
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1root
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1swap
[2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15
127.0.0.1 3484
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['niagras1', '1', 'poweroff']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['niagras1', '1']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [15, '127.0.0.1', 3484]
[2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died
['niagras1', '1']
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=0
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=0
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 2
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1root
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff'] 
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend.log
Row 6077 Col 91  11:06  Ctrl-K H for help
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [16, '127.0.0.1', 3986]
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2  
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains:  failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend-debug.log
Row 171  Col 1   11:02  Ctrl-K H for help
recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c02c9df4
 Init. ramdisk: c02ca000->c02ca000
 Phys-Mach map: c02ca000->c02e2000
 Page tables:   c02e2000->c02e4000
 Start info:    c02e4000->c02e5000
 Boot stack:    c02e5000->c02e6000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}

recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!

File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid            boot.log.1  cron.2   mail       messages    prelink.log
samba     spooler    wtmp            xfrd.log
anaconda.log     boot.log.2  cups     maillog    messages.1  rpmpkgs
secure    spooler.1  xend-debug.log  Xorg.0.log
anaconda.syslog  cron        dmesg    maillog.1  messages.2  rpmpkgs.1
secure.1  spooler.2  xend.log        Xorg.0.log.old
boot.log         cron.1      lastlog  maillog.2  ppp         rpmpkgs.2
secure.2  vbox       xend.log.1      yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done









































    I    xend.log
Row 6157 Col 100 11:07  Ctrl-K H for help
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff'] 
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1 
etc., etc., etc.

Here are the configuration files for the two domUs:
#  -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using
'xm create'.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#============================================================================

#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is 'linux'.
#builder='linux'

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "niagras1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ]

vif = ['mac=02:00:00:01:02:00, bridge=xen-br0' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ 'phy:hda1,hda1,w' ]

disk = [ 'phy:VG1/niagras1root,sda1,w', 'phy:VG1/niagras1swap,sda2,w' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the domain's
# IP config doesn't, e.g. in ifcfg-eth0 or via DHCP.
# You can use 'extra' to set the runlevel and custom environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask=
# Set default gateway.
#gateway=
# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = '169.254.1.0'  
# Root directory on the nfs server.
#nfs_root   = '/full/path/to/root/directory'

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is 'onreboot', which restarts the domain when it shuts
down
# with exit code reboot.
# Other values are 'always', and 'never'.

#restart = 'onreboot'

#============================================================================


#  -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using
'xm create'.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#============================================================================

#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"

# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"

# The domain build function. Default is 'linux'.
#builder='linux'

# Initial memory allocation (in megabytes) for the new domain.
memory = 96

# A name for your domain. All domains must have different names.
name = "vpn1dc1"

# Which CPU to start domain on? 
#cpu = -1   # leave to Xen to pick

#----------------------------------------------------------------------------
# Define network interfaces.

# Number of network interfaces. Default is 1.
nics=2

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ]

vif = ['mac=02:00:00:01:01:00, bridge=xen-br0', 'mac=02:00:00:01:01:01,
bridge=xen-br1' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ 'phy:hda1,hda1,w' ]

disk = [ 'phy:VG1/vpn1dc1root,sda1,w', 'phy:VG1/vpn1dc1swap,sda2,w' ]

#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the domain's
# IP config doesn't, e.g. in ifcfg-eth0 or via DHCP.
# You can use 'extra' to set the runlevel and custom environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).

# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask=
# Set default gateway.
#gateway=
# Set the hostname.
#hostname= "vm%d" % vmid

# Set root device.
root = "/dev/sda1 ro"

# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = '169.254.1.0'  
# Root directory on the nfs server.
#nfs_root   = '/full/path/to/root/directory'

# Sets runlevel 4.
#extra = "4"

#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is 'onreboot', which restarts the domain when it shuts
down
# with exit code reboot.
# Other values are 'always', and 'never'.

#restart = 'onreboot'

#============================================================================


Here is the network script:

# Exit if anything goes wrong.
set -e 

# First arg is the operation.
OP=$1
shift

# Pull variables in args in to environment.
for arg ; do export "${arg}" ; done

bridge=${bridge:-xen-br0}
netdev=${netdev:-eth0}
antispoof=${antispoof:-yes}
SCRIPTDIR=/etc/xen/scripts

echo "network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof"

# Usage: transfer_addrs src dst
# Copy all IP addresses (including aliases) from device $src to device $dst.
transfer_addrs () {
    local src=$1
    local dst=$2
    # Don't bother if $dst already has IP addresses.
    if ip addr show dev ${dst} | egrep -q '^ *inet ' ; then
        return
    fi
    # Address lines start with 'inet' and have the device in them.
    # Replace 'inet' with 'ip addr add' and change the device name $src
    # to 'dev $src'. Remove netmask as we'll add routes later.
    ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${dst}/
" | sh -e
}

# Usage: del_addrs src
del_addrs () {
    local src=$1
    ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr del/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${src}/
" | sh -e
}

# Usage: transfer_routes src dst
# Get all IP routes to device $src, delete them, and
# add the same routes to device $dst.
# The original routes have to be deleted, otherwise adding them
# for $dst fails (duplicate routes).
transfer_routes () {
    local src=$1
    local dst=$2
    # List all routes and grep the ones with $src in.
    # Stick 'ip route del' on the front to delete.
    # Change $src to $dst and use 'ip route add' to add.
    ip route list | grep ${src} | sed -e "
h
s/^/ip route del /
P
g
s/${src}/${dst}/
s/^/ip route add /
P
d
" | sh -e
}

# Usage: create_bridge bridge
create_bridge () {
    local bridge=$1

    # Don't create the bridge if it already exists.
    if ! brctl show | grep -q ${bridge} ; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi
    ifconfig ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2
    # Don't add $dev to $bridge if it's already on a bridge.
    if ! brctl show | grep -q ${dev} ; then
        brctl addif ${bridge} ${dev}
    fi
}

# Usage: antispoofing dev bridge
# Set the default forwarding policy for $dev to drop.
# Allow forwarding to the bridge.
antispoofing () {
    local dev=$1
    local bridge=$2

    iptables -P FORWARD DROP
    iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT
}

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
    local dev=$1
    local bridge=$2
    
    echo '============================================================'
    ifconfig ${dev}
    ifconfig ${bridge}
    echo ' '
    ip route list
    echo ' '
    route -n
    echo '============================================================'
}

op_start () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi
# Create the bridge and give it the interface IP addresses.
# Move the interface routes onto the bridge.
    while read bridge netdev
        do
            create_bridge ${bridge}
            if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then
                transfer_addrs ${netdev} ${bridge}
                transfer_routes ${netdev} ${bridge}
            fi
# Don't add $dev to $bridge if it's already on a bridge.
            if ! brctl show | grep -q ${netdev} ; then
                brctl addif ${bridge} ${netdev}
            fi
            
            if [ ${antispoof} == 'yes' ] ; then
                antispoofing ${netdev} ${bridge}
            fi
    done < ${SCRIPTDIR}/bridges.txt
}
                                                                                
                                                             
op_stop () {
    if [ "${bridge}" == "null" ] ; then
        return
    fi
# Remove the interface from the bridge.
# Move the routes back to the interface.
    while read bridge netdev
    do
        brctl delif ${bridge} ${netdev}              
        if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then
            transfer_routes ${bridge} ${netdev}
        fi
    done < ${SCRIPTDIR}/bridges.txt
# It's not our place to be enabling forwarding...
}

case ${OP} in
  start)
        op_start
        ;;
    
    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
       ;;

    *)
       echo 'Unknown command: ' ${OP}
       echo 'Valid commands are: start, stop, status'
       exit 1
esac

Here is the bridges.txt file:
xen-br0 eth0
xen-br1 eth1
xen-br2 eth2


I'm stumped but it sure looks serious.  Have I produced some dreadful
misconfiguration? This host is being shipped to a remote data center on
Monday and goes into production in a week.  Any help would be greatly
appreciated.  Thanks - John

-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@xxxxxxxxxxxxxxxxxxx

Financially sustainable open source development
http://www.opensourcedevel.com


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>