WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/con

To: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Subject: Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
From: Sander Eikelenboom <linux@xxxxxxxxxxxxxx>
Date: Fri, 19 Nov 2010 19:39:44 +0100
Cc: Ian Campbell <Ian.Campbell@xxxxxxxxxxxxx>, Keir Fraser <keir@xxxxxxx>, "Xen-devel@xxxxxxxxxxxxxxxxxxx" <Xen-devel@xxxxxxxxxxxxxxxxxxx>, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
Delivery-date: Fri, 19 Nov 2010 10:41:54 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4CE6AEDF.9040700@xxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization: Eikelenboom IT services
References: <1341501057.20101117132828@xxxxxxxxxxxxxx> <C909A6EB.A4FB%keir@xxxxxxx> <1945713192.20101117212227@xxxxxxxxxxxxxx> <4CE44FDB.7000508@xxxxxxxx> <1266442973.20101118205715@xxxxxxxxxxxxxx> <4CE6AEDF.9040700@xxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Sorry, something went wrong with copy and pasting, here is the complete output:

(gdb) tar rem :9999
Remote debugging using :9999
[New Thread 0]
[Switching to Thread 0]
smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d 
<stop_self>, info=0x0, wait=true) at kernel/smp.c:104
104             while (data->flags & CSD_FLAG_LOCK)
(gdb) thread apply all bt
[New Thread 1]
[New Thread 2]
[New Thread 3]

Thread 4 (Thread 3):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffffffff02dd in ?? ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=286, action=0xffff88001e8257e0) 
at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=286, desc=0xffff88001ea05500) 
at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9e3e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 3 (Thread 2):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81006fbd in xen_force_evtchn_callback () at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:362
#2  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#3  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#4  0xffffffff8109907b in handle_IRQ_event (irq=291, action=0xffff88001e825600) 
at kernel/irq/handle.c:68
#5  0xffffffff8109ae9c in handle_percpu_irq (irq=291, desc=0xffff88001ea05000) 
at kernel/irq/chip.c:684
#6  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#7  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#8  0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#9  0xffff88001e9e1e28 in ?? ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently asm

Thread 2 (Thread 1):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81ce8e70 in cpu_possible_bits ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=296, action=0xffff88001e825420) 
at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=296, desc=0xffff88001e824b00) 
at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9d7e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0):
#0  smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d 
<stop_self>, info=0x0, wait=true) at kernel/smp.c:104
#1  0xffffffff81077a5a in smp_call_function (func=0x40 <irq_stack_union+64>, 
info=<value optimized out>, wait=<value optimized out>) at kernel/smp.c:506
#2  0xffffffff81007eda in xen_stop_other_cpus (wait=<value optimized out>) at 
arch/x86/xen/smp.c:431
#3  0xffffffff81003172 in xen_reboot () at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/smp.h:81
#4  0xffffffff810031b5 in xen_machine_halt () at arch/x86/xen/enlighten.c:1039
#5  0xffffffff81023795 in machine_halt () at arch/x86/kernel/reboot.c:726
#6  0xffffffff8105e9b3 in kernel_halt () at kernel/sys.c:336
#7  0xffffffff8105eb03 in sys_reboot (magic1=-18751827, magic2=672274793, 
cmd=3454992675, arg=0x0) at kernel/sys.c:407
#8  0xffffffff8100acc2 in system_call_fastpath () at 
arch/x86/kernel/entry_64.S:479
#9  0x0000000000000202 in irq_stack_union ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently c


Friday, November 19, 2010, 6:07:43 PM, you wrote:

> On 11/18/2010 11:57 AM, Sander Eikelenboom wrote:
>> Hi Jeremy/Keir,
>>
>> I have tried to add the following lines:
>>        if (!strcmp(type, "console"))
>>                return 0;
>>
>> But it seems to be a red herring .. although the "the xenbus_dev_shutdown: 
>> device/console/0: Initialising != Connected, skipping" warning is gone.
>> The PV domU guest still doesn't get completely halted:
>> - It stays in a running state
>> - It uses 100% cpu according to xentop
>> - I can still connect to it's console
>>
>> So probably something is stuck in a waiting loop, which doesn't sleep in 
>> between and consumes 100% cpu.

> Hm, I generally see clean shutdowns of my multi-VCPU domains.

> Can you do

> # gdbsx -a <domid> <arch-size> 9999        (arch-size is 32 or 64)

> then

> $ gdb vmlinux
> (gdb) tar rem :9999
> (gdb) thread apply all bt

> to see what's going on?  (You may need to enable CONFIG_DEBUG_INFO to
> get useful info if you haven't already.)

>     J


>> Last lines of domU's console(with some additional printk's in 
>> xenbus_dev_shutdown):
>>
>> Cannot access the Hardware Clock via any known method.
>> Use the --debug option to see the details of our search for an access method.
>> Stopping enhanced syslogd: rsyslogd.
>> Asking all remaining processes to terminate...done.
>> All processes ended within 2 seconds....done.
>> Deconfiguring network interfaces...done.
>> Cleaning up ifupdown....
>> Deactivating swap...done.
>> Unmounting local filesystems...done.
>> Will now halt.
>> [   46.643035] md: stopping all md devices.
>> [   47.643320] xenbus_dev_shutdown:  trying shutdown of device/vif/0: 
>> Connected
>> [   47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: 
>> Closed
>> [   47.716461] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: 
>> Connected
>> [   47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: 
>> Closed
>> [   47.772306] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: 
>> Connected
>> [   47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: 
>> Closed
>> [   47.829415] System halted.
>>
>>
>>
>> it's a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very 
>> special.
>>
>> the only difference that triggers it is:
>> vcpus=1 works fine ..
>> vcpus>1 symptoms above ..
>>
>> I have also added xend.log, with first a boot and shutdown with vpcus=1 and 
>> then a boot and shutdown with vpcus=4
>> I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but 
>> doesn't seem to hit any timeout, and keeps running.
>> On 20:40 it ends up stuck, on 20:51 i'm shooting the domain off with xm 
>> destroy.
>>
>>
>> Any pointers for adding extra debug info ?
>>
>> --
>> Sander
>>
>>
>>
>>
>> Wednesday, November 17, 2010, 10:57:47 PM, you wrote:
>>
>>> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
>>>> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 
>>>> 2.6.37-rc2 not:
>>>>
>>>> -       if (!strcmp(type, "console"))
>>>> -               return 0;
>>>> It seems to be changed,
>>>>
>>>>
>>>> There seems to have been a patch 
>>>> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html 
>>>> but it seems it's not applied to xenbus in linux upstream.
>>> And does this patch fix things for you?
>>> But I have to say that's pretty gross.  Is it really the right thing to do?
>>>     J
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel




-- 
Best regards,
 Sander                            mailto:linux@xxxxxxxxxxxxxx


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>