[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Guest OS boots up slowly


  • To: Masaki Kanno <kanno.masaki@xxxxxxxxxxxxxx>
  • From: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>
  • Date: Wed, 2 Sep 2009 11:48:22 +0100
  • Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
  • Delivery-date: Wed, 02 Sep 2009 03:49:29 -0700
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=rbZzAa4L9LTc8Hdfx53dGsFE0LCSKJLqVWiFhQzLhpdqvIEpc0qB7vCmi1hoqLjKcK yLsWdEt2xkFOVpDx1rkXlb3CiF4Ssx+AYIybl38oB4rUhuFQNOzxC+VjrNYxxHD2cZx+ LvHviueLf5QbYIgzt8km/Chns5IQEbEW9XZIg=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Thanks Masaki!  I've taken a quick look at the trace and haven't seen
anything really obvious.  Unfortunately I have some urgent work to do
for XenServer, so this will have to take a back burner for a week or
so.

If anyone else is willing to investigate the problem, I certainly
won't object. :-)

 -George

2009/9/1 Masaki Kanno <kanno.masaki@xxxxxxxxxxxxxx>:
> Hi George,
>
> I tested xen-unstable (changeset: 20128) and linux-2.6.18-xen
> (changeset: 931) with the following cases.
> debit-accounted-only.diff doesn't fix the problem.
> I will send a trace data file to only you because the size of
> the file is big.
>
> Case1 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:256
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 57 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â 80 Â Â |
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 56 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 51 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | Â Â 93 Â Â |
> Â Â Â| Â 2 Â | Â 100 Â | Â Â 62 Â Â |
> Â Â Â| Â 2 Â | Â 150 Â | Â Â 54 Â Â |
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 50 Â Â |
>
> Case2 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:512
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 57 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â725 Â Â | Slow!!
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 57 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 51 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | 1,000 over | I gave up the measurement.
> Â Â Â| Â 2 Â | Â 100 Â | Â Â784 Â Â | Slow!!
> Â Â Â| Â 2 Â | Â 150 Â | Â Â567 Â Â | Slow!!
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 51 Â Â |
>
> Case3 : No patch, cpu_weight dom0:domU = 256:256
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 57 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â 80 Â Â |
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 57 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 50 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | Â Â 95 Â Â |
> Â Â Â| Â 2 Â | Â 100 Â | Â Â 61 Â Â |
> Â Â Â| Â 2 Â | Â 150 Â | Â Â 53 Â Â |
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 50 Â Â |
>
> Case4 : No patch, cpu_weight dom0:domU = 256:512
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 57 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â575 Â Â | Slow!!
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 57 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 50 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | Â Â594 Â Â | Slow!!
> Â Â Â| Â 2 Â | Â 100 Â | Â Â450 Â Â | Slow!!
> Â Â Â| Â 2 Â | Â 150 Â | Â Â290 Â Â | Slow!!
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 51 Â Â |
>
> Case5 : Without changeset 20122, cpu_weight dom0:domU = 256:256
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 57 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â 80 Â Â |
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 56 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 50 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | Â Â 95 Â Â |
> Â Â Â| Â 2 Â | Â 100 Â | Â Â 61 Â Â |
> Â Â Â| Â 2 Â | Â 150 Â | Â Â 53 Â Â |
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 50 Â Â |
>
> Case6 : Without changeset 20122, cpu_weight dom0:domU = 256:512
>   Â| vcpus | cpu_cap | boot-up  Â|
> Â Â Â| Â Â Â | Â Â Â Â | times[sec] |
> Â Â Â+-------+---------+------------+
> Â Â Â| Â 1 Â | Â Â 0 Â | Â Â 56 Â Â |
> Â Â Â| Â 1 Â | Â Â50 Â | Â Â 80 Â Â |
> Â Â Â| Â 1 Â | Â 100 Â | Â Â 56 Â Â |
> Â Â Â| Â 2 Â | Â Â 0 Â | Â Â 50 Â Â |
> Â Â Â| Â 2 Â | Â Â50 Â | Â Â 95 Â Â |
> Â Â Â| Â 2 Â | Â 100 Â | Â Â 61 Â Â |
> Â Â Â| Â 2 Â | Â 150 Â | Â Â 53 Â Â |
> Â Â Â| Â 2 Â | Â 200 Â | Â Â 50 Â Â |
>
>
> Best regards,
> ÂKan
>
> Tue, 01 Sep 2009 08:41:39 +0900, Masaki Kanno wrote:
>
>>Hi George,
>>
>>I will try them on today.
>>
>> Kan
>>
>>Mon, 31 Aug 2009 14:10:47 +0100, George Dunlap wrote:
>>
>>>Hmm, it appears my patch still debits credits for VMs that aren't
>>>earning credits anymore; such VMs can earn an unlimited amount of
>>>negative credit before becoming active again.
>>>
>>>Try the attached patches; if it doesn't fix the problem, please take a
>>>short trace during boot using the following command and send it to me:
>>>
>>># xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace
>>>
>>>Thanks,
>>> -George
>>>
>>>On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@xxxxxxxxxxxxx>
>>>wrote:
>>>> Hi Maskai,
>>>>
>>>> Thanks for the report. It's a good idea to Cc the patch author when making
>>>> bug reports. In this case it's George Dunlap who I've cc'ed in this reply.
>>>>
>>>> ï-- Keir
>>>>
>>>> On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@xxxxxxxxxxxxxx> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I tested the latest xen-unstable (changeset: 20128) and the latest
>>>>> linux-2.6.18-xen (changeset: 931). ïA guest OS is booted up slower
>>>>> than before. ïAnd the following messages are shown by the guest OS.
>>>>>
>>>>> Switching to new root and running init.
>>>>> unmounting old /dev
>>>>> unmounting old /proc
>>>>> unmounting old /sys
>>>>>
>>>>> ï ***************************************************************
>>>>> ï ***************************************************************
>>>>> ï ** WARNING: Currently emulating unsupported memory accesses ï**
>>>>> ï ** ï ï ï ï ïin /lib/tls glibc libraries. The emulation is ï ï**
>>>>> ï ** ï ï ï ï ïslow. To ensure full performance you should ï ï ï**
>>>>> ï ** ï ï ï ï ïinstall a 'xen-friendly' (nosegneg) version of ï **
>>>>> ï ** ï ï ï ï ïthe library, or disable tls support by executing **
>>>>> ï ** ï ï ï ï ïthe following as root: ï ï ï ï ï ï ï ï ï ï ï ï ï **
>>>>> ï ** ï ï ï ï ïmv /lib/tls /lib/tls.disabled ï ï ï ï ï ï ï ï ï ï**
>>>>> ï ** Offending process: modprobe (pid=761) ï ï ï ï ï ï ï ï ï ï **
>>>>> ï ***************************************************************
>>>>> ï ***************************************************************
>>>>>
>>>>> Pausing... 5<3>BUG: soft lockup detected on CPU#0!
>>>>> ï[<c0151205>] softlockup_tick+0xa5/0xd0
>>>>> ï[<c010978a>] timer_interrupt+0x2fa/0x6c0
>>>>> ï[<c011d8c1>] __activate_task+0x21/0x40
>>>>> ï[<c012fed0>] lock_timer_base+0x20/0x50
>>>>> ï[<c0151563>] handle_IRQ_event+0x33/0xa0
>>>>> ï[<c0151678>] __do_IRQ+0xa8/0x120
>>>>> ï[<c01076e1>] do_IRQ+0x31/0x80
>>>>> ï[<c02af6b0>] neigh_periodic_timer+0x0/0x140
>>>>> ï[<c024ced5>] evtchn_do_upcall+0xe5/0x1f0
>>>>> ï[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170
>>>>> ï[<c0105ba3>] hypervisor_callback+0x33/0x3b
>>>>> ï[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170
>>>>> ï[<c01ed68b>] delay_tsc+0xb/0x20
>>>>> ï[<c01ed6d6>] __delay+0x6/0x10
>>>>> ï[<c01160f0>] do_fixup_4gb_segment+0x140/0x170
>>>>> ï[<c0169597>] do_munmap+0x197/0x200
>>>>> ï[<c01ee0a4>] copy_to_user+0x34/0x70
>>>>> ï[<c0105b6b>] error_code+0x2b/0x30
>>>>> Continuing...
>>>>>
>>>>>
>>>>> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS
>>>>> is booted up as before.
>>>>>
>>>>> Best regards,
>>>>> ïKan
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>>> http://lists.xensource.com/xen-devel
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>> http://lists.xensource.com/xen-devel
>>>>
>>>
>>>-------------------------------text/plain-------------------------------
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>http://lists.xensource.com/xen-devel
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@xxxxxxxxxxxxxxxxxxx
>>http://lists.xensource.com/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.