WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] RE: xenoprof: operation 9 failed for dom0 (status: -1)

To: Dante Cinco <dantecinco@xxxxxxxxx>
Subject: [Xen-users] RE: xenoprof: operation 9 failed for dom0 (status: -1)
From: "Santos, Jose Renato G" <joserenato.santos@xxxxxx>
Date: Tue, 17 Nov 2009 18:16:56 +0000
Accept-language: en-US
Acceptlanguage: en-US
Cc: Xen-users <xen-users@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Tue, 17 Nov 2009 10:18:26 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <8d2cca230911161532u2100d5afhba9a44313bb4c736@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <8d2cca230911051715n5b9c1050g12bdf49c80b59053@xxxxxxxxxxxxxx> <EF547E542C520A4D858CFEF5B404D0536080DFF2CB@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <8d2cca230911101012p8d038d5u8ab095855393ba74@xxxxxxxxxxxxxx> <EF547E542C520A4D858CFEF5B404D0536080DFF70D@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <8d2cca230911101120p7d0af9e8s7302c12dd5d2fc43@xxxxxxxxxxxxxx> <EF547E542C520A4D858CFEF5B404D05360810D45BF@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <8d2cca230911111755v3c652ce6r12500a540a402803@xxxxxxxxxxxxxx> <EF547E542C520A4D858CFEF5B404D05360810D4C8C@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <8d2cca230911121744h611ee5ceu10038151f705abad@xxxxxxxxxxxxxx> <EF547E542C520A4D858CFEF5B404D05360810D53EA@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <8d2cca230911161532u2100d5afhba9a44313bb4c736@xxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AcpnFSH7QJHc1T6xRzWs1QQUjoPNGgAnEXfQ
Thread-topic: xenoprof: operation 9 failed for dom0 (status: -1)
The pvops kernel and Dullor's patch are very new and did not get enough 
testing. I am not surprised that using that did not work.
Your best option is using the 2.6.18-xen kernel. This has been used for a 
while, although I suspect nobody has actually used active mode profiling or we 
would have seen the bug you found earlier. Anyway that is the kernel most 
likely to work for you.

Renato

> -----Original Message-----
> From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]
> Sent: Monday, November 16, 2009 3:33 PM
> To: Santos, Jose Renato G
> Cc: Xen-users
> Subject: Re: xenoprof: operation 9 failed for dom0 (status: -1)
>
> Renato,
>
> Is there a list of Linux guests that have support for
> Xenoprofile or do they all need to be patched? What are the
> components that are needed in the Linux guest to support Xenoprofile?
>
> I saw Dulloor's xenoprof.patch for the pvops kernel
> (http://xen.markmail.org/search/?q=performance+tools+on+xen#qu
> ery:performance%20tools%20on%20xen%20list%3Acom.xensource.list
> s.xen-devel+page:1+mid:mouwof65pvnyan4p+state:results). I
> tried applying it to my 2.6.30.1 domU and most of it applied
> cleanly except for mmu.c and xenoprofile.c. I tried
> reconciling the changes to these 2 files and was able to
> compile the kernel but still could not get OProfile to run
> with --active-domains.
>
> BTW, I tried the kernel I'm using for dom0 and it won't event
> boot as a domU.
>
> Dante
>
>
> On Thu, Nov 12, 2009 at 5:54 PM, Santos, Jose Renato G
> <joserenato.santos@xxxxxx> wrote:
>
>
>       It looks like your domU does not have the right version
> of oprofile.
>       You should have seen a few printks for "dom 1". In
> particular dom1 should have called XENOPROF_enable_virq when
> you run "opcontrol --start" in the guest.
>       That is why is failing.
>       Either your kernel or your user level tools do not have
> Xen support. Did you apply the patch
> "oprofile-0.9.3-xen-r2.patch" to Oprofile in the guest? f so
> your kernel does not have the Xenoprof support.
>
>       Renato
>
>
>
>
> ________________________________
>
>
>               From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]
>
>               Sent: Thursday, November 12, 2009 5:44 PM
>
>               To: Santos, Jose Renato G
>               Cc: Xen-users
>               Subject: Re: xenoprof: operation 9 failed for
> dom0 (status: -1)
>
>
>               I added this line (after the close bracket of
> "switch(op)") to xenoprof.c so that I can see the state
> transitions and the values of the static variables:
>
>                   printk("xenoprof: operation %d for dom %d
> (xenoprof_state : %d, ret : %d, activated : %d, adomains : %d)\n",
>                          op, current->domain->domain_id,
> xenoprof_state, ret, activated, adomains);
>
>               Here's the Xen console output after running
> "opcontrol --start-daemon --active-domains=0,1"
>
>               (XEN) xenoprof: operation 1 for dom 0
> (xenoprof_state : 1, ret : 0, activated : 0, adomains : 0)
>               (XEN) xenoprof: operation 3 for dom 0
> (xenoprof_state : 1, ret : 0, activated : 0, adomains : 1)
>               (XEN) xenoprof: operation 3 for dom 0
> (xenoprof_state : 1, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 14 for dom 0
> (xenoprof_state : 1, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 5 for dom 0
> (xenoprof_state : 2, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 6 for dom 0
> (xenoprof_state : 2, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 6 for dom 0
> (xenoprof_state : 2, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 7 for dom 0
> (xenoprof_state : 3, ret : 0, activated : 0, adomains : 2)
>               (XEN) xenoprof: operation 8 for dom 0
> (xenoprof_state : 3, ret : 0, activated : 1, adomains : 2)
>
>               Notice that "activated" is incremented only to
> 1 and it is the mismatch when compared against "adomains"
> that causes XENOPROF_start to fail after running "opcontrol
> --start" in dom0:
>
>               (XEN) xenoprof: operation 9 for dom 0
> (xenoprof_state : 3, ret : -1, activated : 1, adomains : 2)
>               (XEN) xenoprof: operation 9 failed for dom 0
> (status : -1)
>
>               My question is what is supposed to trigger
> "activated" to go to 2? It seems that XENOPROF_enable_virq
> (calls set_active which increments "activated") needs to be
> called twice (once for each active domain). I found only one
> instance in the dom0 kernel that calls
> "HYPERVISOR_xenoprof_op(XENOPROF_enable_virq, NULL)" and it
> is in drivers/xen/xenoprof/xenoprofile.c inside the function
> xenoprof_setup(void).
>
>               I did run "opcontrol --start" in domU before
> running it in dom0. Here's the sequence:
>
>               domU: opcontrol --shutdown; opcontrol --reset
>               dom0: opcontrol --shutdown; opcontrol --reset;
> opcontrol --start-daemon --active-domains=0,1
>               domU: opcontrol --start
>               dom0: opcontrol --start
>
>               Dante
>
>
>
>               On Wed, Nov 11, 2009 at 6:06 PM, Santos, Jose
> Renato G <joserenato.santos@xxxxxx> wrote:
>
>
>                       This is not the right way to use active domains.
>                       You should specify "--active-domains=1"
> and not "--passive-domains=1".
>                       By specifying passive domains the
> samples for domain 1 are being delivered to dom0.
>                       Probably the samples you are seeing in
> the guest are from a previous run.
>                       This definetely not right.
>                       Also you do not need to specify dom0 as
> an active domain. The option --active-domain=0 should have no effect
>
>                       If "--active-domains=1"  is giving you
> the same error as before, this is probably because you are
> not following the right sequence of commands between dom0 and
> the guest.
>                       It is working fine for me after the patch
>
>                       Renato
>
>
> ________________________________
>
>
>                               From: Dante Cinco
> [mailto:dantecinco@xxxxxxxxx]
>
>                               Sent: Wednesday, November 11,
> 2009 5:55 PM
>
>                               To: Santos, Jose Renato G
>                               Cc: Xen-users
>                               Subject: Re: xenoprof:
> operation 9 failed for dom0 (status: -1)
>
>
>                               Yes, your patch made a big
> difference. I can now get profile data in domU. Thanks.
>
>                               There's a slight twist though
> in order to make this work. I have to use this command line in dom0:
>
>                               opcontrol --start-daemon
> --passive-domains=1 --passive-images=/boot/vmlinux-2.6.30.1
> --active-domains=0 --vmlinux=/boot/vmlinux-2.6.30.3
> --xen=/boot/xen-syms-3.4.1
>
>                               If I try to use
> "--active-domains=1" and not specify --passive-domains and
> --passive-images, I still get the same error message when I
> run "opcontrol --start" in dom0. I thought passive-domains
> are domains that are not running OProfile but I am running
> OProfile in domU (dom ID#1).
>
>                               When I run "opreport -l" in
> dom0, I see samples from domain1-kernel and when I run
> "opreport -l" in domU, I see samples from vmlinux-2.6.30.1.
> My question is how does Xenoprof know which samples to route
> to domain1-kernel in dom0 and which samples to routed in
> vmlinux-2.6.30.1 in domU?
>
>                               Dante
>
>
>                               On Wed, Nov 11, 2009 at 12:16
> AM, Santos, Jose Renato G <joserenato.santos@xxxxxx> wrote:
>
>
>                                       I think I found the bug.
>                                       This patch should fix it.
>                                       Please let me know if it works
>
>                                       Thanks
>
>                                       Renato
>
>                                       diff -r 7422afed66ee
> xen/common/xenoprof.c
>                                       ---
> a/xen/common/xenoprof.c     Mon Sep 07 08:53:07 2009 +0100
>                                       +++
> b/xen/common/xenoprof.c     Tue Nov 10 23:45:48 2009 -0800
>                                       @@ -681,8 +681,9 @@ int
> do_xenoprof_op(int op, XEN_GUEST_HAN
>                                           {
>                                           case XENOPROF_init:
>                                               ret =
> xenoprof_op_init(arg);
>                                       -        if ( !ret )
>                                       -
> xenoprof_state = XENOPROF_INITIALIZED;
>                                       +        if ( (ret == 0) &&
>                                       +
> (current->domain == xenoprof_primary_profiler) )
>                                       +
> xenoprof_state = XENOPROF_INITIALIZED;
>                                               break;
>
>                                           case XENOPROF_get_buffer:
>
>
>
>                                       > -----Original Message-----
>                                       > From: Dante Cinco
> [mailto:dantecinco@xxxxxxxxx]
>
>                                       > Sent: Tuesday,
> November 10, 2009 11:21 AM
>                                       > To: Santos, Jose Renato G
>                                       > Cc: Xen-users
>                                       > Subject: Re:
> xenoprof: operation 9 failed for dom0 (status: -1)
>                                       >
>                                       > As you can see from
> the output of "opreport -l" below, most
>                                       > of the cycles are
> coming from domain1-modules so I do need to
>                                       > focus on
> --active-domains=1 since --passive-domains=1 does
>                                       > not provide the
> profiling details for the domU modules.
>                                       >
>                                       > The steps described
> in the xenoprof tutorial for active
>                                       > domains is pretty
> straightforward but I cannot get past the
>                                       > "write error" when I
> run "opcontrol --start" in dom0.
>                                       >
>                                       > After running
> "opcontrol --start" in domU, I see the response
>                                       > below and I use lsmod
> to verify that oprofile module is
>                                       > loaded. Given the
> results from --passive-domains=1, the dom0
>                                       > and Xenoprof
> interface is working. It's the domU and dom0
>                                       > interface that has
> some problem. Is there some other way I
>                                       > can tell from dom0
> that domU is "ready" before running
>                                       > "opcontrol --start" in dom0?
>                                       >
>                                       > FYI: I'm using the
> same oprofile-0.9.3 with xen-r2.patch in
>                                       > dom0 and domU. If I
> boot the Debian 2.6.30.1 kernel (the same
>                                       > one I'm using in
> domU) in bare-metal (no Xen VMM), I'm able
>                                       > to successfully run
> oprofile-0.9.3.
>                                       >
>                                       > opcontrol --start
> (from domU before "opcontrol --start" in dom0):
>                                       > Using 2.6+ OProfile
> kernel interface.
>                                       > Reading module info.
>                                       > Using log file
> /var/lib/oprofile/samples/oprofiled.log
>                                       > Daemon started.
>                                       > Profiler running.
>                                       >
>                                       > opreport -l (from
> dom0 using --passive-domains=1
>                                       >
> --passive-images=/boot/vmlinux-2.6.30.1):
>                                       > CPU: Core 2, speed
> 2533.51 MHz (estimated) Counted
>                                       > CPU_CLK_UNHALTED
> events (Clock cycles when not halted) with a
>                                       > unit mask of 0x00
> (Unhalted core cycles) count 100000
>                                       > samples  %
> image name               app name
>                                       >       symbol name
>                                       > 617835   38.6156
> domain1-modules          domain1-modules
>                                       >       (no symbols)
>                                       > 448911   28.0576
> domain1-xen-unknown
>                                       > domain1-xen-unknown
>    (no symbols)
>                                       > 72460     4.5289
> domain1-kernel           domain1-kernel
>                                       >       __down
>                                       > 43294     2.7059
> domain1-kernel           domain1-kernel
>                                       >       __down_killable
>                                       > 34145     2.1341
> domain1-kernel           domain1-kernel
>                                       >       validate_slab_slab
>                                       >
>                                       > Dante
>                                       >
>                                       >
>                                       > On Tue, Nov 10, 2009
> at 10:22 AM, Santos, Jose Renato G
>                                       >
> <joserenato.santos@xxxxxx> wrote:
>                                       >
>                                       >
>                                       >       With passive
> domains you cannot have detailed profiling
>                                       > information on
> modules, only in kernel builtin functions and
>                                       > on Xen. All the
> samples associated with modules will be
>                                       > grouped under the
> same symbol "domain1-modules".
>                                       >       If you are
> interested in one particular module you
>                                       > should try to
> recompile the kernel with the associated code
>                                       > builtin (or you can
> use active domains, but follow the steps
>                                       > on the xenoprof
> tutorial to coordinate opcontrol in dom0 and
>                                       > in the guest)
>                                       >
>                                       >       Renato
>                                       >
>                                       >
>                                       >
> ________________________________
>                                       >
>                                       >
>                                       >               From:
> Dante Cinco [mailto:dantecinco@xxxxxxxxx]
>                                       >
>                                       >               Sent:
> Tuesday, November 10, 2009 10:13 AM
>                                       >
>                                       >               To:
> Santos, Jose Renato G
>                                       >               Cc: Xen-users
>                                       >
> Subject: Re: xenoprof: operation 9 failed for
>                                       > dom0 (status: -1)
>                                       >
>                                       >
>                                       >               Renato,
>                                       >
>                                       >               I think
> I'm making progress. I followed your
>                                       > suggestion of using
> --passive-domains and --passive-images.
>                                       > When I run opreport,
> it is unable to find /domain1-modules
>                                       > and
> /domain1-xen-unknown. Where or how do I specify the
>                                       > kernel modules I have
> running in domU/domain1? I tried
>                                       > copying the *.ko
> files in /boot in dom0 and used
>                                       > --image-path=/boot in
> opreport and it is still not finding them.
>                                       >
>                                       >               Thanks.
>                                       >
>                                       >               Dante
>                                       >
>                                       >
>                                       >               On Mon,
> Nov 9, 2009 at 6:44 PM, Santos, Jose
>                                       > Renato G
> <joserenato.santos@xxxxxx> wrote:
>                                       >
>                                       >
>                                       >
> Try replacing "--active-domains=1" with
>                                       > "--passive-domains=1
> passive-images=<domU-kernel-image>" (use
>                                       > the uncompressed
> version of your kernel image for the guest,
>                                       > vmlinux-*)
>                                       >
>                                       >
> To use "active-domains" you need to run
>                                       > opcontrol in the
> guest in addition to running it in dom0 and
>                                       > you need to
> coordinate the execution of both instances.. This
>                                       > require the guest
> opcontrol to be ready before running
>                                       > "opcontrol --start"
> in dom0. That is why it is failing. I
>                                       > suspect you have not
> executed opcontrol in the guest
>                                       >
> Using active-domains is very tricky. I
>                                       > suggest that you use
> --passive-domains, unless you really
>                                       > need active domains
> (it is only useful in case you need
>                                       > detailed profiles for
> user processes running in the guest)
>                                       >
>                                       >                       Renato
>                                       >
>                                       >
>                                       >
> ________________________________
>                                       >
>                                       >
>                                       >
>         From: Dante Cinco
>                                       > [mailto:dantecinco@xxxxxxxxx]
>                                       >
>                                       >
>         Sent: Monday, November 09, 2009 6:13 PM
>                                       >
>                                       >
>         To: Santos, Jose Renato G
>                                       >
>                                       >
>         Cc: Xen-users
>                                       >
>         Subject: Re: xenoprof:
>                                       > operation 9 failed
> for dom0 (status: -1)
>                                       >
>                                       >
>                                       >
>         Renato,
>                                       >
>                                       >
>         I've narrowed down the
>                                       > opcontrol command
> sequence that causes the "write error" I'm
>                                       > having. If I just run
> "opcontrol --start" in dom0, it runs
>                                       > with no error and
> after "opcontrol --shutdown" I can run
>                                       > "opreport" and get a
> real report.
>                                       >
>                                       >
>         If I run "opcontrol
>                                       > --start-daemon
> --active-domains=1" in dom0, run "opcontrol
>                                       > --start" in domU
> (ID#1) and go back to dom0 and run
>                                       > "opcontrol --start" I
> get the "write error" message. It's as
>                                       > if "--start-daemon"
> is grabbing the file handle for
>                                       > /dev/oprofile/enable
> so when "--start" tries to write "1" to
>                                       > /dev/oprofile/enable,
> it is unable to do so because it is
>                                       > already locked.
>                                       >
>                                       >
>         So I can run OProfile in normal
>                                       > (non-Xen) mode but as
> soon I start using "--start-daemon" I
>                                       > have problems. To me
> it seems like a Xenoprofile problem.
>                                       >
>                                       >
>         I'm using OProfile 0.9.3 and
>                                       >
> oprofile-0.9.3-xen-r2.patch applied successfully.
>                                       >
>                                       >
>         Dante
>                                       >
>                                       >
>                                       >
>         On Thu, Nov 5, 2009 at 6:05 PM,
>                                       > Santos, Jose Renato G
> <joserenato.santos@xxxxxx> wrote:
>                                       >
>                                       >
>                                       >
>                 What version of
>                                       > OProfile are you using?
>                                       >
>                 Did you apply the Xen
>                                       > patch available in
> http://xenoprof.sourceforge.net ?
>                                       >
>                                       >
>                 Renato
>                                       >
>                                       >
> ________________________________
>                                       >
>                                       >
>                         From: Dante
>                                       > Cinco
> [mailto:dantecinco@xxxxxxxxx]
>                                       >
>                         Sent: Thursday,
>                                       > November 05, 2009 5:16 PM
>                                       >
>                         To: Santos,
>                                       > Jose Renato G
>                                       >
>                         Cc: Xen-devel
>                                       >
>                         Subject:
>                                       > xenoprof: operation 9
> failed for dom0 (status: -1)
>                                       >
>                                       >
>                         Renato,
>                                       >
>                                       >
>                         When I tried
>                                       > running "opcontrol
> --start" (after previously running
>                                       > "opcontrol
> --start-daemon") in dom0, I get this error message:
>                                       >
>                                       >
>                                       >
> /usr/local/bin/opcontrol: line 1639: echo: write error:
>                                       > Operation not permitted
>                                       >
>                                       >
>                         and this
>                                       > message in the Xen console:
>                                       >
>                                       >
>                         (XEN) xenoprof:
>                                       > operation 9 failed
> for dom 0 (status : -1)
>                                       >
>                                       >
>                         It looks like
>                                       > opcontrol is trying
> to do this: echo 1 > /dev/oprofile/enable
>                                       >
>                                       >
>                         and it is
>                                       > failing. "operation
> 9" maps to XENOPROF_start which is
>                                       > consistent with
> running "opcontrol --start." At first, I
>                                       > ignored the error
> because it gave the indication "Profiler
>                                       > running" but after I
> ran "opcontrol --shutdown" followed by
>                                       > "opreport" in dom0, I
> got this error message:
>                                       >
>                                       >
>                         error: no
>                                       > sample files found:
> profile specification too strict ?
>                                       >
>                                       >
>                         Do you know why
>                                       > the write error is
> occurring? I followed the steps in
>                                       > xenoprof_tutorial.ppt.
>                                       >
>                                       >
>                         Dante
>                                       >
>                                       >
>                                       >
>                                       >
>                                       >
>                                       >
>
>
>
>
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users