Re: [Xen-devel] how to generate a smaller core with xm dump-core

[Zhenzhong Duan <zhenzhong.duan@xxxxxxxxxx>]

å 2015/1/16 0:16, Don Slutz åé:
> On 01/15/15 05:20, Ian Campbell wrote:
> > On Thu, 2015-01-15 at 11:31 +0800, Zhenzhong Duan wrote:
> > > Hi Maintainers,
> > >
> > >
> > > We are facing issue collecting  coredump using the xm dump mechanism
> > > in Dom0.
> > >
> > > We face couple of such issues daily, where the VMs panic s and the SA
> > > team is supposed to collect the core.
> > >
> > > The actual problem is that where the VMs are having huge RAM like 32+
> > > GB RAMs dumping the core using xm dump generated core of almost the
> > > same size.
> > >
> > > And also when the RAM size is huge the time taken to dump is also
> > > huge.
> > >
> > > Transferring this core dump of this huge size to another system for
> > > analysis is really a pain. I am looking for some solution where we can
> > > generate smaller core using xm dump or any other tool.
> > >
> > > Something similar to makedumpfile which eliminates the zero and
> > > unnecessary pages, which results in core of smaller size.
> > >
> > >
> > > This has become a hot issue and we terribly need this feature.
> > >
> > > Can you come across with anything or are you aware of any feature
> > > which could cater our needâ.?
> > I'm not aware of any existing features or efforts in this area.
> >
> > If things like zero pages can be omitted from a core file without making
> > it non-standard then I think we'd be happy to see such patches to the
> > core libxc functions which produce the core files.
> >
> > That wouldn't help much with a heavily loaded VM which is making good
> > use of all its ram though.
> >
> > I've not looked but it's not unlikely that the dumping code itself is
> > not very optimal, so it might be worth having a look to see if more
> > batching etc might help speed things up at the root. Ages ago there were
> > patches floating around which improved the performance of the privcmd
> > mmap ioctl (from Mats Peterson), which might help here too (assuming
> > core dumping uses those interfaces as I expect). AFAIK those patches
> > were abandoned when Mats moved onto other things, resurrecting them
> > would probably help here as well as with migration and other
> > foreign-memory-man heavy workloads.
> >
> > Other random thoughts:
> >        * Automatically compress the core file, doesn't help with time to
> >          produce the dump (quite the opposite) but helps with copying it
> >          off box (but could also compress manually before moving or 
> > do it
> >          as a post processing step)
> >        * Have an option to produce some kind of "minicore", e.g. just
> >          register state and a few pages of the referenced stacks, 
> > perhaps
> >          a few pages of RAM either side of any register which happens to
> >          be a valid looking pointer.
> >                * Care would be needed to retain an acceptable 
> > probability
> >                  that the resulting core will actually be useful.
> >        * Consider including kernel RAM only (which might still be huge
> >          though if it has a 1:1 mapping of all RAM).
> >
> > I don't think any of those would be suitable for enabling by default but
> > they might be useful to have in the arsenal.
> >
> > HTH,
> >
> > Ian.
> If these are Linux guests then the patch:
>
> http://lists.xenproject.org/archives/html/xen-devel/2013-11/msg02351.html
>
> Can be used to enable crash to access the crashed guest and collect
> some basic info. I would also include the output of xen-hvmctx and/or 
> xenctx.
> -- This is the quick way I would do Ian's minicore .
>
> Note: dump-core currently does not include xen-hvmctx output (nor does 
> it include
> xenctx -a output).
>
> Using the results from this may allow you to not need a copy of every 
> dump (
> not to imply that 2 similar minicore's would insure that a copy of 
> each core
> was not needed).
>
> I also think that makedumpfile can process a xen dump-core file and 
> make it
> smaller.
Looks similar as gdbsx.
This patch will help if I could reproduce the issue locally.
But I can't access customer's env in most situation and they will not 
wait me to do online debug remotely.
Yes, makedumpfile may help in size rather than time here.
I am thinking if port some code from makedumpfile is possible and 
acceptable.
zduan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel