[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] A snapshot is not (really) a cow



It's very encouraging to know that I misunderstood the situation. But I do have a problem: It may be that I am genuinely running out of memory, but I am also getting a lot of messages about nbdNNN, and the overall effect is pretty disastrous.

I successfully created 2 new snapshots while another was in use as the root file system of a xenU domain. Previously this had failed immediately. Then the next lvcreate -s failed with a message about running out of memory, as before.

I wonder whether the timing problem I have which affects rpm is also causing some problem?

The relevant bit of /var/log/messages is:

... lots of nbdNNN messages ...
Sep 26 19:47:33 a4 kernel: nbd126: Request when not-ready
Sep 26 19:47:33 a4 kernel: end_request: I/O error, dev nbd126, sector 0
Sep 26 19:47:33 a4 kernel: nbd127: Request when not-ready
Sep 26 19:47:33 a4 kernel: end_request: I/O error, dev nbd127, sector 0
Sep 26 19:48:26 a4 net.agent[7343]: add event not handled
Sep 26 19:48:26 a4 kernel: device vif6.0 entered promiscuous mode
Sep 26 19:48:27 a4 kernel: xen-br0: port 3(vif6.0) entering learning state
Sep 26 19:48:27 a4 kernel: xen-br0: topology change detected, propagating
Sep 26 19:48:27 a4 kernel: xen-br0: port 3(vif6.0) entering forwarding state
Sep 26 19:50:00 a4 CROND[7360]: (mail) CMD (/usr/bin/python -S /usr/lib/mailman/cron/gate_news)
Sep 26 19:53:40 a4 kernel: nbd0: Request when not-ready
Sep 26 19:53:40 a4 kernel: end_request: I/O error, dev nbd0, sector 0
Sep 26 19:53:40 a4 kernel: nbd1: Request when not-ready
Sep 26 19:53:40 a4 kernel: end_request: I/O error, dev nbd1, sector 0
Sep 26 19:53:40 a4 kernel: nbd2: Request when not-ready

... lots more nbdNNN messages ...

Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd124, sector 0
Sep 26 19:53:44 a4 kernel: nbd125: Request when not-ready
Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd125, sector 0
Sep 26 19:53:44 a4 kernel: nbd126: Request when not-ready
Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd126, sector 0
Sep 26 19:53:44 a4 kernel: nbd127: Request when not-ready
Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd127, sector 0
Sep 26 19:53:44 a4 kernel: lvcreate: page allocation failure. order:0, mode:0xd0 Sep 26 19:53:44 a4 kernel: [__alloc_pages+824/842] __alloc_pages+0x338/0x34a
Sep 26 19:53:44 a4 kernel:  [<c013a16f>] __alloc_pages+0x338/0x34a
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [kmem_cache_alloc+89/100] kmem_cache_alloc+0x59/0x64
Sep 26 19:53:44 a4 kernel:  [<c013eb87>] kmem_cache_alloc+0x59/0x64
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [alloc_pl+48/76] alloc_pl+0x30/0x4c
Sep 26 19:53:44 a4 kernel:  [<c0328a54>] alloc_pl+0x30/0x4c
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [client_alloc_pages+40/129] client_alloc_pages+0x28/0x81
Sep 26 19:53:44 a4 kernel:  [<c0328ba7>] client_alloc_pages+0x28/0x81
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [vmalloc+32/36] vmalloc+0x20/0x24
Sep 26 19:53:44 a4 kernel:  [<c014f4d1>] vmalloc+0x20/0x24
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [kcopyd_client_create+104/199] kcopyd_client_create+0x68/0xc7
Sep 26 19:53:44 a4 kernel:  [<c03296cd>] kcopyd_client_create+0x68/0xc7
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [dm_create_persistent+199/320] dm_create_persistent+0xc7/0x140
Sep 26 19:53:44 a4 kernel:  [<c032b769>] dm_create_persistent+0xc7/0x140
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [snapshot_ctr+715/892] snapshot_ctr+0x2cb/0x37c
Sep 26 19:53:44 a4 kernel:  [<c0329f7f>] snapshot_ctr+0x2cb/0x37c
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [dm_table_add_target+323/464] dm_table_add_target+0x143/0x1d0
Sep 26 19:53:44 a4 kernel:  [<c0324aa7>] dm_table_add_target+0x143/0x1d0
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: [populate_table+125/216] populate_table+0x7d/0xd8
Sep 26 19:53:44 a4 kernel:  [<c0327105>] populate_table+0x7d/0xd8
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [table_load+103/309] table_load+0x67/0x135
Sep 26 19:53:44 a4 kernel:  [<c03271c7>] table_load+0x67/0x135
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [ctl_ioctl+236/330] ctl_ioctl+0xec/0x14a
Sep 26 19:53:44 a4 kernel:  [<c03279d3>] ctl_ioctl+0xec/0x14a
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [table_load+0/309] table_load+0x0/0x135
Sep 26 19:53:44 a4 kernel:  [<c0327160>] table_load+0x0/0x135
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [ctl_ioctl+0/330] ctl_ioctl+0x0/0x14a
Sep 26 19:53:44 a4 kernel:  [<c03278e7>] ctl_ioctl+0x0/0x14a
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [sys_ioctl+484/670] sys_ioctl+0x1e4/0x29e
Sep 26 19:53:44 a4 kernel:  [<c0167606>] sys_ioctl+0x1e4/0x29e
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Sep 26 19:53:44 a4 kernel:  [<c010d967>] syscall_call+0x7/0xb
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: device-mapper: : Could not create kcopyd client
Sep 26 19:53:44 a4 kernel:
Sep 26 19:53:44 a4 kernel: device-mapper: error adding target to table

... at this point lvm2 is unusable until after a reboot (unless some service can be restarted?) and xenU domains (or at least those with lvm2 snapshot based file systems) are dead.

I certainly have no intention of writing to  the original image.

Thanks
Peri


Christian Limpach wrote:

On Sun, Sep 26, 2004 at 12:38:06PM +0100, Peri Hankey wrote:
I always found the lvm2 'snapshot' terminology confusing - the thing created as a 'snapshot' is what accepts changes while a backup is made of the original volume.

I don't think that's the terminology the LVM2 people use.  The regular
use is to create a snapshot and backup this snapshot while you keep
using the original.

# drat - I needed another domain
lvcreate -L512M -s -n u4 /dev/vmgroup/root_file_system
... nasty messages .... all xenU domains dead ....
... lmv2 system in inconsistent state ...
... /dev/vmgroup/u4 doesn't exist ...
... /dev/mapper/root_file_system-u4 does exist ...

This should work, if it doesn't then it would seem to be a bug in
LVM2.  Since you mention out of memory error messages, are you sure
that you're not running out of memory in dom0?

The problem is that the 'snapshot' cows hold onto each other's tails - they seem to be held in a list linked (I think) from the original logical volume (here /dev/vmgroup/root_file_system). For their intended use as enabling backup, this seems to be meant to allow writes to the original volume to be propagated to all 'snapshots' created against that volume - there are comments about getting rid of the 'snapshots' after the backup has been done because this propagation of writes hits performance.

For my requirements, and I imagine for most others reading this list, all of this is superfluous. I don't need

   original -> snap1 -> snap2 -> snap3 ...

This is not the layout LVM2 uses.  If you look at the output of
``dmsetup table'', you'll see that each snapshot is independent
and only refers to the device it is a snapshot of and to its cow
device which will hold modifications.

so that I can't create a new snap4 while any of the others are in use.

I just need

   original <- cow1
   original <- cow2
   original <- cow3
   original <- cow4
   ...

where A '<-' B means B is a cow image of A, and where each of the cows is independent of the others so that a new cow can be created at any time, regardless how many others are active.

This is the layout LVM2 uses.  And it is indeed simple (and should be
quite robust) as long as you don't want to write to the original.
If you write to the original, you will have to copy the changed
blocks to every snapshot's cow device.  I think I've seen this
fail when having multiple snapshots and writing to the original.
But since you didn't write to the original (and one generally doesn't
need/want to write to the original in our case), that problem
is unlikely to be relevant to the failure you've seen.

   christian



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel





-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.