[Xen-users] Re: domU corrupt after server crash, help needed trying to recover domU

[Rudi Ahlers <rudiahlers@xxxxxxxxx>]

Does anyone know how to fix this?

On 5/8/09, Rudi Ahlers <rudiahlers@xxxxxxxxx> wrote:
> On Fri, May 8, 2009 at 2:29 PM, Rudi Ahlers <rudiahlers@xxxxxxxxx> wrote:
>
>> Hi all,
>>
>> One of our Dell servers has failed badly, and one of the domU's has been
>> corrupted in the process. It boots up to a point and then gives me a
>> kernel
>> panic:
>>
>> Loading dm-zero.ko module
>> Loading dm-snapshot.ko module
>> Scanning and configuring dmraid supported devices
>> Scanning logical volumes
>>   Reading all physical volumes.  This may take a while...
>>   No volume groups found
>> Activating logical volumes
>>   Volume group "VolGroup00" not found
>> Creating root device.
>> Mounting root filesystem.
>> mount: could not find filesystem '/dev/root'
>> Setting up other filesystems.
>> Setting up new root fs
>> setuproot: moving /dev failed: No such file or directory
>> no fstab.sys, mounting internal defaults
>> setuproot: error mounting /proc: No such file or directory
>> setuproot: error mounting /sys: No such file or directory
>> Switching to new root and running init.
>> unmounting old /dev
>> unmounting old /proc
>> unmounting old /sys
>> switchroot: mount failed: No such file or directory
>> Kernel panic - not syncing: Attempted to kill init!
>>
>>
>> It shows up as a Zombie:
>>
>> [root@xen ~]# xm list
>> Name                                      ID Mem(MiB) VCPUs State
>> Time(s)
>> Domain-0                                   0     1439     1 r-----
>> 329.0
>> Zombie-hfserver2                          15     1024     1 ----c-
>> 0.5
>> hfdns02                                   10      519     2 r-----
>> 1552.8
>>
>>
>> I can't mount either:
>>
>> root@xen ~]# mount /dev/data/hf
>> hfdns02    hfserver2
>> [root@xen ~]# mount /dev/data/hfserver2 /mnt/cpanel/
>> mount: you must specify the filesystem type
>> [root@xen ~]# mount -o loop /dev/data/hfserver2 /mnt/cpanel/
>> mount: you must specify the filesystem type
>>
>> Here's the output of the LVM partitions:
>>
>> [root@xen ~]# lvscan
>>   ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
>>   ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
>>   ACTIVE            '/dev/data/storage' [50.00 GB] inherit
>>   ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
>>   ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
>>   ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
>>   ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
>>   ACTIVE            '/dev/system/root' [39.06 GB] inherit
>>   ACTIVE            '/dev/system/swap' [9.75 GB] inherit
>> [root@xen ~]# vgscan
>>   Reading all physical volumes.  This may take a while...
>>   Found volume group "data" using metadata type lvm2
>>   Found volume group "system" using metadata type lvm2
>> [root@xen ~]#
>>
>>
>> Does anyone know how to fix a LVM like this?
>>
>>
>> --
>>
>
>
>
>  Here's what I've done so far:
>
> [root@xen ~]# losetup /dev/loop4  /dev/data/hfserver2
>
> # This mounts the LVM partition as imaginary physical to /dev/loop4
> # data is the virtual group (VG) name
>
> [root@xen ~]#  kpartx -va /dev/loop4
> add map loop4p1 : 0 208782 linear /dev/loop4 63
> add map loop4p2 : 0 62701695 linear /dev/loop4 208845
>
> # This creates a device map with partitions in /dev/data/hfserver in
> /dev/mapper
>
> [root@xen ~]# vgscan
>   Reading all physical volumes.  This may take a while...
>   Found volume group "VolGroup00" using metadata type lvm2
>   Found volume group "data" using metadata type lvm2
>   Found volume group "system" using metadata type lvm2
>
>
> [root@xen ~]# lvscan
>   inactive          '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
>   inactive          '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
>   ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
>   ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
>   ACTIVE            '/dev/data/storage' [50.00 GB] inherit
>   ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
>   ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
>   ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
>   ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
>   ACTIVE            '/dev/system/root' [39.06 GB] inherit
>   ACTIVE            '/dev/system/swap' [9.75 GB] inherit
>
> [root@xen ~]# lvchange -ay VolGroup00
> [root@xen ~]# lvscan
>   ACTIVE            '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
>   ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
>   ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
>   ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
>   ACTIVE            '/dev/data/storage' [50.00 GB] inherit
>   ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
>   ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
>   ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
>   ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
>   ACTIVE            '/dev/system/root' [39.06 GB] inherit
>   ACTIVE            '/dev/system/swap' [9.75 GB] inherit
>
> [root@xen ~]# e2fsck /dev/VolGroup00/LogVol00
> e2fsck 1.39 (29-May-2006)
> /dev/VolGroup00/LogVol00: clean, 631982/7325696 files, 4512772/7323648
> blocks
>
>
> At first it found a whole lot of damages inodes which I repaired.
>
> Then, I reversed the process:
>
> [root@xen ~]# lvchange -an VolGroup00
> [root@xen ~]# lvscan
>   inactive          '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
>   inactive          '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
>   ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
>   ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
>   ACTIVE            '/dev/data/storage' [50.00 GB] inherit
>   ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
>   ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
>   ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
>   ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
>   ACTIVE            '/dev/system/root' [39.06 GB] inherit
>   ACTIVE            '/dev/system/swap' [9.75 GB] inherit
>
>
>
> [root@xen ~]# vgchange -an VolGroup00
>   0 logical volume(s) in volume group "VolGroup00" now active
> [root@xen ~]# kpartx -d /dev/loop4
> [root@xen ~]# losetup -d /dev/loop4
> [root@xen ~]#
>
> [root@xen ~]# xm create -c /etc/xen/hfserver2
>
>
> And then it dies:
>
>
>  Reading all physical volumes.  This may take a while...
>   Found volume group "VolGroup00" using metadata type lvm2
> Activating logical volumes
>   2 logical volume(s) in volume group "VolGroup00" now active
> Creating root device.
> Mounting root filesystem.
> kjournald starting.  Commit interval 5 seconds
> EXT3-fs: mounted filesystem with ordered data mode.
> Setting up other filesystems.
> Setting up new root fs
> no fstab.sys, mounting internal defaults
> Switching to new root and running init.
> unmounting old /dev
> unmounting old /proc
> unmounting old /sys
> exec of init (/sbin/init) failed!!!: No such file or directory
> Kernel panic - not syncing: Attempted to kill init!
>  [root@xen ~]#
>
>
> --
> Kind Regards
> Rudi Ahlers
> CEO, SoftDux Hosting
> Web: http://www.SoftDux.com
> Office: 087 805 9573
> Cell: 082 554 7532
>


-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users