Testing raidpycovery through mdadm

Hi!

I'm still working on polishing raidpycovery. I started doing real experiments today to recover data from broken RAID5s. In order to do it, I used linux' md. It allowed me to study md a little bit while testing raidpycovery.

So, let's get our hands dirty.

First, let's create a directory where we will work, for example:

$ mkdir raid5
$ cd raid5

Now let's create the separate images where we will create our RAID5. I will use 4 disks 10 MBs each:

$ for i in 0 1 2 3; do dd if=/dev/zero of=disk$i bs=1M count=10; done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0584047 s, 180 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.100258 s, 105 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0691083 s, 152 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0508324 s, 206 MB/s
$

Now we have four empty files that we will feed into md to create our RAID to do our tests.

First, we will "loop" them so that we can use then with md (I don't know if this is really needed, but I'll do it just in case). As I'm working on a live USB, I'll have to link from /dev/loop2 and on. Check the used loop devices with losetup -a (as root... or with sudo):

$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)
$

Now, I loop the files:
$ sudo losetup /dev/loop2 disk0
$ sudo losetup /dev/loop3 disk1
$ sudo losetup /dev/loop4 disk2
$ sudo losetup /dev/loop5 disk3
$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)
/dev/loop2: [000f]:27898 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk0)
/dev/loop3: [000f]:27899 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk1)
/dev/loop4: [000f]:27900 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk2)
/dev/loop5: [000f]:27901 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk3)
$

Great. Now we can use the loop devices to create our RAID device:

$sudo mdadm --create /dev/md0 -l 5 -p ls -n 4 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5
mdadm: array /dev/md0 started.
$

If you read the man page of mdadm you will see that by default the algorithm for the RAID will be left sync and a default stripe/chunk size of 64k. We will need that data later on.

Now, let's format it so we can use it as a normal partition:

$ sudo mkfs.ext3 /dev/md0
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=64 blocks, Stripe width=192 blocks
7648 inodes, 30528 blocks
1526 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=31457280
4 block groups
8192 blocks per group, 8192 fragments per group
1912 inodes per group
Superblock backups stored on blocks:
8193, 24577

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
$

Now, we can mount our just formated RAID device:
$ sudo mount /dev/md0 /mnt/tmp
$ sudo mount
.
.
.
/dev/md0 on /mnt/tmp type ext3 (rw)
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
.
.
.
/dev/md0 29557 1400 26631 5% /mnt/tmp
$

There you go. A new partition with around 25 MBs of data available for mortal users. Now, let's copy some files into that partition:

$ sudo cp blah blah blah /mnt/tmp/

I'm using sudo to do the copy because right now that directory is owned by root.

After I copy the files I wanted, let's check their MD5s:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

Now, let's umount our RAID and stop it:

$ sudo umount /mnt/tmp
$ sudo mdadm --stop /dev/md0$ sudo umount /mnt/tmp/
mdadm: stopped /dev/md0
$

Great... now, let's test try to rebuild the RAID with the raidpycovery tools. I don't have the tools at the same directory, so I'll have to move and use relative names for the disks, keep that in mind:

$ ./Raid5Recovery.py 4 left async 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Asynchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes
$

Now, let's mount it and see if we can get any data from the recovered RAID:

$ sudo mount -o loop,ro wholedisk /mnt/tmp
mount: unknown filesystem type 'linux_raid_member'
$

Not good. From my tests, that's because there's md garbage at the end of the disks that the mount is seeing. I'll force the partition type then:

$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop6,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$

Gotcha! Did you notice that I resembled the RAID using left async? It has to be left sync, remember? Let's try again:

$ ./Raid5Recovery.py 4 left sync 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Synchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes
$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp

No complains. Great! Now let's see if we can see the data in the recovered RAID:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

And there you are. Everything is right there. Using this method today I discovered I was not rebuilding chunks from missing images correctly and that one of the right-handed algorithms has to be corrected. Let's see what I find.

As homework, try running the recovery script providing a missing disk (write none for it) and see if it works.

Hope you liked the read or find it useful.

Update: Have corrected the right algorithm issue. Make sure you get the latest stable release if you are gonna use it:

bzr branch -r tag:2.00.02 lp:~eantoranz/+junk/raidpycovery