sábado, 27 de noviembre de 2010

Testing raidpycovery through mdadm


I'm still working on polishing raidpycovery. I started doing real experiments today to recover data from broken RAID5s. In order to do it, I used linux' md. It allowed me to study md a little bit while testing raidpycovery.

So, let's get our hands dirty.

First, let's create a directory where we will work, for example:

$ mkdir raid5
$ cd raid5

Now let's create the separate images where we will create our RAID5. I will use 4 disks 10 MBs each:

$ for i in 0 1 2 3; do dd if=/dev/zero of=disk$i bs=1M count=10; done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0584047 s, 180 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.100258 s, 105 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0691083 s, 152 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0508324 s, 206 MB/s

Now we have four empty files that we will feed into md to create our RAID to do our tests.

First, we will "loop" them so that we can use then with md (I don't know if this is really needed, but I'll do it just in case). As I'm working on a live USB, I'll have to link from /dev/loop2 and on. Check the used loop devices with losetup -a (as root... or with sudo):

$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)

Now, I loop the files:
$ sudo losetup /dev/loop2 disk0
$ sudo losetup /dev/loop3 disk1
$ sudo losetup /dev/loop4 disk2
$ sudo losetup /dev/loop5 disk3
$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)
/dev/loop2: [000f]:27898 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk0)
/dev/loop3: [000f]:27899 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk1)
/dev/loop4: [000f]:27900 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk2)
/dev/loop5: [000f]:27901 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk3)

Great. Now we can use the loop devices to create our RAID device:

$sudo mdadm --create /dev/md0 -l 5 -p ls -n 4 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5
mdadm: array /dev/md0 started.

If you read the man page of mdadm you will see that by default the algorithm for the RAID will be left sync and a default stripe/chunk size of 64k. We will need that data later on.

Now, let's format it so we can use it as a normal partition:

$ sudo mkfs.ext3 /dev/md0
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=64 blocks, Stripe width=192 blocks
7648 inodes, 30528 blocks
1526 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=31457280
4 block groups
8192 blocks per group, 8192 fragments per group
1912 inodes per group
Superblock backups stored on blocks:
8193, 24577

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Now, we can mount our just formated RAID device:
$ sudo mount /dev/md0 /mnt/tmp
$ sudo mount
/dev/md0 on /mnt/tmp type ext3 (rw)
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 29557 1400 26631 5% /mnt/tmp

There you go. A new partition with around 25 MBs of data available for mortal users. Now, let's copy some files into that partition:

$ sudo cp blah blah blah /mnt/tmp/

I'm using sudo to do the copy because right now that directory is owned by root.

After I copy the files I wanted, let's check their MD5s:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

Now, let's umount our RAID and stop it:

$ sudo umount /mnt/tmp
$ sudo mdadm --stop /dev/md0$ sudo umount /mnt/tmp/
mdadm: stopped /dev/md0

Great... now, let's test try to rebuild the RAID with the raidpycovery tools. I don't have the tools at the same directory, so I'll have to move and use relative names for the disks, keep that in mind:

$ ./Raid5Recovery.py 4 left async 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Asynchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes

Now, let's mount it and see if we can get any data from the recovered RAID:

$ sudo mount -o loop,ro wholedisk /mnt/tmp
mount: unknown filesystem type 'linux_raid_member'

Not good. From my tests, that's because there's md garbage at the end of the disks that the mount is seeing. I'll force the partition type then:

$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop6,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Gotcha! Did you notice that I resembled the RAID using left async? It has to be left sync, remember? Let's try again:

$ ./Raid5Recovery.py 4 left sync 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Synchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes
$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp

No complains. Great! Now let's see if we can see the data in the recovered RAID:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

And there you are. Everything is right there. Using this method today I discovered I was not rebuilding chunks from missing images correctly and that one of the right-handed algorithms has to be corrected. Let's see what I find.

As homework, try running the recovery script providing a missing disk (write none for it) and see if it works.

Hope you liked the read or find it useful.

Update: Have corrected the right algorithm issue. Make sure you get the latest stable release if you are gonna use it:

bzr branch -r tag:2.00.02 lp:~eantoranz/+junk/raidpycovery

domingo, 21 de noviembre de 2010

Broken RAID5 you said? Don't use Java anymore. Go with python instead!

So, you have a broken RAID5 and found the article I wrote for the Free Software Magazine 5 years ago about it and would like to give it a shot.

First off: Good luck! You are gonna need it.

Second: When I solved that problem I didn't have any experience in python whatsoever plus Oracle hadn't bought Sun, sued Google, etc.

During the course of these 5 years I have received some requests to help people out with their situation and it's been a very nice experience to hack on those devices here or there to get the data back. I have received a little money by paypal and even once got a box of whine bottle from Italy (thanks, Marco!) as payments.

Every single time I've had to help someone with it, I always get the question of how to run the .java classes I made. And I even took the task of compiling the .class files to then send them to the person in need.

Given the gripes over oracle/java recently I have decided to translate the library to python for people who want to use it so they don't have to go around trying to learn how to compile/run things in Java (at least in GNU/Linux).

The project is here:


You can get it very easily by using bazaar:

bzr branch -r tag:2.00.00 lp:~eantoranz/+junk/raidpycovery

Or you can just download the files from that same version and use them:
Let's not forget the readme where you get to see how to use it:

There's more stuff in the project (some chunks from the RAID I recovered at the Hospital, mostly... but you never know!) so I'd suggest to use bazaar but....

Warning/disclaimer/whatever: I haven't tested it on a real environment so there might be some glitches (but I did a translation mostly so it should be correct overall).

In case you want to get the original java code:

bzr branch -r tag:1.00.00 lp:~eantoranz/+junk/raidpycovery

Or start browsing here to get the files.

PS Sorry, Larry.... it just had to be done. A little weight off my jumping-off-java shoulders. :-)

PS2: Donations, donations! You found the project useful? How about giving a little contribution? 0,10 US$ / Gb recovered? Sounds like fair to me. Specially considering how fast HDs sizes are growing compared against the inflation.

lunes, 15 de noviembre de 2010

My thoughts on the switch to wayland

Like anybody cares for what I think, right? Anyway... I made a comment in one of these wayland-related news at linuxtoday and Carla Shroder took the time to ask me:

"...why all that extra complexity to go back to where we were in the first place?"

That's a fair question. By the way, my love to you, Carla. You haven't sent me a comment about pythogoras that I asked you for but I still love you and care for you. :-D

Anyways... the thing is this:

Wayland is going to replace X in Ubuntu and Fedora. That's quite a remarkable statement to make. We are talking about X, the same X that has been there since I started using GNU/Linux about 9 years ago and even many years before that cosmic event, if you will.

But all the GUI applications for UNIX at the time use X, so that means it's going to be a really troublesome change, isn't it? Also, there are really cool and extremely useful features in X like Network Transparency (think of export DISPLAY=blablah:0.0 or ssh -X, pleople) that would disappear from the face of the earth (I'm wondering what I would show my Windows-loving friends now when I start showing them the wonders of GNU/Linux if I don't have network transparency. I'll have to think about that).

As I was saying, I'll (try to) tackle those two questions.

First off, this means a major reworking to get all applications to render on wayland, isn't it? I think it can be solved by hacking the lower layer APIs like Qt or GTK or even at a lower level like wrapping the Wayland API for clients around the X API (update: I think it's viceversa... but you get the concept, don't you?). Then not much work would have to be done on the higher layers (or so the fairy tale theories of software development say). That means applications won't be hurt that much. I'm not implying that it's going to be easy but not much work will have to be required from us mere mortals in that case. Also it means that the switch could be done overnight to start using Wayland.

Then there are things built inside X that would be gone once Wayland makes its debut, right (to be read as "Network Transparency")? What I've read around (won't provide any links but it's very logical) is that X could still use Wayland as just another graphics interface and then you will have X running on top of Wayland and (tadaaaaaaaaaaaaaaaaa) you would get Network Transparency back faster than you can say "mi moto alpina derrapante" (that's a nice joke in spanish... people who are learning spanish should give me a call so that I tell you about it and have a laugh).

But then Carla makes the final killing point: Is all this hassle really worth it?

In all honesty, I don't know. It's still too early in the game to make up my mind weather it will be worth it or not. But if X's complexity/overhead can be cut down and get a faster/lighter/snappier/sexier/whatever-you-consider-important-er environment to spend your time onto, then it could be worth it in the end... specially if (as I said) it won't require much work but on the lower layer APIs.

Now I don't know if switching to Wayland will be really worth it, but was it reading this article worth it after all? I truly hope so.

miércoles, 10 de noviembre de 2010

Why don't manufactures get together and get Microsoft out of the loop?

Here comes a crazy thought:

Why don't computer manufactures get in cahoots and get Microsoft out of the demi-god loop? I mean, computer manufactures are struggling because their profit out of every sell are very small... in the meantime, Microsoft is laughing its ass off while making millions upon millions over their sells of Windows 7 (or so they say).

If any of the manufacturers dared getting out of the Microsoft way (you know, let's push Linux a bit harder, let's not say we recommend Windows 7, etc, etc), Microsoft would just need to get the price of Windows 7 licenses a little higher to force the manufacturer into red numbers (if it isn't already, of course) and teach the bully a lesson, right? We have all seen the Halloween Documents and know of Microsoft's business practices, so no surprises there. Microsoft would still have all the other manufactures lined up with lower prices to pay for Windows 7, the rebel manufacturer would have to get their prices a little higher because of the increase in the price of Windows 7 it has to pay now making it more difficult for them to compete against the other manufacturers and Microsoft gets to keep the upper hand in the end. Sounds like a subtle kind of trust, doesn't it? By the way, am I the only one who has the phrase "Divide and conquer" pounding in his head?

But what would happen if the biggest manufactures would get together (say, in a secret deal) and fight Microsoft all at the same time? Is that possible? Is that legal? Would that be enough to get Microsoft out of its ways and start really competing instead of been shoved down everybody's throat? If it is legal (I honestly don't know), then why don't they do it and let Microsoft laugh (very loudly, by the way) at them (and the rest of us)?

Now... where did that smart-ass laugh I'm hearing in surround sound come from? Seattle?