Technology FLOSS

Dude, i'm not the usual PC technician!

2012-01-04T13:55:00.000-08:00

So, you are in the middle of a crisis, you have tons of data stuck in a broken RAID5, shall I say it's critical data, and you need it back.

So you look around the internet and find me, none other, with some articles published on the internet about some successful recoveries, a tool to do the recovery which, by the way, is completely Free (with capital F) to use.... and yet you want to ask for my help.

Well.... if you think this is the kind of job the PC-guy-you-call-when-your-home-PC-gets-busted-with-viruses can do and are willing to pay accordingly then don't bother to ask for my help. This is veeeery hackish stuff which takes some time to get done, a great deal of attention to details and yet another great deal of tricks and experience to solve so be ready to pay through your nose.

You still wanna call me? You have been warned.

How to connect your smartphone through your GNU/Linux box. Part I: With a spare wireless router

2011-11-22T15:28:00.001-08:00

Hi!

So, you forgot to pay your phone bill? Or you are just in the middle of a financial crisis? Whatever the reason, you lost connection of the most important item of communication in your daily life: your smartphone.

But you have a computer that does connect to internet. Well, I got my BlackBerry to connect to internet (and regain simple communication to a very dear group of friends) by connecting through the computer.

You see, as I've always said, GNU/Linux is about fun and flexibility. This is no exception. I connect to my ISP through a USB dongle (GSM connection). This box has a wireless interface but, in this chapter, I will use a spare wireless router I have at home. I'll hack on to connect through the wireless interface directly on a following chapter (if the interface allows me to).

So, first, I set up my wireless router and my box to use static IPs (so that my computer and the router will be able to "see" each other). To make things simple, I used this set up:
My Box: 192.168.2.1/24
Router: 192.168.2.2/24
Router's GW: 192.168.2.1 (your box)
Router's DNS: Whatever my ISP's DNSs are (cat /etc/resolv.conf)
Now, in order for the traffic comming from the router to go through your box, you have to make sure two things are set up.

There not be any rule/policy in netfilter's FORWARD chain keeping the traffic from going through
# iptables -L FORWARD -nvChain FORWARD (policy ACCEPT 1756 packets, 917K bytes)
pkts bytes target prot opt in out source destination

You have a DROP policy? Don't want to get rid of it? Add rules to allow traffic going from the router to internet to pass through, and also the traffic that comes back. That's a whole topic in and of itself so I won't go into it. I have an ACCEPT policy on FORWARD, so I have no problem with that.

The kernel be enabled to forward ipv4 traffic
# sysctl net.ipv4.ip_forwardnet.ipv4.ip_forward = 1

It says 0? You have to enable it:
# sysctl -w net.ipv4.ip_forward=1

That should be it.... or almost. The thing is that the kernel is now letting traffic comimg from the router to get to internet but it's not doing any network address translation on that traffic so it's coimg out not with your network interface's IP address but with the wireless router's (which is probably doing nat on the traffic from its clientes) and that won't hold water so next step is to masquerade all traffic that is going out to internet. Something like this should be enough:

# iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

I'm using ppp0 for my example (as I said, USB dongle) but it has to be the interface you use to connect to internet. Now, your smartphone (if it's already using the wireless network from the router) should be able to get to internet.

I hope it does the trick for you.

raidpycovery supports raid4 and repetitions

2011-11-08T08:29:00.000-08:00

Hi!

Just wanted to announce that I just tagged/pushed version 2.01 of raidpycovery which now supports raid4 and also repetitions (like the one an HP SmartArray P410i displayed).

Repetitions works this way.On a normal RAID5 (left async) you would see something like this:

A0 B0 C0 D0 X0

A1 B1 C1 X1 D0

A2 B2 X2 C2 D2

A3 X3 B3 C3 D3

X4 A4 B4 C4 D4

A5 B5 C5 D5 X5

.

.

.

On a raid with 2 "repetitions" you would see:

A0 B0 C0 D0 X0

A1 B1 C1 D1 X1

A2 B2 C2 X2 D2

A3 B3 C3 X3 D3

A4 B4 X4 C4 D4

A5 B5 X5 C5 D5

.

.

.

You get the idea, right? On the raid5 I broke last weekend the controller did 16 repetitions.

Repository for the project is here.

Edmundo 750 GBs - HP 0 Gbs

2011-11-06T16:31:00.000-08:00

Hi!

Just today I finished one almost perfect data recovery from another RAID5 that its controller spitted onto (another HP controller, yet again).This time around, it was a 6-disk RAID5 (some 130 GBs each).

I knew beforehand that my python library (which I had translated from java about one year ago) was too slow and this task was going to take days if I hadn't done something about it. So, first I took the task of translating it to C++, which I finished some days ago.

When faced with the disks, the HP controller reported that 1 of them was physically dead, one of them had been hotswapped and another was to fail soon (public administration, don't ask about the details... gruesome). Fact of the matter is that the controller didn't want to make it visible so the server would not start. HP support asked me to dip the array in holy whater and forget about it. But that was not to happen, was it?

The disks were mounted on a separate computer one by one (well, not one by one but on a separate computer), and images were made from each one of them (only one of the drives didn't allow itself to be copied). Tips: LiveUSB, ddrescue.

After getting my hands on the images, I started analyzing how it was built to figure out the order of the drives, the chunk size and the algorithm used. Normally for this job the first MBs from each image help to do it.... after taking a long cold look at the images, I had figured out the chunk size (64ks) by looking at some numbered markers present on the ext2 partition that was at the begining of the disks... however, as this was a linux server with no FAT of NTFS stuff on it, there was no data (specially text) to be able to figure out the order... and specially, not crammed at the begining of the disk (does NTFS still work like this? You gotta be kiding me).

After thinking about it (and working on it) for a while I ended up finding whole stripes that would be made out of text (except, most probably) for a single chunk for every stripe. Finding these stripes and their sequence took its time, but eventually I was able to break it. The only thing that bothered me was that for many stripes (one after the other) the checksum chunk was always on the same disk (which I didn't expect) and the sequence remained the same on those stripes. At first I thought that perhaps it had been built as a RAID4 instead of a RAID5, but after checking many stripes the whole thing made itself clear: It's a RAID5 in which row configurations repeat themselves a number of times (in this case, 16 times). All in all, left async 64Ks with 16 repetitions. This took me to make an adjustment to my C++ utility just for this task and on a single shot I was able to rebuild the RAID5 from its pieces. Rebuilding took some 14 hours (probably not because of the utility but because of the bottleneck of having all 5 images on a single drive).

After rebuilding, I was able to do fdisk -lu on the disk image, do the losetup trick, and mount the server's / partition and get data out of it. There were some IO errors (not that many, fortunately.... given that the controller had already reported more than one disk as broken, maybe some pieces of the images were broken or the controller had stopped using one of the drives as a whole some time before crashing). Some other people will take of getting the stuff working but my task is definitely over.

So, next time you have a RAID5 crash , always remember it can take some time/thought breaking it... but in the end, it can pay off.

I'm starting to feel like the Messi of data recovery. Anybody needs a data recovery hacker around there? I'm willing to travel anywhere!

PS And before you ask, it's 750 GBs (or close) for this array plus the one I had recovered in 2005.

Remember the times when I used python for RAID5 Recovery? Not anymore

2011-11-03T23:43:00.000-07:00

Hi!

Remember the times when I translated the raid5 recovery utility from java to python? Well, let me tell you something. It was horrendously SLOW. For example, I used to help a guy who had a problem with his array and the operation to rebuild the array took literally days (and the data wasn't recovered in the end, so...). While doing my tests, a (virtual) 300 MBs RAID5 with a missing disk took about 450 secs to be rebuilt on python (on my netbook, anyway).

Right now I'm facing the recovery from a RAID that has 6 disks 150 GBs each. That is going to take _a lot_ of time to recover. So, just in case we do have to go on with the operation, I decided I was goint to translate it to C++, which I started doing a little more than 24 hours ago (not fully employed on the task so don't complains about me being slow in the process).

So, I'm happy to announce that the translation has worked and after making some tests I can attest that it absolutely kicks ass (at least time-wise :-)). Remember the 300 MBs RAID5 I talked about a couple of paragraphs ago? It's rebuilt in roughly 8 seconds. 8!!!!

The way you call it is the same (same arguments) only that, instead of calling Raid5Recovery.py, you will just write Raid5Recovery (with the accompanying ./, you know the drill).

I don't intend to change its name at the time so it will remain Raidpycovery.

As usual, I have made some tests on it but they were not exhaustive so feel free to use it at your own risk.

The project is hosted on launchpad (no release) here:

https://code.launchpad.net/~eantoranz/+junk/raidpycovery

Last but by no means least, I dedicate this project to the memory of my wife, Lina Marcela "Nena" Salgado.

Open Letter to Neelie Kroes

2011-09-24T10:19:00.000-07:00

Hi, Ms. Kroes.

First, I'd like to say that I'm not an european citizen. I'm
venezuelan and I'm living in Colombia at the time. If that's all it
takes to get this email dropped from your inbox, so be it. If you (or
someone from your staff) is still reeading it, I thank you.

During the last few days it has hit the IT mainstream media that
Microsoft, as part of their "Windows 8 Logo" program, will request
computer manufacturers to support UEFI's "secure boot". Manufacturers
that don't support it, won't get the privilege of putting a "WIndows
8" logo stamp on their computers. It has also been published that
manufacturers are willing to implement this _without_ giving the users
the possibility to disable this option.

In order for "secure boot" to be used, it requires using certification
keys to allow any operating system to boot the computer and that
provides Microsoft with a double advantage:
- Users wouldn't be able to use an older version of Windows
- Users won't be able to use a "non certified" operating system, like
say, GNU/Linux.

Given their strong position on computer manufacturers, whatever
Microsoft comes up with, manufacturers will follow. Unfortunately,
that power, as we can see here, is being used to hinder Microsoft's
competition and force the user to use Microsoft's latest OS instead of
any other OS they so choose to use (even older versions of Microsoft
Windows).

Thank you very much.

Update: The mail has been forwarded to Mr Almunia, on the Competition front. Email Address: CAB-ALMUNIA-COURRIER at ec.europa.eu

Update II: I got a response this morning (oct. 27th) from staff:

Dear Mr. Carmona,

Thank you very much for your messages of 24 and 26 September 2011 expressing concerns regarding the new Microsoft's Windows 8 security requirements. According to these requirements, the computer manufacturers ("OEMs") conforming to the Windows 8 logo program are obliged to use Unified Extensible Firmware Interface ("UEFI") secure boot. This might allegedly lead to anti-competitive effects in the market.

As you are probably aware, Windows 8 is currently under development and therefore not final. We will, however, monitor the further developments in order to ensure the full respect of European competition rules. Whether or not there is a violation of EU competition law, however, depends on a range of factual, legal and economic considerations.

For more information about the competition-related activities of the European Commission we invite you to consult our website: http://europa.eu.int/comm/competition/index_en.html .

Feedback from interested citizens, such as yours, is a very valuable source of information for us which we take seriously, and for which we would like to thank you.

Yours sincerely,

Per Hellström
Head of Unit
European Commission
DG Competition
Markets and cases II: Information, Communication and Media
Antitrust: IT, Internet and Consumer electronics

Will ARM get to sweep X86 (and take WIndows to hell along with it)?

2011-08-28T20:46:00.000-07:00

Hi!

I've been very interested in ARM developments since a couple of years ago when it looked like ARM netbooks were just about to be released and kick x86 out of the market (at least on netbooks). A lot of things didn't happen since then, don't you agree? A lot of speculation about how Microsoft got to influence potential manufacturers has been thrown around... but anyway, fact of the matter is that they were never released.

During the last month a couple of things have happened that are making me think that ARM is going to become a major headache for the (already kind of broken) Wintel monopoly (duopoly? whatever!).

First, I helped solve a printing problem involving three cheap printers. First we tried to solve the problem connecting all 3 printers to a single computer (some PC) and almost got to get the whole thing working... the only problem was that only one of the printers would print reliably and the other 2 would refuse to make it all of the times. Workaround? Bought 3 dreamplug servers, connected a single printer to each one of the servers, ran a couple of commands* and, voilà!, problem solved.

Here's a picture of the whole set up (will upload a higher resolution pic when I get it):

On the left of the desk you can see all 3 printers and each printer's dreamplug master right on top of each one, see?

Now.... this was a rather simple fix, each server is kind of cheap (specially compared with a fully fledged computer) and, on top of that, these beauties will keep the electricity bill low (as they all put together eat a lot less than a computer). To me, this spells trouble ahead for Intel (and Windows, though they are at the top on servers will see a ferocious battle with GNU/Linux on the not-so-short term) as it means that people will have yet another simple comoditized option different from a PC to solve problems.

And if that was not enough, then I see this today:

Full article where I got it from here.

Now, this is not a PC. This is an ARM-based platform that is expected to start selling by the end of the year... and at US$ 25 a pop.

About 2 years ago I thought ARM was about to kick butt (back then)... boy, was I wrong.... but given these two events I told you about, I don't think it's too crazy to believe that this is going to make a major dent on Intel and Microsoft (given that Microsoft is years from releasing/stabilizing their ARM port of Windows... and I'm not talking about WP7 or Windows CE, of course) in the not-so-distant future. Only time will tell.

PS Oh, and did I say that apparently Canonical is throwing a lot of its weight on ARM too? Interesting times ahead of us.

* As a matter of fact, getting the printer to work took a little work. The splix drivers (they are cheap samsung printers) were not available for the model of the printers we were using (on updated debian) but we knew ubuntu did have it available so, after a little trying and trying we ended up compiling the ubuntu splix driver's source on the dreamplugs to get them running (the source of the drivers I got from a normal netbook running kubuntu natty and placed them on the dreamplug for compilation). A little scripting here and there and they were finally printing the way we wanted them to.

Sales of GNU/Linux servers explode on Q2 2011 with an increase of 47%

2011-08-26T13:23:00.000-07:00

Hi!

According to numbers released by IDC, income from server sales for Q2 2011 saw an increase of 47.5% compared to the same quarter 2010 to reach US$ 2.7 billon. This represented 20.5% of all income for the quarter. In comparison, income for Windows servers increased a healthy 12.4% to reach US$ 5.9 billion with represents 45.5% of all income for the quarter. These numbers in a very healthy quarter for all server segments.

If you get the calculater out of your pockets you will notice that GNU/Linux income increased by some US$ 870 million while Windows servers' incresed by some US$ 650 million.

In case you want to see the whole report, check here.

How to add a custom session type to kdm

2011-08-23T21:19:00.000-07:00

Hi!

I was trying to start my "failsafe" session from kdm. Back in the old days, what I would get with that would be a plain session with a terminal on top and nothing else. Unfortunately, when I tried it I would always get back to kdm. Not what I expected.

I figured out that there must be a simple way to create my own custom session... say, where I would only call rxvt for example... and there certainly is a simple way to do it. All you have to do is create a simple file like this under /usr/share/xsessions. Mine is called rxvt.desktop with this content:

[Desktop Entry]
Encoding=UTF-8
Type=XSession
Exec=/usr/bin/rxvt
TryExec=/usr/bin/rxvt
Name=Just RXVT Session
Name[es]=Solo RXVT

After the file is set up, just restart kdm and your new session type will be on the menu.

Given that the location looks pretty neutral and not tied to a desktop environment, I would expect the same trick to work for gdm and xdm as well.

Also, I would recommend people to take a close look at /etc/kde4/kdm/kdmrc as there's a lot of interesting stuff in that file.

Help us defend colombian's digital rights from entertainment lobbies

2011-08-23T10:11:00.000-07:00

Hi!

Colombia has become one of the places where the entertainment industry is trying to overtake things as basic as the right to access internet from people in order to protect their money (as if the law is going to keep big-buck copyright infringers from doing it). You can help us try to stop them from pulling the strings to get some crazy laws (two-strikes??? Not even three? That's because baseball is not a popular sport here, otherwise...) passed here in Colombia by signing this petition online (petition in english is here).

Thanks in advance for your help and support.

My wife has died

2011-08-03T11:04:00.000-07:00

I'm sorry to announce that Lina Marcela "La Nena" Salgado, my wife, died on July 8th. She fought cancer corageously but ultimately the disease defeated her.

Nena: siempre tendrás un lugar muy especial en mi corazón y siempre te recordaré. Te amo!

Kubuntu 11.04: After updating to kde 4.7, things are back to normal

2011-08-01T12:11:00.000-07:00

Hi!

You might remember I wrote quite a rant a few months back about the sorry state of kubuntu 11.04 when it was released. There were so many complains I had.

Well, after I updated to kde 4.7 yesterday things have mostly fallen into place. No more kded4 going crazy when I mess around with my usb dongle, my desktop starts with 3d effects enabled and everything looks fine and dandy. If I notice other things, I'll let you know.

There's something I haven't tested yet but it's not related to kde or kubuntu per se. The driver of my wireless card crashes my netbook (apparently, it's a driver bug... I haven't used the wireless card in a couple of months so I don't know if it's solved... launchpad bug report is here).

Update: Thanks to Sam for advicing me to remove the comment about my wife. I created another blog entry for her.

How to perform automatic actions after a connection is established with wvdial/ppp

2011-05-22T09:17:00.000-07:00

Hi!

I'm on kubuntu Natty and I'm pretty glad with the distro overall. However, there are a couple of bugs that bug me more often than I'd like them to. One of them is that, for whatever reason, when I establish an internet connection with wvdial/ppp, squid doesn't go through it. I have to restart he service once the connection is set up so that traffic starts pouring from squid. What I realized was that it is possible to do this action automatically.

There is one directory where you can put scripts to be run when a new connection is established and when a connection is dropped. I wanted to do something when the connection was established, so had to upt the script in /etc/ppp/ip-up.d:

$ cat /etc/ppp/ip-up.d/9999squid
#!/bin/bash

/etc/init.d/squid3 restart

Pretty straightforward, right? Make sure the script has the executable flag set up and you are done.

Additionally, the script get the following information as parameters (in the order they are received):
Network Interface
TTY
Speed
Local IP address
Peer IP address
Additiona Parameter (as received by pppd).

There are some environment variables that I think are available for the script, but I haven't tested them: PPP_IFACE, PPP_TTY, PPP_SPEED, PPP_LOCAL, PPP_REMOTE y PPP_IPPARAM.

Note: Given that the hack is implemented on ppp, this should probably work with network connections set up on NetworkManager that use ppp (like broadband).

How to install django (or run it without installing it at all) without administrative permissions

2011-05-09T10:45:00.000-07:00

Hi!

I'm giving my first baby steps at learning Django. Right up front I saw the installation procedure and it talked about requiring an administrator to do the installation or copy the django directory inside the Djando tar ball to site-packages.... I personally don't like doing these kind of things so I sat down for a while to see how it could be done without requiring running setup.py at all. It's like this.

In my environment, I unpackaged the Django tarball that created this directory:

$ ls -l /home/antoranz/Downloads/Django/Django-1.3
total 72
-rw-r--r--  1 antoranz antoranz 19166 2011-03-21 20:38 AUTHORS
drwxr-xr-x 17 antoranz antoranz  4096 2011-05-09 09:16 django
drwxr-xr-x 14 antoranz antoranz  4096 2011-03-23 00:08 docs
drwxr-xr-x  2 antoranz antoranz  4096 2011-03-23 00:08 extras
-rw-r--r--  1 antoranz antoranz   592 2009-10-30 03:24 INSTALL
-rw-r--r--  1 antoranz antoranz  1558 2008-08-09 09:40 LICENSE
-rw-r--r--  1 antoranz antoranz  1494 2011-03-16 14:02 MANIFEST.in
-rw-r--r--  1 antoranz antoranz  1228 2011-03-23 00:08 PKG-INFO
-rw-r--r--  1 antoranz antoranz  1786 2011-01-28 17:07 README
drwxr-xr-x  2 antoranz antoranz  4096 2011-03-23 00:08 scripts
-rw-r--r--  1 antoranz antoranz   108 2010-08-05 08:00 setup.cfg
-rw-r--r--  1 antoranz antoranz  4325 2011-03-23 00:06 setup.py
drwxr-xr-x  5 antoranz antoranz  4096 2011-03-23 00:08 tests

This is the place you should run setup.py from to do django's installation, right?.... you can just skip it completely.

Now, without doing anything but just extracting the tarball, you won't be able to do much with it. From a small project I'm working on (following the tutorial):

$ python manage.py runserver

Traceback (most recent call last):
  File "manage.py", line 2, in 
    from django.core.management import execute_manager
ImportError: No module named django.core.management

Too bad. But if you just export the PYTHONPATH to include the django directory you are done:

$ export PYTHONPATH=/home/antoranz/Downloads/Django/Django-1.3/
$ python manage.py runserver
Validating models...

0 errors found
Django version 1.3, using settings 'django1.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

And that's it. If you need to call django's binaries (django-admin.py and so on), use the full path to reach them or add the required path to PATH environment variable so that it's available.

I staill haven't tried to run django from apache without doing the instalation..... but will be publishing it (if possible) once I reach that part of the tutorial.

Will kubuntu natty stabilize? Ever?

2011-05-04T13:19:00.000-07:00

Hi!

First off, let me tell you something before I start my rant on kubuntu: I've been a kubuntu user for 6 years now... and I don't intend to switch to gnome (ubuntu) or xfce (xubuntu) or any other of the other variants anytime soon. I like KDE and I'm willing to put up with the nag that I have to go through in order to continue using it

Back when I installed natty 'bout a couple of weeks ago I had a few complaints but we were in beta, right?... but it's been a few days since release time and things feel worse and worse. At first I didn't use gl-based screensavers because they made the graphics go completely crazy (it's a samsung netbook with intel chipset) so I used a simple screensaver instead (though I had desktops effects turned on and they tend to work pretty well)... then updates started to come in. gl screensavers started to be stable some days ago and I could start/stop it at will... only sometimes it crashed KWin.... but yesterday there was another update and things have gone to hell. Every single time I start the screensaver it crashes KWin... now gkrellm doesn't remember it's on desktop 6 when KDE starts.... When KDE starts it always turns off my desktop effects (which I think didn't happen when I updated to natty), hell... this is just too much for my taste. Are we experimenting here? I haven't tested if kded4 goes crazy when I come back from sleeping/hibernating yet... but something tells me it's not going to be pretty.

Guys, could you please stabilize this whole thing and not make us kde users feel like third class citizens in the ubuntu world and keep us in second class at least?

Updated minibd driver for linux (sbull.c)

2011-04-19T22:16:00.000-07:00

Hi!

I'm embarking myself in a journey to write a block driver for linux (raid5 recovery stuff, anyone? Doesn't ring a bell? Look around this blog).

My firsts steps have involved trying to figure out how linux's block layer api works... so I looked for the simplest example I could find.... and it was in the sbull.c of the minibd drive. It's a dull ram-based block driver where the basics for the API are set in place for people to see how a driver is created.

I started working with it but it was written for older versions of the kernel and the block layer API has received an overhaul that made the sbull.c driver as I got it unusable so I sat down to modify the driver to make it work while trying to figure out how things work... I hopefully did both... but even if I didn't get to understand how the API works, I did make the driver work. How well am I getting the API? Guess will have to wait for my real driver to come out to know for sure.

Here it goes. First, the Makefile so you don't have to reinvent the wheel:

obj-m := sbull.o

PWD := $(shell pwd)
KDIR := /lib/modules/$(shell uname -r)/build

all:
        make -C $(KDIR) M=$(PWD) modules
clean:
        make -C $(KDIR) M=$(PWD) clean

That's it... and now the driver (sbull.c):

/*
* Mini-block driver.
*
* this code is based on one example from LWN.NET: http://lwn.net/Articles/31513/
*
* Copyright 2003 Eklektix, Inc. Redistributable under the terms
* of the GNU GPL.
*
* Update to new block layer api by Edmundo Carmona
* works on linux 2.6.38.3 tested on UML.
*/
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include

MODULE_LICENSE("Dual BSD/GPL");

/*
* some values that could have been configurable... but aren't (at least,
* not in the revision of the driver).
*/
#define KERNEL_SECTOR_SIZE 512
#define MAJOR_NUM 240 /* a free major number, according to devices.txt */
#define MINIBD_MINORS 16 /* number of possible partitions */
#define N_SECTORS 1024 /* number of sectors in our block device */
#define MINIBLOCK_SECTOR_SIZE 512 /* sector size */

/*
* request queue of our mini-block device.
*/
static struct request_queue *miniblock_queue;

/*
* Internal representation of our device.
*/
static struct miniblock_device {
        unsigned long size;
        spinlock_t lock;
        u8 *data;
        struct gendisk *gd;
};

static struct miniblock_device *Miniblock = NULL;

/*
* Real handling in our device of IO Requests.
*/
static void miniblock_transfer(struct miniblock_device *dev, struct request * req)
{
    unsigned long offset = blk_rq_pos(req) * MINIBLOCK_SECTOR_SIZE;
    unsigned long nbytes = blk_rq_cur_bytes(req);

    if ((offset + nbytes) > dev->size) {
        printk (KERN_NOTICE "minibd: working past the end of the block device (%ld %ld)\n", offset, nbytes);
        return;
    }

    if (rq_data_dir(req) == WRITE) {
        memcpy(dev->data + offset, req->buffer, nbytes);
    } else {
        memcpy(req->buffer, dev->data + offset, nbytes);
    }
}

/*
* driver handler of requests.
*/
static void miniblock_request(struct request_queue *q)
{
    struct request *req;

    req = blk_fetch_request(q);
    while (req != NULL) {

        if (req->cmd_type != REQ_TYPE_FS) {
            printk (KERN_NOTICE "Skip non-CMD request\n");
            blk_end_request_all(req, -EIO);
            continue;
        }

        miniblock_transfer(Miniblock, req);

    // if there are no more requests, _do not call_ blk_fetch_request
        if (!blk_end_request_cur(req, 0)) {
        req = blk_fetch_request(q);
    }
    }
}

/*
* custom Ioctls for our device.
*/
int miniblock_ioctl (struct inode *inode, struct file *filp,
                unsigned int cmd, unsigned long arg)
{
        long size;
        struct hd_geometry geo;

        switch(cmd) {
            case HDIO_GETGEO:
                size = Miniblock->size*(MINIBLOCK_SECTOR_SIZE/KERNEL_SECTOR_SIZE);
                geo.cylinders = (size & ~0x3f) >> 6;
                geo.heads = 4;
                geo.sectors = 16;
                geo.start = 4;
                if (copy_to_user((void *) arg, &geo, sizeof(geo)))
                        return -EFAULT;
                return 0;
    }

    return -ENOTTY;
}

/*
* operations associated for our device.
*/
static struct block_device_operations miniblock_ops = {
    .owner          = THIS_MODULE,
    .ioctl          = miniblock_ioctl
};

/*
* Module init.
*/
static int __init miniblock_init(void)
{
        static int ret;

        ret = register_blkdev(MAJOR_NUM, "minibd");
        if (ret < 0) {
                printk(KERN_WARNING "minibd: error assigning major number\n");
                return -EBUSY;
        }

        printk(KERN_DEBUG "minibd: Successfully registeres driver\n");

        Miniblock = kmalloc(sizeof(struct miniblock_device), GFP_KERNEL);
        if (Miniblock == NULL) {
                printk(KERN_WARNING "minidb: error assigning memory with kmalloc\n");
                goto out_unregister;
        }

        /*
        * Initializing our device.
        */
        memset(Miniblock, 0, sizeof(struct miniblock_device));
        Miniblock->size = N_SECTORS*MINIBLOCK_SECTOR_SIZE;
        Miniblock->data = vmalloc(Miniblock->size);
        if (Miniblock->data == NULL) {
                printk(KERN_WARNING "minidb: error assigning memory with vmalloc\n");
                kfree(Miniblock);
                goto out_unregister;
        }
        spin_lock_init(&Miniblock->lock);

        /*
        * Create our request queue (one block per device).
        */
        miniblock_queue = blk_init_queue(miniblock_request, &Miniblock->lock);
        if (miniblock_queue == NULL) {
                printk(KERN_WARNING "minibd: error on blk_init_queue\n");
                goto out_free;
        }

        blk_queue_logical_block_size(miniblock_queue, MINIBLOCK_SECTOR_SIZE);

        /*
        * Fill out our gendisk structure
        */
        Miniblock->gd = alloc_disk(MINIBD_MINORS);
        if (!Miniblock->gd) {
                printk(KERN_WARNING "minibd: error on alloc_disk\n");
                goto out_free;
        }

        Miniblock->gd->major = MAJOR_NUM;
        Miniblock->gd->first_minor = 0;
        Miniblock->gd->fops = &miniblock_ops;
        Miniblock->gd->private_data = Miniblock;
        snprintf(Miniblock->gd->disk_name, 10, "%s", "minibd0");
        set_capacity(Miniblock->gd, N_SECTORS*(MINIBLOCK_SECTOR_SIZE/KERNEL_SECTOR_SIZE));
        Miniblock->gd->queue = miniblock_queue;

        add_disk(Miniblock->gd);

        return 0;

out_free:
        vfree(Miniblock->data);
        kfree(Miniblock);
out_unregister:
        unregister_blkdev(MAJOR_NUM, "minibd");

        return -ENOMEM;
}

/*
* module removal
*/
static void __exit miniblock_exit(void)
{
        del_gendisk(Miniblock->gd);
        put_disk(Miniblock->gd);

        unregister_blkdev(MAJOR_NUM, "minibd");
        blk_cleanup_queue(miniblock_queue);

        vfree(Miniblock->data);
        kfree(Miniblock);

        printk(KERN_DEBUG "minibd: driver removed successfully\n");
}

module_init(miniblock_init);
module_exit(miniblock_exit);

My journey updating to natty

2011-04-18T22:25:00.000-07:00

Updating to a new version of ubuntu always has its glitches.... I hate doing clean installs (this is not Windows after all, right? So we should be able to do updates while working on the box) so I always do dist-upgrades in order to move to a new release of ubuntu (and don't try to change my mind... I'm stubborn and won't do it).

In my case, I wanted to update because I was having a small glitch with my intel graphics related to refreshing of the screen on my Samsung N150P plus netbook. Sometimes I had to switch to a different virtual desktop and come back in order to force the refresh of the whole screen to see things right... and I was tired of it.

I'm using kubuntu, anyway... so here's the journey so far.

First, going to /etc/apt/ and edit sources.list in order to change all "maverick"s in the file for "natty" (as a matter of fact, this is not what I do but the results are exactly that).

Then, performing the actual upgrade (you are on your own if you choose to do it this way, so no whining is accepted if something breaks):

sudo apt-get update

sudo apt-get dist-upgrade -y --force-yes -f

Give it time to download the stuff it has to download (about 800+ MBs for me), then answer a few question for a few packages, then the upgrade is finished (by the way... I normally had to run the dist-upgade a couple of times in order to get all packages installed... there was always a glitch... but not this time... it was flawless).

And here is where things get tricky. I reboot my computer and it fails to boot with the natty kernel with a kernel panic. Damn... it's never going to give itself away easily, is it? Aparently it doesn't understand where the root partition is. So I start booting with the maverick kernel that's still installed. Fortunately it does boot and I'm able to start kde normally..... except that graphics are a MESS (in order to keep the article PG-13). I had to disable desktop effects in order to have something usable. Ugh! And I thought natty would get rid of the refresh problem.

So I file a bug to complain about it (come on, guys! Natty is almost out there and the graphics on my netbook are awful!)... and when I'm about to finish the submission process, it hits me like a truck: old kernel. :-S I better start the new kernel to see if the graphics are better there. I file the bug anywhere with a comment at the end.

And then a new chapter of education begings. After a few attempts of trying to solve the problem by hand, I end up running update-grub to see what happens... and then I see that there's an error when trying to generate a new grub.cfg file:

$ sudo update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.38-8-generic
Found initrd image: /boot/initrd.img-2.6.38-8-generic
Found linux image: /boot/vmlinuz-2.6.35-28-generic
Found initrd image: /boot/initrd.img-2.6.35-28-generic
Found memtest86+ image: /boot/memtest86+.bin
error: syntax error.
error: Incorrect command.
error: syntax error.
error: line no: 143
Syntax errors are detected in generated GRUB config file.
Ensure that there are no errors in /etc/default/grub
and /etc/grub.d/* files or please file a bug report with
/boot/grub/grub.cfg.new file attached.
done

What do we have in the new broken grub.cfg file (grub.cfg.new) around line 143?

submenu "Xen 3.3" {
}

Well, well.... who knows why this is... but this xen stuff is going out. So I chmod -x /etc/grub.d/20_linux_xen and try again:

$ sudo update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.38-8-generic
Found initrd image: /boot/initrd.img-2.6.38-8-generic
Found linux image: /boot/vmlinuz-2.6.35-28-generic
Found initrd image: /boot/initrd.img-2.6.35-28-generic
Found memtest86+ image: /boot/memtest86+.bin
done

Much better now.... Now I'm able to reboot with the natty kernel and we are fine. And let me tell you that graphics are much better now... I have noticed a couple of things so far (I have to restart squid3 after I reboot in order to make it work, when the screensaver starts, sometimes it kills the whole kde session) but I'll look around to see if updates arrive that solve this problems or if there are easy fixes for them (besides not using them, of course).

Internet connection through (GSM) USB Modem with wvdial

2011-04-14T20:09:00.000-07:00

Hi!

Just tonight I'm having problems to start my internet connection with the NetworkManager on kubuntu maverick. The connection has been working like a charm these days so I'd think it's some kind of hiccups.

I had taken the time to install wvdial on my box hoping I wasn't going to need it... normally console based applications are much more verbose on the kind of problems they are facing to do things so they are a very good fallback method when you have problems and so given the current situation, I'll give wvdial a shot.

So, here's the recipe for it. First, set up a simple text file with the configuration you will be using.. it depends on the carrier you are using. I have COMCEL (in colombia) and so this is the content of the file:

[Dialer Defaults]
        Modem = /dev/ttyUSB0
        Phone = *99#
        Username = COMCELWEB
        Password = COMCELWEB

Phone, username and password depend on your carrier but it's fairly simple, right? Say the file is called comcel.txt, then call wvdial supplying this file name as the config file:

$ wvdial -C comcel.txt

A while later, you will be connected.... this is the output for me:

--> WvDial: Internet dialer version 1.60
--> Cannot get information for serial port.
--> Initializing modem.
--> Sending: ATZ
OK
--> Modem initialized.
--> Sending: ATDT*99#
--> Waiting for carrier.
ATDT*99#
CONNECT
--> Carrier detected.  Waiting for prompt.
--> Don't know what to do!  Starting pppd and hoping for the best.
--> Starting pppd at Thu Apr 14 21:46:29 2011
--> Pid of pppd: 2594
--> Using interface ppp0
--> local  IP address x.x.x.x
--> remote IP address y.y.y.y
--> primary   DNS address z.z.z.z
--> secondary DNS address w.w.w.w

And that's how I made this post. Perhaps I could set up a little DB with the configuration in different carriers. Do you use wvdial to set up internet connection through a usb modem (through GSM)? Want to give it up? I'll update this post.

Is Microsoft trying to equate selling computers without Windows to software piracy as a new world policy?

2011-03-26T07:00:00.000-07:00

Hi!

Very recently (as recently as 23rd of March) there was a small event in Mexico. An independent computer builder and a Microsoff legal representative had a meeting at the Legal Direction of Mexico's National Institute of Author's Rights. Apparently Microsoft wanted to make a statement specifying that they could take any legal action Microsoft considered pertinent given the builder's lack of a Microsoft certificate of authenticity or original license included along with a computer built/sold by the independent builder. The builder states that given that they sell their computers with Free Software instead of Windows, the software has licenses and that Microsoft doesn't have anything to complain about given that they don't own copyrights for said software.

If it had been an isolated event, I'd have been tepted to consider this as just a case of a legal/marketing staff messing up but given Microsoft's attempts to equate selling computers without operating system to piracy in other parts of the world, I'm starting to wonder if this is a deliverate attempt to try to equate both in the public mind to spread a little FUD around Free Software?

I'll try to get in touch with the independent builder to see what they have to say. Mary-Jo Foley, is it possible that you take care of Microsoft to see what they have to say about it?

My (first) 10 years of linux usage

2011-03-12T16:16:00.000-08:00

Hi!

Just about these days it's my 10th anniversary as a GNU/Linux user and it all started, irony of ironies, as a consequence of BSA's activities (or so the gossip said at the time). I was doing my internship at a small Java development company in Mérida, Venezuela, a rather small city but with a promising future (at the time) in the technology field. Those days BSA was pretty active arriving at businesses and asking for licenses of which, in what is pretty normal in Latinamerica, businesses didn't have any. Material raided plus fines, etc etc. You know the story.

Given that this small office had licensing problems of their own, it was decided that some of the computers at the office had to be switched to GNU/Linux.... so a few days later I arrived at the office and my computer had been switched to Mandrake. And let me tell you something: it was hard! Fortunately I was working with Francisco Andrades, who we used to call JGuru and not only was he good at java but on GNU/Linux as well. So I bothered him as much as I could to try to learn what I could. A few months went by and I headed back to my beloved Maracaibo to finish my engineering degree stuff.

I didn't install GNU/Linux on my computer for a few months... but I certainly had the will to do it. So when I wasn't doing anything important on my personal computer I installed Mandrake on it (about 6 months later) to try to get used to it. First months I had to force myself into using it... really. When there was something I couldn't do on GNU/Linux, I went to Windows and when I finished it, I headed back to Mandrake. In the beginning, I switched often but as months went by, it was less frequent... eventually I just didn't do it anymore.

From my first experience on GNU/Linux in Mérida, I remember very clearly the sensation of not being under control of the computer (unlike another popular OS for end users were one is at the mercy of it). I felt the power was there, at my fingertips and I just didn't know how to take advantage of it... but exploring the power would take time... and eventually I did learn how to use it.

In these ten years I have learned many many things, hacked a broken RAID to get back from it plus a lot more.

Bottomline: BSA, I want to thank you for putting me on this track... I'm sure that's not the desired outcome you have in mind when you go after license violators everywhere. I just hope more people follow suit.

International Free Software Community has been proposed for Príncipe de Asturias Prize

2011-02-18T11:45:00.000-08:00

Hi!

I just learned that CENATIC has proposed the International Free Software Community as a whole as candidates for the Príncipe de Asturias Prize.

There's a web page where people can support this proposal. In it, it can be read (my personal translation) that:

"The International Free Software Community is the group of people and organizations from around the world that contribute to free software through a collaborative and open model based on freedom, sharing of knowledge, collaboration, meritocracy and respect for authors' rights with actions like development, disclosure, promotion, documentation, testing, organization, support and marketing of free software. Free Software has allowed generalization of knowledge and access to technology worldwide getting rid of economical, social, cultural, and language barriers".

It'd be nice if you provided your support for this proposal. On the left side of the page, there's a blue square bellow a text that says "Apoya la candidatura". The boxes are:

- Name
- Lastname
- ID Card (???)
- Entity (organization, I suppose)
- Country
- email (required)
- Brief comment
- Then a checkbox specifying that you have read the legal warning.
- Submission button.

Hope you find it worthy of your support. Remember, it's here.

It took a little more than a year.... but Novell crumbled nevertheless

2011-01-24T10:30:00.000-08:00

Do you remember when you opened your browser to get the latest news on Nov 3rd 2006? I do. I just couldn't believed my eyes when I learned that Novell had signed this deal with Microsoft..... and if that's not enough, how about bad mouth developers of FLOSS everywhere (the same guys who largely develop the product Novell was trying to get money from) saying that if you (we... I still do a little FLOSS development as personal side projects) didn't want to get into trouble you better do your work for free? Ain't that lovely? As a summary, it was a very deceptive turn of events.

I remember back then that someone from Red Hat made a comment somewhere (it's not a press release, I think.... I checked on their site for the press releases around the time) that they felt comfortable with the move. That within a year Novell would have been zapped into oblivion as each and everyone of the previous partners Microsoft had gone to bed with had.

I found this article (from Dan Lyon's) while trying to find the quote I'm telling you about and it's basically the same thing. Well... it took a little more that one year but Novell did crumble after all.

I wonder if there are other companies out there that are still willing to change their luck with a Microsoft partnership.

PS In the Dan Lyon's article, there's a note for Red Hat: When companies start talking about Microsoft "validating" their market, they're usually about to be validated out of existence. I say, it sounds like it's going to take a little more than RH talking about Microsoft validating their market in order to throw the company down.

Stop complaining about Unity and suck it up!

2011-01-04T13:49:00.000-08:00

It's been a couple of months since Mark Shuttleworth dropped the bomb that Canonical intends to use Unity instead of the GNOME shell as the default shell in Ubuntu.... and I'm still reading about how the GNOME community could be torn apart by this decision.

Have these same people sat down to think how the people involved with KDE feel about Canonical's push for GNOME instead of KDE in Ubuntu? It's no secret that Kubuntu is not the best KDE-based distribution out there (I'll give you up that I'm a Kubuntu user myself, I use the PPA for KDE for Kubuntu and I don't use GNOME that much). Then why so much hair pulling because of Canonical's push for Unity instead of the GNOME shell? These people might just as well suck it up, get off the ground, remove the dust from their shoulders and continue working on their beloved environment.... you know, the way the developers of KDE do.

Testing raidpycovery through mdadm

2010-11-27T13:52:00.000-08:00

Hi!

I'm still working on polishing raidpycovery. I started doing real experiments today to recover data from broken RAID5s. In order to do it, I used linux' md. It allowed me to study md a little bit while testing raidpycovery.

So, let's get our hands dirty.

First, let's create a directory where we will work, for example:

$ mkdir raid5
$ cd raid5

Now let's create the separate images where we will create our RAID5. I will use 4 disks 10 MBs each:

$ for i in 0 1 2 3; do dd if=/dev/zero of=disk$i bs=1M count=10; done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0584047 s, 180 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.100258 s, 105 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0691083 s, 152 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0508324 s, 206 MB/s
$

Now we have four empty files that we will feed into md to create our RAID to do our tests.

First, we will "loop" them so that we can use then with md (I don't know if this is really needed, but I'll do it just in case). As I'm working on a live USB, I'll have to link from /dev/loop2 and on. Check the used loop devices with losetup -a (as root... or with sudo):

$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)
$

Now, I loop the files:
$ sudo losetup /dev/loop2 disk0
$ sudo losetup /dev/loop3 disk1
$ sudo losetup /dev/loop4 disk2
$ sudo losetup /dev/loop5 disk3
$ sudo losetup -a
/dev/loop0: [0811]:31 (/cdrom/casper/filesystem.squashfs)
/dev/loop1: [0811]:38 (/casper-rw-backing/casper-rw)
/dev/loop2: [000f]:27898 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk0)
/dev/loop3: [000f]:27899 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk1)
/dev/loop4: [000f]:27900 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk2)
/dev/loop5: [000f]:27901 (/home/ubuntu/raid5/raidpycovery/bin/raid5/disk3)
$

Great. Now we can use the loop devices to create our RAID device:

$sudo mdadm --create /dev/md0 -l 5 -p ls -n 4 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5
mdadm: array /dev/md0 started.
$

If you read the man page of mdadm you will see that by default the algorithm for the RAID will be left sync and a default stripe/chunk size of 64k. We will need that data later on.

Now, let's format it so we can use it as a normal partition:

$ sudo mkfs.ext3 /dev/md0
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=64 blocks, Stripe width=192 blocks
7648 inodes, 30528 blocks
1526 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=31457280
4 block groups
8192 blocks per group, 8192 fragments per group
1912 inodes per group
Superblock backups stored on blocks:
8193, 24577

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
$

Now, we can mount our just formated RAID device:
$ sudo mount /dev/md0 /mnt/tmp
$ sudo mount
.
.
.
/dev/md0 on /mnt/tmp type ext3 (rw)
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
.
.
.
/dev/md0 29557 1400 26631 5% /mnt/tmp
$

There you go. A new partition with around 25 MBs of data available for mortal users. Now, let's copy some files into that partition:

$ sudo cp blah blah blah /mnt/tmp/

I'm using sudo to do the copy because right now that directory is owned by root.

After I copy the files I wanted, let's check their MD5s:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

Now, let's umount our RAID and stop it:

$ sudo umount /mnt/tmp
$ sudo mdadm --stop /dev/md0$ sudo umount /mnt/tmp/
mdadm: stopped /dev/md0
$

Great... now, let's test try to rebuild the RAID with the raidpycovery tools. I don't have the tools at the same directory, so I'll have to move and use relative names for the disks, keep that in mind:

$ ./Raid5Recovery.py 4 left async 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Asynchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes
$

Now, let's mount it and see if we can get any data from the recovered RAID:

$ sudo mount -o loop,ro wholedisk /mnt/tmp
mount: unknown filesystem type 'linux_raid_member'
$

Not good. From my tests, that's because there's md garbage at the end of the disks that the mount is seeing. I'll force the partition type then:

$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp
mount: wrong fs type, bad option, bad superblock on /dev/loop6,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
$

Gotcha! Did you notice that I resembled the RAID using left async? It has to be left sync, remember? Let's try again:

$ ./Raid5Recovery.py 4 left sync 65536 raid5/disk0 raid5/disk1 raid5/disk2 raid5/disk3 > wholedisk
Number of disks: 4
Algorithm: Left Synchronous
Chunk size: 65536 bytes
Skip 0 bytes from the begining of the files
Finished! Output size: 31457280 bytes
$ sudo mount -t ext3 -o loop,ro wholedisk /mnt/tmp

No complains. Great! Now let's see if we can see the data in the recovered RAID:

$ md5sum /mnt/tmp/*
a27ebcacc64644dba00936abc758486e /mnt/tmp/IMSLP32718-PMLP01458-Beethoven_Sonaten_Piano_Band1_Peters_9452_14_Op27_No2_1200dpi.pdf
e68fabdcda296ef4a76d834a11a6f1df /mnt/tmp/IMSLP44764-PMLP48640-Mahler-Sym9.TimpPerc.pdf
md5sum: /mnt/tmp/lost+found: Permission denied
d7bfe06473430aad5ca0025598111556 /mnt/tmp/putty.log
670536c55ae9c77b04c85f98459c0cd8 /mnt/tmp/Resume Edmundo Carmona.pdf
8727e8ff88739feca15eb82b4d9cb09b /mnt/tmp/Titulo Ingenieria.png

And there you are. Everything is right there. Using this method today I discovered I was not rebuilding chunks from missing images correctly and that one of the right-handed algorithms has to be corrected. Let's see what I find.

As homework, try running the recovery script providing a missing disk (write none for it) and see if it works.

Hope you liked the read or find it useful.

Update: Have corrected the right algorithm issue. Make sure you get the latest stable release if you are gonna use it:

bzr branch -r tag:2.00.02 lp:~eantoranz/+junk/raidpycovery

Broken RAID5 you said? Don't use Java anymore. Go with python instead!

2010-11-21T05:39:00.000-08:00

So, you have a broken RAID5 and found the article I wrote for the Free Software Magazine 5 years ago about it and would like to give it a shot.

First off: Good luck! You are gonna need it.

Second: When I solved that problem I didn't have any experience in python whatsoever plus Oracle hadn't bought Sun, sued Google, etc.

During the course of these 5 years I have received some requests to help people out with their situation and it's been a very nice experience to hack on those devices here or there to get the data back. I have received a little money by paypal and even once got a box of whine bottle from Italy (thanks, Marco!) as payments.

Every single time I've had to help someone with it, I always get the question of how to run the .java classes I made. And I even took the task of compiling the .class files to then send them to the person in need.

Given the gripes over oracle/java recently I have decided to translate the library to python for people who want to use it so they don't have to go around trying to learn how to compile/run things in Java (at least in GNU/Linux).

The project is here:

https://code.launchpad.net/~eantoranz/+junk/raidpycovery

You can get it very easily by using bazaar:

bzr branch -r tag:2.00.00 lp:~eantoranz/+junk/raidpycovery

Or you can just download the files from that same version and use them:
Chunk.py
Raid5Recovery.py
RaidImageRecovery.py
Let's not forget the readme where you get to see how to use it:
readme.txt

There's more stuff in the project (some chunks from the RAID I recovered at the Hospital, mostly... but you never know!) so I'd suggest to use bazaar but....

Warning/disclaimer/whatever: I haven't tested it on a real environment so there might be some glitches (but I did a translation mostly so it should be correct overall).

In case you want to get the original java code:

bzr branch -r tag:1.00.00 lp:~eantoranz/+junk/raidpycovery

Or start browsing here to get the files.

PS Sorry, Larry.... it just had to be done. A little weight off my jumping-off-java shoulders. :-)

PS2: Donations, donations! You found the project useful? How about giving a little contribution? 0,10 US$ / Gb recovered? Sounds like fair to me. Specially considering how fast HDs sizes are growing compared against the inflation.

My thoughts on the switch to wayland

2010-11-15T12:27:00.000-08:00

Like anybody cares for what I think, right? Anyway... I made a comment in one of these wayland-related news at linuxtoday and Carla Shroder took the time to ask me:

"...why all that extra complexity to go back to where we were in the first place?"

That's a fair question. By the way, my love to you, Carla. You haven't sent me a comment about pythogoras that I asked you for but I still love you and care for you. :-D

Anyways... the thing is this:

Wayland is going to replace X in Ubuntu and Fedora. That's quite a remarkable statement to make. We are talking about X, the same X that has been there since I started using GNU/Linux about 9 years ago and even many years before that cosmic event, if you will.

But all the GUI applications for UNIX at the time use X, so that means it's going to be a really troublesome change, isn't it? Also, there are really cool and extremely useful features in X like Network Transparency (think of export DISPLAY=blablah:0.0 or ssh -X, pleople) that would disappear from the face of the earth (I'm wondering what I would show my Windows-loving friends now when I start showing them the wonders of GNU/Linux if I don't have network transparency. I'll have to think about that).

As I was saying, I'll (try to) tackle those two questions.

First off, this means a major reworking to get all applications to render on wayland, isn't it? I think it can be solved by hacking the lower layer APIs like Qt or GTK or even at a lower level like wrapping the Wayland API for clients around the X API (update: I think it's viceversa... but you get the concept, don't you?). Then not much work would have to be done on the higher layers (or so the fairy tale theories of software development say). That means applications won't be hurt that much. I'm not implying that it's going to be easy but not much work will have to be required from us mere mortals in that case. Also it means that the switch could be done overnight to start using Wayland.

Then there are things built inside X that would be gone once Wayland makes its debut, right (to be read as "Network Transparency")? What I've read around (won't provide any links but it's very logical) is that X could still use Wayland as just another graphics interface and then you will have X running on top of Wayland and (tadaaaaaaaaaaaaaaaaa) you would get Network Transparency back faster than you can say "mi moto alpina derrapante" (that's a nice joke in spanish... people who are learning spanish should give me a call so that I tell you about it and have a laugh).

But then Carla makes the final killing point: Is all this hassle really worth it?

In all honesty, I don't know. It's still too early in the game to make up my mind weather it will be worth it or not. But if X's complexity/overhead can be cut down and get a faster/lighter/snappier/sexier/whatever-you-consider-important-er environment to spend your time onto, then it could be worth it in the end... specially if (as I said) it won't require much work but on the lower layer APIs.

Now I don't know if switching to Wayland will be really worth it, but was it reading this article worth it after all? I truly hope so.

Why don't manufactures get together and get Microsoft out of the loop?

2010-11-10T00:46:00.001-08:00

Here comes a crazy thought:

Why don't computer manufactures get in cahoots and get Microsoft out of the demi-god loop? I mean, computer manufactures are struggling because their profit out of every sell are very small... in the meantime, Microsoft is laughing its ass off while making millions upon millions over their sells of Windows 7 (or so they say).

If any of the manufacturers dared getting out of the Microsoft way (you know, let's push Linux a bit harder, let's not say we recommend Windows 7, etc, etc), Microsoft would just need to get the price of Windows 7 licenses a little higher to force the manufacturer into red numbers (if it isn't already, of course) and teach the bully a lesson, right? We have all seen the Halloween Documents and know of Microsoft's business practices, so no surprises there. Microsoft would still have all the other manufactures lined up with lower prices to pay for Windows 7, the rebel manufacturer would have to get their prices a little higher because of the increase in the price of Windows 7 it has to pay now making it more difficult for them to compete against the other manufacturers and Microsoft gets to keep the upper hand in the end. Sounds like a subtle kind of trust, doesn't it? By the way, am I the only one who has the phrase "Divide and conquer" pounding in his head?

But what would happen if the biggest manufactures would get together (say, in a secret deal) and fight Microsoft all at the same time? Is that possible? Is that legal? Would that be enough to get Microsoft out of its ways and start really competing instead of been shoved down everybody's throat? If it is legal (I honestly don't know), then why don't they do it and let Microsoft laugh (very loudly, by the way) at them (and the rest of us)?

Now... where did that smart-ass laugh I'm hearing in surround sound come from? Seattle?

Introducing Pythogoras.... or what is musical tuning?

2010-10-02T15:48:00.000-07:00

Music is everywhere.

A major part of culture is driven by it. Dancing without music? Movies without music? Going to a restaurant and having no background music? You get the idea, right? It's, just like John Milton said in "The Devil's Advocate", EVERYWHERE!

In terms of western music (and that's what I intend to talk about in the article, keep that in mind), a loooot of music theory has been created and published for many centuries. Yet much of it based on experience.

And there's a major part of it where I believe a lot of research is yet to be made: Tuning. Have you ever heard someone play with an instrument that's "out of tune"? But then, this begs the question: What is "Tuning"? Well... the answer can be anything but simple. Go take a look at wikipedia to make yourself an idea.

The fact is that for us humans, or rather, our human ears, being in tune will mean using the simplest ratios possible for notes/chords. But that is in direct contrast with tuning instruments... specially keyboards..... let me explain myself a little bit better:

I play the flute and it's very simple to assume that whenever you play, say, an A, you will always get the same frequency that you used to tune the flute (be it 440, 441, whatever)... but it's not like that. The tuning of the flute can be adjusted a little bit "on the fly" by readjusting the angle of the wind that is coming out of your mouth towards the flute or by _turning_ the flute either forward or backward (backward makes the pitch a little bit lower). Even more, the flute will play the forte sounds sharper than piano ones and just in case this is not enough, higher notes will also produce higher pitches for each note... in other words, a flute player is _always_ tuning his instrument according to what's being played. Most instruments, like the flute, can be tuned on the fly: Violins, Violas, Cellos, Basses, Oboes, Clarinets, etc etc. Most of the instruments in the orchestra can do that... most notorious exception: The Piano. It's tuned by a "tuner" and will remain like that for years to come, no on the fly adjustments.

The thing is that there are many ways to tune notes, in other words, many Tuning Systems are available and each has advantages and disadvantages. Normally there are trade-offs between "tuning experience" and "practicality of tuning the instrument to different tonalities".

I believe a lot of research has to be done on the way we perceive being in tune, but where to start? Well, I'm a programmer (among other things) so I developed one application to _try_ to play music with different tuning systems (I'm sorry for the long introduction).

It's still very rough but I hope I can make it better in the following months (sponsoring is welcome, by the way).

So, instead of describing what it can or can't do, I want you to listen to three files I produced from the application. Here you have the Air on the G String by J. S. Bach played in three different tuning systems:
- Just
- Pythagorean
- Equal Temperament

Files are available both in MP3 format and OGG (which I prefer). If you'd like to get more info, don't hesitate to email me.

Oh, and by the way, the application is called "Pythogoras" and can be found here. It's released under the terms of the Affero GPLv3... and then again, keep in mind it's still veeeery rough.

Who dares using Windows in a critical environment? (updated)

2010-08-24T18:38:00.000-07:00

I just can't believe what my eyes just saw. You are telling me that a commercial plane (a real plane, the ones that take off from land at hundred of kilometers per hour carrying hundreds of people in them) had Windows running on one of its central processing nodes? You gotta be kidding me! Who dares doing something like that? Come on, people! We're talking about a life-or-death situation here, not the normal pop up that at its worst will nag me to hell asking me to date someone on the other side of the planet.

This begs the questions: Will there be consequences for the people involved in taking the decision to put Windows on board? Maybe consequences for Microsoft? Will there be prosecution for the guys who developed the malware for (unintended?) murder? Nice things to talk about in the following months, I guess.

Absolutely terrible.

Update: From what I've read about it, Windows is not used on the plane but on computers used to get information from the plane. Unfortunately it's still part of the causes that generated the accident which had an outcome of several lives lost. Why was Windows used in the first place?

So the patent mess started from Java and not .NET.. what an irony!

2010-08-21T12:20:00.000-07:00

You know how many times I complained about .NET for being backed up by Microsoft and so I distrusted it even if they had made pledges (under certain conditions) not to sue and having made it into an ECMA standard (portions of it)?

Not that I had put all my developing faith into Java, anyway, I've worked with PHP and Python during the last couple of years but it's certainly an irony seeing the patent mess to come from Java instead.

Will this mean there will be less development to Java? Will this push development to other languages like Python? Perhaps Go? I'm just wondering?

I guess ORACLE will lose a lot of FLOSS support after this. Will we start seeing people from ORACLE jump ship the way some did (like Samba's Jeremy Allison) when Novell signed off their agreement with Microsoft back in 2006?

I finally know when ARM netbooks will come out

2010-08-14T17:34:00.000-07:00

Hi!

Before I tell you when they will come out, let me tell you how I figured it out.

I'm used to having to face disappointments.... disappointment after disappointment. I want to have something? I just can't go out there seeking for it. No, no, no... that'd be the wrong approach. It'd never happen. I have to move slowly towards my goal. As one of my teachers told me in computer engineering classes when talking about the career: It's resistance race, not a speed one. Well, that's the approach I have to take with most things in life. And even then, when things happen the way I want, I worry that Murphy will somehow learn about how well things are going for me and mess up.

So, after waiting for over 1 year since I heard for the first time that ARM-based netbooks would come out, I'm still waiting... and I'm fed up of waiting already so I'm ready to give up on them and I'll end up buying an Atom-based one. Disappointment, see?

And so, when will they come out? A couple of days after I buy my netbook.... as it was supposed to be since day one. I better hurry and buy it so that they come out cause otherwise they will never come out.

RMS is liberal, not a libertarian

2010-07-31T08:40:00.000-07:00

Hello!

I was reading the latest responses by RMS to 25 very important questions he was asked recently. It's been linked from the two IT news site I read most often (OSnews, LinuxToday), so it was like unavoidable to read it.

While reading, I found a rather interesting statement by RMS: That seems to describe the viewpoint called "laissez-faire" or "Libertarian". Where business is concerned, I disagree with it very throughly, because I'm a Liberal, not a Libertarian.

I'm venezuelan, you see? And the things that are going on in Venezuela are despicable. I just can't understand how so may people can still follow a government so bad after 11 years just because 20 years ago a guy showed up on TV in a 30-second clip to say that he was guilty for a failed coup d'état (whereas all the other cells of their movement were successful in getting their tasks done...in other words, the only one who actually failed!). I don't support coup d'états, let's make it clear.

Venezuela's government has become a theocracy lately. Who's the god? Chávez? Not at all. The god in Venezuela is called 'Socialism' (old-fashioned very far left-wing socialism.. the one where the leader has to be protected from criticisms of all kinds and so on ). The government is certain that socialism is going to solve each and every problem faced by Venezuelans. But when will that happen? There's not such certainty on this point. It's such a shame to see this high-level venezuelan politicians say how socialism is the only way to solve x or y problem... it's offensive, really.

Sorry for the break and going back to RMS: In Venezuela the government pushed for FLOSS (and I'm thankful for that). And there's people (government followers) that believe that FLOSS is somehow related to 'Socialism'. I bet they'd go as far as to present RMS as one of the nowadays socialist heroes and the Free Software Movement as the the equivalent of Khmer Rough for IT. I'd also bet more than one guy out there is more than willing to paint RMS together with Fidel, Che and Marx (well, they have the guts to have Simón Bolívar and Jesus Christ in that crowd having the last supper, so no surprises) or RMS holding a rifle in his hands (to defend Revolution), just like it's been done with Jesus Christ in walls in Caracas. I just can't see how something where there's no central control (as FLOSS is) can be somehow related to Socialism (far left-wing, as I said). Perhaps the ways of the community that's associated to FLOSS could be somehow related to the ideal socialist community's behaviour, but Free Software per se? So it's always reassuring when RMS defines himself as a liberal. No more doubts about that anymore.

Perhaps Free Software Development model could be related to examples of almost pure liberal environments and used in scientific/social investigation. I think some investigations have already been made, right?

PS I don't have anything against socialism per se. It's a political philosophy and there's nothing wrong (if you ask me) with applying some of its principles in public policy (the ones that have been tried and have had a good result in different places). But when it's used as the excuse to apply bad public policies that have been tried in the past (and failed miserably... both in the past and in their actual instance), that won't leave a good trail for it in the future.

How to (easily) fool your host into thinking a name is mapped to a certain IP address

2010-07-20T09:33:00.001-07:00

Hi!

Very recently I've been involved in setting up a site that uses Google Maps API. When you want to use the API you have to create a key that is generated according to the name of the site but, as of now, the DNSs of the name we want to use are not pointing to our hosting. I could use the IP address of the host where the site is, right? Well, no. It's a shared hosting so requests have to be made by name.

In this case I have to fool my host into thinking that the name we want to use is mapped to an IP without going through public DNS resolution. I could set up a DNS service just to serve this name and forward everything else but it sounds like an overkill, doesn't it?

In GNU/Linux (and I'd dare to say in any POSIX compliant OS) there's a simpler way to do this:

/etc/hosts

In this file we can map names to IP addresses at will. So this thing I want to do could be performed by adding a single line to my /etc/hosts (not a real example... IP address and name are bogus):

85.23.54.87 mydomain.com

Then I could go to http://mydomain.com and I'd be able to see the site that uses Google Maps, even if the name is not "officially" public.

Notice this trick (surprise, surprise) also works in Windows (which is POSIX compliant, right?), only the file that has to be edited is System32\drivers\etc\hosts

Additional information:
man hosts
man nsswitch.conf

Is moving to IPv6 all that important, really?

2010-07-16T11:05:00.000-07:00

I'm reading this article where we are told how important it is moving to IPv6 (and I'm not saying it isn't), but I can't help but wonder... will enterprises (for example) really have to move to IPv6? Let's suppose you work in a not so large enterprise where basically the only internet need is to reach internet as a client (which I guess would be most of the needs of a large portion of internet users). Does it make sense to move the whole infrastructure to IPv6? I don't think so. With having the proxy/nat server set up with IPv6 to gain internet access, the inner networking for said enterprise could remain using IPv4, or isn't it?

Well over half of the most reliable hosting companies run on Linux

2010-07-15T14:07:00.000-07:00

Hi!

I like to follow statistics on market share of browsers, os, web server and so on. They have to be taken with a big grain of salt for sure but they do give us a look at trends more or less accurate (or so I think).

One of the most interesting statistics I've been following are the ones presented by Netcraft regarding web server market share and OS used by hosting companies. The Web Server market share (Apache Vs IIS and so on) is the one you'll see more often talked about on the web. But the one about reliability is rarely (if ever) talked about... so I'll take a couple of lines to talk about latest statistics that correspond to June'10 puts it like this:

- Over two thirds (29 out of 42) of the most reliable hosting companies use Linux (would they use GNU along with it?)
- 14.2% use BSD (FreeBSD to be more precise)
- A little less than 10% use Windows
- 3 out of 42 are a big question mark

How about that? You think the numbers are accurate... or are they skewed?

PS Did you see the uptime chart for microsoft.com? It's kind of shameful, you know? No wonder it's used by less than 10% of the most reliable hosting providers. :-)

Definition of XOR for multiple variables.... or my little contribution to theory

2010-06-27T09:32:00.000-07:00

I was remembering one of my classes from computer engineering when I was just starting it at the university. The definition of the XOR operator.

One of the ways to define XOR is:

A ⊕ B = ~A . B + A . ~B

In other words, one and only one of the variables is true. Great... couldn't be simpler.

When you consider operators like and and or, you find that you can break these operations into smaller pieces when you are considering more than two variable... like this:

A + B + C = ( A + B ) + C = A + ( B + C )

A . B . C = ( A . B ) . C = A . ( B . C )

All is fine and dandy.... but then let's say we find a case of exclusive or where all the variables (more than two) have to be considered at the same time.

Say you are considering a definition where you have three variables where one and only a single one of them is true to consider the result to be true. If we try the same logic that was applied to and or or, you would end up with this:

A ⊕ B ⊕ C = A ⊕ ( B ⊕ C )

If none of them is true, it will be false. If only one of the variables is true, it will be true, if two of them are true, it will be false but (and here's the problem) if all of them are true, it will be true.... and according to the problem definition, it will be wrong cause one and only one of them had to be true for the result to be true in the end. In other words, all operators have to be considered at the same time to say whether the result is true or not.

For this case I think there should be a different operator called "MultiXOR" or something like that. The problem with this operator is that there's no easy way to break it into smaller pieces. Here's a shot using a recursive definition:

x1 m⊕ x2 m⊕ x3 ... m⊕ xn = x1 . ~( x2 + x3 ... + xn ) + ~x1 . ( x2 m⊕ x3 ... m⊕ xn )

⊕m being the MultiXOR operator.

There you have it.

GNU/Linux: Flexibility is the name of the game

2010-05-09T12:14:00.001-07:00

Some things are meant to happen:

- We will all eventually die
- Windows gets filled with viruses over time (if we don't consider Windows a virus in itself, of course)
- Chavez will rather be talking crap on TV than taking care of doing a real government that will make people happier... or, at least, safer (in the sense of personal safety).

You get the idea... those are facts of nature. But there are others where the planets align in such a way that you are able to sense when they will happen a couple of seconds before they do happen. I had one of those last wednesday.

I was working on a project that had required that I left Bogotá to another city for a week... I was working at a client's installation and was completely isolated from internet. I remember a thought came to my mind: I need to do a backup of all the things I have of the project and send it to my coworkers. 10 seconds later, my laptop was heading to a crash with the floor. Oops! Not good.

The working day was already closing so I had sent the computer into hibernation but the fall happened before the computer shutdown. I tried to restart it (Windows.... but the GNU/Linux part is coming, don't worry... read on). It would fail to start because of a bad checksum of one of its dlls.

See, I'm forced to use Windows on my computer at work but I do must of my work on a Virtual Machine that I cheerfully use on VirtualBox. In the virtual machine is where I keep must of the information of the projects.

Thursday was my last day outside of Bogotá (I was traveling back on friday) so I had to finish all of the tasks I had been assigned to do in 24 hours... with my computer brain-dead, it doesn't look like I'm gonna get a congratulation at the end from my boss.

I always keep a LiveCD with me just in case things like this happen (on other people's computer, normally). I fire it up and the computer responds normally... so I'm able to work on it... even if I don't have all the information about the project. I check and see that the D partition (where the data is) is usable... at least I get to see many files I'm working with and md5sum them.

I head back to the hotel cause I gas leaving already and try to connect to the wireless... but I just can't (It's Kubuntu Jaunty, not lucid.... hope it's better with lucid now). I ask for permission to connect to the router through UTP which I'm granted so I head to the lobby and mails start going back and forth about the support questions I have regarding the project. One of my coworkers through mail tells me to backup the files... damn, why didn't I think of that? I had brought with me a pen drive that came with the hardware of the project so I copy many of its files but not all of them as they are inside a VirtualBox virtual hard drive (switches configurations in time... thanks to version control, plus many other things). I stay there until about 3 or 4 AM and head to bed to try to get some sleep.

8 AM... Here's when the real hacking begins. I start wpa_supplicant by hand to try to see what's going on with the wireless connection. I see a message about non-wpa networks not being allowed to connect through wpa_supplicant. I think we can try something different in this case: /etc/network/interfaces (on debian based distros). I edit it to include the wireless I'm trying to connnect to and its key... something like:

iface eth1 inet dhcp
wireless-essid this is the wireless name
wireless-key this is the key

Save it and try to connect: sudo ifup eth1

I get connected in a couple of seconds... oh, well.... :-S One less problem. Now the data of the project. I have the home of the virtual machine in partition D of the HD (as a matter of fact, I use the whole virtual HD as the home... there's no partition table... the joys of using GNU/Linux) so I need to be able to start VirtualBox so that I get the information of the project out of the virtual machine. I install openssh-server and VirtualBox from repos. I try to start VirtualBox with the home virtual HD and the LiveCD I'm using.... the boot process begins but VirtualBox dies because of memory issues (I have 1 Gg of RAM and running Kubuntu plus virtual box... I knew it easn't going to hold water). I download DSL and try to run the virtual machine with just 64 MBs of ram in "single" mode (dsl single on the boot menu).... and I'm up in a couple of minutes. I mount the home partition on /mnt inside DSL:

mount /dev/sda /mnt

See I didn't use a device name that includes a partition number? That's because, as I said, I'm using the hole HD as a partition instead of using a real partition of the HD. I tar the project and, as you should rememer, I had installed the openssh-server on the liveCD session, right? I sftp it out of the virtual machine inside the liveCD session... and now I'm able to forget about the HD cause I won't need it anymore. Put the project in the Pen Drive and the liveCD + pen drive becomes my new work environment and I move on to do my final work day on site (I did have time to finish all my tasks thanks to GNU/Linux... as usual). Get the congratulations from my boss and go to have a good nigh sleep which I'm already m,issing.

Conclusion: Flexibility is the name of the game... that's why (among other things) I enjoy using GNU/Linux.

PS for those of you wondering how I was able to install software on the liveCD session once I was on-site with no internet connection, should take a look at this article.

Panama is on the road to MS hell

2010-05-01T23:04:00.000-07:00

If South Korea is an example of what Panama is attempting to do by following the Microsoft guide on how a country can jump into technological advancement, then things sound like it's going to be a bumpy road to get there.

Apparently Panama's President has signed a treaty with Microsoft to push technological development in the centralamerican country.

And what's the recipe?
- A computer for each and every one of the students of Panama (I'm wondering if they will get to use other things what were not Windows and other Microsoft stuff)
- You will be able to do all things government related on-line (wanna bet that it will be IE-only?)
- Plus other things.

Feels so nice to see that governments still see Microsoft as the cornerstone of technology (or even worse, sell out).... what a shame!

Too depressing, really to keep on writing about.

How does GPL licensing affect projects that don't involve linking/compiling?

2010-04-10T08:58:00.000-07:00

This is a question that I've been trying to figure out for a while already.

I'm working on a php project that I will distribute (or whatever word you want to use) under the terms of the Affero GPLv3.

Now I included a piece of code (PHP Simple HTML DOM Parser) that's not under that license (MIT kind of license, apparently) and I started wondering how GPL code gets affected by the code I included... I mean, it's already settled matter that you become a GPL sinner (so to speak) not if you choose to use GPL code mixed with other (proprietary, for example) code.... the sin happens at the moment of redistributing (or propagating or reciprocating or whatever it's called now) the binaries resulting from linking GPL code with code licensed/released under other terms.

But how is it handled when there's no linking? At least no linking to redistribute... think of projects made in PHP or Python, where no binaries are released.

SCO sends patches to the linux kernel

2010-04-01T07:58:00.000-07:00

In a move that will surely leave many in the FLOSS community shocked, SCO, the company that just three days ago was claiming that they and not Novell owned UNIX's copyright and that the Linux community were a bunch of free loaders, after getting their claims rejected in a court of law have decided, in a classical "in you can't beat them, join them" fashion, to start sending patches to the linux kernel.

Here's a sample taken from their first submission (from init/main.c):

4c4,7
< * Copyright (C) 1991, 1992 Linus Torvalds
---
> * Copyright (C) 2010 SCO
> *
> * This file is released by SCO under terms that forbid it to be part of any
> * project released under the GPL (as the GPL is famous for being inconstitutional)

Who would have believed it?

Networking is a little more than IPs and netmasks

2010-03-27T11:04:00.000-07:00

Hi!

Case one

Very recently I was asking this questions (which is still open) at www.linuxquestions.org (the first place I hit when I have a question regarding linux or gnu, by the way) and took a brief look at the questions open on the networking forum and I hit this beauty.

It's a guy who has set up DNAT on netfilter to forward packets that are sent to one host to another server that does the real work. Think of it as a proxy. In his example, he wanted to forward packets that arrive at his host on port 3306 to port 3197 on another host (let's use IP a.a.a.a). So, he set up a simple rule on (nat) PREROUTING:

$ iptables -t nat -A PREROUTING -p tcp --dport 3306 -j DNAT --to a.a.a.a:3197

What this rule is doing is telling the kernel to change the destination IP address of any packet that arrives at his host through any network interface to IP address a.a.a.a (reachable from his server, maybe not from the host that originated said packages) and the destination port to 3197 (the port where the real service is working on the real server). When the routing decision is made on those packages a while later the destination IP address will be a.a.a.a and so the packets are sent to the real server. Source address/port of those packages remains the same (unless a little more natting is done, of course). Nice and dandy.

Then, when the packets arrive at server a.a.a.a port 3197 the response will be sent to the originating host/port and the "networking cycle is complete". A word of caution: this works if the packets that are sent back from the real server go through the same host that is doing the natting. If the real server is sending the packets to the originating host through another host, the trick is broken as packets arriving from a.a.a.a:3197 to the originating host don't match the IP:port he sent traffic to, so the connection is not established. This can be solved by SNATting this same traffic on the server that does that DNAT before the traffic is sent to the real server (making sure traffic will come back through it on the way back).

He tests it and it's working. Traffic is reaching the real server and going back to clients.

He then tried to replicate that same behavior but using localhost instead. So he added a rule that looks very much the same on OUTPUT, like this:

$ iptables - t nat -A OUTPUT -p tcp --dport 3306 -i lo -j DNAT --to a.a.a.a:3197

It should make it, shouldn't it? Try telnet to localhost port 3306 and nothing happens. No connection is established. Doesn't work. But why? Using a sniffer it's seen that when the -t nat OUTPUT rule is not set, traffic to localhost port 3306 is moving through interface lo, nothing wrong with that, but when the rule is set up again, traffic gets lost. It doesn't go through lo or any other network address.... so the IP stack is descarding it. Weird. Counter for the -t nat OUTPUT rule is increasing so it's doing its job as required... still, no traffic.

So what's going on? Let's think of what's going on with the traffic. When it reaches -t nat OUTPUT, this packets have source address 127.0.0.1 port whatever, destination address 127.0.0.1 port 3306. Then, after the rule is applied, source address is 127.0.0.1 port whatever, destination host is a.a.a.a port 3197. As the packet is changed in nat, a second routing decision is made on it. As the destination host is a.a.a.a, traffic should be sent to the real server but IP source address is 127.0.0.1. If there were a DNAT rule on POSTROUTING, it should take care of this problem (there was a MASQUERADE rule in place, so no problem). The problem (which is a little buried in the networking stack of linux) is that by the time the second routing decision is made, the source IP address is not consistent. Let me show you my routing table:

$ ip route show
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 1
169.254.0.0/16 dev eth0 scope link metric 1000
default via 10.0.2.2 dev eth0 proto static

Nothing about 127.0.0.1. Why is that? It's because this is set up at another routing table (linux supports multiple routing tables, in case you didn't know). You can see the routing tables available by taking a look at file /etc/iproute2/rt_tables. I have default, main and local. Let's take a look at them:

$ ip route show table default
$ ip route show table main
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 1
169.254.0.0/16 dev eth0 scope link metric 1000
default via 10.0.2.2 dev eth0 proto static
$ ip route show table local
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 10.0.2.0 dev eth0 proto kernel scope link src 10.0.2.15
local 10.0.2.15 dev eth0 proto kernel scope host src 10.0.2.15
broadcast 10.0.2.255 dev eth0 proto kernel scope link src 10.0.2.15
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1

And this is where things start to make sense. If you see carefully, with src address 127.0.0.1, all routes there have a local scope, which means that they are not used outside of the scope of the actual host. In our case the dest address is a.a.a.a and with src address 127.0.0.1, it's impossible to route this traffic... so it gets dumped.

But it fails because we attempted on address 127.0.0.1, but if you tried to telnet to the ip address of your intranet address instead, the test would be successful (our traffic will go through interface lo, the kernel can figure that out, and so the filter will apply). The src address will be that same address and the DNAT will change dest address to a.a.a.a and the trick will work.

Hope you find this trick useful.

Case two

Think of a situation where you have two internet connections through two different ISPs. You get two ethernet cables from them, they provide you with two static addresses/netmasks/default gateways/dns etc.

You connect each cable to a different box, set up networking and everything works like a charm.

Now, you want to get a little wacky and connect those two cables to a single switch (layer two) and connect those two boxes to the switch as well.

Connections should work fine, right? And they do... but then, what happens if you try to send traffic between those two boxes? Say, from box A you ping box B. In this case box A checks it's routing table and realizes there's no network defined for such host so it goes through its gateway. ARP request to get the mac of its gateway, gateway responds with its mac address, packets go out with mac address of the gateway, src address A, dest addres B and the traffic is heading to internet through one ISP. Then traffic comes through the other ISP to box B, box B gets it. It's going to respond to host A, there's no route for it, sends it through its gateway, goes through same ISP that sent the request to host B, comes back through first ISP to host A and we see a reply on host A. Great.

But wasn't that too long a trip to reach a host that is two ethernet connections away from host A? There should be a way to make the trip shorter, right? And sure there is. You can set up routes to be reached through gateways (layer 3 routing) but also through devices (layer two routing). How does it work?

Let's add a layer two route for host B on host A:

ip route add b.b.b.b dev ethx

ethx being the interface we use to connect to switch. And that's it.

Now what happens when host A tries to ping B? Now, there's a route to reach B through interface ethx so an ARP request for IP b.b.b.b is sent through said interface. Traffic is sent to the switch. The switch broadcasts this ARP request and reaches B, B responds to the ARP request with its mac address. A learns Bs mac address and sends traffic to it. Source IP address is A's, dest address is Bs, dest mac address is Bs. B is able to see this traffic (it's got its mac as destination) and sees As ping request. Now to respond to A it checks its routing table. Remember you didn't change anything on B? Well, there's no route to A so have to go through gateway. Traffic is sent to gateway, ISPs and then it reaches A.

To get the trick working to avoid using ISPs at all, you have to do the same thing on B:

ip route add a.a.a.a dev ethx

ethx being the interface B uses to connect to switches.

After not writing for so long, I had to have something interesting to write about, right?

Have fun!

Space is being consumed too fast? Find where!

2010-02-14T17:09:00.000-08:00

You've found hat some process (don't know which) is eating up too much space too fast in one of your partitions and don't know where? Just yesterday I found a simple way to figure it out.

Suppose you want to check the partition where you have /home (which is in a separate partition from /, right?) where space is been eaten up too fast.

Run this simple command:

find /home -mount -type f -exec ls -s {} ';' > list_of_files1.txt

After it finishes running, wait for a little while (30 seconds, a minute, your call) and run the same thing outputting to a different file:

find /home -mount -type f -exec ls -s {} ';' > list_of_files2.txt

To find out what's going on, run a simple diff (a tool present in I guess every GNU/Linux system... if not unix) between both files and you should be able to see what's going on.

diff list_of_files1.txt list_of_files2.txt

You're welcome!

ARM will fly without windows? Then bring it on!

2010-02-04T05:49:00.001-08:00

I was reading yesterday this article of an interview to Warren East, one of the top guys at ARM.

He goes on about how ARM will succeed with or without Windows (not ME) supporting it once it starts being pumped into markets in the shape of a new architecture for netbooks.

I've being begging for this beauties to come out for months already. Last year I saw forecasts saying how thew would start selling for roughly 200US$ and coming out on the 3rd quarter 2009... then the 4th... we are already past the 1st month of the 1st quarter of 2010 and only prototypes are what I've seen. I'm already fed up with it.

So, Warren, please... instead of forecasting doom for Windows if they don't support ARM (and I hope both things do happen), tell me when the machines will finally be out, who will put them out and the prices. I've already had my fare share of predictions about ARM netbooks. I want to actually see them and buy them (just 5 of them for me... I want to see the faces of my brother and sister when they get theirs being unable to run Windows on them).

Thanks in advance!

There's nothing wrong with being thankful (or why I say GNU/Linux)

2010-01-30T08:48:00.000-08:00

When I was a little kid (not that I'm too big, anyway... 5'10/~150lbs or 1,75/74Ks.... whatever you get better) I was taught to be thankful for things that I get. There's nothing wrong with saying "thank you" when someone has fulfilled one's need/wish for something... even more if the person who fulfilled it was not in any way forced to do it for us.

In the FLOSS community there's this old argument about whether we should call the OS that people usually call "linux" as GNU/Linux or plain linux.

The ones who defend "plain linux" say it's out of simplicity, being more catchy, undoubtfully easier to pronounce than GNU (at least in english... in spanish we convert GN to Ñ... or at least, I do it), and a long etc.

But there are many sides to this story that, at least to me, don't add up.

For example, simplicity for newcomers: What about when you have a distro that doesn't have Linux inside of it? Say, Debian's hurd or kFreeBSD or NetBSD ports? Those are distros, but Linux is not to be found inside cause it (the kernel) has been replaced for another kernel. What are we gonna call them? Debian Non-Linux? Go figure how you will explain that to newcomers ("sure... it's Debian Linux... but it has no linux... yet it is linux". Priceless).

Some people have said that it's out of Stallman's big ego that he wants everybody to call it GNU. Well, I think Stallman hits the nail (at least on the funny part) when he says that "sure... and that's why I ask people to call it Stallmanix". So I think it's not out of ego... but maybe if he had named the OS Stallmanix in the first place, we wouldn't be having this argument as it (too) is more catchy than GNU. :-)

Then we have the people who say that then we should call distros Kubuntu KDE/X/GNU/Linux, for example... but I think there's a line where we can say that KDE/X/GNU/Linux is just too much and GNU/Linux is ok: The minimum usable machine by a user would require GNU/Linux (I can work perfectly well on a GNU/Linux computer with bash and no KDE/X, so that makes the basic machine for me) while a machine with just linux (the kernel) would be pretty pointless as there's laking a system so that I could interact with it (the shell of the OS, at least).

Also I don't like to call things something they are not. For example, I wouldn't say "I like driving my wife's V4 16-valve 1.4 engine to work" as just the engine doesn't make up the whole car (yeah right, like she actually allows me to drive her car :-)). Of course, you can hear people bragging about their 5.0s, but sure as hell these people actually want to talk about _the engine_, not the whole car.

Linux (the kernel) is quite a nice thing. I'm still overwhelmed by its capability to run on the tiniest machines and the biggest supercomputers as well. How it's capable of running on all this different architectures, how you can basically hack it whatever ways you fill like to fit whatever need you have. To all of that (and more), I take my hat off and fill humble (and, trust me, that's not something I can say of many things or people:-)). But I think (that's me, personally, I'm not asking anyone to do something agains their will) GNU deserves being named alongside linux.

I'm thankful to all the people who have helped develop GNU (and linux) into what it is and so I call it GNU/Linux just like, though I use Kubuntu, I proudly wear a Debian cap (cause I know where Etcbuntu gets a lot of what makes it what it is)... and you will pry it off my cold dead hands.

And finally, to bring this chapter to an end from where it started: Thank you!

PS The cat in the picture is Tomás (after Tom from Tom & Jerry), my wife's pet.

FF3.6 on ubuntu is not a reason why GNU/Linux is not ready for the mass-market

2010-01-22T14:21:00.000-08:00

Hi!

I have become quite a replicator lately, right?

Well, yet another article from an IT journalist/commentator I have to disagree with.

In the article the writer states that it's too difficult to get FF3.6 installed on Ubuntu and that it's reason enough to call GNU/Linux dead on its tracks to get to the mass market. That sounds compelling at first sight.... but

First: I bet the users of software that aren't quite up to date and that make up those huge botnets differ with the writer of the article. They all make up a part of the mass market as we know it, don't they? So it's OK to have outdated* software, isn't it? (I know, I know... it's not OK... but we are talking about mass market here, so go with the flow!).

Second: Remember that the way software is installed/maintained in the GNU/Linux world is completely different from Windows'. In Windows, as the writer said, you grab the software from internet (hopefully form a reliable location.... but we know that's not always the case, is it?), click on it, maybe will have to restart your computer.... a couple times (why the hell installing Adobe Reader requires you to reboot Windows? Is Adobe Reader the equivalent for Windows of glibc or something?) and then finally you are done with the software. In GNU/Linux, at least in Ubuntu (and every other distro that prides itself of being such), you have to wait for the maintainers of Ubuntu to review software to make it available. That's right.... they do that job for you, the user. And it's not just firefox that they maintain... they take care of thousands (literally) pieces of software to make them fit together and not mess with each other when you installed them on your beloved Ubuntu-powered box. And that not only sounds like a dauntin task... it really is. And what would be the equivalent of that in the Windows world? It would be like waiting for Microsoft to review the software when it's made available by its developers (have you seen how long it takes Microsoft to work on their own bugs? How long would it take them if they had to review other people's software as well?) and make it available to you through the centralized software they provided Windows with so that their beloved customers don't have to go leaping from site to site to grab the latest piece of malware-infested piece of software... oh, but there's no such thing for Windows, is there? Such a shame, you know.

So, in other words, FF3.6 is not made available in the stable Ubuntu release because it's going to be a major work to get it merged, but that doesn't mean there is no way to get it packaged so that our dear writer can use it. It didn't take me too long to find unstable/unsupported repos for FF3.6 (probably its stable enough, don't know for sure) for Ubuntu:

https://launchpad.net/~ubuntu-mozilla-daily/+archive/ppa

I'll personally wait for Ubuntu to make 3.6 available through their standard repositories... which I hope will happen for Jaunty... but maybe they won't and will make it available for lynx only... will have to wait and see.

Just so that it's crystal clear. This article doesn't state that GNU/Linux is ready for the mass market. I'm just stating that the writer-of-the-article's difficulty to install FF3.6 on Ubuntu is not an excuse to dismiss GNU/Linux's readiness for the mass market. Also, I do think GNU/Linux is ready for the mass market, but that's another quite different story.

That's it.

* FF 3.5 is not outdated, by the way. It will be maintained (at least, security-wise, by the Mozilla foundation for a while).

How can people blame on GRUB if Windows doesn't like another bootloader?

2010-01-21T10:25:00.000-08:00

I was reading this very interesting article on a guy who noticed that when going from Vista SP1 to SP2 windows would almost finish the process (taking quite a while, apparently) and then it would report an unknown error and rollback all the things that it had done (wasting CPU and real time, by the way). After seeing the problem show up a couple of times the person realized that grub was there in the MBR. Replaced the MBR with Windows', tried to update it it was done. Great.

So.... a very interesting read, I have to say. Then I hit the comments and what do we find? None other than people saying that it's grub's fault. Say what?

What's there for Windows to see that belongs to grub? Not much, really. The first phase bootloader, located in the MBR, in other words it's comprised within the first 512 bytes of the HD. The second phase bootloader (which is called from the first phase bootloader) is located somewhere within the GNU/Linux partitions set up at the box. So, the only thing from GRUB that Windows can actually see (unless Windows is capable of reading out of the box ext2, ext3, ext4 and the other gazillion FS that we have available in GNU/Linux) is the first phase bootloader.

In my opinion, it's something as simple as old Microsoft's motto in action: "It's the Microsoft way or the highway". The update process is taking a look at the MBR and notices that's it's not Windows' bootloader. "Who in their right mind would dare install something on the MBR that's not made by Microsoft?" I bet they think there at Redmond. End of the game, let's stop the update process... _and_ (specially) not tell the user what's going on. It wouldn't be as insulting if at least they would suggest the user to replace the MBR with Microsoft's tools. You know, it can be replaced with GRUB a couple of minutes later after shutting Windows down after the upgrade process is done... but what do we expect from a OS that was made to resemble black magic, anyway?

As I have already said before:
Windows equals esotericism
GNU/Linux equals determinism

A little bug fix for ADOdb (php) for MySQL

2010-01-05T09:27:00.000-08:00

Hi!

I'm using ADOdb to connect to more than one MySQL db at the same time but then i noticed that queries where being done on the last DB connection that was established. After researching for a while, i ended up modifying MySQL driver of ADOdb. It's a rather simple trick, so be free to use it as needed:

On adodb-mysql.inc.php, go to the definition of _query($sql, $inputarr=false) and change it to:

@mysql_select_db($this->database, $this->_connectionID);
return mysql_query($sql,$this->_connectionID);

That should do. By the way, I've only tested it connecting to a single mysql server (with two different DBs, obviously) but I think it should work with multiple DB servers as well. Also, It works like a charm when doing ->Execute() so it could be necessary to add that mysql_select in other places so that it worked with other functions that deal with the DB, but Execute() is enough for what I need so....

I did this patch on flisys, and I'll upload it in the following days.

My wishes for 2010

2009-12-31T12:40:00.000-08:00

Hi!

As just about everybody else does, I will be making my wish list for next year, which is just a few hours ahead here in Colombia. Most of the topics are IT related, but not all of them are.

- Get to grab one of those cheap, never-ending battery life ARM based netbooks (the sooner, the better).

- See Firefox (and now Chrome) get to grab more market share than IE.

- See FLISoL explode in assistance and installations (in Bogotá, Colombia and everywhere).

- This is a big one: See a 60 minutes article about FLOSS. That'd be so cool! Here in Colombia, one "Especiales Pirry" or "Séptimo Día" article would make the day.

- Convince the bosses at the company where I work to allow me to use GNU/Linux on my laptop computer (haven't tried yet... have to think of a way to do it). We use a lot of GNU/Linux but not on our desktops which is a shame, if you ask me.

- Get to see my wife come out of her health problems and make it back to her normal life (love you, Honey!).

- Get to spend next Xmas and new year (that'd be 2011) with my family (I mean, the blood related one), be it in Maracaibo (where I'm from) or here in Bogotá... better here in Bogotá (I just spent a weekend at a hot place and I couldn't bear it. Don't want to think what it'll be when I put my feet back in Maracaibo. :-S)

I guess that'd make it for next year. Hope at least half of them come true... specially the last two.

Happy MMX! (no relation to Intel)

How to see files hidden behind a mount

2009-12-24T11:51:00.001-08:00

Hi!

It's been a while since I did my last technical post. Yesterday I saw myself in a situation where I had to see some files in a directory that was used to mount a partition so I wasn't able to see the files I needed, so to speak. After some hacking, I was able to see the files. Here's how it's done.

First, let's set an environment for our tests. I have some files in /mnt/D/ and /mnt/D/ is in the root partition:

# mount
/dev/sda6 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
# ls /mnt/D/ -l
total 8
-rw-r--r-- 1 root root 6 2009-12-24 14:55 test2.txt
-rw-r--r-- 1 root root 5 2009-12-24 14:55 test.txt
# cat /mnt/D/test2.txt
adios

Now, I use this partition to mount a drive from a windows host, so after I do that mount:
# ls /mnt/D/ -l
total 884802
-rwxr-xr-x 1 antoranz root 1883 2009-11-27 09:42 20091125.html
-rwxr-xr-x 1 antoranz root 2150 2009-11-27 10:59 20091126.html
-rwxr-xr-x 1 antoranz root 110230 2009-10-17 19:56 3D400055.WAV
-rwxr-xr-x 1 antoranz root 0 2009-12-23 15:10 algodon.txt
drwxr-xr-x 1 antoranz root 0 2009-11-04 15:34 billnew
etc

so the files that were in /mnt/D are now hidden because of the new mount. What I did to see the files was to remount the root partition (as that's the partition where the files I need really are) in another mount point. Let's try to do it directly. As you can see from the mount I showed before, my root partition is in /dev/sda6 so:

# mount -o ro /dev/sda6 /mnt/tmp
mount: /dev/sda6 already mounted or /mnt/tmp busy
mount: according to mtab, /dev/sda6 is mounted on /

I knew it wasn't going to give itself up so easily. But don't panic... there's a way to fool the SO... really! Have you heard of loop devices? Now's a great moment to learn about them... but I won't go into the details. Let's just say they are very useful. Do your homework and find out about them. Let's link /dev/sda6 to a loop device:

# losetup -f -v /dev/sda6
Loop device is /dev/loop0

So /dev/sda6 is now linked to /dev/loop0. Let's try to remount the root partition again to see what happens:

# mount -o ro /dev/loop0 /mnt/tmp
#

The prompt is staring at me and the command didn't complain. Seems like it's done. Let's see what's in /mnt/tmp:

# ls -l /mnt/tmp/
total 108
drwxr-xr-x 2 root root 4096 2009-12-11 21:01 bin
drwxr-xr-x 3 root root 4096 2009-12-24 11:10 boot
lrwxrwxrwx 1 root root 11 2009-08-20 12:06 cdrom -> media/cdrom
etc

The content of the root partition... as expected. Well... let's see if the files I need are still there:
# ls -l /mnt/tmp/mnt/D
total 8
-rw-r--r-- 1 root root 6 2009-12-24 14:55 test2.txt
-rw-r--r-- 1 root root 5 2009-12-24 14:55 test.txt

As you can see, the new mount point is not fooled by the fact that there's something else mounted in /mnt/D. And now finally let's see if we can get the original content of the files:

# cat /mnt/tmp/mnt/D/test2.txt
adios

And there you are. Now, a little word of caution. I was able to see the files but the content of the other file appears to be corrupt (have to find out why) so be careful with what you get at the end. Take it as a nice start.

Merry Xmas, everyone!

Venezuelan Justice System used to be David Morales Bellos' before the revolution... Now it's Hugo Chavez's. At least David Morales Bello was never the president of Venezuela.

A tip for the next interview with Steve Ballmer

2009-11-15T07:44:00.000-08:00

Hi, guys!

In case you are an IT journalist and were to interview Steve Ballmer, Microsoft's CEO, please, I beg you, ask him this question:

- Mr. Ballmer, given your hard stance on GNU/Linux violating Microsoft's IP (whatever that means), what's your take on Microsoft being caught twice in a matter of a few months in violations of the General Public License? Is it official policy at Microsoft that personnel can throwing all kinds of (unproven) accusations at the Free Software community yet, at the same time, treat GPL software with total disregard for the terms they demand?

I just can't believe a company that prides itself of "respecting IP" can be caught twice in a GPL violation. Even worse if all they do is point fingers at other people saying other's are violation Microsoft's IP (without showing proof).

- http://news.slashdot.org/story/09/07/23/1327205/Microsofts-Code-Contribution-Due-To-GPL-Violation
- http://www.pcmag.com/article2/0,2817,2355892,00.asp

man date: %C: century; like %Y, except omit last two digits (e.g., 20)?

2009-11-09T19:03:00.000-08:00

Oh, really? The only problem that I find in that statement is that it would be correct on the very last year of a century. I mean We are in 2009 and our century is the 21st, right?
Well, let's see what date says about our century:

$ date +%C
20

Lovely! What about in 1999?

$ date -d 19990303 +%C
19

So in 1999 being the 19th century, the closest thing I was to using a computer was an abacus (considering I was a windows user at the time we can agree I was actually using something next to an abacus).

And what about the last year of the 20th century?

$ date -d 20000303 +%C
20

Only then the definition provided by 'man date' does match the actual century for a certain year. And before people start complaining about 2000 being the first year of the 21st century instead of the last of the 20th century:
The year 0 doesn't exist in our calendar. Remember the gregorian calendar is roman based and romans didn't know the concept of 0, therefore no year 0. Theoretically speaking, from Dec 31st 1 BC, you'd have jumped into
Jan 1st 1 AD. If the first year of the first century is 1 AD and the last is 100, the first year of the second century is 101, the first year of the 20th century would be 1901 and the last would be 2000 and the first of the 21st century would be 2001 and so on. I used to discuss about this very same point with my Music History professor at the Conservatory in Maracaibo and we'd get into all kinds of philosophical questions about when it was 0, the limit between year
1 BC, 1 AD and so on and we'd never get to a point. My best wishes for my dear professor Osvaldo Nolé, by the way.

More info:
- http://en.wikipedia.org/wiki/Year_zero
- http://en.wikipedia.org/wiki/Gregorian_calendar

3com 3CRWER101U-75: Not working.

2009-10-03T13:00:00.000-07:00

Hi!

I went shopping to the mall that is around my house for a wireless router this morning. I didn't want to spend a lot of time on it so I basically bought the (almost) first router that I saw. It's a 3com router. A 3CRWER101U-75 (model WL-550, I think). I bought it thinking that it'd be a reliable router, even if it wasn't very powerful. As long as my wife could be connected to the net from our room, it'd be more than enough.

So, I arrived home about 4 hours ago. Since then I've been trying to get the damn thing to work properly and I just haven't been able to get it to work.

If I try pingin the router from a box that is connected to it by wire, the connection is intermittent:

64 bytes from 192.168.200.1: icmp_seq=3594 ttl=64 time=0.384 ms
64 bytes from 192.168.200.1: icmp_seq=3595 ttl=64 time=1.14 ms
64 bytes from 192.168.200.1: icmp_seq=3596 ttl=64 time=0.678 ms
64 bytes from 192.168.200.1: icmp_seq=3597 ttl=64 time=0.467 ms
64 bytes from 192.168.200.1: icmp_seq=3598 ttl=64 time=0.563 ms
64 bytes from 192.168.200.1: icmp_seq=3599 ttl=64 time=0.789 ms
64 bytes from 192.168.200.1: icmp_seq=3609 ttl=64 time=9770 ms
64 bytes from 192.168.200.1: icmp_seq=3619 ttl=64 time=0.435 ms
64 bytes from 192.168.200.1: icmp_seq=3620 ttl=64 time=0.401 ms
64 bytes from 192.168.200.1: icmp_seq=3621 ttl=64 time=0.421 ms

See the wholes between 3599 and 3619? How about that?

Talking about the wireless, the only setting that is close to working is setting it to use WPA/WPA2 Only with TKIP+AES. And I say close because I only get to connect to the wireless on GNU/Linux with wpa_supplicant (running on the console) and even then it disconnects quite often. On windows, it's a no-go. And I'm talking about boxes that are less than 2 mts from the router while I do my testing.

Is it that I'm putting it under too much of a load while I do my testing? It's a very small router so I don't expect it to behave as a high-end router.... but even then... what's the firmware based on? Windows Vista?

I just can't believe it, really. I thought 3com products where reliable. At least this router doesn't make the company look good. What should I do? Is there a firmware update or something? Somebody help me, please!

PS I should have heard recommendations from a friend of mine who told me to get a WRT54G router instead, but I didn't get to see any at the mall, so....

Update 1
The wired connection has been working much more reliably today. So far, I haven't had any disconnections so far. However, the wireless is another story. Though the wireless can be seen by hosts, I can't connect to it:

CTRL-EVENT-SCAN-RESULTS
Trying to associate with 00:1e:c1:a2:3f:ec (SSID='Enfermitos' freq=2422 MHz)
Association request to the driver failed
CTRL-EVENT-DISCONNECTED - Disconnect event - remove keys
Authentication with 00:00:00:00:00:00 timed out.
CTRL-EVENT-SCAN-RESULTS
Trying to associate with 00:1e:c1:a2:3f:ec (SSID='Enfermitos' freq=2422 MHz)
Association request to the driver failed
CTRL-EVENT-DISCONNECTED - Disconnect event - remove keys
Authentication with 00:00:00:00:00:00 timed out.
CTRL-EVENT-SCAN-RESULTS
Trying to associate with 00:1e:c1:a2:3f:ec (SSID='Enfermitos' freq=2422 MHz)
Association request to the driver failed

What should I try?

dhcp-lb 0.02

2009-09-09T17:54:00.000-07:00

Hello!

You might have read about dhcp-lb when I announced it a couple of weeks ago. It's a suite (sui generis :-) that is used to keep multi-path routing configured when you use dhcp up-links.

After it was tested at the Hospital where I used to work before, we noticed a problem and I decided to enhance the script over that discovery so I'm announcing dhcp-lb 0.02. Go get it from here.

Not much has changed, as a matter of fact. The thing is that you can tell dhcp-lb what static interfaces you are using besides the up-link. Those static interfaces are not involved in the multi-path routing (just yet), by the way... so if you have a provider that has a static configuration and another that uses dhcp, the script is not ready to handle that... but it will come... someday (hopefully in the rather not too distant future).

So, say you have interfaces eth1 and eth2 used for internet (dhcp) where eth2 has double the bandwidth of eth1 and a static connection to your intranet on eth0, the configuration file will look like this now:

dhcp-lb eth1 table-eth1 200 1
dhcp-lb eth2 table-eth2 201 2
static eth0

I also included some checkups for the configuration file (more than one space or tab can be used between the components of a configuration line, empty lines are welcome, etc.. but more checkups are still missing, I'm sure). Feel free to use it (and I thank you for using it if you do) and tell me how it goes.

Bye!

PS Thanks go to Overt Barreto at the Hospital of Pediatric Specialties in Maracaibo, Venezuela for taking the time to test the script... he had to anyway but it's always nice to have a person tell you when something goes wrong.

Carta abierta a rector de la UTN (Argentina)

2009-09-04T07:16:00.000-07:00

Buenos días!

Soy un ingeniero de computación venezolano que vive en colombia. Acabo de enterarme de que la UTN canceló una charla del Sr. Richard Matthew Stallman a darse en su Universidad. Dicha charla, según la fuente, fue cancelada por contratos que la UTN tiene con Microsoft que impiden que se hagan críticas sobre dicha empresa.

Creo que de más está decir que debería darles _pena_ haber cancelado la charla de una persona tan importante dentro del mundo de las tecnologías como RMS por razón de dichos contratos. Cualquier mortal se vería tentado a pensar que la universidad debería ser un ambiente para el desarrollo libre de las ideas/expresión.... claro, siempre y cuando dicha universidad no tenga un contrato con alguna compañía que se lo impida.

Ojalá se den cuenta de su error y decidan llevar adelante la charla... incluso si eso implica la cancelación de los contratos (total... hay bastante software libre por ahí que les servirá para las mismas funciones y sin un contrato restrictivo). Creo que en este caso hay suficientes bases desde el punto de vista moral (como se acaba de ver con este ejemplo) como para que su universidad decida rescindir de ellos.

Muchas gracias por su atención.

Popping GNU/Linux out of the Virtual Machine

2009-08-29T09:48:00.000-07:00

Hi!

At work I'm tied to a windows machine, however I have been able to use GNU/Linux which is where I'm most productive.

First, I tried working with Portable Ubuntu. It works pretty well, however the latency can be a little high sometimes (Firefox being the most obvious case that I noticed).

About a week ago I started working on a clustering experiment and wanted to give it a shot inside Portable Ubuntu. However, I discovered it wasn't possible because its kernel doesn't support bridging (I was going to use some qemu virtual machines for the experiment).

Well, It seemed like I was going to need a virtual machine to host the experiment after all. I have used qemu and love it because of all the networking tweaking it allows you to do, however it can be very slow... and inside the virtual machine I was going to host more virtual machines, so I decided to use something else, at least on the hosting virtual machine.

I tried VirtualBox and was gladly surprised by its performance. It was veeeeeeeeeery fast.

Got Kubuntu installed on it. It works acceptably. Now the problem with virtual machines where you have a window that represents the computer's monitor is that you have to use key combinations to get in and out of the virtual machine environments.... plus the windows in the virtual machine are inside a window that looks not integrated to the real environment.

But then a truck hit me. How about finding a way to get the windows out of the virtual machine? How about using an X server on Windows (XMing, for example) and configure the Virtual Machine to use that X server. kdm can be configured to use a remote X server.

If you check /etc/kde4/kdm/kdmrc (remember it's Kubuntu, so I'm using KDE... gdm must have something similar), in the [general] section, there's StaticServer and it's set to :0 (in other words, the X server of the host). I changed that to say 10.0.2.2:0.0 (the address of the windows box from VirtualBox):

StaticServer=10.0.2.2:0.0

Then I started XMing (XMing included in Portable Ubuntu, by the way) with -rootless, so that I got no decoration for the X server window, then I restarted the kdm service on the virtual machine and there it is! kdm is displayed on the windows environment. Log in to kde and after a moment of not seen anything on the X server screen I get to see KDE's background, plasma panels and everything else. Cool! The windows are not integrated in windows, but I don't have to get in/out of the Virtual Machine screen anymore.

The only thing I'd have to complain about so far is that sometimes the latency on KDE gets too high and I don't know how to avoid it. I tried with the windows XMing and cygwin's XWing but I got the same latency issue. How can that be avoided?

Picture
The picture I'm including is the same thing running off a Kubuntu 9.04 LiveCD on my wife's box (shhhhhhhhhhh, she better not hear about it).

Sidenote
When I log into kde, depending on the resolution of Windows (the real OS of the machine), KDE's splashscreen is seen or not.

dhcp-lb: Load balance with DHCP links

2009-08-22T13:03:00.000-07:00

Hi!

Some years ago, when I worked at the Hospital of Pediatric Specialties in Maracaibo, I did a project that would update routing for our multi-link internet connection when there was a DHCP event. I decided to redo the whole thing using a simpler approach plus I would release the code without asking for permission (as it's mine now) under Affero GPLv3.

Introduction
So, you have read lartc's guide on 'Routing for multiple uplinks/providers' and it all makes sense (and does work). However, there's a catch for you: Instead of having static network configurations, your ISPs use DHCP to set your network connections and you don't intend to sit all day long waiting for DHCP event to happen to reconfigure the whole thing, do you? That's what I thought.

dhcp-lb
In order to solve the problem I just described I created dhcp-lb. It's a python script that, once configured and linked as a dhclient exit hook, can listen to dhcp events and reconfigure networking accordingly.

What I need
You need python, dhclient, iproute2 (that I think will come by default in almost every distro) and a multipath-enabled kernel (check for CONFIG_IP_ROUTE_MULTIPATH, I think).

How to get it?
Right now, the only way to get it is through bazaar from launchpad's branch I set for it. I have created a PPA, but I have to learn how to use it so be patient (maybe someone can help me in this journey). If you want it but don't want to use bazaar to get it, email me and I'll gladly send you a tgz with it.

Configuration
Configuring it is fairly simple. You need to create a file called /etc/dhcp-lb with the configuration of each link involved in the multilink default route. Each line has the configuration of each link and it has four fields (so far):
- Network Interface
- Routing Table for that interface
- Routing Table ID (numeric, less than 255... check /etc/iproute2/rt_tables)
- Weight (numeric, more than 0)
All 4 fields have to be separated by exactly a single white space at this time. One example /etc/dhcp-lb file would look like this:

# interface routing-table table-id weight
eth0 table-eth0 200 1
wlan0 table-wlan0 201 2

In this example I set wlan0 to have twice the routing weight of eth0 (for example, eth0 is 512 kbps and wlan0 is 1 mbps). Also, if the routing tables are not set in /etc/iproute2/rt_tables, dhcp-lb will take care to add them to that file, so no need to change rt_tables beforehand.

After you have done that, all you need to do is link /usr/bin/dhcp-lb into the exit hooks directory (/etc/dhcp3/dhclient-exit-hooks.d in my case) and you are done. Restart the network service and it should start working right away.

Keep in mind
dhcp-lb only configures routing with multiple uplinks and it doesn't correct any of the shortcomings that come along having multiple uplinks. It only reconfigures routing with dhcp events. What are the shortcomings? That would require a blog posting of its own, but this is the most noticeable (at least, to me):

- Balancing works beautifully for multiple short connections from your LAN. Few long connections? It won't look like it's balancing at all. It's because of the way routing works. The kernel doesn't make a routing decision for every packet that goes out of a box. Instead, it uses a routing cache to associate a target IP with the network interface it chose to use to get to it so it doesn't have to make a routing decision again... and the cache lasts for a few minutes (run this command: ip route show cache to take a look at the actual cache). After a while, it can decide to send packets to that same host through another network interface possibly breaking connections established to that host cause the other host will start receiving packets from a different address.

Conclusion
I hope this project helps you. However, this project is (as you can see from the logs) very young so expect it to have some problems (that I'll be working to solve). Have patches for it? Please, send them without hesitation. I'm open to suggestions and donations if you consider I deserve any.

Update
I just released dhcp-lb 0.02 (sep 9th 2009). Check this post to learn more about it.

[OT] Adsense: Get scientology out of my site!

2009-08-16T16:36:00.000-07:00

I set up Google's adsense on this blog. It's not being making all the truckloads of money I expected from it, but it's better than nothing nevertheless.

However, since a couple of weeks ago, the only advertisements I've seen on those spaces are dedicated to Scientology... at least on the computers I've been able to check for it.

Now, I've got a problem with that... and it's not because I don't like Scientology per se. It's just that I don't have any sympathy for any religion in particular. And come on, if there's an advertisement of Scientology every once in a while, I can let it slip through... but EVERY AD for the last couple of weeks? That is a little too much for my religious tolerance.

So, guys at Adsense, get rid of the Scientology ads on my site.... or even better, set up a page so that I can ban some ads I don't want to see on my site (like religious ones, for example).

Thanks in advance to anyone who can help in this regard.

Installing openSuSE: grub complaining on a hardware RAID

2009-08-11T17:46:00.000-07:00

Hi!

I'm a Kubuntu/Debian user (and intend to stay that way), however a few days ago I set myself to install openSuSE 11.1 (LiveCD based on KDE4). The thing is that this particular box has a hardware RAID set up.

0f:08.0 RAID bus controller: Hewlett-Packard Company Smart Array E200i (SAS Controller)

I had no problem installing other OSs on top of it so I thought it was going to be painless.... It wasn't.

As a matter of fact I didn't have too much of a problem (and the installation was pretty fast, congratulations for openSuSE for that).... except for the fact that when the installer was about to finish, on the part where grub was going to be installed on the MBR, it would fail miserably with an error 21. It would complain about not being able to find a certain device. I was given the option to retry installing grub without going though the whole installation procedure all over again. I tried a couple more times with a few option changes but I always got the same error.

I then went into grub (by calling grub as root on the terminal) to see what I could do. I tried to change the root device, but when I wrote root and then hit the tab key to see the available devices, no one showed up. Could it be because I'm working on a live CD and some of those partitions were just created?

I went openSuSE's IRC channel on freenode.net and the guys there were very interested on my problem. After a while someone (sorry not to remember your nick, man.... or woman... but thanks for showing me the light) told me that it could be a problem with grub's device.map file.

I went there (/boot/grub/device.map of the running LiveCD instance) and only saw a line for (fd0). It looks like it's missing all the partitions I have set up on the disk after the installation (Important Notice: see the update to the article at the end). I write a mapping of all the devices that I needed (including the HD I was using and all the other partitions with other operating systems) with what I would guess partitions would be to grub. Something like:

/dev/cciss/c0d0 (hd0)
/dev/cciss/c0d0p1 (hd0,0)
/dev/cciss/c0d0p2 (hd0,1)

And so on. Saved the file then I went into grub again. root tab..... and there are all the partitions I needed. So I do the grub installation procedure which ends up being like this:

root (hd0,4)
setup (hd0)

(hd0,4) would be the / partition I was installing on (with /boot not in a separate partition). Remember I was trying to install it on the MBR, that's why I say (hd0) on the setup command. I get the output that the installation went right and I'm done.... almost. I mount my root partition (that would be / of the just installed openSuSE) and see the content of its /boot/grub/device.map and it's the same old broken one. I replace it with the one I edited on the running LiveCD. Unmount, just in case so the installer (which is still waiting for me) doesn't go crazy.

Then I let the installer finish telling it that I don't want to install any bootloader. It finishes installing. Reboot, I get openSuSE running.

Hope you don't need to apply this but... it could be a little too late. Otherwise, why would you be reading this, right? Good luck anyway!

UPDATE
I'm looking at the device.map file on my box at home with Kubuntu kinky... I mean, karmic (that openSuSE box was at work) and I see that the only line there is a mapping to the hard drive and not all the partitions of my box:

(hd0) /dev/hda

So perhaps I just needed to add a line to map /dev/cciss/c0d0 to (hd0) and that would be it... but it could also be because karmic is using grub2, so keep that in mind when you try. In any case, I'm not going to retry doing the installation just to figure it out.

Are we too naive by believing that GNU/Linux is more secure by design?

2009-07-26T16:00:00.000-07:00

Hi!

I've been wondering for the last couple of days about the proposition that I use where I state that GNU/Linux is far more secure than Windows among other things because it's designed to be so (a real muti-user OS by design, real Networked OS by design, etc) plus other customs that we *NIX users have like not using the root account to play Frozen Bubble and so on.

Now, there are people that say that it's just that GNU/Linux is less attractive to malware software because there are so few of us GNU/Linux users. I have always thought that this is crap but anyway....

Now, think about the things that FLOSS developers get to do:
- Crack encrypted DVDs
- Allow for communication between Microsoft Windows hosts (with a twisted SMB protocol) and *NIX hosts before Microsoft (reluctantly... but with a lot of PR spin, as usual) released the documentation about it
- Synchronize with iTunes
- Running GNU/Linux on basically any piece of equipment worthy of running it (with or without support by the vendor).. and some others that aren't worthy but....
- Brake every DRM mechanism ever built

And the list goes on and on. No matter what the developers wanted to restrict, there has always being a way to crack it.

Now... if FLOSS developers are able to do basically just everything they set themselves to do, wouldn't it be possible that malware developers will get get to do the same with the security barriers set on a GNU/Linux (or *NIX for that matter) no matter how hard we try to restrain them?

I just wonder

Random musings on GPL and Microsoft

2009-07-25T18:10:00.000-07:00

This has been a very remarkable week.

We had Microsoft releasing some 20,000 LOC of Linux drivers so that Linux can run faster on their Hyper-V solution. A lot of MS PR saying how they love interoperability and how cool they are. I will be the first to say that I was in shock (and I bet I wasn't the only one). After all, we are talking about a license that their managers explicitly hate so why release code under that license then? I just couldn't help seeing a little hypocrisy involved.... to say the least. It's always cool to say that they had to eat their own words anyway so I didn't take it as a bad thing, after all, as Linus says, we are all developing scratching our own itches.

But a couple of days later things got even more interesting when we hear that there was an issue with some GPL code that they were using in their solution that was improperly linked and so they had to release the code... BUT it was not a GPL violation (or so they say). Now, wait a minute... if there was a situation that had to be called upon (which people involved say they wanted it to be handled quietly) and ended up with them releasing the 20 KLOC, then how can that not be a GPL violation? If there was no violation then there was no situation to be called upon (which people involved wouldn't have had to say that they would have wanted it to be handled quietly) in the first place, was there? I don't know, but something smells fishy here.

Now, let's make something clear. Microsoft representatives have stated in the past that GPL is viral and that it attaches itself in their IP. Look, pal... GPL doesn't attach itself. It's not a living thing. It's attached by developers when they choose to use it. Their own in-house developers do it (as Cisco via Linksys and others have had to learn the hard way... now perhaps Microsoft joined that Hall of Shame). After all, nobody is forcing anyone to use GPL code in their solutions. Well, at least I haven't heard of anyone complaining of having RMS threating them with using GPLed code in their solutions or.... So if you don't like the GPL, then don't use code released under its terms. Stop wining about how bad the GPL is, do your homework and write your own code instead. I know.... there's excellent GPL code out there and it's gonna take time to reproduce it but... if you choose to use it, then abide by its rules. That's all their creators asked for when they released the code under its terms after all, right?

Miguel: You, the man! - Open letter to Miguel de Icaza

2009-07-05T17:48:00.000-07:00

Dear Miguel:

During the last couple of weeks there has been a tremendous amount of information pouring about Mono, the free (speechwise) implementation of .NET started by you, Novell's vice-president of Development Platform, and how it should/would be handled by distros. We even had RMS himself come into the fray and tell us his take on it (which he had been mum about till now).

Mono is a development framework that (more or less) implements .NET (yes, that framework from Microsoft). The thing is that while some parts of .NET are open standards approved by ECMA and released under RAND terms, there has to be a solid word from Microsoft about what these RAND terms would be for Mono. The biggest fear of it is having a patent related claim later on when Mono-dependent applications become more "established" and harder to be replaced.

We have seen that when you are asked on it, you point at ECMA and when they are asked they do an HTTP redirect to Microsoft instead.. and they are just as mum as RMS was (conveniently so for them, I think.. they must be laughing their asses off by seen how we debate on the topic with them moving not a single finger).

So, coming back to the point, what we need, Miguel, is a statement from Microsoft, and I mean from someone that has gained a little confidence from the FLOSS community like Sam Ramji or Lawrence Crumpton who have at least shown their faces and stared at the beast (us) straight into the eye and not some random PR representative, telling us how the thing with Mono is.

Now you are the guy with the contacts at Microsoft, you are in charge of Mono, so just like in "The Ladykillers" (do I love that movie!) when the quarterback grabs Lump by the helmet's grill and tells him: "Hey, butt-head! You, the man!", now it's my time to tell you: You, the man! Get us out of this FUD feast (from both sides of the debate) and make them say something we can eat.

Glad to be of service

Named Pipes... or how to get two separate applications to interact

2009-06-27T09:03:00.000-07:00

Recently, I've been working on an application (bash based) that could gather some information that I need from a host (network interfaces configuration, arp neighborgs, routing policy, pinging some other hosts, etc). Then I wondered that it would be good if I were able to connect to some hosts through SSH and run some commands on those hosts and save the output of those commands as part of the information of the first host. Like an information gatherer of sorts.

I started working on this part of the project and hit a brick wall. When I connect to a host using openSSH's server, I had no problem throwing a bunch of commands at the server and wait for the output to come from ssh and save it. Say, something like:

$ ssh ubuntu@ubuntu <<EOF
> ip link show
> ip addr show
> EOF
ubuntu@ubuntu's password:
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:

mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:21:70:94:08:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:44:d3:f4:bf brd ff:ff:ff:ff:ff:ff
4: pan0: mtu 1500 qdisc noop state DOWN
link/ether 96:4d:e1:83:8a:4d brd ff:ff:ff:ff:ff:ff
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:

mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:21:70:94:08:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:44:d3:f4:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.123.127/24 brd 192.168.123.255 scope global eth1
inet6 fe80::216:44ff:fed3:f4bf/64 scope link
valid_lft forever preferred_lft forever
4: pan0: mtu 1500 qdisc noop state DOWN
link/ether 96:4d:e1:83:8a:4d brd ff:ff:ff:ff:ff:ff

Great. However, I didn't' want to connect to a host with one openSSH service. I had to connect to a router (a HW router, so to speak) that, whenever I sent it more than one command, would break my connection.

After several tries and experiments, I thought about creating one application that would use the ssh client to send commands to the router and get the output of the ssh client. It would wait for the router's prompt to show up before sending another command. Now, that would require not only getting the output of the ssh client, as that is a piece of cake:

$ ssh user@host | ./ssh_handler

That would allow my_ssh_handler to get the output of ssh (in other words, the router) to process it, but I also need to send commands to the ssh, somehow. That's when named pipes show up.

Named pipes allow you to send/receive data from streams that are not the standard input/outputs we get with every process (standard input, standard output, standard error).

Say you have two terminals sessions sitting on the same directory:

Session 1:
$ pwd
/home/ubuntu/pipe experiment

Session 2:
$ pwd
/home/ubuntu/pipe experiment

Let's create a named pipe in this directory in one of those sessions:
$ mkfifo my_pipe
$ ls -l
total 0
prw-r--r-- 1 ubuntu ubuntu 0 2009-06-28 18:00 my_pipe

We now have a pipe in the directory (see the leftmost p in the listing, that means it's a named pipe).

Now, let's try to send something from session 1 to session 2 through the pipe:

Session 1:
$ echo "Hello" > my_pipe

Notice how the process is blocked and doesn't exit. Let's read the content of the file with cat on session 2:

Session 2:
$ cat my_pipe
Hello

And if you go to Session 1, you will see that the echo has finished executing.

Now, let's create two scripts that will exchange information through two pipes. Script 1 will read lines from its standard input sending them and then will receive a line of information. The second will send back exactly the same line prepending: "You said".

Script 1:
# read from the standard input
while read input; do
    echo $input > pipe1
    # read from the other session
    read input < pipe2
    echo $input
done

Script 2:
#read from the other session
while true; do
    cat pipe1 | sed 's/^/You said /' > pipe2
done

When you run script2, it will stay there forever waiting for processes to dump stuff into pipe1

Then we run script1 like this:

$ ( echo HELLO; echo BYE; ) | ./script1
You said HELLO
You said BYE
$

Right on target. Now, I want to explain a very tiny detail. Why did I use an infinite loop on script2? Because if you try with a while read, it would only read a single line from pipe1 and then get an EOF and finish the loop (something related to the way the echo > pipe1 in script1 works, I think).

And then, going back to my problem, how did I get to make the handler? One simple way to put it is:

$ ./ssh_handler | ssh -i certificate user@host > a_pipe

ssh_handler uses its standard output to send commands to ssh. ssh is using a certificate so that I don't have to use password authentication, it gets the commands from its standard input and writes whatever comes out of the ssh server to a_pipe (you guess it, a named pipe). a_pipe is used by ssh_handler to read whatever comes from the ssh server and that's it: Two interacting applications.

SSH Tunnels: Using a service from a nated (twice) box

2009-06-06T09:09:00.000-07:00

Hi!

Recently I have being managing a box using a 3rd party application that allowed me to handle a windows box where I could use putty to get SSH access to a linux box. It had to be done this way because both my box and the linux box are nated, so they can't reach each other. Let me say it was a real PITA. The keyboard layouts were getting on my nerves. Some important keys didn't work sometimes... or at all (like ; or @ or ', etc). After a while I was encouraged enough to dig for a solution to get access to the SSH service of the linux box directly (or almost) instead of depending on this mess I was using.

First, let me introduce SSH tunnels before I dig into the actual solution to my problem.

SSH Tunnels

SSH tunnels are used between an ssh client and a server so that there is one parallel trusted (encrypted) connection using SSH as its transport.

When the tunnel is set up, there will be a passive listening side and one active connecting side. On the passive end we set up a port so that the tunnel waits for connections of clients to this port. When a client connects to this port, on the active side there is another connection to (potentially) another host/port and so the tunnel connects the client that used the passive port to the new connection on the active side.

The tunnels can be set up so that either our client is the listening side or the active side. But, it will never be both in a single side. It's either listening or connecting, and the tunneled connection is always established from the listening side to the active side.

So, how do this work? Well, let's do some simple examples.

L Tunnel (client side is the listening end)

On a local tunnel, we set up a port on our side and the SSH server will be the connecting side.

Let's say we want to get access to a HTTP service that is on the SSH server, but we want to use one encrypted transport for the transmission.

Say we will use our local port 8080, and on the other end the HTTP service is listening on port 80, the user to connect to the SSH service is sshuser and the host is sshhost. So, we set up the connection like this:

ssh -nNT -L 8080:localhost:80 sshuser@sshhost

After the tunnel is set up, we can use a web browser to use the http server:

http://localhost:8080

Ok, let's explain the details so we can get the devil out of the equation.

-nNT is used so that SSH doesn't start a SSH terminal session besides the tunnel (as I don't want to use it).

-L 8080:localhost:80 Here is where the tunnel is set up. The first parameter (8080) is the port we want to set up on the listening end (our host for a L Tunnel). Then the interesting part, localhost:80... with this we are telling the active side (on the SSH server for a L Tunnel) that when a client connects to our listening port (8080) we want the other end to connect to host localhost (localhost to the other end, the SSH server, that is) port 80 (http service).

After running that command on our box, we can see this with netstat:

tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 3310/ssh

As you can see, we set up a listening port on our host on port 8080 and it's only available to processes running on our host (I guess it's possible to fool this a little with a little nating, but it's out of the scope of this article).

Now, we just have to use this port on our host to use the HTTP service on the other end. That's why we say http://localhost:8080.

In this case we used the HTTP service of the same SSH server we used to set up the tunnel. But we could use the HTTP service of yet another host that's accessible to the SSH server. Say there is a host that's accessible to the SSH server through IP 192.168.25.3 (as you can see, it's a private network that could be only accessible to the SSH server and not us from our host). In that case:

ssh -nNT -L 8080:192.168.25.3:80 sshuser@sshhost

Then we use our browser:

http://localhost:8080
R Tunnel (ssh server is the listening end)

Say we want to set up port 2000 on the other end of the tunnel so that when clients connect to it, we will let those clients use our HTTP service on our host. We do basically the same we did before:

ssh -nNT -R 2000:localhost:80 sshuser@sshhost

As you see, the only real change is that we said -R instead of -L. All it does is invert the direction the tunnel is set up (listening side on the SSH server).

After we set it up, on the other end we can use netstat to check if we are listening:

tcp 0 0 127.0.0.1:2000 0.0.0.0:* LISTEN

Then we should be able to browse with a client from the other end of the connection by using port 2000:

http://localhost:2000

As in the case of L Tunnels, the order of the parameters of the tunnel is always the same: port on the listening side:server host on the active side:port on the active side.

And just like in the case of the L Tunnel, we could use a R Tunnel to connect to a host different from the active host of the tunnel. So say I want to enable access to a remote desktop service of a windows box that's on my private network accessible (to me, that is) through IP 172.17.32.67. Let's say I'll use port 3000 on the other side:

ssh -nNT -R 3000:172.17.32.67:3389 sshuser@sshhost

Then on the other side:
rdesktop localhost:3000

And it's done!

Now, let's work on our problem.

Access to a service on a host that's nated from a box that's nated too

Well.... as both boxes are nated, then it's impossible to get them in touch with each other.... directly, that is. But I bet you can use a box that has a SSH service that's accessible to the original two boxes, can't you? I bet you do! And then, we can do this:

On the side of the box that has the SSH service we want to get access to:

ssh -nNT -R 2000:localhost:22 sshuser@sshhost

What we do there is set up port 2000 on the middle box so that when a client connects to it, it will be connecting to the SSH service of the host we are running the command from. In other words, we have forwarded the ssh service of this host to port 2000 of the middle box.

Then, on the box we want to run SSH from to get access to the other box:

ssh -nNT -L 4000:localhost:2000 sshuser@sshhost

What we do is set up port 4000 or our host so that when a client connects to it, there will be a connection on the middle box to its port 2000 (which is the forwarded SSH service of the ending box). In other words we have forwarded the SSH service of the ending box to our port 4000.

Then we can use a ssh client to get in touch with the service we are interested in:

ssh -p 4000 remoteuser@localhost

And it's done! What do you think?

Bash Tricks II: repetitive tasks on files

2009-06-06T08:14:00.000-07:00

It's been a while since I wrote for the last time. I found a job (finally) and it's eating up most of my time.

Anyway, I had already written a piece on repetitive tasks before. Yesterday I had to do a thing that required another set of repetitive tricks. I had to find a file that could be included in a number (huge number) of compressed files. Some where named .tar.gz, others where tgz. I didn't want to spend the next month checking each compressed file to see if my target file was there. So I made a one-liner that did the whole thing for me.

First Attempt

( find /mnt/tmp/ -iname '*'.tgz; find /mnt/tmp/ -iname '*'.tar.gz; ) | while read filename; do lines=`tar tzf $filename | grep -i file-pattern | wc -l`; if [ $lines -gt 0 ]; then echo $filename; fi; done

First we have the ()s. These little kids let you run various commands and tie together their outputs so that they make up a single output.

Second we have the while read variable; do x; y; z; done. This construct allows us to read from the standard input line by line placing the content of each line in a variable (multiple variables can be used, in that case a single word from the standard input will be placed in each variable). In our case, we used $filename as our variable (be careful not to use $ on the while read).

Then the ``s. These kids allow us to run a command so that its output can be assigned. In our case, we are listing the files of a tgz file, grepping to find the pattern of the file we are looking for and then counting the lines that come out of grep. The number of lines is what is saved in the variable $lines.

Finally, we are testing to see if the number is lines is greater than 0. If it is, we print the name of the file where we found the file pattern we were looking for.

Second Attempt

Now let's try something a little bit different (though with the same pattern of file search). I have a number of ISOs saved in a box and each one of them has a number of RPMs inside of them. I have to look for this same file I was looking for before.

Basically, it's the same thing we did before, the only thing that's changing is that we will use another level of nesting so that we can mount/umount the iso files. Let's see:

find /var/isos/ -iname '*'.iso | while read iso; do mount -o loop,ro $iso /mnt/tmp; find /mnt/tmp/ -iname '*.rpm' | while read rpm; do lines=`rpm -qlp $rpm | grep -i file-pattern | wc -l`; if [ $lines -gt 0 ]; then echo $iso $rpm; fi; done; umount $iso; done

And that's it! Neat, isn't it?

Now, keep in mind that if you want to do rather simple things with the files, you can ask find to execute some commands on the files it finds. In my case it would have been a little tricky (at least) to write the actions I wanted to do on each file in find's syntax, so I went for the piping solution.

GNU/Linux up 60% YoY, Windows down 4.02%

2009-05-04T12:53:00.001-07:00

Well well.... I couldn't blog on friday when the news were fresh (and I knew right away everybody was going to comment on it). GNU/Linux has hit a 1% market share during April 2009 according to Hitslink's statistics. Alongside these statistics we can see that IE keeps bleeding market share while FF and Chrome continue to go up. Great news as well.

But then I find Adrian Kingsley-Hughes' article on the subject where he makes a comparison of market shares the way they were a year ago.

GNU/Linux had a market share of 0.63% in april 2008. One year later, it's (finally) reached 1,02%. If we look at the sheer number it's still laughable, right? Well, It's not laughable for two reasons:
- It's an increase of over 60% year over year for GNU/Linux (the brighter side of having such a low number to start with, for sure).
- Also, I bet they are not laughing at this at Redmond, Washington. Windows is one of two cash cows of Microsoft and losing a hundredth of the (potential) income these days plus having to almost give away Windows to be present on one of the few markets that's healthy nowadays (netbooks) is costing them hard cash.

Year over year, Windows had a decrease of 4.02% (from 91.58% to 87.9%). Some months ago it had to face being under 90% in years (probably more than a decade), and still going down.... very slowly, but down nevertheless.

Also, if we consider how FF has been going against IE, we find these numbers: FF was at 17.76% in april 2008 and it's at 22,48% last april which translates in an increase a little over 26% YoY. Not bad. IE's numbers are a little different. One year ago it was at 74.86% and last april it's at 66.1% which means it had a decrease of over 11% year over year.

I see two menaces for Microsoft here. Microsoft's own survival depends (completely, I think) on Windows' survival. Windows doesn't survive, Microsoft will follow through next shortly, so loosing Windows' market share directly is bad news no matter how you see it. But also, remember that a little over 10 years ago Microsoft had to basically kill one of the biggest threats to Windows as a platform: web browsers. The new push for alternative web browsers means they have a direct threat to Windows yet again, plus loosing their chance to leverage their stuff through EEE. And having FF (which is a direct legacy of Netscape's browser) spearheading this wave is more than poetic justice to me.

Overall, not good news for Microsoft, but good news for everyone else. Let's hope next year I can still comment on news as good as this one.

FLISoL Bogotá 2009 - The good, the bad and the ugly

2009-04-28T09:56:00.000-07:00

Hello!

Last saturday we had a run of FLISoL. I was present to Bogotá's instance as an installer. It's so wonderful to see all those guys going somewhere with the only purpose of helping other people (plus hanging out afterwards).

The Good
There were roughly 30 installers overall. 121 machines got worked on. Roughly 100 of them got a GNU/Linux installed/updated on them. 55 of them were Ubuntus (and such), 21 of them got Debians (no love for RPMs here in Bogotá, guys.... sorry), 12 of them got Mandriva, 4 of them got OpenSuSE, 2 got Fedoras plus a few others.

I got to install Kubuntu 9.04 64 on a machine (cause I didn't have Ubuntu 64 at hand), plus another Ubuntu 9.04 32. Unfortunatly I had to leave early cause of family matters. Till the moment I left things were flowing normally.

The environment was cool though we were a little crammed.

The Bad
We were going to have some servers with mirrors of the repositories of the major distributions. Unfortunately the mirrors weren't up till a couple of hours later which meant that the first installations that went through had to download stuff from internet repositories.... and with so many installations going on at once, you can guess how fast that went through. Unfortunately not having the mirrors from the start could have easily added 20 minutes (being optimistic) to the installation times of that first round. For my second installation the mirrors were already in place and that went much faster.

The thing to keep in mind for every installation you do is trying to get it well done in as short amount of time as possible, and having the "customer" understand the basic things (most important of all: package manager). That can be a little tricky to balance.

Another letdown was that we couldn't use the application I had worked on (along with Jorge) to gather all the information about the boxes once they were finished. It was probably because of the same reason we didn't have the mirrors up when we started. Fortunately that was not a show stopper.. I hope we'll be using it for other coming events.

The Ugly
In one of our previous installers' mettings we had been told not to bring anything to the Festival (other than the strictly necessary). Last year one computer got lost and they didn't want the same to happen again.

Well.... the Festival wasn't even started and not one but two laptops were already missing. Apparently there's a guy who has this custom of fooling around at the library to see which unwary person he can try to con. One person was candid enough to hand him with two laptops he/she was carrying while he/she did something else. A while later the person who got the boxes was missing... along with the two boxes. A strong reminder for me that, even if Bogotá is not as insecure as Maracaibo or Caracas, it's not Vienna (don't want to start a war here, guys, it's just my appreciation).

Some pictures
Here, here and here.

Bottomline
I liked the experience very much. I hope I can work on it next year as well. I still haven't seen statistics of Colombia or Latin America overall... Let's just hope they are as good (or even better) than Bogotá's.

Sidenote
I already learned that Maracaibo (my hometown in Venezuela) will be having its FLISoL next saturday (don't know why).

And for those who are curious: Nope, Microsoft didn't hand out anything for FLISoL. But given their last quarter's results, who can blame them?

Una ayudita para FLISoL - Carta abierta a Microsoft

2009-04-20T12:45:00.001-07:00

Querido Microsoft

Antes de hacer mi petición quiero dejar claro quien soy. Mi nombre es Edmundo Carmona. Soy un ingeniero de computación venezolano viviendo en Bogotá. Desde hace varios años soy un usuario/administrador de GNU/Linux y, a diferencia de otras personalidades de la comunidad de FLOSS (muchos de ellos con muchos más logros que yo), no confío en ustedes y despotrico de ustedes cada vez que puedo (ítem 940).

Durante las útimas dos semanas he estado colaborando en la organización de FLISoL, el Festival Latinoamericano de Instalación de Software Libre, que tendrá lugar este sábado. Vamos a estar instalando (de forma gratuita) Software Libre en los equipos que nos traigan a los sitios dispuesto para ello para el festival. En cierta forma, somos cuidadanos responsables tratando de eliminar las drogas tecnológicas de las calles.

Durante nuestra última reunion surgió la pregunta de qué hacer con los niños que vengan acompañando los padres que traigan sus equipos para la instalación? Ahí es donde entran ustedes.

Todos sabemos que Vista fue un gran desastre (probablemente comparable a Windows Me). Me imagino que ustedes deben tener pilas con miles y miles de CDs/DVDs de instalación de Windows Vista disponibles esperando que algún incauto compre alguno.

Serían tan amables de facilitarnos unos 15 o 20 CDs/DVDs de instalación de WIndows Vista? No los vamos a estar instalando en los equipos, no se preocupen. Yo personalmente se los entregaría a los niños para que se entretengan con ellos (usándolos como frisbies o rayando su superficie). mientras estamos haciendo nuestra labor. Yo se perfectamente que su sistema operativo es un juguete y por ello lo deberíamos instalar en los equipos de los niños pero, la verdad, me rehúso.

Atentamente
Edmundo Carmona
Ingeniero de Computación

ACLARATORIA
Ya que no tengo idea de con qué puede salir Microsoft a raiz de este correo, quiero dejar claro que esto es un chiste y que por ningún motivo puede que ser interpretado como que está relacionado de alguna forma a FLISoL. Es mi creación exclusiva y nadie más de FLISoL participó en su creación.

English Version

A little help for FLISoL - Open Letter to Microsoft

2009-04-19T12:12:00.000-07:00

Dear Microsoft

Before I make my request, let me first state who I am. I'm Edmundo Carmona, a venezuelan computer engineer living in Colombia. I've been a GNU/Linux user/administrator for years and, unlike others in the FLOSS community (many with far greater accomplishments than mine), don't trust you a tiny little bit and I bash you every time I can (940).

During the last couple of weeks I've been helping out in the organization of FLISoL. It's the Latinamerican Free Software Installation Festival, which will take place next saturday. We will be installing Free Software on computers that people take to the locations we set up for free (as in no monetary charge). In a sense, we are just responsible citizens trying to keep technological drugs off the streets.

During our last meeting there was a question about what we will be doing with people that carry their kids along with them to the location. That's where you come into the scene.

See, given the fact that Windows Vista has been such a mess (probably comparable to Windows Me), I thought that you probably have thousands upon thousands upon thousands of Vista installation CDs/DVDs stacked one on top of the other waiting for the first poor soul to buy one of them.

Would you be kind enough to provide us at FLISoL Bogotá with say, 15 or 20 of those installation CDs/DVDs? We won't be using them to install Windows on the computers, don't have to worry about it. I will personally hand them out to the kids so that they use them to play around (as frisbies or just to scratch on their surface) while we are on our stuff. I know that yours is a toy OS and so we should install them on the kids computers, but I refuse to.

Yours truly...
Edmundo Carmona
Computer Engineer

DISCLAIMER
As I don't know what Microsoft could be up to after this letter, I will state that it is a joke and that by no means it should be interpreted as being somehow related to FLISoL. It's my sole creation and no one else from FLISoL (besides me, of course) helped in its creation.

Versión en español

Are macs more insecure than Windows / GNU/Linux?

2009-04-17T16:24:00.001-07:00

I've been very busy lately studying python to create a tool for FLISoL, so expect to hear about that within the next days (the event is to take place sat. April 25th).

I hit this article about security on Macs by Preston Gralla where he states that some security experts say that Macs are easier to crack than Windows and GNU/Linux.... now, easier to crack than GNU/Linux, I find that believable... but more than Güindous? I doubt it. Anyway, they are the experts and I don't mean to contradict them.

However, Macs are just another closed platform. It can't be verified to be secure by any third party, so perhaps it really is that they be more insecure than Güindous... given that there's no source code to review, I can't really tell (not that I would check the code, anyway).

How much DRM code is in OSX these days? Does it spy its users "a la güindous"? Does anybody know? No source code, so nobody can't really tell.

I have developed this analogy of someone who wants to buy a nuclear reactor. There are two organizations interested in providing you with their nuclear reactor.

- Provider # 1 gives you the reactor plus all the design information, all blue prints, everything but the kitchen sink!

- Provider # 2 gives you the reactor and doesn't give you a clue as to how it is built inside. It's a black box (or a massive gray one). All you have is the control panels and the documentation that this provider is kind enough to provide with (you know.... they can't give you everything for security reasons).

Given those two choices... which would you consider to be more secure/stable/reliable? Which one would you choose? I'd personally go for Provider 1. At least I know what I'm getting. And the guys are so comfortable with their design that they even give it away to buyers. Perhaps the guys at Chernobyl chose provider # 2.

And finally, what's a virus for Macs called? An iVirus?

Latinamerican Free Software Installation Festival'2K9

2009-04-07T09:11:00.001-07:00

Hi!

I have registered myself as an installer at the Latinamerican Free Software Installation Festival, a.k.a FLISoL (first time to work on a FLISoL ever, by the way). This year this event is going to take place on April 25th on many locations all around latinamerica.

I will be working in Bogotá, Colombia. In my location the number of registered installers is around 30 people and we expect to make 150 installations. We will be working at the Biblioteca Virgilio Barco, in case you want to show up. The registration process if you want to get GNU/Linux installed in your computer is already open and, in case you are not insterested in installing GNU/Linux, we will be doing installation of Free Software on other platforms as well (Firefox and OpenOffice come to mind).

I'm more than sure that organizers will welcome any kind of help you think you could provide. Interviews? Donations? Working hours? Anything goes! I'll gladly work as a translator, in case it's needed.

FLISoL
FLISoL Colombia
FLISoL Bogotá

One DVCM to rule them all (follow up)

2009-04-04T07:47:00.000-07:00

Hi!

There were some comments in my original story about the performance of some DVCMs where I was told that bazaar degrades pretty much when you have thousands upon thousands of revisions and that the repositories could be packed. I decided to follow suit and see where git and bzr would stand having some thousands of revisions in them.

First I used git-svn to import some 20,000 revisions of a project into git (I got the first 20,696 revisions from kde... there were roughly a million, but I thought that would be enough... as a matter of fact I spent a couple of days getting to those 20,696 revisions).

Importing
I exported the content of git and imported it into the separate VCMs to see how they would match up on that task.

Bazaar took hours to complete this import. The first 2000 revisions where imported in about 6 minutes... but by the end, every 100 (hundred, not thousand) revisions were imported in roughly 10 minutes (one commit every 6 seconds?). The repository would be like 554 MBs in size (after packing).

Git made the import (so that I matched apples to apples) in less than 5 minutes and ended up with a repository like 283 Mbs in size (after gc).

Operations
Halfway diff of the project to where it is in the last revision took bazaar some 9 minutes and 15 seconds. Git made it in about 28 seconds. I think bzr won't recover after that liver hook.

When I tried to move to that halfway revision, git took 17 seconds to do it (reset --hard revid), bzr took.... well, to tell you the truth, I forgot about it... I went for lunch, came back and it was still working on it. In Tenchu terms, git got a Grand Master (by the way... I'd love to play Tenchu!).

Conclusion
Well... git did mop the floor with bzr on a big repo after all, both in terms of repository size and performance.

Sidenote
Should I include mercurial? Could it withstand git? How do I make the import to begin with? I tried with hg import -, but it was using massive amounts of memory (bzr did too, by the way... I barely made it to import with the memory I had) and I didn't know if it was the right way to do it.

Update
bzr finally reverted. It took 46 minutes.

One DVCM to rule them all

2009-03-28T09:46:00.000-07:00

Hi!

Given the recent news that GNOME development will be moving to git soon, I decided to take a look at three DVCMs to see how they matched each other. Not that I made exhaustive tests. I just thought of a series of operations to perform on a small and a big project to see how they matched against each other.

I compared git, mercurial and bazaar (the one I use).

The small project I used was a project of mine (some thousand lines of code.... not too big). I used a sequence of ten revisions taken from the project and stuffed them in the three VCMs.

The big project was linux (the kernel). Given the time that some operations took (plus the room that it took on my already mostly filled up box) I only tried with revisions 2.6.27 (327 MB) and 2.8.28.8 (348 MB).

I measured performance both in time taken for the operations and room taken by the repository. I'm using my dated box (4 years old?) on jaunty using the repository packages. Verbigracia: git 1.6.0.4, hg 1.1.2 and bzr 1.13

Here are the results:

Time Performance
Small Project
On the small project the absolute winner was git. Second was mercurial and third bazaar. Git made most of the operations 3 to 10 times faster than mercurial and the later did them mostly three times faster than bazaar. Of course, as it's a smal project, what git could do in the blink of an eye, bazaar could do it in a longer blink of an eye. The slower operation on all three VCMs was a revert after having deleted the whole project. It took bazaar 1,72 seconds on the biggest revision (the 10th), mercurial made it in 0,56 and git took 0,13 (on the 8th revision... I was already bored with seeing git kicking a55es).

Big project
Here I expected git to mop the floor with the other VCMs. Given Linus' dislike for slowness (at least on VCMs subjects) and that at least bazaar recommends not to use it for big projects, I expected to see git go faster than Ussain Bolt yet here I saw mixed results amazingly having bazaar (the slowest of them all on a small project) be the fastest sometimes (not that I did things people would normally do on a project.... I don't think you will work with a 20 MB patch between one revision an another, anyway).

On the first add (a 300+MB add) results were veeeeery strange. Here git dragged itself to let the others pass over it. Mercurial took ~6 seconds, Bazaar made it in ~41 seconds, but git.... well, it made it in ~106 seconds. That shattered my expectations... and that was for starters.

Status after an add: On the first revision, bazaar and git were very close around 3-4 seconds... bazaar made it at over 10 seconds.... but on the add of the second revision we had very different results: ~3 seconds for mercurial, ~14 seconds for bazaar and git made it at ~191 seconds.

For the first commit, git won easily. It took git ~55 seconds, Mercurial did it in ~131 seconds and bazaar arrived when the party was already over at ~203 seconds (mercurial was already drunk and git was on the way to the hospital)... however on the second commit (remember, it's over a 20 MBs difference), they were all much closer. Mercurial arrived last with ~168 seconds, git second with ~144 seconds and bazaar (oh, my!) first with ~130 seconds. Now I didn't expect that.

After the commit, (again) I expected git to fly when asked for a status. But it didn't deliver. Both times mercurial was first, bazaar was second with about double the time and git was third with about 5 times the time for mercurial. By the way, the status after the first commit was about 3 times slower than the second on all VCMs.

Now, when I removed all the content of the project (rm * -dfR) and asked for a status, we went back to normal: git first, mercurial second bazaar third. For both revisions, bazaar stayed around 8-9 seconds. Mercurial made it in ~8 seconds for the first and ~4 seconds for the second. Git made it in ~2 for the first and ~4 for the second.

Then I tried to revert (in bazaar and mercurial, reset --hard HEAD for git) after having removed everything. Here the result were strange again. Bazaar made it first with 213 and ~212. Mercurial and git exchanged places between the two revisions. First revision, mercurial made it in ~356 and git made it in ~435. Second revision, git made it in ~455 and mercurial in ~459.

Then I tried to go to 2.6.27. Mercurial took ~88 seconds and bazaar was waaaaaaaaaay behind at ~507. When i revert back to the last revision, mercurial made it in ~186 and bazaar made it in ~393. When I tried it in git (with a checkout) when I reverted to the first revision, I destroyed (I think) the second revision, so I decided to not include the time it took to revert/revert.

So, I thought that git would be the clear winner and the fact is that I got mixed results. Perhaps people can join in and give me some insights about why it was like that.

Repository Size Performance
Here there were no mixed results. Both for a small project and a big project, bazaar was the clear winner. Mercurial made it second and git arrived last.

Bazaar's repository size for the small project was around 10-40% smaller than mercurial's. Also bazaar was around a third the size of git's.

On the big project, here are the sizes:
Bazaar: 87152k for the first revision and 99456k for the second.
Mercurial: 148844k and 166122k
Git: 158912k and 228544k

Bottomline
Well.... I would have loved to give you a clear winner, but the fact is that there wasn't. For a small project you already know who the best is in terms of user time and repository size... but for a big project, it's a little more blury. I urge you to jump into the comments area to give me your thoughts on it.

As usual, keep in mind that other than performance, there are other differences between all of them in terms of features. It's not just about performance to decide what DVCM to use for a project.

Disclaimer
I could have made a mistake when working with mercurial or git, as I didn't know how to use them before writing this article (and I still don't). So if you think I could try to do something a little different, then go ahead, mail me or post a comment. Perhaps we could create a test suite or something to compare their performance.

Microsoft uses git for version control

2009-03-25T17:51:00.000-07:00

Not only has GNOME decided to switch to using git. We have learned that development at Microsoft is based on git too.

Development for Windows 7 is handled on a modified (in-house) version of git that allows it to run natively on Windows (the blessings of the GPL, I believe they must think at Redmond). Why reinvent the wheel when one of the best tools for the job is out there for free?

According to our sources, at first, some developers introduced git behind the backs of their managers by using it on top of cygwin. As they felt more comfortable with the tool and presented their benchmarks to management, it was obvious that git allowed development to be carried out in a much more efficient fashion. Then they started the process of making git run natively on windows without needing an intermediate layer, which allowed it to run even faster. It has been reported that the patches to git will be sent upstream once Windows 7 reaches RTM.

This may sound as shocking news for some people, but over the last couple of years Microsoft has been trying to get cozier with open source... and this is yet another example of it. Unfortunately Microsoft's Horacio Gutierrez wasn't available for comments, nor Lawrence Crumpton.

Having said that, did you notice that it's April 1st? It's the only way something that far fetched could ever be published.

My opinion: I hope they are using RCS... or even better: no source control at all. :-)

"Get the facts" on browsers

2009-03-25T10:57:00.000-07:00

When I was heading to download IE8 to test its JS performance, I saw this link about "Getting the facts" on web browsers. Having the knowledge of what "Get the facts" meant back in time when that was devoted to trying to put GNU/Linux in bad light against Windows-et-all, I just couldn't help myself and went in there to see what was showing up.

They have this video where they show how IE8 renders "index" pages of popular sites. Apparently IE8 kicks ass on that (as a matter of fact the differences between rendering times of the browsers they compare is very little, sometimes the blink of an eye). And so they have the guts to affirm that IE8 is one of the fastest browsers on earth. Is that really? Is rendering times for pages the real measure for browser speed? Come on! As applications are more and more "functional" and so becoming more Javascript intensive (yes, not VBScript intensive, but Javascript... thanks heaven) then I think Javascript performance is just as important.. or even more so that rendering times (specially if the difference in terms of rendering is so close between the browsers they showcase).

And how does IE8 do on that front (on my box)?
V8: 34.4
SunSpider: 9164.4

And, as you can see here, that's a looooooooong shot from being fast. So Microsoft, why don't you try to get the facts right at least once?

PS Were they saving that video for April Fool's day? I guess I just blew it.

IBM(+Sun) Vs Microsoft

2009-03-19T08:13:00.000-07:00

Well well.... "History repeats itself" is something we hear sometimes.

So IBM appears to be headed to buy Sun Microsystems. That's quite a move.... I have seen comments from people noticing how Solaris/OpenSolaris will affect/be affected by the merge... but I haven't seen people taking these other three things in consideration:

- Java
- MySQL
- OpenOffice

Lotus Symphony is based on OpenOffice so I guess it will be easily assimilated inside IBM. How will that affect OpenOffice development? It's being criticized before for being too Sun-centric. How will this affect the MS Office / OpenOffice saga?

I don't know how MySQL will play out inside IBM. Now IBM would be in charge of DB2, Informix (bought in 2001) and MySQL. Will that be enough to wipe SQL Server out of this world?

But Java! Java vs .Net is old news. And Java's development (as OpenOffice) had being criticized for being too Sun-centric. But now Java being pushed forward by IBM (in house).... that will be interesting to see.

Are these things gonna take the IBM/Microsoft battle into another chapter?

Sidenote
I wonder what Darl McBride would have to say about this. :-)

Give a presentation? Use a LiveCD!

2009-03-14T08:54:00.000-07:00

Hi!

Yesterday I had to give a presentation on the way (or say, my way) to develop a small (so far) web system. It's built on JSP and I'm working on Eclipse (no fancy thing... just yet). So I wanted to go there and show my workflow in terms of branches (plain application, application+design, etc) plus I would have liked to control my home computer remotely (through ssh remotely... if possible).

On top of eclipse (and java), I would have had to use bzr and qbzr... now... I wasn't going to be able to carry a computer with me, so I was going to get a box from the company I was doing the presentation for. Well.... what are my options? Expect to get a Güindous Computer ad start installing things on a computer that's not mine. That didn't sound cool. Another was to get a GNU/Linux box (perhaps) and start installing stuff (not cool either). But how about taking a LiveCD with me and carry along all things I'd need for the presentation (in case there's no internet access)? Let's give it a try.

I use a Kubuntu Intrepid LiveCD (but the procedure will wok like that on any debian-based distro). I have one standalone installer of the JRE from sun and Eclipse in a compressed format, so that side of the story is covered. But then bzr/qbzr? I'd have to get them from packages with apt... but how do I know what packages I will need? Get your notebook.

Home Procedure
Start the LiveCD, wait for it to finish starting. Then, add the stable repositories of bzr and qbzr in sources.list (that's because the normal repos of ubuntu intrepid have very dated packages.. at least of qbzr). So far, so good. Now, the normal procedure to install bzr/qbzr:

$ sudo apt-get update
$ sudo apt-get install bzr qbzr

By now, you have both packages installed in your LiveCD session.... but you want to be able to install them later on (like, at the company without an internet connection, right?). So, what can you do? Well.... let me tell you a little secret: If you didn't clean after you installed the packages, you can grab all those beautiful .deb packages (bzr, qbzr and their dependencies) from this location: /var/cache/apt/archives/

So, hook up a pen drive, mount it and copy those .deb packages on it:

cp /var/cache/apt/archives/*.deb /somewhere/

Now head for your presentation.

Company Procedure (or home test.... just in case)
You arrive at the company and are presented with the computer you will give your presentation on. So let's get our hands dirty right away. Start your LiveCD, hook up your pen drive and mount it. Then, install all those .deb packages:

sudo dpkg -i /somewhere/*.deb

And you are done. bzr and qbzr are installed. You had the jre and eclipse files in the pendrive too.. two branches of the project too, so you do the installation procedure and can start my presentation right away. Finish your presentation, turn of the box, give it back.... nothing happened. End of Story.

A little rant.... back in time...
I learned about this trick some time ago. About 3 years ago, I was going to a Cisco Accademy course (CCNA preparation) and I remember that when we reached the DHCP part, our teacher (by the way, and old friend of mine from my going-to-the-youth-orchestra days, Juan Jakymec) told us that one of our fellow students was going to have one of the boxes converted into a Güindous Server so that we could see how it worked. I volunteered to have GNU/Linux used as a DHCP server too.

Comes the day.. the other guy had showed up at the classroom who knows when in order to install Güindous Server on the computer. So, I'm told to go first, I use this trick that I described before to install the DHCP server, try to start it but it fails.... so I dig a little on the logs (or perhaps running it as a normal process to see its output... can't remember). I get the error cause of the IP addresses that the DHCP service is told to provide and the IP address of the network interface. So I change the configuration of dhcpd, start it.... we're ready to go. I even start tcpdump so that we can see what's going on between the server and the clients. All lovely. Less than 10 minutes after I was given the chance to do the show, I'm heading to my chair with a big ear-to-ear smile on my face.

Time for the Güindous Server to show off.... well... let's just say that it wasn't pretty. You know how these things work in güindous when something fails. You get a small window telling you that something failed... no details. So he starts going (guessing, basically) here and there to see what happened... tries and retries.... well... some minutes go by and then he is able to start the service... no tcpdump, though. Bottom line:

Güindous = esoterism, GNU/Linux = determinism.

Come on.... think about it... I was able to do the whole thing in 10 minutes.... I left the computer as it was before I had a finger on it.... the other guy? he had to install a whole operating system beforehand in order to see a dhcp service working. Does it even make sense at all?

Jaunty: Apt is broken? Move to another country!

2009-03-12T08:53:00.001-07:00

Hi!

You might remember that I had switched to Jaunty to give it a test drive. Well, since yesterday I was having this problem with apt. It was throwing a segmentation fault whenever I tried to work with its repos (upgrade would seg fault, and dist-upgrade, and the same for install). In other words.... a broken apt = an un-upgradable system. Not good. That's not something I wanted to face at all. Having X not start once in a while is something... but having a broken package manager is a whole different story.

After submitting the bug to launchpad and spending a little time at #ubuntu+1 one guy there (Thanks, IntuitiveNipple) noticed that one of the files that apt handles appeared to be broken (/var/cache/apt/pkgcache.bin). I removed the file, but the file stayed there on the FS (go figure!). After some attempts at it, I was able to remove the file. But then I tried to update again... and BOOM! SEGFAULT. It's not going to give itself away so easily.

So.. after about 24 hours I'm stuck with no clue of what to do. I even had the crazy thought of reinstalling kubuntu from scratch (hell, I hated that thought.... made me think of my long forgotten days of formatting every 6 months to reinstall güindous). But I did a simple test instead: Switched my repos to use the venezuelan repositories instead of the colombian ones:

$ sudo su
# cd /etc/apt/
# sed 's/co\./ve\./' sources.list > sources.list.ve
# mv sources.list sources.list.co
# mv sources.list.ve sources.list
# exit
$

And voilà, apt is back on track. So the problem was the repos for colombia after all. Hope they solve it soon.... and why didn't apt complain about a discrepancy or something instead of exploding? Weird.

Oops! Removed all kernels!

2009-03-06T05:49:00.000-08:00

Hi!

My last post on this blog was about how to switch from intrepid to jaunty without having much room on hard drive. One accident I had while doing this was that I removed all my linux kernels and so I ended up with a box that could do mem tests all day long or boot into windows (god forbid). So, here's how I got it back. I did it on kubuntu jaunty, but it could work almost the same if you tried on debian or any other distribution that has a package manager (or even without it, though the steps will change, of course).

First thing, I booted the box with a Kubuntu LiveCD. After having booted, I mounted the root partition of my busted GNU/Linux:

$ sudo su
# mkdir /mnt/tmp
# mount /dev/sda5 /mnt/tmp

After having mounted, let's get into a chroot cage so I can manage packages from that mounted partition as if it were root:

# chroot /mnt/tmp

Well... it's basically like being on a terminal on the GNU/Linux that has no kernel. I'm at the root of that box, I have a working internet link..... so, let's just install the missing kernel package:

# apt-get update

# apt-get install linux-image-generic

We let it do its stuff and reboot and tadaaaaaaaaaaaaaaaa, we're back on track. See? It wasn't that difficult, was it?

PS If you have separate boot partitions or perhaps a separate /usr, perhaps you'll have to do more mounting before you do the chroot, but the concept remains the same.

PS2 One reader left a comment where he said that he hated the terminal and wanted a GUI based application to get it done. I was going to post a picture of such an application: konsole. :-D I don't know if such application really exists. I think the situation I described is 1 fairly uncommon, 2 rather easy to fix (if you know where to look). But if you feel really lost, then email me. I'm always available to help. As a matter of fact, back in december I got a box of 9 bottles of italian wine for helping an italian guy recover some data from a broken RAID5 (I wish all people thanked that way... thanks, Marco!).

From intrepid to jaunty.... with no room left

2009-03-04T14:07:00.000-08:00

Hi!

Today I decided to move my desktop to jaunty. It's still an alpha but I'm not afraid of having to face a non-working Xorg once in a while and having to use the VTs instead. Given the speed improvements I've read about... the time is now... there's just a tiny little problem. I have no room left to do the dist-upgrade:

$ df
S.ficheros Bloques de 1K Usado Dispon Uso% Montado en
/dev/sda5 5162796 4783244 117296 98% /
tmpfs 508248 0 508248 0% /lib/init/rw
varrun 508248 228 508020 1% /var/run
varlock 508248 4 508244 1% /var/lock
udev 508248 2824 505424 1% /dev
tmpfs 508248 12 508236 1% /dev/shm
lrm 508248 2004 506244 1% /lib/modules/2.6.27-11-generic/volatile
/dev/sda8 6190664 5793332 82864 99% /home

Anyway... let's just try, ok? This is the second time i have to do this process. Why? Cause it's my wife's computer and I have a little space set up for GNU/Linux on it (and if people don't start using www.cdfleamarket.com, I'll have to repeat the process for Kinky Kangaroo... I mean, Karmic Koala as well.... so why don't you give me a hand and use the site? It's free anyway!).

First, let's set up apt to use the jaunty repositories instead:
$ sudo su
# cd /etc/apt
# sed 's/intrepid/jaunty/' sources.list > sources.list.jaunty
# rm sources.list
# ln -s sources.list.jaunty sources.list
# exit

I like to keep the different versions of sources.list so I have a version of each ubuntu release sin edgy.

Now, let's try to dist-upgrade to see what happens:

$ sudo apt-get update
$ sudo apt-get dist-upgrade
1010 upgraded, 98 newly installed, 7 to remove and 1 not upgraded.
Need to get 1073MB of archives.
After this operation, 639MB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/

Well.... that's a long no.

Ok... here's the deal: As long as you keep the package ubuntu-standard installed, you can remove basically everything else and the system will keep on working. So let's remove all this desktop packages (a lot of them at least) so that we have a system basic enough that we could dist-upgrade.

You have to know how to use a VT before you proceed, cause you won't be able to use the graphical environment for a while.

If you are ready, just close your graphical session and head to one of the VTs.

I use kubuntu, so the first thing is to remove the kubuntu-desktop package so that I get "permission" to automagically remove all the other packages that depend on it (if you use Ubuntu instead, then it's ubuntu-desktop):

$ sudo apt-get remove --purge kubuntu-desktop
roughly 50 Ks freed. :-S
$ sudo apt-get autoremove --purge
Nothing :'(

Oh well.... I didn't expect that. Seems like kubuntu-desktop has no dependencies... that could be because I use the repositories for KDE 4.2. So let's try with something a little more straight-forward:

$ sudo apt-get remove --purge kde'*' openoffice'*' xfce'*' firefox'*'
168 packages to be removed
771 MBs to be freed
3 Mbs of data to install right away

Much better now.

$ sudo apt-get autoremove --purge
another 153 packages to eliminate
235 Mbs to be freed

Now I have about 1Gb of space, but apt-get update still needs to download
718 MBs of data plus take another 530 MBs. I have to keep on cutting stuff out.

Let's go with some other things that take a lot of room till I get enough free space:
$sudo apt-get autoremove texlive'*' lilypond'*' --purge
I didn't know I could ask autoremove directly to remove packages and their unnecessary dependencies. You never know when knowledge will hit at your screen.
200+ MBs freed
Now the dist-upgrade would barely fit in there, but let's make a little more room just in case:

$ sudo apt-get autoremove xorg --purge
About 1.4 GBs are free by now.

There are still a lot of packages that are not basic still installed... but I won't care about them now.

Let's try to dist-upgrade to see how it goes now.
sudo apt-get dist-upgrade

Now it's a 500 MBs download plus 357 Mbs to take for installation. I'll let it do it like that, then I'll clean and then I will install kubuntu-desktop so I get the environment back.

I'll use lynx to read the latest news on another VT while the download takes place to see what I'll do next.

After a text round of www.linuxtoday.com, www.osnews.com, www.eltiempo.com, www.noticias24.com, www.talcualdigital.com, prelude of the Suite Antillaise of Francis Kleynjans (that'd be me playing guitar), Aire de Zamba and a little more guitar fiddling, it's downloaded. Now I have to see if the upgrading process asks me something (some more guitar fiddling).

dist-upgrade is done. Let's run it one more time just in case. Ran it. Nothing is left to do. So let's reboot so I can uninstall this kernel I'm running from intrepid and continue the upgrading process.

sudo reboot

Now... booting was FAST! At least to the console login. Let's see if this wonder can keep that way after I install the desktop ;-).

Let's remove the old kernel then:

ls /boot/vmlinuz*
There I see that I have a 2.6.27-x kernel.

Let's remove the 2.6.27-whatever:

sudo apt-get autoremove --purge linux-image-2.6.27'*'

Some more packages went away along with it. A total of over 280 Mbs are freed.

Sidenote: I just discovered (next morning) that the new kernel went away with it, so perhaps it's a good idea to remove the linux-image-2.6.27-x-generic instead (providing the exact number in x instead... so watch out!). Now... how do I install a new kernel on a non booting system? I'll be back!

Now, let's remove all the .deb files I've downloaded so far that are already installed so I free up that space as well (that won't uninstall the packages... just remove the .deb files that were used to install those packages). Before I clean, I have about 800 Mbs free.

sudo apt-get clean

Now, I have about 1.3 GBs free.

Next step? Let's bring back the desktop, shall we?

sudo apt-get install kubuntu-desktop

257 Mbs to be downloaded, 825 MBs to be used on top of that. So be it.

Here, I just hit my first problem. The network is down. Why? I just don't know right now. ifconfig -a shows me eth1 instead of the usual eth0, tried reconfiguring the interface using /etc/network/interfaces but still won't come up. So I take it to the next lower level:

sudo dhclient eth1

I get the configuration from my ISP. So let's go on installing so I can focus on this problem later (and I won't publish the solution in this post.. perhaps in another post tomorrow or so).

Again:
sudo apt-get install kubuntu-desktop

It does work now. Install finishes and now let's go back to graphics mode.

sudo /etc/init.d/kdm restart

And we're back in track. Now I have to install the packages that I use that were uninstalled in the process... but I'll do it as I need them.

Now, see if you can do that from Vista to 7. :-)

PS: I was able to reinstall the new kernel after having removed it. The keywords to fix it are: LiveCD + chroot.

Bill Gates suffers from digital phobia

2009-03-02T17:10:00.000-08:00

Yep... the same guy who is credited with creating the "desktop computer" (or so they say) must be suffering from phobia to anything electronic.

How can we tell? Well... I inferred that after knowing that Bill Gates has banned iPods and iPhones at home... in the same page where we find that Steve Ballmer banned using google at home. We already know that Bill Gates hates GPL. But then, if everything that is against Microsoft || Windows won't be used by these two (and their families), I can't help but wonder... will they really try to escape using GNU/Linux? Come on... it's like escaping big brother. It's everywhere... even if you can't see it. Does Steve Ballmer use (did Bill Gates use) Microsoft's Wireless? If he does, shame on him. Do they ban sites powered by anything but IIS? If that's the case, then poor guys.... they can only use about 30% of the web.

So.... are Bill Gates and Steve Ballmer's lives going to be increasingly restricted in terms of access to digital resources as time goes by and GNU/Linux makes more and more inroads into our daily lives in unnoticeable ways? I wish they were. It'll be cool to have them both (specially Steve) recluded in one of those monasteries that time has basically forgotten about.

Server sales: If you're falling, dont' fall so hard

2009-02-26T15:47:00.000-08:00

And I thought I wasn't going to post anything today.

I just saw the article of ServerWatch on server sales on Q4'2008. I had been waiting for it for a while to see how bad the quarter had been given the economy.. but mostly I wanted to see how the GNU/Linux vs Windows battle on the server space is going.

For Q2 2008 (the IT sector not in decline yet): Windows had a YoY gain of revenue of 1.38% (with market share down to 36.5% from 38.2 one year earlier), UNIX gained 8.22% (market share up to 32.7% from 32.06%) and GNU/Linux had a revenue gain of 4.55% (market share down to 13.4% from 13.6%). GNU/Linux + UNIX make up 46.1% of revenue market share.

For Q3 2008: Windows had a loss of revenue of 2.86% (reaching 40.8% from 40.4%), UNIX lost -8.15% (ouch!.. down to 29.7% from 31.1%) and GNU/Linux barely made it even with a gain of 0.49% (reaching 14% from 13.4%) . GNU/Linux + UNIX held 43.7% of the market.

Q4 was a poem (if you ask me... I doubt Steve Ballmer will agree): Windows nosedived with a loss of 17.07% (double ouch!!... down to 35.3% of the market from 36.6%), GNU/Linux had a loss of 7.92% (reaching 13.6% from 12.7%) and UNIX lost 6.52% (reaching 36.2% from 33.3%). GNU/Linux + UNIX got 49.8% of the market.

The numbers and percentages can be a little misleading... so let me try to mislead you a little more: If you look at the amounts of gross losses/gains of revenue, for example, you will see that in Q2 2008, Windows had an increase of YoY revenue of roughly US$ 70 million, GNU/Linux made a little more with 80 million (it's hard to know cause the numbers are rounded, so perhaps the difference could have been a little less or more) but UNIX had an increase of 350 millions (wow!). Q3 is another story: Windows had a decrease of revenue of 150 million, GNU/Linux broke even and UNIX had a decrease of 330 million. And Q4... well, this is not for the faint of heart: Windows had a YoY decrease of revenue of..... 980 million, GNU/Linux had a decrease of 150 million and UNIX had a decrease of 340 million.

Bottom line: If on good times you are growing well, and on bad times you are shrinking less than your enemy... I think you aren't doing so bad after all, don't you think?

The second bottom line: These are statistics and we know that they can say whatever we want them to say.... so take those numbers with a grain of salt.

Disclaimer: The numbers, though I guess are right, could be wrong. I had to put together numbers from different sources (mostly IDC and Gartner) so that there could be a mistake here or there (though I wouldn't expect it to be so dramatic). Want to get the file I have with the numbers? Let me know and I'll email it to you.. perhaps we could improve it.

Windows = Antivirus = Pollution?

2009-02-25T15:08:00.000-08:00

I just read an article by the University of Calgary where the author claims (and I think he's correct) that IT is a huge pollutant. For example, we have hardware that becomes obsolete, we have to produce electricity to pump into our gadgets, etc.

Not long ago I read another article where it's calculated (or so they say) how much pollution is produced by each search on google.

But, man.... I just couldn't resist the temptation of asking myself "... then how much power is spent on Windows implementing DRM protection mechanisms?". It has been disputed that it be a lot of energy to implement DRM mechanisms. It has been argued that this is not too much of an effort... that DRM in Vista is rougly a couple of LOC on the whole system. I just couldn't care less about it.. but then the next even more obvious question was "then how much pollution is produced by the usage of antivirus?". And here you won't tell me that it's just a little effort. Antiviruses checking whole computers (millions of them) weekly (at the very least), an operation that can take a while to complete, plus the effort of checking every jpg file that gets into a system. And using the antivirus is no low-cpu-ussage activity. I know that when a computer running windows is dragging behind a turtle for no apparent reason, I could just check processes to see if the antivirus is doing its stuff, if the box hasn't already being been invaded by the random virus that's hot at the time and that is eating all of the CPU sending all those beautiful Xmas mails.

What bothers me the most is that windows users are still paying for the ultimately bad design that was implemented on Windows early on (every .exe you downloaded from internet could be executed right away, default user is administrator, programs that won't run unless the user is an administrator, Firewall? What's that?, the usually long etc.). Vista is barely trying to fix all those problems, and we all know the backslash that things like UAC has been for Windows Vista (at least in its inception).... but we know where Vista is staying in user's preferences... so people are sticking with XP design flaws instead... and seems like it will be a little longer till it fades away into oblivion.

So... coming back to the question: Windows = Antivirus = Pollution? Can anybody try to make a wild guess about how much pollution is produced by antiviruses?

PS And I didn't mention hardware that's not capable of running today's systems. How many times have you being forced to buy more hardware (or another computer) just to get the latest incarnation of Windows to work acceptably well turning your perfectly working system (so far) into digital trash? That's another thing where at least GNU/Linux will help you avoid as well. As a matter of fact, I'm using the very latest release of Kubuntu, patched to use KDE 4.2 (using some of its 3D eyecandy, by the way) in a computer that's a little dated (I guess 4 or 5 years is a safe guess) on a box with a D865GVHZ motherboard (4 years old? Maybe 5?). I wonder if I could run Vista with Aero on this box. I guess that makes up another equation: Windows = New Hardware = Pollution? By the way, I'm sure other OSs will help you avoid those update cycles as well... but my experience is with GNU/Linux, so I won't speak for other OSs.

Browser Wars: JS performance on my dated box

2009-02-20T09:10:00.000-08:00

Well.... let me put it in simple terms: What I'm about to write is not gospel. It's just the result of running a couple of tests on a number of Web Browsers on my rather dated computer. So I don't intend to convince you of dropping your usage of one browser and start using another. It's just another post in the already long (long, long, loooooooong) list of posts comparing the performance of said browsers. Hope you find it helpful &&|| informative (in any way).

So... let's go down to our subject.

I'm comparing IE8b1, IE7, Opera9 on Windows, Opera10 alpha on Windows, Opera9 on GNU/Linux, Opera10 alpha on GNU/Linux, FF3 on Windows, FF3.1 on Windows, FF3 on GNU/Linux (from packages), FF3.1 on GNU/Linux (downloaded from Mozilla's site) and Konqueror (from packages... updated to KDE 4.2). Unfortunately I didn't test Chrome cause when I tried to download its installer, I got the installer of Google Earth instead (go figure!).

The hardware is like this:
- Very dated PC with one Intel(R) Pentium(R) 4 CPU 2.80GHz (hyperthreading disabled), 1 GB of RAM. Do you need to know something else?

The software environments are like this:
- Windows XP SP2 (pretty much unpatched... as a matter of fact, I couldn't care less about it... and don't start nagging me saying that I have to keep it updated. As I just said, I don't care for it. I don't use it).
- Patched Kubuntu intrepid running on xfce for the tests.

So, let's get down to the matter.

I ran the SunSpider and V8 benchmark suites on all of those browsers.

Here are my comments:
- IE8 has been a tremendous improvement over IE7. Still, IE8 is behind every single browser, except for IE7 and Opera9 on GNU/Linux on the SunSpider test. So.... given that every new browser is ahead of IE8, I can't help but wonder if Microsoft is going to provide IE8 with a digital casket along with it when it be released.
- Opera has made an improvement in both tests from 9 to 10 on both GNU/Linux and Windows. But then, some days ago, there was a report on FF being slower on GNU/Linux than on Windows and we see exactly the same thing going on with Opera. Is there a reason why this could happen on both browsers? I don't think the widget toolkits are to blame, as FF is built on Gtk+ and Opera is built on Qt (version 3, by the way... is it possible that building Opera on Qt4 would improve its performance on GNU/Linux?). On both platforms, Opera10 was the best performer on the V8 test.
- FF3.1 had a better performance on both platforms on the SunSpider test (being the improvement more dramatic on Windows), however on V8 the results were mixed, on Windows there was an improvement, but on GNU/Linux there was a performance decrease (not too big, but still a decrease).
- Konqueror certainly has to make great improvements to catch up with the other browsers.

Let the flamefest begin! It's not my intention to start a flame war but I think everyone will want to add their own pepper to the mix. You're welcome to do so on the comments area... just keep it in the "readable by kids" rhetoric.

Has IE lost the hearts of IT people?

2009-02-19T08:11:00.001-08:00

I like to see statistics, specially where FLOSS is gaining ground. I try to take them with the usual dose of a grain of salt (or a handful of salt... depending on the source).

- Web browser market share
- OS market share
- Web server survey
- Most reliable hosters

They all have their problems to be measured, but still they do give an idea of trends, at least.

It's not breaking news that FF has gained a lot of momentum on the browsers front. Some people even say that we are in the "Browsers Wars" all over again. Hitslink provides us with some very attractive statistics on that. During January, IE reached 67.55% of browsing (down from 68.15%), FF reached 21.53% (up from 21.34), Safari reached 8.29% (up from 7.63%), Chrome reached 1.12% (up from 1.04%) and the remaining browsers had less than 1% each. That's fine and dandy, but that comes from a huge market of thousands of sites that are not particularly inclined to IT subjects (my guess). I have a hunch that in sites that are inclined to IT, usage statistics of FF are much higher than those provided by
hitslink.

I have been tracking one site that's devoted to web subjects and that's not inclined towards one browser or another. It's www.w3schools.com and the statistics from it are... well, very different from hitslink's.

Here we find that FF has reached over 45% and, as a matter of fact, for the first time, FF reached more market share than all versions of IE they display in the statistics.... combined. We see an ever growing usage of FF and a steady decline of IE.

There are some things that I have noticed that would explain such a decline:
- It's easier to find people that have heard of/used FF in IT circles.
- People in IT would be more inclined to install other applications besides the ones that come bundled with whatever OS they get (I don't install other applications besides the ones bundled with the OS I get.... I install another OS).
- It's no secret that IE is a resource hog when compared with other browsers. Hell, even IE8 (which is in RC status) comes way behind in terms of performance when compared with any other mainstream browser. Side note: Would FF on GNU/Linux be close to IE8? I guess there's material for an article there. :-)

And that leads me to the question: Has IE lost the hearts of IT people?

I'm more than willing to see other statistics from sites that are technology inclined and that have platform agnostic (or multiplatform) content. Do you have a site like that and would like to share those statistics to the world? Then the comment area is waiting for you to guide us.... or guide me, at the very least.

Quickfix: My ISP is blocking connections to one of my ports

2009-02-18T05:36:00.000-08:00

Hi!

Recently, one friend of mine wanted to show me something he was creating that he had running on a web service at his own box. He was connected directly to the internet and so he provided me with his address like this:

http://x.x.x.x/resource/

x.x.x.x being his public IP address. It failed miserably. I couldn't see what he had up there. He then told me that people could see it remotely when he was at home, but that he was at his workplace and didn't know what was going on. I told him that probably his ISP was blocking requests to port tcp 80 and that's why it didn't work. I told him (to make sure) to do a brief tcpdump to make sure that traffic wasn't arriving at his box on that port:

tcpdump -i eth0 -n tcp and port 80 and host y.y.y.y

y.y.y.y being my public address, so that only the traffic I was about to send to his box would be displayed. -n is used so that tcpdump doesn't try to do a "reverse name resolution" of our IP addresses. I do this from my box:

telnet x.x.x.x 80

After some seconds, his tcpdump output is completely mute. Well.... it's his ISP the source of the problem after all. I know that normally one ISP won't block all new requests to a destination port, only some ports are banned (for security reasons, I guess). So I wonder that we could simply use another port. I tell him to redo the tcpdump but listening on port 8080. But (without having tried) he complained saying that he didn't want to change the configuration of apache to listen on another port. I tell him to relax and let go. He won't have to do that.

tcpdump -i eth0 -n tcp and port 8080 and host y.y.y.y

Then I redo the telnet to the new port:

telnet y.y.y.y 8080

And there he had some output on tcpdump. I get a connection refused message (because he had no service running on that port). What it all means is that his ISP is not blocking that port. Tip: tcpdump will show you traffic that arrives at a box, doesn't matter if you have netfilter rules on FORWARD or INPUT that will block that traffic or if there's a service running or not on that port.

Now comes the trick: How to make apache listen on that port without changing its configuration?

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8080 -j REDIRECT --to-port 80

The REDIRECT target of iptables is used to tell iptables to change the target of those packets, no matter what the target of those packets be, to the box that's processing those packets to a given port. And there you have it. After running that command, I could see what he wanted to show me without having to change anything on his apache configuration.

http://x.x.x.x:8080/resource/

And the content was there.

Oh, you are running windows and want to do the same thing, you say? I guess you have to get yourself an ISA Server to be able to do NAT (though I could be wrong, of course) or download some virus-ridden piece of freeware that you could find out there that will do that... plus turning your computer into a zombie, as a gift feature. In other words: Why don't you get a nice LiveCD (didn't say what distro or what operating system) and start tinkering with a real OS? If you were able to figure out that it was your ISP that was blocking the requests to a port in your own box, it means you already have the basic ingredients.

Take care!

A couple of things on PHP

2009-02-17T10:59:00.000-08:00

Hi!

As I promised yesterday on Bash Tricks I, I would be making a spin off article on things related to PHP.

I want to talk about two things, actually.

1 - Performance: Are constants faster than variables?
2 - Security: My usage of a FS_ROOT constant (that could become a variable depending of the results of point 1) to hide all non-starting scripts from apache.

Variables vs Constants
Yesterday I ran a script 1000 times to see if a script that handled a variable or a constant was faster. The one with the constant was a little faster. Let's redo the test but with 10000 times instead:

variable.php (without the php starting tags):
$VARIABLE = 5;
echo $VARIABLE . "\n";

constant.php:
define('CONSTANT', 5);
echo CONSTANT . "\n";

Let's run them then:
echo Variable; time ( i=0; while [ $i -lt 10000 ]; do php variable.php > /dev/null; i=$(( $i + 1)); done ); echo Constant; time ( i=0; while [ $i -lt 10000 ]; do php constant.php > /dev/null; i=$(( $i + 1)); done )
Variable

real 12m49.269s
user 4m31.665s
sys 2m23.161s
Constant

real 12m36.780s
user 4m28.013s
sys 2m27.241s

Well.. not much difference, really.

Now.... does this difference translate into a php script running on apache? To pull it off with bash, I will have to use one of my favorite (and most basic) tricks I've come to use when doing web development: Acting as a web client from a terminal. As a matter of fact, bash won't be the client... but I will certainly use bash in order to run the request a number of times. How does it work? As web developers most probably know, we can use telnet to make requests on a web server. Let's do a basic request: www.yahoo.com but on a yahoo.com and let's ask for the default web page to see what it says:

telnet yahoo.com 80
Trying 68.180.206.184...
Connected to yahoo.com.
Escape character is '^]'.
GET http://www.yahoo.com HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Feb 2009 19:15:04 GMT
Location: http://www.yahoo.akadns.net/
Connection: close
Content-Type: text/html; charset=utf-8

The document has moved here.

Connection closed by foreign host.

After I connected to the web service successfully, I made the requests GET http://www.yahoo.com HTTP/1.0 followed by an empty line, and then the web service replied with the headers, an empty line and then the content of the web page.

Cool.... but I won't be doing that 10000 times to test my scripts on apache, right? Instead of telnet, let's use another tool: netcat. Then we can send netcat the web request to its input stream, effectively sending it to the web server. Like this:
{ echo GET http://www.yahoo.com/ HTTP/1.0; echo; } | netcat yahoo.com 80
HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Feb 2009 19:20:15 GMT
Location: http://www.yahoo.akadns.net/
Connection: close
Content-Type: text/html; charset=utf-8

The document has moved here.

Cool, now I can make as many requests as I want one after the other, and so I can see how long it takes to run to each of the scripts. So, let's see:

echo Variable; time ( i=0; while [ $i -lt 10000 ]; do { echo GET http://localhost/variable.php HTTP/1.0; echo; } | netcat 127.0.0.1 80 > /dev/null; i=$(( $i + 1)); done ); echo Constant; time ( i=0; while [ $i -lt 10000 ]; do { echo GET http://localhost/constant.php; echo; } | netcat 127.0.0.1 80 > /dev/null; i=$(( $i + 1)); done )
Variable

real 2m17.096s
user 0m58.724s
sys 1m8.092s
Constant

real 2m7.158s
user 0m55.163s
sys 1m2.608s

Well.... I notice two things here:
1 There was a roughly 10% reduction when using constants
2 This puts in perspective the Spawning processes is expensive mantra, doesn't it? A reduction of roughly 75% when compared with running the script with the PHP binary.

Parenthesis: I tried with 1000 times instead of 10000 and something weird happened. The tests ran in under 7 seconds each (always with the constants as a winner), but that's almost 20 times faster (instead of the expected 10 times). Any explanations for it? End of parenthesis.

Now, if you wanted to use the value of the variable inside a function without passing it as a variable, you would have to use a "global" directive and then use the value, but there's no need to do that with a constant. Let's see how that changes the result:
variable.php
$VARIABLE = 5;

function printValue() {
global $VARIABLE;
echo $VARIABLE . "\n";
}

printValue();

constant.php
define('CONSTANT', 5);

function printValue() {
echo CONSTANT . "\n";
}

printValue();

Le's run the scripts 1000 times:
echo Variable; time ( i=0; while [ $i -lt 10000 ]; do { echo GET http://localhost/variable.php HTTP/1.0; echo; } | netcat 127.0.0.1 80 > /dev/null; i=$(( $i + 1)); done ); echo Constant; time ( i=0; while [ $i -lt 10000 ]; do { echo GET http://localhost/constant.php; echo; } | netcat 127.0.0.1 80 > /dev/null; i=$(( $i + 1)); done )
Variable

real 2m20.279s
user 0m59.952s
sys 1m7.548s
Constant

real 2m11.901s
user 0m57.440s
sys 1m3.176s

Again the constants are winners with a little less than 10% reduction in time. So I guess that's it on this subject. Constants are winners over and over again.

My usage of FS_ROOT

After so much time dealing with PHP and requires/includes, I came to use a constant that always tells me where the root of the project is on the File System. This value along with a number of others (HTTP_ROOT, DB settings and so on) is in a single script (strangely enough, it's called conf.php). Now, no matter where I start the execution of the script, I know where to find conf.php relative to this starting point and then I just don't care about where the others scripts are... I always call them using FS_ROOT as the root directory of the others scripts:
require_once FS_ROOT . "/model/one_model.php";
require_once FS_ROOT . "/utilities.php";

You get the idea. I guess that's not rocket science.... but then something weird happened. I was integrating phpweby's ip2country into my new project. This module has a script that can update the ip2country information on the DB that has to be deleted for security reasons... and then, it hit me: That script, which is not supposed to be called from the web, could be outside of the space mapped by apache and then I wouldn't have to delete it, would I? And that brought me to another thought: How about if I didn't have to publish ANY of the scripts that I use in my project, besides the starting scripts? And that's when FS_ROOT becomes vital. By using FS_ROOT to locate all the other scripts, they could just be outside of the apache mapped files and so the project is "safer", don't you think?

So, now the project I'm working with has like 20 scripts "inside" apache, and all the other scripts (a whole bunch of them) are safely protected outside of apache's reach. Now, I've never used root cages, so I don't know if a cage would allow this kind of behavior. What do you think of the trick?

Well. That's it for today. I hope you can take advantage of this information.