Open Source Adventures

Our mailserver had a hard time last weekend
keeping up with the loads of emails thrown at it. It took 2 minutes to get an
ssh connection, because all other processes than qmail and spamd were kicked
out of memory and swap space. This is when I decided to try out greylisting. What a
success!

But what I wanted to show here are two lists:

> sqlite3 /var/db/greylite/greylite.db "select ip from pending" | xargs -n 1 geoiplookup | sort | uniq -c | sort -nr | head -n 15
389 GeoIP Country Edition: US, United States
285 GeoIP Country Edition: KR, Korea, Republic of
209 GeoIP Country Edition: TR, Turkey
208 GeoIP Country Edition: RU, Russian Federation
208 GeoIP Country Edition: IT, Italy
194 GeoIP Country Edition: RO, Romania
179 GeoIP Country Edition: BR, Brazil
172 GeoIP Country Edition: ES, Spain
167 GeoIP Country Edition: CO, Colombia
166 GeoIP Country Edition: AR, Argentina
163 GeoIP Country Edition: PL, Poland
134 GeoIP Country Edition: --, N/A
120 GeoIP Country Edition: CL, Chile
118 GeoIP Country Edition: CN, China
115 GeoIP Country Edition: MX, Mexico

and

> sqlite3 /var/db/greylite/greylite.db "select ip from verified" | xargs -n 1 geoiplookup | sort | uniq -c | sort -nr | head -n 15
141 GeoIP Country Edition: US, United States
 28 GeoIP Country Edition: TW, Taiwan
 22 GeoIP Country Edition: GB, United Kingdom
 20 GeoIP Country Edition: BE, Belgium
 14 GeoIP Country Edition: RU, Russian Federation
 14 GeoIP Country Edition: FR, France
 14 GeoIP Country Edition: DE, Germany
 12 GeoIP Country Edition: NL, Netherlands
 10 GeoIP Country Edition: JP, Japan
 10 GeoIP Country Edition: CN, China
  9 GeoIP Country Edition: IN, India
  8 GeoIP Country Edition: PE, Peru
  7 GeoIP Country Edition: TR, Turkey
  6 GeoIP Country Edition: UA, Ukraine
  6 GeoIP Country Edition: CA, Canada

Some questions and remarks I have here are

• Who is sending us all this email? :-)
• This database is only in use for something like 24 hours now.
  Wonder what it will look like in a year or so…
• Why is Taiwan second in the verified list? We don’t have any
  customers there. Is it a popular relaying country or do they have
  intelligent SPAM bots?
• I’m happy to see we seem to be blocking off South Korea
  mostly.

If anybody else can make more sense out of this data, please let me know.
Thank you. :-)

Earlier this week, I was present at a HOT or NOT session, organised
by Sioux. This session included a talk by
Alan Cox on The Future of Linux.

Philippe,
it was definitely worth it! :-)

It was the first time I attended a talk by Alan, and I must say I was greatly
surprised. He had a very nice presentation, a clear voice and overall a pretty
entertaining way of presenting his material. The content of the presentation
was fairly technical and dry, which made it extra hard to present it in the
way he did.

One nice thing I’ll remember form his talk is about the discrepancy in
computer hardware evolution. Intel and AMD keep on putting more and more cores
into their processors. Each time you buy a new computer, your amount of
pre-installed RAM has at least doubled. But when speaking about hard disks, we
only see the size increase — 40MB to 1GB tot 500GB to even more
today — but their speed is still the same! Sure, we went from plain
ATA to SATA and PATA (never mind SCSI), but that’s only a performance increase
of roughly 20%. This introduces many challenges to software land, as people
expect to be able to download stuff at more MB per second as their hard disk
can handle. Or data centres are more often than not waiting longer for a 1TB
backup to complete than their mean time between backups. Alan says: “Hard
disks today are more and more like tapes. Great for linear access, awful for
random access”.

Another interesting thought was about the evolution of the GPU back onto the
CPU. This puts Intel ahead, and ATI is “safe” too, under the hood of AMD.
But what will nVidia do?

I was also happy to hear that he believes Europe is doing a good job in the
battle against American Software Patents lobbyists.

Overall, an interesting night. Thank you Sioux.

Look at the following screenshot:

What’s so funny here?
First of all, when searching
for “Open Source” on Google (google.be, the Belgian version to which I was
redirected automatically), the first (sponsored) result is a website from Microsoft. I thought this was quite ironic by
itself, but when you think about it a bit more… this means that Microsoft actually must have a Google Adwords account and pay for the
“Open Source” keyword!

Isn’t that ironic?

Funny thing is that their link (http://www.Microsoft.be/DefyAllChallenges
(mind the upper case characters)) doesn’t work: “We’re sorry, but
there is no Microsoft.com Web page that matches your entry. It is possible you
typed the address incorrectly, or the page may no longer exist. You may wish
to try another entry or choose from the links below, which we hope will help
you find what you’re looking for.”

Oh well…

What can
say
more than these guys? :-)

FOSDEM 2008 has been a great edition,
for me too. I spent most of my time at the BSD+PostgreSQL
Developer Room. One nice thing I found out is that EnterpriseDB has a development centre
in Islamabad at only a few kilometres from our
office and, apparently, we’re both member of the industry advisory board
of NUST. Nice meeting you,
guys!

It must be said though: the ULB is not the
best location for FOSDEM. At least not any
more. For several talks, I had to sit on the floor or stand up against the
wall at the back of the auditorium. The Mozilla dev room was a named
sauna.fosdem.org. The hallway with the project booths was so crowded, it was
nearly impossible to take your time to visit the individual projects. I know
Brussels has a nice international aura hanging around it, but if you would
just turn Leuven into “Brussels East” or something, I’m sure you won’t see
the difference in the amount of people showing up. But, dear FOSDEM
organisers, at Campus Arenberg
III of the K.U.Leuven, I’m sure you
can provide all of us a nicer geek experience.

And, oh yes, FOSDEM is not free. I spent over € 100 this
weekend on donations only! ;-)

I refer to Jan’s
smiley system. I like the idea, and tried to implement something similar
in [t]csh… with only limited success.

First of all, I needed an equivalent of that bash function inspecting the
return value of the previous command and printing a corresponding smiley.
[t]csh doesn’t support functions, so I needed an alias. But in aliases, you
can’t use if-then-else. Or so I thought.
But this works:

set echo_style = both
alias smiley 'eval "if (\$?) then \\
echo -n \\e[1m\\e[31m- $?\\e[m" " \\
else \\
echo -n \\e[1m\\e[32m+\\e[m" " \\
endif"'

Watch your quoting; remember that the shell strips off one level of quoting
when you set the alias and another during the first pass of the eval. The
echo_style could also be just ``sysv'', but not ``none'' or ``bsd'' if you
want some colour instead of a scrambled shell. And, oh yes. Smileys won't
work, as there is no way I could make this statement not
evaluate ('s or )'s correctly (unless maybe I try it with makealias or
quote one day), so I opted for the +
and - signs instead.

Now after figuring this out, I needed a way to get an alias call into my
$prompt variable. Bummer. No support for this in [t]csh. Yikes!

So now what? Well, you could just set the printexitvalue shell variable to
print the exit status of commands which exit with a status other than zero,
which is supported by default by [t]csh.

So, the same functionality can be gotten (even far simpler than in bash), but
not the same coolness factor. Too bad for those [t]csh users out there. :-)

How to rescue a remote, headless FreeBSD system from a Linux rescue image
which cannot mount ufs2 read-write? Impossible, you think? I just did it. :-)

We’re running (amongst others) a FreeBSD 7.0-RC1 installation as a web and mail
server. On a sad sad day, I rebooted the machine and ping told me it came up
just fine, but nmap told me that there was no way in any more. Hmm..

Our hosting provider has this service which
allows us to reboot our servers into a debian rescue image by sending an
properly formatted email to a special email address. Of course, from that
image, I couldn’t do much to fix the FreeBSD installation on the disk. This is
what I did:

1. make an image of the first 100MB of the primary master hard
   disk
2. copy that image to another server, or your local computer
3. now place a FreeBSD image on the first 100MB of that hard disk (I
   prepared one with
   The Depenguinator, version 2.0)
4. reboot
5. log in to the FreeBSD image, which is loaded into memory
6. get back those 100MB you just copied over to a safe place and cat
   them onto the same hard disk (different device name now of
   course)
7. mount your FreeBSD installation’s file system hierarchy and try to
   fix stuff
8. reboot
9. repeat if necessary…

I must say I was a bit nervous and had to concentrate not to accidentally
mistype anything or cat or dd stuff onto the wrong place. But hey! It worked
out just fine in the end.

I’ve got my BSD machine back. /me happy!

Today I found out the best way to find out what /var really means on
a Gentoo system: I removed it.

Something strange happened. There was a shortage of disk space on the
/ partition, which also housed /var, so I created
/home/var on a different partition, did

tar -c --atime-preserve /var/* | tar -x -C /home

, added

/var /home/var auto bind 0 0

to /etc/fstab and rebooted into single user mode. There, umounted
/var to get back to the original partition (of course, the new
fstab-rule as followed already), and rm -rf‘d /var.

So far, so good.

Reboot…

Empty /var!

This means that almost no services can start up (no ssh, no syslog, no
portmap, no nothing…) as things like /var/spool, /var/log,
/var/run and /var/lock are gone an away…

But what’s worse: /var/www is bye-bye (luckily it contained only
localhost-websites) and /var/lib/portage/world as well. Now, portage
thinks nothing is installed and wants to re-emerge every tiny little dependency
out there. regenworld, I hear you think? Won’t work.
/var/log/emerge.log is gone, remember?

Yup, me too:

I wont be there on Saturday though, but you can expect me at the
pre-FOSDEM-social-event and on Sunday!

And you know what’s so cool? I can categorise this post in “Business”,
“Open Source Adventures” ánd “Life, the Universe, and Everything”.
Man, do I love my job! :-)

Just got back from another nice Geekdinner, in Leuven this time. It was
especially pleased to see two of my fellow students again (hello Jan and
Koen!). I also got the chance to try one of these things much beloved in the
geek community these days: a “plooifiets”. Not sure if it’s something for me
though.

Anyway, Wouter convinced me to subscribe
to Planet Grep with my still freshly
smelling weblog. I guess that’s an honour, so… here we go!

One of the things he said was that “having an audience” might help keeping
up the blogging. I don’t really believe this. Although “having an audience”
might spur some short-term enthusiasm, I really believe that blogging is
something you need to do for yourself. Because you want to do it yourself.
Has something to do with internal and external motivators.

Anyway, maybe I should tell “my audience” something now: I don’t allow
comments or messages on my blog. It’s my blog after all,
isn’t it. If you really feel like reacting, please do! But use your own blog
for that and link back to my post’s permalink when doing so. I feel that if
it’s too easy to react to my writings on my own space, I will (1) loose too much
time keeping up with the comments and probably cleaning up spam and other
unwanted things, and (2) more importantly, I guess I’d feel restricted from
writing exactly what I want to write. Now, there’s absolutely no reason for me
to stay (politically or otherwise) correct!

One last thing. I’ll probably always blog in English, but the odd Dutch or
French word might pop up once in a while.

Had a server crash yesterday. Don’t know if it was due to faulty hardware or
something with the OS, but it came at
the exact wrong moment.

I was installing OpenBSD 4.2 on a remote
system to which I have absolutely no physical access. I created a 64 bit image
with qemu on my laptop, installed OpenBSD in it, and deployed it on the remote
system. After that, I partitioned and labelled the remote hard disk and
started moving /home, /usr and /var from the initial image to their brand new
filesystems/partitions.


mount /dev/wd0e /mnt
cd /usr; dump 0f - . | (cd /mnt; restore -rf - )
cd /mnt/usr
mv * ..
cd ..
rmdir usr
cd /
umount /mnt
mv usr usr.old
mkdir usr
mount /dev/dw0e /usr


I fired up $EDITOR to add my new /usr to /etc/fstab and at that exact moment
the machine crashed and kicked me out…

No problem you’d say. Just reboot it and continue from there. Think again.

Where does sshd live?