A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================
 
set -o nounset                              # Treat unset variables as an error
 
CSVFile="GeoIPCountryWhois.csv"
 
if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi
 
if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} && rm ${tmpfile}
fi
 
echo -e "\ngenerating geoip.sql"
 
mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"
 
echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
);
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql -D temp
 
mysqldump --opt temp geo_ip > geoip.sql
 
echo -e "\ndone generating geoip.sql"

The package arrived last week and it was a matter of minutes plugging the devices in and having everything working.  Everything worked without any setup, took me longer to get them out of the box than to hook them up.

Unfortunately our network storage (NAS) is also on the other end of this wireless bridge, and I noticed that when I move large files around (>2GB) or while streaming video/audio off the NAS the connection was dropping out. I don’t mean “ups and downs in the speeed” that is to be expected over wireless, I mean “connections resetting, copy actions aborting with error messages”. Not fun. Unfortunately since the devices are geared toward the “no configuration necessary, just unpack and hook up” crowd, there is no webinterface to see a syslog of what is happening or changing settings. Nada.

After this happening a few times it got really frustrating. I can live with slow, but connections dropping is out of the question. My original plan was to just reset the devices, flash them with a WL-330gE firmware and reconfigure the bridging (the only difference I could find was that the WL-330gE_M is black and not white, and comes preconfigured, and probably has a slightly different firmware without management capabilities).  While I was looking at different options and possibilities I went over to dd-wrt and happily saw that the WL-330gE was supported in the router database. So I decided if I was going to mess around with firmware, I could just as well throw dd-wrt on it.

Even though I am a system administrator, I don’t have the urge to have every device in the house running on Linux with a shell I can ssh in to. I’m perfectly fine with a simple interface that does what I want it to. But the wireless settings I can fine tune in dd-wrt are priceless (especially since I wanted to debug and fix the connection dropouts), normally you only get these options with cisco grade hardware.

The firmware upgrade process of the devices is simple and straightforward. Pull and reapply power with the reset button pressed until the power LED starts flashing, then shove the new firmware onto the device via tftp. Either with the “Firmware Restoration” tool from asus, or with a normal tftp client. I used later. Since this is so straightforward I guess I could also switch over to the official firmware if I wanted to, making two WL-330gE out of the WL-330gE_M pair (saves money since the pair is cheaper that buying two separately).

When in recovery mode (waiting for someone to tftp a new firmware onto it), the device has the IP 192.168.1.220 by default. This is just a rough summary of the steps, anyone wanting to do this should really read through the whole process of deploying dd-wrt with asus, there is important information there (even if the example is a WL500, the WL330 is similar). Just because it worked for my hardware,firmware,setup doesn’t mean you have the same hardware or are deploying the same version I did. Read the dd-wrt documentation before you brick your device.

Clear current settings from the nvram:

# tftp 192.168.1.220
tftp> mode binary
tftp> put wl500g-clear-nvram.trx

Wait 5 min, reboot into recovery, throw a dd-wrt firmware on the device ( I used DD-WRT v24-sp2 (08/12/10) mini – build 14929, standard works fine too).

# tftp 192.168.1.220
tftp> mode binary
tftp> put dd-wrt.v24_mini_generic.bin

Wait 5 mins, reboot and open http://192.168.1.1 To be on the safe side feel free to navigate to Administration -> Factory Defaults to make sure no junk was left behind.  To get bridging configured there are multiple possibilites depending on your needs. For plain LAN bridging you will probably want WDS or one device setup as a AP and the second as a Client Bridge (I used the latter option). One thing you will want to do is go to Setup -> Networking and set the WAN port to “disabled” since the device only has LAN and Wireless.

The rest is fairly ease, set up one device as an AP, chose WPA2 with a good long strong PSK. After testing if the AP works with e.g. a laptop, you can set up the 2nd device as a Client Bridge, just make sure you are on the same channel, same SSID, same security settings.  After everything is up and running now would be a good time to pull backups from the configuration. Might as well tweak around in the wireless advanced settings. If you mess up anything badly enough that it won’t connect again … well that is why you made the configuration backups

As you probably guessed by now, the connection drops are gone, connection is smooth and stable. Peak speed is not quite as fast as before because I throttled some things and tweaked settings for stability, but still good. Turning the TX antenna output power from 71 down to 65 helped a lot and got the maximum out of the connection (probably less crap pulling my SNR down). And now I can see what the access point is doing and where problems are when they arise

I didn’t get around to installing my video and picture software on the new windows until this morning, and it turned out to be lots of fun. Due to pure luck I found the license key for sony vegas (it is shown in the splash screen when starting up, shortly before it crashes due to my harddisk malfunction). Any Photomatix was where I keep most licenses stored. But my Photoshop license was more of a challenge. Adobe only allows 2 activated copies of the software per license, activated copies are bound to hardware … you probably see where this is going. I couldn’t deactivate the old installation since the harddrive was kinda dead, and the new installation says “different hardware (new harddrives), must be a different computer”. Yay, fun. The bright side was that the support was easily contacted and they could reset the activation counter (after lecturing me about using it on “2 computers” and deactivating, bla bla bla). I learned one thing: the more expensive the software, the more problems you have with licenses. A shame I never liked Gimp for photo editing.

Anyway, anyone using “lenny” or “testing” and wanting to update xen; here are a few tips to make the transition a breeze:

• The debian wiki is full of useful information regarding xen
• backports.org has a current xen kernel (linux-image-2.6.32-bpo.4-xen-amd64) that works fine with xen 3.4, just follow the instructions on backports.org on how to add it to apt.
• I had to deactivate the “vfb” (VNC console) setting in the domU config files to get my guests to boot, browsing the Internet I saw people having the same problem with the “dhcp” setting.
• If you are planning on updating the guests kernels too (advised), remember to change the “kernel” and “ramdisk” settings in the domU config files accordingly

With these few points in mind, the update is a breeze.

The offer is good as long as supplies last, so with some luck I may still get one. Cool. Have a look for yourself http://www.domain-karte.de

An order key is required, just have a look at the tags of this posting …. one of them is a bit “strange”

In my last posting I forgot to mention I set up an account over at Deviant Art where I am posting pictures of mine I actually find worthwhile looking at. Not that I find the rest of my pictures outright bad, but they are more average and sorted by events/vacations. The ones I post at Deviant Art are pictures I find are above (my) average and worth the time to have a look. Currently not too many, but the collection will hopefully grow.

I started plaing a new browsergame this week, Kings Age. It’s a simple”build cities, expand, fight” thing. But the graphics are ok, and it’s easy to play (log in once a day and put some actions in the queue).

So I decided to go for it and merge my custom stuff into the Debain config. Greylisting worked out-of-the-box, spamassassin needed some minor tweaks, exim was (more or less) easy. The p.i.t.a. with exim is when you know exactly what is missing and where it would be configured, but because of “smart debian scripts” you have to find some config in an unrelated script and put the value there so it gets put in the right placeholder.

Here is a little summary in case I ever do this again and need to see if I forgot something:
- cyrus: copy /var/spool/cyrus/mail/ , /var/lib/cyrus/user/ , use cyradm to add the user.blargh account and /usr/sbin/cyrreconstruct -rf user (don’t forget the sieve filters)
- getmail: nothing special here, just copy config and add cronjob
- spamassassin: alter exim acl to set noscan for auth’d connections and have spamassassin scan everything not “noscan” (because per default local mail isn’t scanned, that includes everything we pick up via getmail)
- exim: check update-exim4.conf.conf for stupid entries, remember to turn on TLS (imap can use the same certificates), since we are using sasl for imap, have smtp auth use the same database (plain_saslauthd_server), turn on TLS by creating a file conf.d/main/00_exim4-config_localmacros with “MAIN_TLS_ENABLE = true” in it

Now that I’m done I found a pretty detailed German website with steps to set up such a system E-Mail-Server mit Debian, Exim und Cyrus. I did the exim router/transports a bit differently to have a bit more control over what goes where when. Still, defiantly worth reading if you are thinking about building such a system.