A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================
 
set -o nounset                              # Treat unset variables as an error
 
CSVFile="GeoIPCountryWhois.csv"
 
if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi
 
if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} && rm ${tmpfile}
fi
 
echo -e "\ngenerating geoip.sql"
 
mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"
 
echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
);
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql -D temp
 
mysqldump --opt temp geo_ip > geoip.sql
 
echo -e "\ndone generating geoip.sql"

About a month or two ago I built myself a new PC (old one was dying) and installed Win7 on it.  Since I was previously still on WinXP there were a few things to get used to, but all in all I enjoyed the update. There were of course some strange Windows pitfalls involved, moving your users directory to a different drive really isn’t straightforward, but works if you use junctions to link the directories.

Since Virtualbox drops the guests in a subfolder of the users directory I decided to move my users directory to one of the larger mechanical disks in my system, to keep the windows drive (a SSD disk) with plenty of space.

Now fast forward to yesterday, where some software I installed decided to clear out most of my Start->All Programs folder, leaving me with lots of installed softawre, and no Start Menu links. After trying some stuff out I finally caved in and used System Restore to get my Start Menu working again. System Restore actually worked fine, I got my Start Menu back … unfortunately it also corrupted my profile, and while googling to find out how to fix it I found a whole lot of people with the same problem.

Junction + User directory + System Restore = corrupt user, non repairable. Hello Microsoft, this is all original system software, no 3rd party stuff, breaking my user is totally uncool.

So after trying all kinds of crazy tips I found I gave up and followed the official instructions (a.k.a. You are fucked, haha bye bye user registry). And right now I am copying stuff from my old profile to my new one and setting up all the software that had stored stuff in the registry. This time my Profile stays on C: without any junctions, it’s easier to repair Virtualbox if it breaks over the images being in a non-standard path than worrying about windows corrupting my whole profile if I touch System Restore.

I am fully aware that SVN may be a bit overkill and that I am only using a very small subset of the SVN features, but on the other hand it is very easy to set up, and there are clients for all the major OS out there that integrate seamlessly into the file managers. So even if I don’t need all the bells and whistles it does what I need it to do without any worries.

As with all workarounds this of course brings some limitations and implications with it.

• First of all you will only have the history of a file on systems with a svn client, so no history if you are accessing the files via webinterface.
• Situations where multiple people have access to the same files must be set up in the same consellation in SVN to ensure all users have access to file histories
• Due to the usage of two systems (iFolder and SVN) the files will be stored in both systems

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript

#!/bin/bash
#===============================================================================
#
#          FILE:  install_ruby_1.9.sh
# 
#         USAGE:  ./install_ruby_1.9.sh 
# 
#        AUTHOR: Ryan Schulze (rs), ryan@dopefish.de
#       CREATED: 07/07/2011 11:59:37 AM CDT
#===============================================================================
 
Version="1.9.2-p180"
GZFile="ruby-${Version}.tar.gz"
Download="http://ftp.ruby-lang.org/pub/ruby/1.9/${GZFile}"
 
if [[ "$(id -u)" != "0" ]]
then
	echo "You need root permission to execute this script"
	exit
fi
 
apt-get -q update
apt-get -qy upgrade
apt-get install -qy build-essential wget zlib1g-dev libssl-dev libffi-dev autoconf
 
cd /usr/local/src/
test -e ${GZFile} || wget ${Download}
tar -xzf ${GZFile}
cd ruby-${Version}
 
autoconf
./configure --with-ruby-version=${Version} --prefix=/usr --program-suffix=${Version} 
make
make install
 
mkdir -p /usr/lib/ruby/gems/${Version}/bin
 
update-alternatives \
	--install /usr/bin/ruby ruby /usr/bin/ruby${Version} $(echo ${Version//./}|cut -d- -f1) \
	--slave   /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby${Version}.1 \
	--slave   /usr/lib/ruby/gems/bin        gem-bin   /usr/lib/ruby/gems/${Version}/bin \
	--slave   /usr/bin/erb  erb  /usr/bin/erb${Version} \
	--slave   /usr/bin/irb  irb  /usr/bin/irb${Version} \
	--slave   /usr/bin/rdoc rdoc /usr/bin/rdoc${Version} \
	--slave   /usr/bin/ri   ri   /usr/bin/ri${Version} \
	--slave   /usr/bin/gem  gem  /usr/bin/gem${Version} \
 
update-alternatives --config ruby
update-alternatives --display gem >/dev/null 2>&1 && update-alternatives --remove-all gem
 
echo "[+] Done installing"
ruby -v

Currently I’m using SVN as a workaround. I just sometimes forget to commit, which leads to me having old data and no access to updated files. Instead of starting from scratch and build strange stuff with rsync I had a look if there were any open source solutions out there. And I was pleased to find a few. Unfortunately they are almost all in the “I have an idea and a few mockups” stage, maybe even some beta version that does something, but nothing really “production” level. Another downside was that most of them either supported Linux or Windows, but often not both.

One open source solution I stumbled across is iFolder,  seasoned software supported by Novell, with clients for Windows, Linux and Mac. And if all else fails you can use the web interface to access your files. Admin and User web interface are both easy to use. I don’t remember how I stumbled across it, but I do know that it’s not one of the main products you see named when searching for alternatives to Dropbox.

If you have SuSE as an OS, then installing the software is immensely simplified due to SuSE belonging to Novell, just follow the SuSE instructions on the website. I prefer debian based installations and found Ubuntu instructions at help.ubuntu.com that were written mainly for Ubuntu, but should work with any debian deviate.

I’m not going to repeat anything from the installation instructions, they already did a fine job of documenting everything. One thing is important however: the Client 3.8.0.0 just plain doesn’t work. unfortunately it is the one linked from the website. Go to the sourceforge repository, and download the newer 3.8.03 client. That one works like a charm.

. colortext.sh

It contains one simple function called text with the syntax text <color> “text to be output”. Color can be red, green, yellow, blue or grey. The function does not automatically add a linebreak to the putput, so pop a \n in there if you need it. I prefer using it together with printf for clean and easy color output.

Here are some examples of how the function can be used, and below the corresponding output:

. colortext.sh
 
echo "normal text"
text blue "blue text, "
text yellow "yellow text\n"
 
# Using it together with echo
echo -n "Status of script: [";text red "ERROR";echo "]"
 
# more elegant usage with printf
printf "Status of script: [%s]\n" $(text green "OK")

Output:

normal text
blue text, yellow text
Status of script: [ERROR]
Status of script: [OK]

The last few days I have been upgrading my servers to current distributions, today the Voice servers were on the list to get Debian 6 / Ubuntu 11.04. And again I ran into problems with Teamspeak, turns out they won’t work with libmysqlclient 16 libraries and require the good old 15 version (which isn’t available out-of-the-box in the latest Debian and Ubuntu release).

So anybody running into the same problem (do a ldd libts3db_mysql.so to check), can hop on over to http://packages.debian.org/lenny/libmysqlclient15off and download the package for your architecture and install it with dpkg -i <filename>

Example output:

Plugin Installed:		wp-content/plugins/hello.php
	URL			http://www.dopefish.de/wp-content/plugins/hello.php
	SVN			http://svn.wp-plugins.org/wp-content/plugins/hello.phptrunk/
	http://osvdb.org/22654	WordPress wp-content/plugins/hello.php Direct Request Path Disclosure
	http://osvdb.org/62684	WordPress wp-content/plugins/hello.php add_action() Function Path Disclosure
Plugin Installed:		wp-content/plugins/devformatter/
	URL			http://www.dopefish.de/wp-content/plugins/devformatter/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/devformatter/trunk/

Running it against my own webspace revealed a possible SQL injection I was unaware of. *) Fixed that, will probably replace that plugin completely this week, anything that has stuff so obviously bad in it is generally not all too sane.

*) I normally look at plugins before I install them, must have missed this one. @ PHP programmers: anyone who passes on the content of a $_REQUEST directly to a SQL query without any sanity checking deserves to be flogged with his own code.

. debug.sh

It contains 4 simple functions that will make your bash coding easier.
debug and breakpoint both print the argument with a timestamp to STDERR
You can turn off all the functions by adding a DEBUG=false into your code

Example:

#!/bin/bash
. debug.sh
 
debug_on
echo foo
debug_off
 
debug "I am a test"
 
breakpoint "wait for a bit"
 
echo done

Output:

+ echo foo
foo
+ debug_off
+ true
+ set +x
[16:37:47] I am a test
[16:37:47] -breakpoint- wait for a bit
press any key to continue
 
done

http://www.dopefish.de/projects/disemvoweling/