dopefish.de » bash

How to build an efficient GeoIP SQL table

dopefish — Mon, 22 Aug 2011 19:17:48 +0000

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

Select All Code:

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================
 
set -o nounset                              # Treat unset variables as an error
 
CSVFile="GeoIPCountryWhois.csv"
 
if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi
 
if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} && rm ${tmpfile}
fi
 
echo -e "\ngenerating geoip.sql"
 
mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"
 
echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
);
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql -D temp
 
mysqldump --opt temp geo_ip > geoip.sql
 
echo -e "\ndone generating geoip.sql"

How to install ruby 1.9.2 on Ubuntu 10.04

dopefish — Thu, 07 Jul 2011 18:49:52 +0000

The current LTS version of Ubuntu is 10.04 and the most current version of ruby it ships with is 1.9.1. Unfurtunately 1.9.1 wasn’t that great of a release and anyone using the 1.9 branch really should use the stable 1.9.2.

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript

Select All Code:

#!/bin/bash
#===============================================================================
#
#          FILE:  install_ruby_1.9.sh
# 
#         USAGE:  ./install_ruby_1.9.sh 
# 
#        AUTHOR: Ryan Schulze (rs), ryan@dopefish.de
#       CREATED: 07/07/2011 11:59:37 AM CDT
#===============================================================================
 
Version="1.9.2-p180"
GZFile="ruby-${Version}.tar.gz"
Download="http://ftp.ruby-lang.org/pub/ruby/1.9/${GZFile}"
 
if [[ "$(id -u)" != "0" ]]
then
	echo "You need root permission to execute this script"
	exit
fi
 
apt-get -q update
apt-get -qy upgrade
apt-get install -qy build-essential wget zlib1g-dev libssl-dev libffi-dev autoconf
 
cd /usr/local/src/
test -e ${GZFile} || wget ${Download}
tar -xzf ${GZFile}
cd ruby-${Version}
 
autoconf
./configure --with-ruby-version=${Version} --prefix=/usr --program-suffix=${Version} 
make
make install
 
mkdir -p /usr/lib/ruby/gems/${Version}/bin
 
update-alternatives \
	--install /usr/bin/ruby ruby /usr/bin/ruby${Version} $(echo ${Version//./}|cut -d- -f1) \
	--slave   /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby${Version}.1 \
	--slave   /usr/lib/ruby/gems/bin        gem-bin   /usr/lib/ruby/gems/${Version}/bin \
	--slave   /usr/bin/erb  erb  /usr/bin/erb${Version} \
	--slave   /usr/bin/irb  irb  /usr/bin/irb${Version} \
	--slave   /usr/bin/rdoc rdoc /usr/bin/rdoc${Version} \
	--slave   /usr/bin/ri   ri   /usr/bin/ri${Version} \
	--slave   /usr/bin/gem  gem  /usr/bin/gem${Version} \
 
update-alternatives --config ruby
update-alternatives --display gem >/dev/null 2>&1 && update-alternatives --remove-all gem
 
echo "[+] Done installing"
ruby -v

How to easily add colored text output in bash scripts

dopefish — Tue, 31 May 2011 22:29:36 +0000

Here is small snippet that can give your shell scripts some nice output: colortext.sh As with the debug.sh script, just download it to the same directory as your own script and add it with

Select All Code:

. colortext.sh

It contains one simple function called text with the syntax text “text to be output”. Color can be red, green, yellow, blue or grey. The function does not automatically add a linebreak to the putput, so pop a \n in there if you need it. I prefer using it together with printf for clean and easy color output.

Here are some examples of how the function can be used, and below the corresponding output:

Select All Code:

. colortext.sh
 
echo "normal text"
text blue "blue text, "
text yellow "yellow text\n"
 
# Using it together with echo
echo -n "Status of script: [";text red "ERROR";echo "]"
 
# more elegant usage with printf
printf "Status of script: [%s]\n" $(text green "OK")

Output:

normal text
blue text, yellow text
Status of script: [ERROR]
Status of script: [OK]

How to add debugging to shellscripts

dopefish — Wed, 04 May 2011 21:44:13 +0000

Debugging bash scripts is pretty straightforward, throwing around a couple echo and set -x quickly gives you what you need. But what if you want to add a nice breakpoint, debugging to lots of paces in the code or turn all debugging on or off at once? Then this little script I wrote is the right thing for you: debug.sh just download it to the same directory as your script and include it with the following line:

Select All Code:

. debug.sh

It contains 4 simple functions that will make your bash coding easier.
debug and breakpoint both print the argument with a timestamp to STDERR
You can turn off all the functions by adding a DEBUG=false into your code

Example:

Select All Code:

#!/bin/bash
. debug.sh
 
debug_on
echo foo
debug_off
 
debug "I am a test"
 
breakpoint "wait for a bit"
 
echo done

Output:

Select All Code:

+ echo foo
foo
+ debug_off
+ true
+ set +x
[16:37:47] I am a test
[16:37:47] -breakpoint- wait for a bit
press any key to continue
 
done

Wireshark remote capturing

dopefish — Tue, 08 Mar 2011 19:27:50 +0000

yeah, this is real simple stuff, not really worth writing a script for it. but on the other hand it saves me from remembering how to do it every time I need it (which isn’t often). So here is a little script to setup remote capturing with wireshark.
All it basically does is ssh to the remote host and tcpdump sucking the output via stdout through the ssh connection to a local pipe, that is then used by wireshark to display the stream. Because of this you may want to make sure you aren’t capturing your own ssh data when doing this

Select All Code:

#!/bin/bash
 
destination="1.2.3.4" # server to run the remote capture on
filter="port 80"      # tcpdump filter options
interface="eth0"      # interface to listen on on remote server
opts="-n -p -s 0"     # other tcmpdump options
 
##### let the fun begin #####
 
mypipe="/tmp/remotecap.$$.cap"
mkfifo ${mypipe}
 
ssh root@${destination} "tcpdump ${opts} -i ${interface} -w - ${filter}" &gt; ${mypipe} &amp;
pipepid=$!
 
wireshark -k -N ntC -t a -i ${mypipe}
 
kill ${pipepid}
rm -f ${mypipe}

XEN 3.4 with ipv6 routing

dopefish — Wed, 01 Sep 2010 20:41:54 +0000

Yes, there are a few postings out there about getting ipv6 routing running with XEN. But I’ll throw this online anyway since there are a few changes I had to make for it to work on my server. This text is intended for people who know their way around Linux and XEN so it will be a bit technical and won’t spell out every single step you have to make.

Most of the changes are based off scripts and information from BenV and wnagele (latter is interesting for me since I am also running XEN on a hetzner server). Have a look at the two links if anything is unclear. Now let’s start the fun

First of all we need IPv6 up and running on the host (dom0). Add the IP and gateway to your /etc/network/interfaces
This is what mine looks like:
iface eth0 inet6 static address 2a01:4f8:100:1123::2 netmask 64 gateway 2a01:4f8:100:1120::1 pre-up ip -6 route add 2a01:4f8:100:1120::1 dev eth0
Check if the IP address is responding to the outside world (e.g. with wiberg.nu/iptools.php), if everything looks ok, proceed …
Now we need to enable a few things to get routing and neighbor discovery running on the host (dom0). Edit your /etc/sysctl.conf and add/change these 2 entries (and while you are at it, set them with “sysctl -w” too):
net.ipv6.conf.all.forwarding=1 net.ipv6.conf.all.proxy_ndp=1

So, your host should by now be online with ipv6 and soon be able to route packets to it’s guests. By default XEN will only take care of IPv4 when a guest is created, so here is a small patchfile that adds support for IPv6: xen-ipv6-vif-route.patch. The patch changes vif-route and vif-common.sh, while these files may be in different places depending on your distribution, /etc/xen/scripts/ is where they can commonly be found. Download the patch to the directory with the scripts to be changed and execute a “patch -p0 < xen-ipv6-vif-route.patch" (vif-common.sh gets a few new IPv6 functions, and iptables now won't try to change stuff for IPv6 IPs. vif-route changes are: ndp is enabled for the vif device and the route/neighbor IPv6 settings are set)

So, now that the scripts know how to setup all our IPv6 needs, we need to add the IPv6 IP to our guest settings (.cfg file typically found in /etc/xen/). What we want to change is the "vif" setting. Add the IPv6 IP of the guest to the IPv4 IP (just the IP without the trailing /network, space separated form the IPv4 IP):
vif = [ 'mac=B1:A3:3F:25:11:B8, ip=2a01:4f8:100:1123::5 188.40.34.101' ]

Now you can create the guest(domU) and add the IPv6 IP to the /etc/network/interfaces of the guest if you haven't so already (it uses the host (dom0) as the gateway).
iface eth0 inet6 static address 2a01:4f8:100:1123::5 netmask 64 gateway 2a01:4f8:100:1123::2
Restart the networking on the guest (or reboot it) and you should now be able to ping the guest from the internet. See, easy wasn't it

Script of the day – clean up stale .ssh/known_hosts

dopefish — Thu, 19 Aug 2010 10:24:12 +0000

This little script takes an IP or hostname as a parameter, and if there is an offending key in the .ssh/known_hosts it removes it and replaces it with the current valid one useful if you are moving/reinstalling a large amount of servers …

Select All Code:

#!/bin/bash
#===============================================================================
# FILE: ssh-cleankey.sh
# USAGE: ./ssh-cleankey.sh 
#
# DESCRIPTION: deletes stale ssh known_hosts entries
#===============================================================================
 
# true or fasle
VERBOSE=false
 
#=== Exit codes ================================================================
# 1 - Not a valid IP or not reachable
#===============================================================================
 
#=== FUNCTION ================================================================
# NAME: print_help
# DESCRIPTION: Prints help and exits
#===============================================================================
print_help() { #{{{
echo "Usage: `basename $0` "
echo ""
echo "e.g. ./`basename $0` 1.2.3.4"
echo ""
exit 0
} #}}}
 
if [[ $# -eq 1 ]]
then
HOST="${1}"
else
print_help
fi
 
ping -w1 -c1 $HOST >/dev/null 2>&1
if [[ $? != 0 ]]
then
$VERBOSE && echo "ERROR: $HOST is either not a valid IP/hostname, or is not reachable via ping"
exit 1
fi
 
Check=$(ssh -o connecttimeout=10 -o stricthostkeychecking=no $HOST true 2>&1|grep -c "Offending key")
if [[ $Check -gt 0 ]]
then
$VERBOSE && echo "$HOST is stale, updating known_hosts"
ssh-keygen -R $HOST >/dev/null 2>&1
ssh -o connecttimeout=10 -o stricthostkeychecking=no $HOST true >/dev/null 2>&1
else
$VERBOSE && echo "$HOST is OK"
fi
 
exit 0

Checking a list of IPs against RBL

dopefish — Tue, 10 Aug 2010 09:52:07 +0000

This is more a reminder to myself than anything else … this is small snippet that takes a list of IPs and does a whois on all that aren’t in a RBL

Lets say we have al list of IPs in a file “iplist.txt”:
12.172.121.171 24.149.208.68 38.105.100.9 58.185.207.86 61.201.51.66 64.78.164.169

Snippet that checks the IPs (can of course be easily changed to check IPs that are IN a RBL)

Select All Code:

for ip in $(cat iplist.txt)
do
test $(/usr/bin/rblcheck -qm ${ip}) && continue || clear
whois $ip
printf "###########################\n##### %15s #####\n###########################\n" "${ip}"
read
done

bash: using the content of a variable as variable name

dopefish — Thu, 05 Aug 2010 11:14:21 +0000

Since the implementation of Arrays in Bash is somewhat lacking compared to higher level programming languages (only one-dimensional), and hash lists require a bit of work to set up, you may run into a situation where you have a small list of key/value pairs that are both variable and you need to store.
There are various solutions for the problem, e.g. creating two arrays (one for the keys, one for the values, and combining them by using the same index values for the entries), or using the functions from the link above to build a hash list. For me the easiest way to solve the problem, if I only have a few variables and don’t want to bloat the code, is to (mis)use declare. declare is intended for setting the type of a variable (constant, array, integer,…), but has the nice side affect that you can use variables in the key name, and you can set the value of the variable.

usage:
declare ${Key}=${Value}

Example:
$File_Config is variable holding the name of a configfile, the content of the file could look like this:
foo:one bar:three foobar:bignumber

Snippet:

Select All Code:

if [[ -e "${File_Config}" ]]
then
while read line
do
Key="${line%%:*}" # chop off everything after the first :
Value="${line#*:}" # chop off everything before the first :
Value="$(echo ${Value}|sed 's#^ +##g;s# +$##g')" # chop off any excess whitespace
if [[ "${Key}" != "" && "${Value}" != "" ]] # only continue if we have something to do
then
declare Configuration_${Key}="${Value}"
fi
done < "${File_Config}"
fi

after the snippet has read the configfile, you can use $Configuration_foo, $Configuration_bar and $Configuration_foobar in your script. The keynames could also have came from a mysql query, array, command line args, …

Mythtv and IR Blaster

dopefish — Sun, 24 Jan 2010 15:19:45 +0000

We use a set-top-box as the video source for mythtv. And while this works perfectly fine, we have to manually make sure the right channel is set for what mythtv wants to record. This can turn out to being a pain at time, especially when multiple things are planned to be recorded on different channels. So I went and ordered a IR transmitter from irblaster.info to give mythtv the possibility to change channels itself.

Since a bunch of the infos I found online were a bit out dated, here are a few tips for anyone using current versions of the software (I’ve got mythtv-backend 0.22 and lirc 0.8.4a running on a mythbuntu based system when I wrote this)

Hardware is easy to set up, just plug it into a free serial port.
1st we will head over to http://lirc.sourceforge.net/remotes/ and grab the file for our reciever set-top-box if we don’t already have them.
Then go edit /etc/lirc/hardware.conf and add the transmitter settings, these were mine:
TRANSMITTER="Skymaster_XL10" TRANSMITTER_MODULES="lirc_dev lirc_serial" TRANSMITTER_DRIVER="" TRANSMITTER_DEVICE="/dev/lirc1" TRANSMITTER_LIRCD_CONF="skymaster/XL10" TRANSMITTER_LIRCD_ARGS=""
now include the reciever settings to /etc/lirc/lircd.conf
I edited my /etc/init.d/lirc to execute the following line before starting the daemon (was needed to ensure the lirc_serial module can be loaded):
/bin/setserial /dev/ttyS0 uart none
Restart lircd and you can then start testing if transmitting codes work with irsend works. here is a little wrapper script I wrote around irsend for mythtv to use. You may have to twiddle around with the sleeps and change the $Remote to your reciever. If the sleeps are too long (-> the script takes to long to finish switching channels), then mythtv will timeout and not display live tv.

Select All Code:

#!/bin/bash
 
Remote="Skymaster_XL10"
 
send="irsend -d /dev/lircd1 SEND_ONCE"
 
if [[ "$(echo $1|grep -c "^[0-9]")" = "0" ]]
then # non-numeric command
$send $Remote $1
else # numeric
for digit in $(echo $1 | sed -e 's/./& /g'); do
$send $Remote $digit
sleep 1.5
done
fi
sleep 2

If everything is working fine, then go ahead and tell mythtv to use the script to change channels. This is done in the tuner card setup of mythtv-backend (mythtv-setup).