Tag Archives: CLI

Making DD even more awesome

Deputy Director? No…the Unix dd command!

The dd command is one of the most versatile and powerful tools you will find in a Linux box, it is an awesome command! Disk wiping is only one of the more common uses for it. There’re some other tricks that I use dd for, but those will have to be left for another post…

You may have noticed that there’s a tip on monitoring the current status of the dd command right at the end of the --help output:

$ dd --help

{...}
Sending a USR1 signal to a running `dd' process makes it
print I/O statistics to standard error and then resume copying.

  $ dd if=/dev/zero of=/dev/null& pid=$!
  $ kill -USR1 $pid; sleep 1; kill $pid
  18335302+0 records in
  18335302+0 records out
  9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s
{...}

That works as stated, but what if you want to get a continuous status update on the state of the dd command for long tasks such as wiping a hard drive? Well, you use a loop to do that.

As for keeping the loop running for only as long as the dd process is alive and working, I use the -a test to check that the cmdline file for that process id exists. When the process dies, the proc folder is deleted shortly after, and the cmdline file ceases to exist. That will cause the while loop to exit too. No more flooding the screen with useless output!

$ sudo dd if=/dev/zero of=/dev/sdc & pid=$!
$ sudo while [ -a /proc/$pid/cmdline ]; do echo; date; sudo kill -USR1 $pid; sleep 1; done

Thu Sep  6 02:00:12 SGT 2012
63637313+0 records in
63637313+0 records out
32582304256 bytes (33 GB) copied, 6814.37 s, 4.8 MB/s

Thu Sep  6 02:00:14 SGT 2012
63643513+0 records in
63643513+0 records out
32585478656 bytes (33 GB) copied, 6815.34 s, 4.8 MB/s

Thu Sep  6 02:00:15 SGT 2012
63649217+0 records in
63649217+0 records out
32588399104 bytes (33 GB) copied, 6816.46 s, 4.8 MB/s

Thu Sep  6 02:00:16 SGT 2012
63657193+0 records in
63657193+0 records out
32592482816 bytes (33 GB) copied, 6817.42 s, 4.8 MB/s

HTH.

Advertisements

Taking BASH automation deeper down the rabbit hole

Automaton at your service!

The BASH (or rather, any Linux) shell is like a programming environment in itself: you can do pretty simple stuff with it, and you can do increasingly crazy stuff with it if you wish to venture there… Here’s one way to do some simple automating of stuff using the for-do-done construct in BASH:

Here we have for our example three folders, each containing three files.

$ ls -l
total 12
drwxr-xr-x 2 user user 4096 2012-08-31 13:58 folder1
drwxr-xr-x 2 user user 4096 2012-08-31 13:58 folder2
drwxr-xr-x 2 user user 4096 2012-08-31 13:58 folder3

$ ls -l *
folder1:
total 36
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file1
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file2
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file3

folder2:
total 36
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file4
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file5
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file6

folder3:
total 36
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file7
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file8
-rw-r--r-- 1 user user 0 2012-08-31 13:58 file9

A simple loop to do something (like echoing out their names here) on each of the items in the working folder. This is the basic structure of a for-do-done loop in BASH:

$ for i in *;
>     do echo $i;
> done
folder1
folder2
folder3

This can be condensed into a single line for easy copy-and-pasting:

$ for i in *; do echo $i; done
folder1
folder2
folder3

Now, let’s go a little deeper and work with each file within each subdirectory here. For that we just nest another for loop within the outer one. Note that there’re commands to change the working directory in and out of the subdirectory each round:

$ for i in *;
>     do echo $i;
>     cd $i;
>     for j in *;
>         do echo "$i - $j";
>     done;
>     cd ..;
> done
folder1
folder1 - file1
folder1 - file2
folder1 - file3
folder2
folder2 - file4
folder2 - file5
folder2 - file6
folder3
folder3 - file7
folder3 - file8
folder3 - file9

And as you might have guessed, combining all these into a single line is possible too…

$ for i in *; do echo $i; cd $i; for j in *; do echo "$i - $j"; done; cd ..; done
folder1
folder1 - file1
folder1 - file2
folder1 - file3
folder2
folder2 - file4
folder2 - file5
folder2 - file6
folder3
folder3 - file7
folder3 - file8
folder3 - file9

How deep you nest the for loops depends on what you need, and how crazymotivated you are to create and troubleshoot such a monster. Good luck!

HTH.

Geolocation lookups in Linux (/Ubuntu)

Have written a short post on this before, but it seems that I’ve only scratched the surface 🙂

For Ubuntu/Debian users, the APT package to install would be:

$ sudo apt-get install geoip-bin

In addition to its commercial offerings, MaxMind has free Country and City, and AS number lookup databases that can be queried using these command line tools in Linux. Installing the geoip-bin package installs the free version of the country database, but you don’t need to stop there!

By default, the free IP-Country database is situated at /usr/share/GeoIP/GeoIP.dat. Do note that the APT package for it is NOT updated automatically, so you will need to update it yourself.

Grabbing hold of the other two free databases (they’re updated monthly I think) and placing them the shared folder. IP-ASN is a nice way to quickly determine the ownership of an IP address, which you can follow up with actually looking through the WHOIS info should that be too generic. IP-City info comes with geolocation (lat-long coordinates!) info, which is very nice for plotting IP address lists on nice maps for analysis, or for the less technically inclined (or your bosses :P).

$ ls /usr/share/GeoIP/
GeoIPASNum.dat  GeoIP.dat  GeoLiteCity.dat

It appears that GeoIP and GeoIPASNum are queried automatically by default

$ geoiplookup 8.8.8.8
GeoIP Country Edition: US, United States
GeoIP ASNum Edition: AS15169 Google Inc.

Now let’s try querying for basic location information:

$ geoiplookup 8.8.8.8 -f /usr/share/GeoIP/GeoLiteCity.dat 
GeoIP City Edition, Rev 1: US, N/A, N/A, N/A, 38.000000, -97.000000, 0, 0

What are the MaxMind database versions currently “installed”?

$ geoiplookup 8.8.8.8 -v
GeoIP Country Edition: GEO-106FREE 20120403 Build 1 Copyright (c) 2012 MaxMind Inc All Rights Reserved
GeoIP ASNum Edition: GEO-117 20120402 Build 1 Copyright (c) 2012 MaxMind Inc All Rights Reserved

$ geoiplookup 8.8.8.8 -f /usr/share/GeoIP/GeoLiteCity.dat -v
GeoIP City Edition, Rev 1: GEO-533LITE 20120403 Build 1 Copyright (c) 2012 MaxMind Inc All Rights Reserved

If you want more verbose reporting (shows the IP address block that matched the query):

$ geoiplookup 8.8.8.8 -i
GeoIP Country Edition: US, United States
  ipaddr: 8.8.8.8
  range_by_ip:  8.7.211.0 - 8.14.223.255
  network:      8.8.0.0 - 8.11.255.255 ::14
  ipnum: 134744072
  range_by_num: 134730496 - 135192575
  network num:  134742016 - 135004159 ::14
GeoIP ASNum Edition: AS15169 Google Inc.
  ipaddr: 8.8.8.8
  range_by_ip:  8.8.8.0 - 8.8.8.255
  network:      8.8.8.0 - 8.8.8.255 ::24
  ipnum: 134744072
  range_by_num: 134744064 - 134744319
  network num:  134744064 - 134744319 ::24

$ geoiplookup 8.8.8.8 -f /usr/share/GeoIP/GeoLiteCity.dat -i
GeoIP City Edition, Rev 1: US, N/A, N/A, N/A, 38.000000, -97.000000, 0, 0
  ipaddr: 8.8.8.8
  range_by_ip:  8.7.228.0 - 8.8.37.255
  network:      8.8.0.0 - 8.8.31.255 ::19
  ipnum: 134744072
  range_by_num: 134734848 - 134751743
  network num:  134742016 - 134750207 ::19

Cooking all of this with a little CLI script-fu for mass lookups!

$ output=outputfile.csv; echo "ip,country" > $output; for  i in $( cat /path/to/list-of-ips.txt ); do echo "$i,\"$( geoiplookup -f /usr/share/GeoIP/GeoIP.dat $i | cut -d' ' -f4-99 )\"" >> $output; done

HTH, and have fun!