Visualizing Data (using Processing)

Visualizing Data, by Ben Fry (O’Reilly)
[Amazon] [Google Books] [O’Reilly] [Google Search]

One of my interests has always been in data visualization (makes data more understandable, and is one step towards easier interaction with it). Chanced upon this book at the library today, certainly one thing I’d like to look into in more detail at a later point in time.

Why this book caught my interest was the fact that there was another book on such a topic in itself. Other than Applied Security Visualization by Raffael Marty, I’ve yet to chance upon anything else.

A quick browse of the book showed that it’s very possible to use Processing (yet another good reason to take up this book: simple programming!) to implement many of the data visualization concepts. Though many people would say that this is “raw” and “slow” as compared to having a tool to do this simply and quickly, I’d say that doing it this way would certainly give the user a great understanding of the data visualization process itself. Furthermore, who’s to say that Processing’s not the tool itself! 😛 Also, the author has helpfully made the source code examples available online at his blog too.

Will keep this book in mind to look at later. Have other books to go through first… :}

DNS tracking with Python

For those who have need to keep track of changes (IP address changes and “status”) of a list of DNS hosts/addresses, I wrote a Python script to help with this task.

Please help by providing comments/ideas for improvement/thanks/death-threats in the comments section below.  Ok, maybe not the death-threats. 😉

Download and License

The current version of this script is v1.03, released under the GPL license.  Click here to download it.


This software is licensed under the CC-GNU GPL version 2.0 or later.

What it can do/Features:

• Keep track of changes (IP addresses, status: existent/gone) to a list of specified domains
• Resolve many many many domains into a greppable format for you!
• Multithreaded in Python
• Does logging to a logfile, and to console at the same time
• Configurable to a certain extent

How to get and use:

• Install the PyDNS library found in the Python Package Index, hosted at SourceForge
• Grab a copy from the download section above and extract the script out
• Configure the script if you need to (see below)
• Create a text file named dns-list.txt in the same folder as the dns_tracker.py script, list the DNS hosts that you want to track in the dns-list.txt file, one host per line
• Run the script from command line: python dns_tracker.py.  The log file written to would be called dns-track.log
• Profit!
• You can stop the script by pressing Ctrl-C in the console ONCE.  It will attempt to kill off the waiting threads and exit gracefully.

What you can configure:

• All the editable options are in the section marked ##Editable options.  Some of the options are…
• Use system configured resolvers: set ‘use_server’: False,
• Use a DNS resolver of your choice: set ‘use_server’: True, and also ‘server’: ‘<your resolver>’,
• The logfile name can be changed too ‘logfile’: ‘dns-track.log’,

Changelog:

• v1.03 (15 Aug 2010)

• fixed logging hierarchy!  now we can control console and file log levels!

• v1.02 (10 Aug 2010)

• added SERVFAIL to recognise as possible status
• changed DNS resolving fail behaviour: retry 1min later
• changed monitoring start behaviour (faster by using threads)
• orphaned threads will also stop themselves if the main thread’s killed

• v1.01 (3 Aug 2010)

• demarcated editable options section (“##Editable options”)
• changed logging to append instead of overwriting existing log
• remember that CNAMEs are also extracted for comparison of changes
• added in minimum delay checks to account for CNAMEs’ TTL being 0
• adjusted logging levels for logfile, console still outputs everything

• v1.00 (2 Aug 2010)

• initial release!

TODO:

• Perhaps migrating configurable options out to command line parameters
  or a separate config file?

DNS lookups using Python

Using the PyDNS library found in Python Package Index and hosted at SourceForge, some code snippets for usage:

import DNS
DNS.DiscoverNameServers()
reqobj = DNS.Request(name="blog.rayfoo.info")
resp = reqobj.req()

for i in resp.answers: print i
...
{'name': 'blog.rayfoo.info', 'data': '74.207.229.183',
'typename': 'A', 'classstr': 'IN', 'ttl': 1790, 'type': 1,
'class': 1, 'rdlength': 4}

resp.show();
PDG.py 1.0 - blog.rayfoo.info 1
;; options: recurs
;; got answer:
;; -HEADER- opcode 0, status NOERROR, id 60299
;; flags: qr rd ra; Ques: 1, Ans: 1, Auth: 0, Addit: 0
;; QUESTIONS:
;;      blog.rayfoo.info, type = A, class = IN

;; ANSWERS:
blog.rayfoo.info        1790    A       74.207.229.183

;; AUTHORITY RECORDS:

;; ADDITIONAL RECORDS:

;; Total query time: 367 msec
;; To SERVER: <sanitized>
;; WHEN: Sat Jul 31 11:06:24 2010

>>> resp.args
{'protocol': 'udp', 'name': 'blog.rayfoo.info',
'server': 'sanitized', 'rd': 1, 'opcode': 0,
'timeout': 30, 'timing': 1, 'elapsed': 367.52486228942871,
'qtype': 1, 'port': 53}