Tag Archives: Logs

Troubleshooting Splunk

Have been fiddling around with Splunk lately.  Splunk’s a really good tool to use for log collection and analysis (and that’s oversimplifying it, I believe it can even do event correlation…), which really made my love for data mining go crazy of late:P  Best part is that it has a perpetual free license, nice!

One of the things I encountered when using Splunk was that it didn’t seem to be indexing all the log files that it was set to monitor.  After some reading up and experimenting the reason became clear: Splunk will not work properly if you set it to monitor too many files.

How many is too many?  For example, setting it to monitor a logfile directory which only has one active log and 100+++ rotated logs, is too many.  What should be done instead is to set it to monitor the active logfile only, and use oneshot adding of the other logfiles to the index you want.

Gonna do some more sharing/writeups about this crazily great tool.  There’s really a lot that this thing can do man.

Advertisements

Weird web server access log entries

Don’t have the answer to this yet, but it sure makes me really really curious as to the cause.

In my web server access logs, I get plenty of entries that look like this:
69.64.58.126 - - [02/Oct/2009:14:20:55 +0800] "-" 400 0 "-" "-"

That means that these IP addresses have been connecting to my server via a particular domain, without sending any request whatsoever or so it seems. And I get a LOT of accesses from Singnet IP addresses (220.255.0.0/16 range) daily.

Have contacted them on this, but no answers as yet. And the only answer from a forum was that it’s probably load balancer generated traffic (doesn’t explain why I get many many requests from an entire Class C worth of IP addresses).

Not sure whether it was this thing that caused my 1Portfolio account to overuse server resources, since Apache is known to fork a new process for every new incoming connection, as opposed to web servers like Nginx. (And I’m still pissed at how they did NOT contact me whatsoever when they had to terminate my account immediately due to the resource overuse)

If anyone has answers/possibilities I’d sure like to hear and discuss on them!