Finding the needles in a log file haystack

15.02.2011

Then you get the notion that the mail server generally sends a lot of messages to one location. When you see it explode into sending many e-mails to many other locations, maybe some of which are outside of the country where your company doesn't even do business, this might be an indication you have found a helpful nugget of information.

Key to this entire review is ensuring that you have kept good documentation of the interesting entries. Likely, you will be reviewing quite a bit of material in this process and it's easy to forget what you have already looked at. Cutting and pasting into another file is one way to achieve this easily.

* Work small, get bigger. But how do you find that golden needle in the mountain of hay? This is where you start to narrow the field a bit. Now you can sort based on the types of messages in the log file. How many instances are there of a certain error? If you find that the regular e-mail message happens 10,000 times and this interesting login error happened 10, then perhaps the 10 messages are worth a deeper look into.

* Trace your steps. You have reviewed the main file, you have reviewed the interesting messages. You feel pretty confident that everything in that particular log file has been reviewed that is worthy of being reviewed. Have the interesting entries directed you to look somewhere else? Perhaps you found several entries where it appears a user tried to log in to another system from your mail server. There were only a few entries, so this seems odd. At this time, you should seize log files from these other systems as well. With any luck, you can piece together a trail of the incident.

Forensics in general is a long process that can be somewhat involved. Log file forensics is no exception. While there are many tools that can help you find those elusive needles, doing due diligence and having patience will help properly analyze a log file and reveal helpful supporting evidence.