Finding the needles in a log file haystack

You have just been presented with a daunting task: Here are several gigabytes of log files; let us know if they can tell you anything.

OK, where do you start? Luckily there are a few free tools that can help you find the proverbial needle in this kind of digital haystack, but the process is just as important. Any forensic procedure should be methodical so you don't end up duplicating your own work or the efforts of anyone you may be working with. We'll review such a process, but it is by no means the only way to proceed. 


But before you even get to that, start with some fact finding. Forensic investigators need to know what they are looking for, so narrowing down "anything" into pertinent facts will be important. The next step is to confirm that you have the correct evidence files or logs to examine. If you have both of those nailed down you're ready to select your tool set.

One of the main tools is PyFLAG (Forensic and Log Analysis GUI), which is preloaded on another excellent tool: SIFT (SANS Investigative Workstation). SIFT is a fully contained forensic tool environment, with everything a forensic examiner would need from acquisition to analysis and reporting. SIFT is built on Ubuntu and comes in a DVD image or a VMWare image.

A consideration prior to using SIFT and PyFLAG: you will need to copy the log file over to the virtual environment. This can be tricky, but will typically work with sharing on Windows platforms or Samba on . Once you have loaded the file, start with the Case Management module to document case details. Create a new case, which simply consists of adding a case name and a time zone. The next option presented is two links: the Load a Disk Image and the Load a Preset Log File. Skip this for now; you need to do one more step before you can look at the log file.