Website Reporting with Webalizer

Introduction

In the beginning, there was the site; and it was good. Then Doug looked down and said "who visits this?" A quick Google search later brought me to a program called Webalizer.

Webalizer processes log files, and computes all sorts of information and generates lovely graphs. All of this data is then outputted as HTML and PNG files. Perfect for viewing on the web!

Alas, the original version does not seem to properly support IIS 5.0. Fortunately, Webalizer was released under the GNU GPL. Allowing other companies/individuals the ability to modify the source. Which led me a Canadian company called Stone Steps. They have a version that processes IIS log files.

Victory is mine! Well, Almost.

The version of Webalizer from Stone Steps seems to process IIS logs very well. The documentation says that Webalizer reconfigures its parser to handle whatever fields are in the log files. It's been my experience (YMMV) that Webalizer works best when the log files contain every single field made available by IIS in the extended log format.

IIS, by default, logs requests by date. Meaning that any request for November, 14, 2005 will be stored in a file called ex051114.log. Unfortunately, Webalizer cannot just be given a folder full of logs to process. You have to incrementally process each file in the folder. Fortunately, this is available via the -p argument. The incremental data is stored, by default, in a file called webalizer.hist. This has the side benefit that future runs against this file (unless the file has changed) are ignored, since it has already been processed.

The use of the incremental argument is very easy. Stone Steps even has an answer in their FAQ for this very purpose.

Customization

What I like most about Webalizer is the ability to customize the output. Options can be fed via the commandline, or via a configuration file. Perfect for scheduling. The documentation that comes with Webalizer, plus the Stone Steps website is invaluable for coming up with exactly what you want to see.

Set Up

My directory stucture is as follows (this is important for the automation I use later):

sites\
- dougjenkinson.net\
  - logs\ (IIS logs.)
    - W3SVC1\
      - ex051114.log
  - report\ (Webalizer Output)
    - webalizer.hist (Incremental Processing File)
  - dougjenkinson.net.conf (Configuration File)
- dbcache.db (DNS Resolution database created/used by Webalizer)
webalizer\ (Webalizer's Install Location)
Process Logs.bat (Automation File, See Below)
Process Logs.wsf (Automation File, See Below)

Automation

First we'll start off with a simple BAT file that processes the log files. (Word-wrapping may occur.)

Process Logs.bat


						  set site="dougjenkinson.net"

						  for /F %%f in ('dir /O:N /B sites\%site%\logs\W3SVC1\*.log') do (

						      webalizer -p -F iis -c sites\%site%\%site%.conf -o sites\%site%\report sites\%site%\logs\W3SVC1\%%f

						  )

						  unset site

Command Line Arguments:

-p: Incremental processing.
-F iis: Log Format iis.
-c sites\%site%\%site%.conf: Use a specific configuration file.
-o sites\%site%\report: Output director.
sites\%site%\logs\%%f: The log file to process.

It's pretty easy to set this up with a batch file. But, when you manage multiple sites, you need a more elegant solution that scales to additional sites.

Windows Scripting Host Jobs

Process Logs.wsf


						  <package>

						    <job id="WickedStrategery">

						      <script language="VBScript">

						        Option Explicit

						        Dim strSiteName: strSiteName = "dougjenkinson.net"

						        Dim strLogName: strLogName = "W3SVC1"

						        Dim strSiteFolder: strSiteFolder = "sites\" & strSiteName

						        Dim strConfigFile: strConfigFile = strSiteFolder & "\" & strSiteName & ".conf"

						        Dim strOutputFolder: strOutputFolder = strSiteFolder & "\report\"

						        Dim strDNSFile: strDNSFile = "../../dbcache.db"

						   

						        Dim fso, objLogFolder, objLogFile, objShell

						        Set fso = WScript.CreateObject("Scripting.FileSystemObject")

						        Set objShell = WScript.CreateObject("WScript.Shell")

						        Set objLogFolder = fso.GetFolder("sites\" & strSiteName & "\logs\" & strLogName & "\")

						        For Each objLogFile In objLogFolder.Files

						          'Call WScript.Echo(strSiteName & " - " & objLogFile.Name)

						          Call objShell.Run("webalizer\webalizer -p -Q -F iis -c " & strConfigFile & " -D " & strDNSFile & " -N 10 -o " & strOutputFolder & " """ & objLogFile.Path & """", 0, True)

						        Next: Set objLogFile = Nothing

						        Set objLogFolder = Nothing

						        Set objShell = Nothing

						        Set fso = Nothing

						        Call WScript.Echo("Processing Completed.")

						      </script>

						    </job>

						  </package>

Command Line Arguments Used:

-p: Incremental processing.
-Q: Super-quiet, no output to console.
-F iis: Log Format iis.
-c strConfigFile: Use a specific configuration file.
-D strDNSFile: Use a specific DNS resolution database.
-N 10: Use 10 child processes for DNS resolution.
-o strOutputFolder: Output director.
objLogFile.Path: The log file to process.

Conclusion

Webalizer is a great tool. Hopefully, either of the above scripts will prove useful for processing IIS log files.

Metadata

Revisions

v1.0 (14 Nov 2005) - Article published.