h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Hi all

In an attempt to save money I didn't take my hosts option of web based server stats, so left with a http site with loads of log files, one per day, that need analysing.

So - is there any good, free, software (many Google searches haven't yielded any) that can take these many files direct of my site and crunch them - Oh non-CGI based.

AND/OR is there an easy way to download these file en-mass? I've seen various packages that will merge them and one that will analyze single files held locally (analog).

Cheers

CH

when I had alook a few months back, most of the things I found were shareware or worse, for my CGI isn't that bad anyway. scince my whole site is based upon perl.

-- DoctorMO --

Analog is a good program, especially if you take some time to configure it.

For moving your log files, is this a standard unix server that you have telnet or ssh access to? If so, you can tar the files using something like "tar -vcf archivename.tar *.log" (assuming all your log files end with ".log"), then compress the archive using gzip "gzip archivename.tar" which will result in a file called "archivename.tar.gz" which is the unix equivalent of a zip file. Or you could try running analog right in your user account, by installing there instead of on your pc, in which case you don't have to tar, gzip, or transfer the log files at all.

Cheers - I believe it is a MS server - not sure about how I'd go about running analog at the server, especially as there are daily files, couldn't see (OK only had a brief look) in the instructions how to load more than one file at a time.

Back to Google for a bit, whilst seeing if I can strike a deal with the hosts.

CH

Well, if you do end up using Analog, all you do is use a wildcard in the log file name line of the configuration file. Analog will then grab all the matching files and process them at once.

What are you using on your desktop? Another option (if you are using something unixy) is to write a simple shell script that uses something like curl or wget to retrieve the files for you and pop them right into analog. Then put your shell script into your crontab, and bingo - instant daily (or hourly, weekly, whatever) web log analysis. I suppose you could use VBScript or something to do it on Windows too, but at that. Maybe perl is your solution.

W2K on desktop, but thanks for tip on Analog, makes it a bit more of a contender.

CH

My suggestion would be to find or put together a script in a language that has regular expression support (e.g. Perl). Parsing web logs is pretty easy if they're a standard format (e.g. W3C format) and there's probably a module for Perl that will do it for you somewhere on CPAN.

Do you have FTP access to the server? If so, just download the logs with FTP and parse them locally.

I wonder why have a W3C standard for web logs, are we talking your basic website logs? like the ones you get from the apache?

I had to make my own logs because the web logs we'rnt reporting enough information. all it was, was date/time, error number (200) and the file opened, which is a bit daft on the website I'm doing because they all said cgi-bin/index.pm

-- DoctorMO --

currently just http access to the logs, may try and get that changed, or spend the cash and get the hosts to analyse - with 5 independant sites to look after, I'd be doing nothing but downloading and analysing.

Although there is a Dreamweaver extension that builds yuor own logs, like the free StatsforAll I guess, but that (again guessin) involves adding extra code to yuor page, increasing time to complete a page and irrates people if they think you are spying on them. I use the StatsForAll on my personal home page OK.

Cheers

CH

*shudder*

what happened to the good old days of notepad and emacs?

-- DoctorMO --

I believe Apache's log files are W3C standard by default. Even IIS (!!) uses the W3C standard for logs, even if it starts configured with an idiotic set of logging defaults (honestly, who doesn't record the Referer header??).

What do the log file names look like? Downloading files from http sites using Perl is fairly trivial. I can put together a quick script to connect to the server and download all the files that look like log files if you want.

You would need a copy of ActivePerl (which is free): http://www.activestate.com

Yes but if it's a windows machine and he's running apache then, er... I had problems anyway. thats why I had to convince the bods at work to let me install SuSE on my PC.

-- DoctorMO --

Your problems likely had to do with CR/LF (windows) versus LF (Unix) end of line characters. That's (mostly) transparent to Perl and it will save according to the local Operating System's newline character.

ActivePerl? How is that different from the standard Perl from http://www.cpan.org/ ?

ActivePerl == ActiveState's version of Perl. It's built specifically for Windows and it's much nicer to install. Plus no messing with compilation (unless you feel so inclined). The only gotcha (that I've found) is that you have to either use their precompiled CPAN modules or have a copy of MS VC++ to install them from source. It's not a big deal because most of the modules are available through their package manager anyway. The only one I'm wishing I could get is SAX::Expat but it's in the package repository for Perl 5.8.0 (which is currently in beta) already.

Life is easier now, hosts changed to give me FTP access to them.

Testing Analog now, and then maybe any of the paid for ones.



CH