arclog 3.04

arclog is a Perl module that archives the log files monthly. It strips off log entries that belongs to previous months, and then compresses and saves them to archived files named logfile.yyyymm.gz.

Currently, arclog supports Apache access log, Syslog, NTP, Apache 1 SSL engine log and my own bracketed, modified ISO date/time log file formats, and gzip and bzip2 compression methods. Several software projects log (or can log) in a format compatible with the Apache access log, like CUPS, ProFTPD, Pure-FTPd... etc., and arclog can archive their Apache-like log files, too.

Notice: Archiving takes time. To reduce the time occupying the source log file, arclog copies the content of the source log file to a temporary working file and restart the source log file first. Then arclog can take its time working on the temporary working file. However, please note:

1. If you have a huge log file (several hundreds of MBs), merely copying still takes a lot of time. In that case, you had better stop logging first, archive the log file and restart logging, to avoid racing condition in writing. If you archive the log file periodly, it shall not grow too big.

2. If arclog stops in the middle of the execution, it will leave a temporary working file. The next time arclog runs, it will stop when it sees that temporary working file. You have to process that temporary working file first. That temporary working file is merely a copy of the original log file. You can rename and archive it like an ordinary log file to solve this.

Do not sort unless you have a particular reason. Sorting has the following potential problem:

1. Sorting may eat huge memory on large log files. The amount of the memory required depends on the number of records in each archived month. Modern Linux and MSWin32 have memory consuming protection by killing processes that eats too much memory, but it still takes minutes, and your system will hang during that time. I do not know the memory consuming protection on other operating systems. If you try, you are at your own risk.

2. The time units of all recognized log formats are second. Log records happen in a same second will be sorted by the log file order (if you are archiving several log files at a time) and then the log record order. I try to ensure that the sorted archived records are in a correct order of the happening events, but I cannot guarantee. You have to watch out if the order in a second is important.

Be careful on the Syslog(2) and NTP log files: Syslog(2) and NTP does not record the year. arclog uses Date::Parse(3) to parse the date, which assumes the year between this month and last next month if the year is missing. For ex., if today is 2001-06-08, it will then assume the year between 2001-06-30 back to 2000-07-01 if the year is missing. I think this is smart enough. However, if you do have a Syslog(2) or NTP log file that has records older than one year, do not use arclog. It will destroy your log file.

If read from STDIN, please note:

1. You MUST specify the output prefix if you want to read from STDIN, since what it needs is an output pathname prefix, not an output file.

2. STDIN cannot be deleted, restarted or partially kept. If you read from STDIN, the keep mode will fall back to keep all. if you archive several source log files including STDIN, the keep mode will fall back to keep all for all source log files, to prevent disaster.

3. The answers of the ask mode is obtained from STDIN, too. Since you have only one STDIN, you cannot specify the ask mode while reading from STDIN. It will fall back to the fail mode in that case.

I suggest you to install File::MMagic(3) instead of counting on the file executable. The internal magic file of File::MMagic(3) seems to work better than the file(1) executable. arclog treats everything not gzip(1) nor bzip2(1) compressed as plain text. When a compressed log file is wrongly recognized as an image, arclog will treat it as plain text, read log records directly from it and fail. This failure does not hurt the source log files, but is still annoying.

SYNOPSIS

 arclog [options] logfile... [output]
 arclog [-h|-v]


Product's homepage

Requirements:

· Perl