savelogs - save/rotate/delete log files nicely
savelogs saves your log files in a nice way (by default).
savelogs --postmovehook='/usr/local/bin/restart_apache' \ --apacheconf=/www/conf/httpd.conf /var/log/messages savelogs `cat list_of_logs_to_process.txt` savelogs --loglevel=2 /var/log/maillog /var/log/cronlog \ /var/log/messages savelogs --config=/etc/savelogs.conf savelogs --period=15 /var/log/messages savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com
savelogs is a flexible and robust log file archival system. Its logic is simple: move (rename) the log file, filter data from the log file, store the log file in an archive (via tar or gtar), and compress the archive (via gzip or compress). After successful compression, the original log file is deleted.
All of the above phases are optional. This means that you may simply delete files if you wish. Or you may simply compress existing log files. Or you may move files and add them to a tar file but leave the tar file uncompressed, etc. You pick ;o)
(If you just want to cut to the chase and don't care how savelogs works, see the "EXAMPLES" section near the bottom of this document.)
The processing order may be abbreviated into these five phases:
move -> filter -> archive -> compress -> delete
any of which may be subtracted from the process order. In addition to these phases are some intermediate 'hooks' where you may supply an external program or shell command to invoke. This is useful if you're rotating web server log files and you need to HUP (restart) your web server after moving the logs (for example).
Subtracting phases is done in one of two possible ways. The first way is to specify it in the configuration file:
Process move,archive,delete
which will move log files and archive them (but not filter or compress them). After successful archival, the original log files will be deleted.
The second way is to specify it on the command-line:
--process=compress,delete
which will simply compress log files (but not move, filter, or archive them).
In addition to the five phase processing options above, you may also employ the following abbreviations:
If you specify no process option, the default is move,compress.
Do none of the phases. This isn't a very useful option.
Do all of the phases.
A typical savelogs session might begin by typing this command:
savelogs /var/log/messages
After which the following phases will execute:
The log file is renamed:
/var/log/messages --> /var/log/messages.010523
The log file is compressed
/var/log/messages.010523 --> /var/log/messages.010523.gz
All paths you specify to savelogs should be relative to your home directory (if you're the root user--uid 0--your home directory is set to '/'). You do not need to use a tilde (~). You may assume that savelogs runs from your home directory and knows how to handle absolute paths.
If my real home directory were located in /usr/home/joe and I wanted to rotate the log file located in /usr/home/joe/var/log/messages, I would do something like this:
and savelogs would Do What I Mean.
The only exception to this are external commands given to postmovehook, postfilterhook and other such options. Paths you specify here really are full paths.
savelogs will read its configuration options from a configuration file (which you must supply) or from the command-line. Creating a configuration file is easy: it is a simple Apache-style plaintext file that has options specified in this format:
Option value
where Option is one of the options below and value is either a true/false; yes/no; on/off combination or some string value, such as a pathname and file (depending on the nature of the option).
Your distribution of savelogs may have included a sample configuration file for you to edit as you wish in ~/etc/savelogs.conf.sample.
Configuration options are first read from savelogs internal defaults, which are sprinkled throughout this document. Next savelogs reads its configuration file, if any is specified. Lastly, savelogs uses options from the command-line.
For example, the default value for the period directive is 10. If you ran savelogs like this:
savelogs --period /var/log/messages
every day for 10 days, you would have 10 archived log files.
If you have in your configuration file:
Period 5
and ran the same command above every day for 10 days, you'd only have 5 archived log files because the configuration file overrides savelogs internal defaults. Finally, if you had a configuration file with the previously mentioned value for period and ran this command:
savelogs --period=7 /var/log/messages
every day for 10 days, you would have 7 archived log files because command-line options override configuration file options (which override internal default values).
All options you may specify on the command-line you may also sepcify in a configuration file (except the configuration file directive itself). For example, if you had a cronjob that did this:
savelogs --process=delete \ --postmovehook="/usr/local/apache/bin/apachectl graceful" \ --apacheconf=/www/conf/httpd.conf
(which deletes all apache logs) you could make a nice configuration file (call it ~/etc/savelogs1.conf) that would do the same thing:
Process delete PostMoveHook /usr/local/apache/bin/apachectl graceful ApacheConf /www/conf/httpd.conf
and then invoke your cron like this:
savelogs --config=/etc/savelogs1.conf
A sample configuration file may be located in ~/etc/savelogs.conf.sample which you may copy and edit as you wish. Configuration file directives are case-insensitive:
Process move,compress PROCESS move,compress process move,compress pROceSs move,compress
are all the same directive to savelogs. Configuration file values ARE case-sensitive.
ApacheConf /www/conf/httpd.conf ApacheConf /WWW/conf/httpd.conf ApacheConf /www/CONF/httpd.conf ApacheConf /www/conf/Httpd.conf
are four distinct files, depending on the case-sensitivity of your operating system.
When you are testing new configuration directives, use the dry-run option and watch the log output using the loglevel and logfile directives. This will help you avoid losing data unnecessarily.
You may also specify the settings option which will show you all the current settings after defaults, configuration file, and command-line options have been processed.
Options given below are configuration directives that may appear in a configuration file or on the command-line. Options are case-insensitive, i.e., ApacheLog is the same as apachelog, though the values associated with the options are often case-sensitive (e.g., paths, filenames, etc.)
Options specified on the command-line should be prefixed with two hyphens. Some options do not make sense in a configuration file or need to occur before the configuration file is parsed such as config, help, or home.
Shows a brief usage statement and exits.
Example:
savelogs --help
Shows the current version of savelogs and exits.
savelogs --version
Shows the current settings of savelogs and exits.
savelogs --settings --apacheconf=/www/conf/httpd.conf \ /var/log/messages
When used with the logfile and loglevel settings, dry-run will show you what will happen if you were to run savelogs with the current settings without actually doing it. This is a useful option to specify if you want to see what effect some changes might have, or to see which files are going to get archived with the current settings.
Note that savelogs running under the dry-run directive will sometimes produce errors that wouldn't occur during normal use. This can happen for a variety of reasons, mostly related to savelogs looking for files that don't yet exist, or archives that don't yet exist because they weren't actually created. In this respect, dry-run doesn't give you precisely what will happen, but it does give you a good idea. Use it with a grain of salt.
savelogs --dry-run --loglevel=2 /var/log/foo
Changes the default base location of where savelogs runs from. This is mostly a debugging utility. Consider this an advanced feature which should probably be ignored. Defaults to the process owner's home directory (which is almost always what you want).
Home /usr/home/joe/usr/home/bob
Changes the default configuration file savelogs reads at startup. This should be done from the command-line or it won't have any effect.
savelogs --config=/etc/my_other_savelogs.conf
Tells savelogs which phases to execute. If you just want to move (rename) logs, do this:
savelogs --process=move /var/log/messages
and ~/var/log/messages will become ~/var/log/messages.yymmdd (where yymmdd are today's date).
For just removing logs, specify the delete option. You can get fancy:
savelogs --process=move,compress /var/log/messages
which renames the log file ~/var/log/messages to ~/var/log/messages.yymmdd and then compresses it, not filtering, archiving (i.e. putting into a separate tar file), or deleting it (the compress option also renames the file so that delete becomes useless since the file as it was no longer exists).
move,compress is the default value for the process option, so the above directive could have also been specified:
You may also specify all or none as shortcuts for
savelogs --process=move,filter,archive,compress,delete
and
savelogs --process=
respectively.
savelogs has an internal logging facility that helps you diagnose problems, or just see what's going on. By default, savelogs writes to stdout (i.e., your screen if you're running this from a tty).
Determines how verbose savelogs is when it's writing its own internal messages. Valid values are between 0 and 5 inclusive.
LogLevel 3
The general rule of thumb for savelogs logging is this:
Level What will be logged ===== =================== 0 no output except fatal errors 1 Level 0 + start/finish stats, errors 2 Level 1 + warnings, logfiles to process 3 Level 2 + chdir, filter, phase completion 4 Level 3 + phase core actions, phase beginning 5 Level 4 + everything else
The first few times you run savelogs, try a higher loglevel value to see what's happening with your log files. Once you're comfortable with how savelogs works, you may turn it down a few notches (level 1 is usually fine) so at least you can check to see if your cronjob actually ran ;o)
savelogs, depending on the loglevel you've specified, writes what it's doing, such as moving, archiving, or deleting, etc. The logfile directive tells savelogs where to write all these messages.
The default value for logfile is stdout which means that your output will go to the screen unless you've redirected stdout. You may also specify stderr or the path to a file where you'd like messages to be appended.
LogFile /var/log/savelogs.log
savelogs processes the logs you specify on the command-line (items on the command-line that are not recognized as options are assumed to be log files to process).
If no logs are specified (either on the command-line or in a configuration file using the following directives), savelogs will complain and show a 'usage' statement. To turn off the usage statement, use the gripe directive (to gripe is the default behavior):
savelogs --nogripe /no/such/log
To save wear on your finger tips and phosphor in your monitor, we recommend liberal use of the following configuration directives.
This works just like adding a file on the command-line, but is included so that you can put log files you want processed in a configuration file. It will also work on the command-line:
savelogs --log=/var/log/messages
is the same as:
which is also equivalent to a config file named '~/etc/savelogs.conf' with this single line:
Log /var/log/messages
and invoked like this:
savelogs --config=/etc/savelogs.conf
The log directive also accepts any standard csh-ish wildcard (e.g., *, ?, [n-m], etc.) for globbing. Globbing is where you specify a wilcard pattern and the argument list is expanded to all filenames that match the pattern. In savelogs, this pattern implicitly excludes files whose names end in '.tar', '.gz', or '.tgz' (so you don't have to worry about compressing already-compressed files).
This is useful if you have log files that are created dynamically and whose names you may not know precisely. For example, say you have a list of files in a directory:
somelog.010909.gz somelog.01090a.gz somelog.01090b.gz somelog.01090c.gz somelog.01090d.gz somelog.01090e.gz somelog.01090f.gz somelog.011001 somelog.011002 somelog.011003 somelog.011004 somelog.011005 somelog.011006 somelog.011007 somelog.011008 somelog.011009 somelog.01100a somelog.01100b somelog.01100c
To compress these files (the ones that have not already been compressed) you can simply do this:
savelogs --log='/path/to/somelog.*'
The files that end in '.gz' are skipped (savelogs skips them internally).
Be sure to protect the asterisk (*) with quotes so that the current shell doesn't try to expand them. You could also do this in a configuration file:
Log /path/to/somelog.*
This is the compliment to the Log directive: it removes logs from the list of logs to process. This is useful if you have a log or set of logs that is handled by a separate rotation program or needs special treatment at another time.
For example, if you have many log files listed in your Apache configuration file, you'll want to take advantage of the ApacheConf directive (see below). This will make your savelogs configuration file small and easy to understand:
## rename and compress all logs found in httpd.conf ApacheConf /www/conf/httpd.conf
This is great, except that there's this one log that you don't want savelogs to process. Before savelogs version 1.40, the only option you had was to list each log individually with the Log directive (i.e., you couldn't use the ApacheConf directive at all in such cases). Now, however, you can use the NoLog directive to exclude logs that have already been added to the list:
## rename and compress all logs found in httpd.conf ApacheConf /www/conf/httpd.conf ## ... and exclude joe's logs (joe-*_log matches ## joe-access_log and joe-error_log) NoLog /www/logs/joe-*_log
You may use full paths or you may use shell wildcard patterns, just like the Log directive.
If you have both Log and NoLog directives, the NoLog directive is processed last. This means that:
Log /var/log/messages NoLog /var/log/messages
NoLog /var/log/messages Log /var/log/messages
and that /var/log/messages will not be processed in either case.
NoGripe tells savelogs not to complain about not finding any log files to process. By default, savelogs gripes about not finding any log files: if you forget to specify any logs (or any directives such as ApacheConf that find logs for you) savelogs will complain. Also, if you specify log files to process but none of them exist, savelogs will similarly complain.
When you turn on NoGripe, the complaining is stopped and savelogs exits happily.
savelogs --nogripe
or in your configuration file:
Gripe no
If you specify this option, giving it a valid httpd.conf file, savelogs will parse your Apache configuration file looking for standard log file directives. Any files found will be processed.
savelogs --apacheconf=/www/conf/httpd.conf
ApacheConf /usr/local/etc/httpd/conf/httpd.conf
Using the ApacheConf directive will tell savelogs to search through httpd.conf looking for all files associated with TransferLog, ErrorLog, AgentLog, etc. (all those listed in the ApacheLog directive) and process them.
This option tells savelogs which logs to select out of the Apache configuration file (as specified by the ApacheConf directive) based on the Apache ServerName directive in the VirtualHost block. If this option is set, only logs for matching hosts will be rotated (this applies only to logs found in the Apache configuration file; other logs specified in other ways (e.g., on the command-line or via the Log directive) will be processed as usual).
savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com
ApacheConf /www/conf/httpd.conf ApacheHost foo.com
The ApacheHost directive may be specified multiple times to process logs for multiple virtual hosts. If no logs are found in the Apache VirtualHost block, no logs will be rotated for that virtual host.
This option allows you to tell savelogs which logs to process in the httpd.conf file specified by apacheconf. The default value for the apachelog directive is:
TransferLog|ErrorLog|AgentLog|RefererLog|CustomLog
You may do something clever like this:
savelogs --apacheconf=/www/conf/httpd.conf --apachelog=TransferLog
which would archive all of your access_log files. Then after running this command, you could do this:
savelogs --process=delete --apacheconf=/www/conf/httpd.conf \ --apachelog=ErrorLog
which would delete your error_log files.
In general, the fewer values you specify in the apachelog directive the faster savelogs will find your log files (though the speedup really is negligible, it may also save you from rotating logs you didn't want to).
ApacheLog TransferLog|ErrorLog
would be sufficient for most people using combined Apache logs.
For those who want somewhat finer control of which logs get processed in their Apache configuration file, the apachelogexclude directive allows you to specify a Perl regular expression (which may simply be a string like 'error') of log files to exclude when processing logs. This way you could do something like this:
savelogs --apacheconf=/www/conf/httpd.conf --apachelogexclude=logs/bob
which would process all logs found in your httpd.conf file except log files whose names contain the string 'logs/bob'. Maybe Bob likes to rotate his logs using another program or system (conceivable, though unlikely).
Multiple occurances of apachelogexclude are allowed:
ApacheLogExclude /dev/null ApacheLogExclude \| ApacheLogExclude logs/bob
which would exclude log files whose names contained '/dev/null' or 'logs/bob' from being processed. Any valid Perl regular expression will work, so:
ApacheLogExclude ^/dev/null$
is not the same as the previous example. This example will only match log files whose name is exactly /dev/null, no more, no less.
By default, savelogs uses the following patterns to determine logs to exclude:
^/dev/null$ \| (this is a literal pipe character)
This means that by default savelogs will not attempt to archive a log whose name is '/dev/null' or whose name contains a pipe (|). If for some bizarre reason you wish to remove these defaults when you run savelogs, you can give an empty apachelogexclude option on the command-line:
% savelogs --apachelogexclude= --config=/etc/savelogs.conf
Logs that Apache writes via a pipe must be specified separately using the Log directive or on the command-line.
When specified, this directive tells savelogs to read the Apache configuration file (httpd.conf) and follow Inlcude directives (see http://httpd.apache.org/docs/mod/core.html#include for details).
It will work just like Apache does: it will look in directories (when a directory is given as the Include option), it will expand path wildcards (e.g., httpd_[bcd].conf will expand to httpd_b.conf, httpd_c.conf, and httpd_d.conf), and it will work with simple include files as well (e.g., virtual_hosts.conf).
httpd_[bcd].conf
virtual_hosts.conf
Log files found in Include'ed configuration files will also be processed. savelogs has internal consistency checks to ensure that logs are not processed twice, neither are configuration files read twice (thus avoiding those annoying infinite loops).
% savelogs --apacheconf=/www/conf/httpd.conf --apacheinclude
ApacheConf /www/conf/httpd.conf ApacheInclude yes
Beginning with savelogs version 1.90, you can apply directives to groups of log files with the <Group> directive in a savelogs configuration file (the directive has no command-line equivalent). The following directives apply in a Group block:
period, ext, sep, datefmt, hourly, touch, chown, chmod, clobber, apachehost, log, disabled
As of version 1.90, the following directives are not honored, but may be at some point in the future:
size, filter, stem, stemhook, stemlink, nolog, apachelog, apachelogexclude
Group settings override any other settings found in the file, but are applied only to log files found within the Group block.
ApacheConf /www/conf/httpd.conf PostMoveHook apachectl restart Period 30 <Group> ApacheHost www.sample1.tld ApacheHost www.sample2.tld Period 10 Chown roger:staff </Group> <Group> ApacheHost www.sample3.tld ApacheHost www.sample4.tld Chown fonzie </Group>
Explanation:
The first Group block will rotate Apache logs for sample1.tld and sample2.tld, but it will only rotate 10 days worth of logs, since the Period directive inside the block is set to 10.
sample1.tld
sample2.tld
The second Group block will rotate Apache logs for sample3.tld and sample4.tld, and the full 30 day Period setting will apply to them.
sample3.tld
sample4.tld
Both groups will search /www/conf/httpd.conf and restart with apachectl restart.
apachectl restart
Group blocks have a special directive Disabled that when set to true, will skip this block. Griping is also disabled when Disabled is set to true:
## this group will be completely skipped. <Group> Disabled 1 ApacheHost www.foo.com Chown roberto </Group>
The purpose of the Disabled directive is to prevent ApacheConf from reverting to its behavior in the absence of any ApacheHost directives (i.e., it will parse the entire configuration file and process all hosts found). If this is what you desire, then remove or comment out your Group blocks instead of Disabled'ing them.
Known bugs related to the Group directive as of 1.91:
If a log file name found inside of a Group block also matches a NoLog directive outside of the Group block, it will be skipped as if it were outside the block.
If a log referenced inside of a Group block is processed outside of it also (found by some other directive, for example), the first one found will have precedence and the settings inside the Group block may be ignored.
If there are many files being processed, it is possible that in rare circumstances (namely, when processing spans midnight) that one set of logs may be named using one date and another set using another date. This only applies to non-periodic rotation.
Touches the original file, creating it if necessary. This is useful for programs that log in "append-only" mode and do not create the log file if it is missing. Once savelogs has renamed a log file, touch will create the file if it does not exist, or reset the timestamp if it does.
Logs smaller than kbytes will not be included in any processing. To override a default setting, specify --size with no arguments on savelogs command-line.
Size 5000
will skip all log files smaller than 5 megs, regardless of other settings.
Allows you to change how dates are formatted using the standard strftime system call. See strftime(1) for format string options. The default string is '%y%m%d'.
## renames logs like this: access_log.02-12-25 ## Merry Christmas! DateFmt %y-%m-%d
Some popular options are:
## 20020626 (26 June 2002) DateFmt %Y%m%d ## 1.Mar.2002 DateFmt %e.%b.%Y
Set the filename extension to 'string'. When a file is moved, it is renamed to the original filename plus the extension you specify. If no extension is specified, today's date is used in 'yymmdd' format. Options include today and yesterday.
You may not use ext in a configuration file with a value in backticks (e.g., the line:
## this directive will not work: don't use it! Ext `/bin/date`
in a configuration file will not work). Any other (static) value for ext in a configuration file will work.
See also hourly below for information on how to modify ext even further. See datefmt above if you want to format your dates differently. ext is provided chiefly for completeness; its usefulness is limited except in special circumstances where savelogs can't offer a reasonable name for your log.
The separator character to use when moving files. By default this character is a dot ('.'). Other favorites are underscore ('_') and hyphen/minus sign ('-').
Use an underscore character:
savelogs --sep='_' --process=move /var/log/foo
will rename ~/var/log/foo to ~/var/log/foo_yymmdd (where yymmdd is today's date).
Use no separator (just concatenate the extension with the filename):
savelogs --sep= --process=move /var/log/foo
will rename ~/var/log/foo to ~/var/log/fooyymmdd.
Adds a letter of the alphabet to the back of the filename extension. This is useful if you are rotating logs several times a day. For example, if you specified your extension (via ext) as 'foo', any log files rotated in the 10am hour will be named 'log.fook'. At 11am, all logs will be called 'log.fool' and so forth.
savelogs /path/to/log_file
This will rename the log file to log_file.yymmdd where yymmdd are today's year, month, and day of month.
savelogs --hourly /path/to/log_file
If you specify the hourly option (or in your configuration file, the Hourly directive), the log will be renamed to log_file.yymmddz where the z represents the current hour as a letter of the alphabet (0 = a, 1 = b, 2 = c, etc.).
You could also use the datefmt directive to get similar results (with much more flexibility to boot). If you rotate more often than once and hour, use the datefmt directive or use period to rotate logs with a unique number as the extension. period, discussed below, renames log files with a simple integer: log becomes log.0, the former log.0 becomes log.1 and so forth.
Renames the file based on a period, which is how frequently you run savelogs. If you specify period you may also optionally specify a 'count', which is how many log files to save using the period option:
savelogs --period=8 /var/log/messages
You may also use the count option, which is deprecated for backward compatibility and some possible future enhancements:
savelogs --period --count=8 /var/log/messages
If you do not specify a count value (either in period or count), a count of 10 is assumed.
The period option will rename the current log to logfile.0, the log that was previously named logfile.0 to logfile.1 and so on, much like newsyslog(8).
When you specify the period option, the process phases move and compress are assumed. If you also specify a process phase of filter, that will be honored also.
The period option will override any other sep and ext options specified, using the default dot ('.') for the separator and an integer for the extension.
The author also recommends you don't try to mix period named log files with other log files in the same directory, since savelogs may not be able to tell which logs are oldest based on the filename extension and destroy the wrong files. You can safely use any default savelogs extensions (e.g., the default 'today' or 'yesterday' extensions) or your own extension if your own extension contain at least one non-digit (0-9) or your own extension has 5 or more digits. If you meet either of these criteria in your own extension, you may feel confident about mixing logs.
For those familiar with Unix system adminstration, period works like newsyslog(8) with the B option specified in the newsyslog configuration file (the B option tells newsyslog to treat logs as binary files and not append the status message to the log).
An example use of the period option:
savelogs --touch --period=15 /var/log/maillog /var/log/messages
will move any existing ~/var/log/maillog and ~/var/log/messages to ~/var/log/maillog.0 and ~/var/log/messages.0 and compress them. By specifying the touch option, the original ~/var/log/maillog and ~/var/log/messages will be 'touched', recreating the files. When this command is run again, ~/var/log/maillog.0 is moved to ~/var/log/maillog.1 and ~/var/log/maillog is moved to ~/var/log/maillog.0.
Limits the number of logs saved using the period option. The internal default value for count is 10.
If you are using the period option, as of version 1.21 you may now simply specify the count as part of the period option:
savelogs --period=5 /var/log/messages
Runs an arbitrary system command you specify after moving files. If you are rotating Apache log files, you should use a command that will tell your web server to close its log file descriptors and re-open them (e.g., 'restart_apache').
savelogs --apacheconf=/www/conf/httpd.conf \ --postmovehook='/usr/local/bin/restart_apache'
or even nicer in your configuration file:
PostMoveHook /usr/local/bin/restart_apache
Paths in postmovehook are NOT relative to your home directory, as most other paths are. You should specify the full path to the file if the file is not in your environment's $PATH. The exception to this rule is the $LOG macro which, specified, will automatically be replaced with the current log file path. See Variables below for details.
postmovehook is one of two ideal phases to analyze your data before it is archived away (the other phase is the postfilterhook phase).
Variables
Some internal savelogs variables are available during the postmovehook phase. These variables are automatically interpolated by savelogs during execution and are guaranteed to contain some useful value.
Contains the full path to the Apache configuration file as specified with the apacheconf option.
savelogs --apacheconf=/www/conf/httpd.conf \ --postmovehook='touch $APACHE_CONF'
Contains the path to your home directory.
savelogs --postmovehook='$HOME/bin/myprogram'
Contains the current log file being processed. If you wanted to run a command on each log file after it is moved, you may enter that command here (on one line). If the line is really long, consider putting it into a shell script that "wraps" all of your options.
PostMoveHook /usr/local/bin/do_something_with_every $LOG
Executes the postmovehook command even if there are no logs to process. By default, savelogs will not execute the postmovehook command if there are no logs.
After savelogs has renamed your log(s), you may wish to chown them to a new user or group. Use the chown option for this. If you don't specify a user, the user will not change. If you don't specify a group, the group won't change.
## chown the logs to joe:joegroup Chown joe:joegroup ## make the log files owned by the group with gid 500, keep the ## current owner Chown :500 ## change the owner to root, leave the group alone Chown root:
Notice that the colon is necessary at all times.
The chown option executes after the PostMoveHook phase has completed.
After savelogs has renamed your log(s), you may wish to change the permissions of the logfile. Use the chmod option for this. Permissions should be specified in octal.
## make readable only by the owner Chmod 0600
The chmod option executes after the PostMoveHook phase has completed.
Like ext except the stem is used in addition to ext. After the move and postmovehook phases have completed, savelogs checks to see if you have defined a stemhook. If a stemhook has been defined, a symbolic link to the log is made using stem.
As an example, say you were processing ~/var/log/messages. During the stem phase, savelogs would do this:
messages.today -> messages.<ext> (where <ext> is the extension you specified or today's date by default)
Once the stemhook command has executed, the symbolic link (or hard link or copy, as specified by stemlink) is removed.
The stem related options were added in version 1.28.
A command to execute, much like postmovehook, except that this phase is suited for log file analysis tools that require a predictably named, dead (i.e., no logging is currently being done to it) log file. urchin and analog are good examples of programs that require such logs.
An example use for stemhook would be:
% savelogs --stemhook="$HOME/usr/local/urchin/urchin" \ /www/logs/access_log /www/logs/error_log
urchin should be instructed to operate on /www/logs/access_log.today and /www/logs/error_log.today. urchin should also be instructed to do nothing to the log files since we're allowing savelogs to manage them for us. Any changes resulting from the stemhook command to the log file will occur in the original log file.
The same variables for postmovehook are available for stemhook.
Specify the type of link that should be made during the stem phase. The default is symlink. Other options are hard which creates hard links and copy which creates a copy of the original file. hard links are useful for stemhook commands that cannot process symbolic links. If your stemhook command modifies the log file, you may wish to choose the copy option which will be discarded after the stemhook command is executed.
A filter is simply a program that generates something on STDOUT. They may be pipelines or other programs.
If the filter process option is specified (via the process directive), you should supply a program to filter your log (via the filter command), such as egrep or perl (or a pipeline or a shell script containing your commands, etc.).
If no filter command is given, the filter phase will be skipped (even if you specify --process=filter as the process directive). Consider following filter command:
--filter='/usr/bin/egrep -v "/images/" \$LOG'
When savelogs gets to its filter phase, it will open a pipe to the above command. Output from this command will be saved to a temporary file, then the temporary file will be renamed to replace the original log file.
Notice the strange $LOG variable. This is an internal savelogs variable that refers to the location of the log file savelogs is currently working on. It is automatically replaced with the right file during execution.
If you are supplying a filter command on the command-line, the backslash (\) in front of $LOG is necessary to tell the shell to not interpolate <$LOG> as a shell variable, but instead pass it along untouched to savelogs. The backslash is not necessary if you are specifying a filter directive in the configuration file:
Filter /usr/bin/egrep -v "/images/" $LOG
For the sake of completeness, you can also chain filters via a pipeline, like this:
Filter egrep -v "/images/" $LOG | egrep -v "(root|cmd)\.exe" -
The final '-' tells egrep to use stdin from the previous pipe for its input.
Runs an arbitrary system command you specify after filtering files. See postmovehook for examples, including the Variables section.
Like force-pmh, this forces execution of the postfilterhook command even if there are no logs (and assuming savelogs has reached this phase).
Specifies the location of the tar program to use. This defaults to whatever it can find on your system. You usually don't need to modify this option unless your tar or gtar program is not in your path.
Gtar /usr/sbin/gtar
If both gtar and tar are specified, gtar will be used.
Specifies the name of the archive to which files will be appended.
This directive is somewhat tricky to understand. Under normal use, savelogs uses the name of the file being archived as the archive name. For example, if you were archiving a file named ~/var/log/messages, the name of the archive would be ~/var/log/messages.tar.
If you are archiving multiple files (which is common), each file will be stored in its own archive by name in the directory where the file is located. For example, if you had several files located in ~/var/log, each file would be stored in its own archive named filename.tar in the ~/var/log directory.
If you want to lump together all files in a particular directory into one archive, use the archive directive without any path information:
savelogs --archive=system.tar \ /var/log/messages /var/log/proftpd /var/log/foo \ /www/logs/access_log /www/logs/error_log
This will archive ~/var/log/messages, ~/var/log/proftpd, and ~/var/log/foo in a single file named ~/var/log/system.tar (which may later be compressed if you've so specified). ~/www/logs/access_log and ~/www/logs/error_log will also be lumped together in a file called ~/www/logs/system.tar.
If you want to lump together all files for this savelogs session into one archive, use the archive directive and specify the full path to the archive:
savelogs --archive=/var/tmp/logs.tar /var/log/messages \ --apacheconf=/www/conf/httpd.conf
This will archive ~/var/log/messages and all log files found in the Apache configuration file into a single archive named ~/var/tmp/logs.tar (which may later be compressed).
If you wish to place the archive in your home directory, you may be tempted to just do this:
savelogs --archive=/logs.tar
This won't work. Because of the way savelogs tries to simplify things relative to your home directory, the leading slash is dropped and savelogs doesn't find any path information (and therefore places the archive in where the files are).
To put files in your home directory, preceed the archive command with a dot-slash:
savelogs --archive=./logs.tar
This won't put the archive in your current working directory, as some are wont to assume, but in your home directory. savelogs has no notion of a current working directory because it is always changing directories from your home to the directory where the log files are and back.
Specifies whether files stored in archives are full paths relative to your home directory or relative paths relative to the directory in which the file is found.
If full-path is not specified, paths are stored in the tar file relative to their parent directory.
will create an archive with the following file in it:
messages
while
savelogs --full-path /var/log/messages
var/log/messages
When you extract this file later on, the paths will be created for you if they don't exist, which may not be what you want (but, of course, it may be what you want which is why we have this directive).
Specifies the location of the gzip/compress/uncompress binaries for decompressing files (files ending with '.gz' or '.Z'). Use of compress and uncompress is deprecated. If your system has a gzip program in a directory that is not in your $PATH variable, specify its location with this directive. If gzip and compress are specified, gzip will be used.
By default (if none of the above options are specified), savelogs will search for a gzip binary in your path and use it.
If a compressed archive already exists along side a non-compressed archive (e.g., archive.tar and archive.tar.gz), and you've instructed savelogs to compress archive.tar, some compression programs (like gzip) will ask you for confirmation before overwriting existing files.
To get around this, savelogs by default enables the 'force' option on compression programs (usually -f). This way, if you're running savelogs from a cron job or another method where there is no controlling terminal, savelogs keeps running.
If you're running savelogs interactively (i.e., from a tty) and want savelogs to prompt you to overwrite existing compressed files, specify the noclobber option:
savelogs --noclobber
Clobber no
The author recommends liberal use of the dry-run option when testing these examples or when making big changes to your configuration file or command-line options. Doubly-so when you have the delete process option enabled. There's no 'undelete' for UNIX.
By default, savelogs will move, and compress a log file. This is its simplest use. If this command is run daily, the result will be a file name like the old file with a yymmdd extension. This file will be compressed.
You can use savelogs (contrary to its name) to just wipe out log files and reclaim the disk space. If you've got a couple of files that from time to time just get too big and there's really no valuable information in them, do something like this:
savelogs --process=delete /path/to/log_file1 /path/to/log_file2
If you want to nuke all Apache log files, do something like this:
savelogs --process=delete --postmovehook=/usr/local/bin/restart_apache \ --apacheconf=/www/conf/httpd.conf
When you specify only the delete as a process option, no logs are moved, archived, or compressed. They're just deleted.
Compressing logs daily is easy:
will make:
-rw-r--r-- 1 test vuser 751327 Jul 6 12:48 messages
become:
-rw-r--r-- 1 test vuser 84625 Jul 6 12:48 messages.010706.gz
savelogs --size=5000 /var/log/messages
will only compress the log if the size of the file is 5000 kilobytes (5 megabytes) or larger.
You want to save 3 days worth of Apache log files and 5 days worth of system log files. You might try the following lines in your crontab:
1 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \ --postmovehook=/usr/local/bin/restart_apache --period=3 \ --apacheconf=/www/conf/httpd.conf 5 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \ --period=5 /var/log/messages /var/log/ftp.log
Most crontab files require that no lines wrap, so you'd need to make sure to keep everything on one line.
Your ~/www/logs directory may look something like this after a week:
access_log access_log.0.gz access_log.1.gz access_log.2.gz error_log error_log.0.gz error_log.1.gz error_log.2.gz
and your system logs directory:
messages messages.0.gz messages.1.gz messages.2.gz messages.3.gz messages.4.gz ftp.log ftp.log.0.gz ftp.log.1.gz ftp.log.2.gz ftp.log.3.gz ftp.log.4.gz
Most people want to group their log files in an archive. This makes storing them and retrieving them later for post-processing simple and efficient. Your directory tree might look like this:
usr/local/etc/httpd/logs usr/local/etc/httpd/logs/bar usr/local/etc/httpd/logs/bar/access_log usr/local/etc/httpd/logs/bar/error_log usr/local/etc/httpd/logs/baz usr/local/etc/httpd/logs/baz/access_log usr/local/etc/httpd/logs/baz/error_log usr/local/etc/httpd/logs/biz usr/local/etc/httpd/logs/biz/access_log usr/local/etc/httpd/logs/biz/error_log usr/local/etc/httpd/logs/buz usr/local/etc/httpd/logs/buz/access_log usr/local/etc/httpd/logs/buz/error_log usr/local/etc/httpd/logs/foo usr/local/etc/httpd/logs/foo/access_log usr/local/etc/httpd/logs/foo/error_log
Issue this command:
savelogs --process=all --apacheconf=/www/conf/httpd.conf
and your directory tree now looks like this:
usr/local/etc/httpd/logs usr/local/etc/httpd/logs/bar usr/local/etc/httpd/logs/bar/access_log.tar.gz usr/local/etc/httpd/logs/bar/error_log.tar.gz usr/local/etc/httpd/logs/baz usr/local/etc/httpd/logs/baz/access_log.tar.gz usr/local/etc/httpd/logs/baz/error_log.tar.gz usr/local/etc/httpd/logs/biz usr/local/etc/httpd/logs/biz/access_log.tar.gz usr/local/etc/httpd/logs/biz/error_log.tar.gz usr/local/etc/httpd/logs/buz usr/local/etc/httpd/logs/buz/access_log.tar.gz usr/local/etc/httpd/logs/buz/error_log.tar.gz usr/local/etc/httpd/logs/foo usr/local/etc/httpd/logs/foo/access_log.tar.gz usr/local/etc/httpd/logs/foo/error_log.tar.gz
Inside of each compressed archive is a single file:
access_log.010523
which is the old log renamed with today's date. When you run this command again (e.g., from a cron job) tomorrow, you'll see the same list of files above, except that inside each compressed archive is an additional file:
access_log.010523 access_log.010524
Say you want to group together all logs in a single directory in one archive. Your directory tree might look like this:
Try this:
savelogs --process=all --archive=logs.tar --apacheconf=/www/conf/httpd.conf
usr/local/etc/httpd/logs usr/local/etc/httpd/logs/bar usr/local/etc/httpd/logs/bar/logs.tar.gz usr/local/etc/httpd/logs/baz usr/local/etc/httpd/logs/baz/logs.tar.gz usr/local/etc/httpd/logs/biz usr/local/etc/httpd/logs/biz/logs.tar.gz usr/local/etc/httpd/logs/buz usr/local/etc/httpd/logs/buz/logs.tar.gz usr/local/etc/httpd/logs/foo usr/local/etc/httpd/logs/foo/logs.tar.gz
Inside each 'logs.tar.gz' are two files:
access_log.010523 error_log.010523
Say you want to lump all logs in a savelogs session into a single archive. Specify the archive option with a full path, like this:
savelogs --process=all --archive=/tmp/all_logs.tar \ --apacheconf=/www/conf/httpd.conf /var/log/messages
This will create a single file /tmp/all_logs.tar.gz that contains all logs found in httpd.conf as well as /var/log/messages.
Maybe your log file fills up with garbage entries that you want to clean out daily. You can use the filter option to trim your log in place:
savelogs --process=filter \ --filter='/usr/bin/egrep -v "(root|cmd)\.exe" \$LOG' \ --postfilterhook='/usr/local/bin/restart_apache' \ --apacheconf=/www/conf/httpd.conf
This will clean out some Windows worm requests from your Apache log files.
savelogs will not properly interpolate backticks inside the configuration file. Because backticks are shell operators, they must be visible to the shell (meaning they have to be done on the command-line). An example is the ext command where the command-line option might look like this:
savelogs --ext=`smalldate -m`
there is no corresponding configuration file entry.
You must obey shell escaping rules when you're doing filter commands on the command-line. Sometimes these can be tricky. For example, double quotes will do different things than single quotes (I recommend single quotes because you can then use the $LOG variable without big headaches). Using a config file sometimes helps.
You can't have multiline commands in the configuration file. This is on the "to do" list.
Would be nice to use the VirtualHost 'User' directive for the chown option. Maybe a 'ApacheChown' option.
Optimize the ApacheHost directive to jump out of the Apache config parsing once all hosts have been found (can you have a VirtualHost block appear twice with the same ServerName directive?)
Thanks to Jeroen Latour (cpantester@calaquendi.net) for working with me on getting all the tests to run cleanly on his Cobalt box. It should now test cleanly on many other platforms because of his patience.
Scott Wiersdorf, <scott@perlcode.org>
Copyright (c) 2001-2004 Scott Wiersdorf. All rights reserved.
1 POD Error
The following errors were encountered while parsing the POD:
=back doesn't take any parameters, but you said =back 4
To install savelogs, copy and paste the appropriate command in to your terminal.
cpanm
cpanm savelogs
CPAN shell
perl -MCPAN -e shell install savelogs
For more information on module installation, please visit the detailed CPAN module installation guide.