Log::Unrotate - Incremental log reader with a transparent rotation handling
version 1.31
use Log::Unrotate; my $reader = Log::Unrotate->new({ log => 'xxx.log', pos => 'xxx.pos', }); my $line = $reader->read(); my $another_line = $reader->read(); $reader->commit(); # serialize the position on disk into 'pos' file my $position = $reader->position(); $reader->read(); $reader->read(); $reader->commit($position); # rollback the last 2 reads my $lag = $reader->lag();
Log::Unrotate allows you to read any log file incrementally and transparently.
Log::Unrotate
Incrementally means that you can store store the reading position to the special file ("pos-file") using commit(), restart the process, and then continue from where you left.
commit()
Transparently means that Log::Unrotate automatically jumps from one log to the next. For example, if you were reading foo.log, then stored the position and left for a day, and then while you were away, foo.log got renamed to foo.log.1, while the new foo.log got some new content in it, Log::Unrotate will find the right log and give you the remaining lines from foo.log.1 before moving to foo.log.
Even better, it will do the right thing even if the log rotation happens while you were reading the log.
Log::Unrotate tries really hard to never skip any data from logs. If it's not sure about what to do, it throws an exception. This is an extremely rare situation, and it is a good default for building a simple and robust message queue on top of this class, but if you prefer a quick-and-dirty recovering, you can enable autofix_cursor option.
Creates a new unrotate object.
Name of a file to store log reading position. Will be created automatically if missing.
Value '-' means not to use a position file. I.e., pretend it doesn't exist at the start and ignore commit calls.
Instead of pos file, you can specify any custom cursor. See Log::Unrotate::Cursor for the cursor API details.
pos
Log::Unrotate::Cursor
Recreate a cursor if it's broken.
Warning will be printed on recovery.
Time period in seconds. If rollback_period is greater than 0, commit method will save some positions history (at least one previous position older then rollback_period would be preserved) to allow recovery when the last position is broken for some reason. (Position may sometimes become invalid because of the host's hard reboot.)
commit
The feature is enabled by default (with value 300), set to 0 to disable, or set to some greater value if your heavily-loaded host is not flushing its filesystem buffers on disk this often.
Name of a log file. Value - means standard input stream.
-
Describes the initialization behavior of new cursors. Allowed values: begin (default), end, first.
begin
end
first
When start is begin, we'll read current log from the beginning.
When start is end, we'll put current position in log at the end (useful for big files when some new script don't need to read everything).
log
When start is first, Log::Unrotate will start from the oldest log file available.
I.e., if there are foo.log, foo.log.1, and foo.log.2, begin will start from the top of foo.log, end will skip to the bottom of foo.log, while first will start from the top of foo.log.2.
Describes the reading behavior when we reach the end of a log. Allowed values: fixed (default), future.
fixed
future
When end is fixed, the log is read up to the position where it ended when the reader object was created. This is the default, so you don't wait in a reading loop indefinitely because somebody keeps adding new lines to the log.
When end is future, it allows the reading of the part of the log that was appended after the reader was created (useful for reading from stdin).
Describes the locking behaviour. Allowed values: none (default), blocking, nonblocking.
none
blocking
nonblocking
When lock is blocking, lock named pos.lock will be acquired in the blocking mode.
When lock is nonblocking, lock named pos.lock will be acquired in the nonblocking mode; if lock file is already locked, exception will be raised.
This flag is set by default. It enables content checks when detecting log rotations. There is actually no reason to disable this option.
Enable inode checks when detecting log rotations. This option should not be enabled when retrieving logs via rsync or some other way which modifies inodes.
This flag is disabled by default, because check_lastline is superior and should be enough for finding the right file.
Read a line from the log file.
Get your current position in log as an object passible to commit().
Save the current position to the pos-file. You can also save some other position, previosly obtained with position().
position()
Pos-file gets commited using a temporary file, so it won't be lost if disk space is depleted.
Get the size of data remaining to be read, in bytes.
It takes all log files into account, so if you're in the middle of foo.log.1, it will return the size of remaining data in it, plus the size of foo.log (if it exists).
Get the current log's number.
Get the log's name. Doesn't contain .N postfix even if cursor points to old log file.
.N
To find and open correct log is a race-condition-prone task.
This module was used in production environment for many years, and many bugs were found and fixed. The only known case when position file can become broken is when logrotate is invoked twice in *very* short amount of time, which should never be a case.
Don't set the check_inode option on virtual hosts, especially on openvz-based ones. If you move your data, inodes of files will change and your position file will become broken. In fact, don't set check_inode at all, it's deprecated.
The logrotate config should not use the compress option to make that module function properly. If you need to compress logs, set delaycompress option too.
compress
delaycompress
This module expects the logs to be named foo.log, foo.log.1, foo.log.2, etc. Skipping some numbers in the sequence is ok, but postfixes should be *positive integers* to be properly sorted. If you use some other naming scheme, for example, foo.log.20130604-140500, you're out of luck. Patches welcome!
Andrei Mishchenko druxa@yandex-team.ru, Vyacheslav Matjukhin me@berekuk.ru.
druxa@yandex-team.ru
me@berekuk.ru
File::LogReader - another implementation of the same idea.
unrotate - console script for reading logs.
Copyright (c) 2006-2013 Yandex LTD. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
See <http://www.perl.com/perl/misc/Artistic.html>
To install Log::Unrotate, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Log::Unrotate
CPAN shell
perl -MCPAN -e shell install Log::Unrotate
For more information on module installation, please visit the detailed CPAN module installation guide.