Tail n mail

From Bucardo

Jump to: navigation, search

tail_n_mail (sometimes abbreviated TNM or tnm) is a Perl script for automatically detecting interesting items that appear in log files and mailing them out to interested parties. It is primarily aimed at Postgres log files but can be used for any files. It was developed at End Point Corporation by Greg Sabino Mullane.

Contents

Download

The latest version, 1.7.1, can be downloaded here:

Basic Usage

To use this script, simply run and provide it with the name of a config file as the first argument. A sample config file can be generated by running: 

 tail tail_n_mail.pl > tnm.config.txt

Edit this file so that it points to your particular logfile and change the email address, then test it out by running: 

 perl tail_n_mail.pl tnm.config.txt

Once it is working properly, have it run periodically by adding an entry to your crontab.

How It Works

When tail_n_mail is run, it first finds which log file or log files to look at. Then it travels to the point in the file (if any) that it left off of last time, extracts all the lines of interest by applying inclusion and exclusion rules, and generates an email message which is then sent to the addresses in the EMAIL line of the config file. Finally, it rewrites the config file itself so that it knows where to start looking the next time it is run.

Command Line Options

There is only one mandatory command line option, and that is the name of a config file. All other options start with a "double dash".

Basic options:

  • --version
    • Returns the version number
  • --verbose
    • Increases the level of verbosity
  • --quiet
    • Turns off minor warnings, e.g. the log file not existing
  • --reset
    • Resets the file size stored in the config file to the current file size. This has the effect of marking the log file as "up to date" so that the next run will only contain entries that occurred after running tail_n_mail with the --reset option.
  • --duration=XXX
    • If running in duration mode, specified the minimum number of milliseconds a query has run before it is reported. It is usually better to put this into the config file with the DURATION: option.
  • --flatten
    • Attempt to normalize similar queries by replacing things such as "VALUES (123, 'abc')" with "VALUES (?,?)". This is on by default, so use --noflatten to turn it off
  • --pgmode
    • On by default, this indicates that the log files are coming from Postgres. In other words, if you are not analyzing Postgres log files, it's best to use --pgmode=0

Debugging options:

  • --dryrun
    • Does everything except send email and rewrite the config file. The contents of the mail message are sent to stdout.
  • --nomail
    • Prevents the sending of email, and outputs the command it would have used.
  • --debug
    • Greatly increases the level of verbosity
  • --file=XXX
    • Use the specified file rather than determining the log file from the config file.
  • --offset=XXX
    • Specifies the exact byte position to start at in the log file. Used for debugging.
  • --rewind=XXX
    • Seeks backwards in the log file by the given number of bytes before processing it. Used for debugging.

Config File Format

Note: the config file is rewritten by tail_n_mail each time it is run, so items inside of it may get reordered and reformatted. However, the basic information will stay the same. Blank lines and lines starting with a hash (#) are ignored. All other items are of the format NAME: value where NAME is always a single uppercase word. The items are:

  • EMAIL:
    • (mandatory) This item consists of one or more email addresses, comma-separated
  • FILE:
    • (mandatory) This is the name of the file to be scanned. In addition to an absolute file name, the following formats are supported:
      • POSIX escape syntax, for example: FILE: /var/log/pg_log/postgresql-%Y-%m-%d.log
      • The special word LATEST. which will cause the program to use the file in that directory that was most recently changed. For example: FILE: /var/lib/postgresql/8.4/main/pg_log/LATEST
  • INCLUDE:
    • A regular expression indicating which lines match for possible mailing out. More than one INCLUDE item line can be used.
  • EXCLUDE:
    • A regular expression indicating which lines to exclude. This is applied after the INCLUDE rules above. More than one EXCLUDE item line can be used.
  • MAILSUBJECT:
    • The subject line for the email to be sent. Special uppercase words can be used that will be replaced each time:
      • HOST - replaced with the current hostname
      • FILE - replaced with the current log file name
  • FIND_LINE_NUMBER:
    • Indicates whether an effort is made to report the line number that each line of interest appears at. Valid values are 1 and 0, and this item is on by default. Counting line numbers can be expensive for very large log files, so this item should be turned off if tail_n_mail is running very often (e.g. once a minute) and the line number is not important.
  • OFFSET:
    • How many bytes to skip in the current log file before beginning to scan. This is set automatically by the program and should not be changed manually.
  • LASTFILE:
    • The name of the last file to be checked. Used in case the file has rotated to another one since the last time tail_n_mail was run. Set automatically, and should not be changed manually.
  • MAXSIZE:
    • The maximum number of bytes in the file to check. This is set to 80 million bytes by default.
  • TYPE:
    • What type of file this is. Currently, the only two choices are:
      • normal: the default, no special processing
      • duration: the duration times are extracted from the log file, allowing the DURATION item of --duration command line flag to be used. This mode is automatically chosen if the name of the config file contains the string 'duration'.
  • DURATION:
    • The minimum number of milliseconds a query is run before it will be reported. Only works when the mode is set to 'duration'.
  • PGFORMAT:
    • Indicates the format of the log file, so that the date can be extracted from each line. The value should be a single number, and defaults to choice "1" below. Each number corresponds to a regular expression as follows:
      • 1: (.+?\[\d+\]): \[\d+\-\d+\]
      • 2: (.+?\d\d:\d\d:\d\d \w\w\w)
      • 3: (.+?\d\d:\d\d:\d\d \w\w\w)

Config File Example

Here is a complete sample config file that grabs all "fatal" errors from today's Postgres log, while excluding any errors caused by trying to connect to a non-existent database: 

 ## Config file for the tail_n_mail.pl program
 ## This file is automatically updated
 EMAIL: greg@endpoint.com
 
 FILE: /var/log/postgresql-%Y-%m-%d.log
 INCLUDE: FATAL:  
 EXCLUDE: database ".+" does not exist
 MAILSUBJECT: HOST Postgres fatal errors (FILE)

Note that there are two spaces after the final colon in the INCLUDE line. This regex might pick up some false positives, but it's good enough for everyday use. If the name of the file above was "tnm.config.fatals.txt", it could be run every five minutes by adding the following line to your crontab: 

 */5 * * * * perl tail_n_mail.pl tnm.config.fatals.txt

Bugs and Feature Requests

Bugs should be reported through the bugzilla bug tracking site. Feature requests are welcome there as well, or send us an email.

Development

Everyone is encouraged to look over and make improvements to the code. The latest development version can be obtained by running: 

 git clone git://bucardo.org/tail_n_mail.git

There is also a GitHub mirror for easy patch contribution by the general public.

Retrieved from "http://bucardo.org/wiki/Tail_n_mail"
Personal tools