mailtoe - a train-on-error simulator for use with dbacl.
SYNOPSIS
mailtoe command [
command_arguments ]
DESCRIPTION
mailtoe automates the task of testing email filtering and classification
programs such as
dbacl(1) . Given a set of categorized documents, mailtoe initiates test runs
to estimate the classification errors and thereby permit fine tuning
of the parameters of the classifier.
Train-on-error (TOE) is a learning method which is sometimes advocated for
email classifiers. Given an incoming email stream, the method consists
in reusing a fixed set of category databases until the first misclassification
occurs. At that point, the offending email is used to relearn the relevant
category, until the next misclassification. In this way, categories are
only updated when errors occur. This directly models the way that some email
classifiers are used in practice.
TOE's error rates depend directly on the order in which emails are seen.
A small change in ordering, as might happen due to networking delays,
can have a large impact on the number of misclassifications.
Consequently,
mailtoe does not give meaningful results, unless the sample emails are chosen carefully.
However, as this method is commonly used by spam filters, it is still worth
computing to foster comparisons. Other methods (see
mailcross(1) , mailfoot(1) ) attempt to capture the behaviour of classification errors in other ways.
To improve and stabilize the error rate calculation,
mailtoe performs the TOE simulations several times on slightly reordered email streams, and
averages the results. The reorderings occur by multiplexing the emails from each
category mailbox in random order. Thus if there are three categories, the first email
classified is chosen randomly from the front of the sample email streams of each type.
The second email is also chosen randomly among the three types, from the front of the
streams after the first email was removed. Simulation stops when all sample streams
are exhausted.
mailtoe uses the environment variable MAILTOE_FILTER when
executing, which permits the simulation of arbitrary filters, provided
these satisfy the compatibility conditions stated in the
ENVIRONMENT section below.
For convenience,
mailtoe implements a
testsuite framework with predefined wrappers for several open
source classifiers. This permits the direct comparison of
dbacl(1) with competing classifiers on the same set of email samples. See the USAGE section below.
During preparation,
mailtoe builds a subdirectory named mailtoe.d in the current working directory.
All needed calculations are performed inside this subdirectory.
EXIT STATUS
mailtoe returns 0 on success, 1 if a problem occurred.
COMMANDS
"prepare
Prepares a subdirectory named mailtoe.d in the current working directory, and
populates it with empty subdirectories for exactly
size subsets.
"add
Takes a set of emails from either FILE if specified, or STDIN, and
associates them with
category .The ordering of emails within FILE is preserved, and subsequent FILEs are appended
to the first in each category.
This command can be repeated several times,
but should be executed at least once.
"clean"
Deletes the directory mailtoe.d and all its contents.
"run"
Multiplexes randomly from the email streams added earlier, and relearns
categories only when a misclassification occurs. The simulation is repeated
size times.
"summarize"
Prints average error rates for the simulations.
"plot
Plots the number of errors over simulation time. The "ps" option, if present,
writes the plot to a postscript file in the directory mailtoe/plots, instead of
being shown on-screen. The "logscale" option, if present, causes the plot to
be on the log scale for both ordinates.
"review
Scans the last run statistics and extracts all the messages which belong to category
truecat but have been classified into category
predcat . The extracted messages are copied to the directory
mailtoe.d/review for perusal.
"testsuite
Shows a list of available filters/wrapper scripts which can
be selected.
"testsuite
Prepares the filter(s) named
FILTER to be used for simulation. The filter name is the name of
a wrapper script located in the directory
/usr/share/dbacl/testsuite . Each filter has a rigid interface documented below, and the act of selecting
it copies it to the
mailtoe.d/filters directory. Only filters located there
are used in the simulations.
"testsuite
Removes the named filter(s) from the directory
mailtoe.d/filters so that they are not used in the simulation.
"testsuite
Invokes every selected filter on the datasets added previously, and
calculates misclassification rates. If the "plots" option is present,
each filter simulation is plotted as a postscript file in the directory
mailtoe.d/plots .
"testsuite
Describes the scheduled simulations.
"testsuite
Shows the cross validation results for all filters. Only makes sense
after the
run command.
USAGE
The normal usage pattern is the following: first, you should separate your email
collection into several categories (manually or otherwise). Each category should
be associated with one or more folders, but each folder should not contain
more than one category. Next, you should decide how many runs to use, say 10.
The more runs you use, the better the predicted error rates. However, more runs take more time.
Now you can type
% mailtoe prepare 10
Next, for every category, you must add every folder associated with this
category. Suppose you have three categories named
spam , work , and
play , which are associated with the mbox files
spam.mbox , work.mbox , and
play.mbox respectively. You would type
% mailtoe add spam spam.mbox
% mailtoe add work work.mbox
% mailtoe add play play.mbox
You should aim for a similar number of emails in each category, as the random
multiplexing will be unbalanced otherwise. The ordering of the email messages
in each
*.mbox file is important, and is preserved during each simulation. If you repeatedly
add to the same category, the later mailboxes will be appended to the first, preserving
the implied ordering.
You can now perform as many TOE simulations as desired. The multiplexed emails
are classified and learned one at a time, by executing the command given in the
environment variable MAILTOE_FILTER. If not set, a default value is used.
% mailtoe run
% mailtoe summarize
The testsuite commands are designed to simplify the above steps and allow comparison
of a wide range of email classifiers, including but not limited to
dbacl . Classifiers are supported through wrapper scripts, which are located in the
/usr/share/dbacl/testsuite directory.
The first stage when using the testsuite is deciding which classifiers to compare.
You can view a list of available wrappers by typing:
% mailtoe testsuite list
Note that the wrapper scripts are NOT the actual email classifiers, which must
be installed separately by your system administrator or otherwise.
Once this is done, you can select one or more wrappers for the simulation
by typing, for example:
% mailtoe testsuite select dbaclA ifile
If some of the selected classifiers cannot be found on the system, they
are not selected. Note also that some wrappers
can have hard-coded category names, e.g. if the classifier only supports binary
classification. Heed the warning messages.
It remains only to run the simulation. Beware, this can take a long time
(several hours depending on the classifier).
% mailtoe testsuite run
% mailtoe testsuite summarize
Once you are all done, you can delete the working files, log
files etc. by typing
% mailtoe clean
SCRIPT INTERFACE
mailtoe testsuite takes care of learning and classifying your prepared email corpora for each
selected classifier. Since classifiers have widely varying interfaces, this
is only possible by wrapping those interfaces individually into a standard
form which can be used by
"mailtoe testsuite" .
Each wrapper script is a command line tool which accepts a single command
followed by zero or more optional arguments, in the standard form:
wrapper command [argument]...
Each wrapper script also makes use of STDIN and STDOUT in a well defined
way. If no behaviour is described, then no output or input should be used.
The possible commands are described below:
filter
In this case, a single email is expected on STDIN, and a list of
category filenames is expected in $2, $3, etc. The script writes the
category name corresponding to the input email on STDOUT. No trailing newline
is required or expected.
learn
In this case, a standard mbox stream is expected on STDIN, while a
suitable category file name is expected in $2. No output is written to
STDOUT.
clean
In this case, a directory is expected in $2, which is examined for old
database information. If any old databases are found, they are purged or
reset. No output is written to STDOUT.
describe
IN this case, a single line of text is written to STDOUT, describing the filter's
functionality. The line should be kept short to prevent line wrapping on a terminal.
bootstrap
In this case, a directory is expected in $2. The wrapper script first checks for
the existence of its associated classifier, and other prerequisites. If the
check is successful, then the wrapper is cloned into the supplied directory.
A courtesy notification should be given on STDOUT to express success or failure.
It is also permissible to give longer descriptions caveats.
toe
In this case, a list of categories is expected in $3, $4, etc. Every possible
category must be listed. Preceding this list, the true category is given in $2.
Right after loading,
mailtoe reads the hidden file .mailtoerc in the $HOME directory, if it exists, so
this would be a good place to define custom values for environment variables.
MAILTOE_FILTER
This variable contains a shell command to be executed repeatedly
during the running stage.
The command should accept an email message on STDIN and output a
resulting category name. On the command line, it should also accept
first the true category name, then a list of all possible category
file names. If the output category does not match the true category,
then the relevant categories are assumed to have been silently
updated/relearned.
If MAILTOE_FILTER is undefined,
mailtoe uses a default value.
TEMPDIR
This directory is exported for the benefit of wrapper scripts. Scripts which
need to create temporary files should place them a the location given in TEMPDIR.
NOTES
The subdirectory mailtoe.d can grow quite large. It
contains a full copy of the training corpora, as well as learning files for
size times all the added categories, and various log files.
While TOE simulations for
dbacl(1) can be used to compare with other classifiers, TOE should not be used
for real world classifications. This is because, unlike many other filters,
dbacl(1) learns evidence weights in a nonlinear way, and does not preserve relative
weights between tokens, even if those tokens aren't seen in new emails.
WARNING
Because the ordering of emails within the added mailboxes matters, the estimated
error rates are not well defined or even meaningful in an objective sense.
However, if the sample emails represent an actual snapshot of a user's incoming email,
then the error rates are somewhat meaningful. The simulations can then be interpreted
as alternate realities where a given classifier would have intercepted the incoming mail.
SOURCE
The source code for the latest version of this program is available at the
following locations:
