xferstats is a utility that parses wu-ftpd, ncftpd, and (special) apache
xferlogs (although it's not difficult to add support for other log types) and gives
various information and totals about the data transferred.
OPTIONS
+
The single plus tells
xferstats to get the
xferlog from stdin. For example:
cat xferlog | xferstats +
-
The single hyphen tells
xferstats to get the
xferlog from a file. While this is the default behaviour, the configuration file may
change the default, and this option allows you to override it. For example:
xferstats - /var/log/xferlog
-c configuration file
Specify an alternate path and filename for the configuration file. The packaged
default is
/etc/xferstats.cfg but your administrator may have changed this. A new default can be set by
changing the appropriate variable in xferstats.h and recompiling.
-T log type
Specify the type of log file to be parsed. Currently, only three types are
defined,
wu-ftpd for wu-ftpd-style logs,
ncftpd for ncftpd-style logs, and
apache for special Apache logs. Note: you can also use "wu-ftp", "wuftp", "wuftpd", or
"ncftp".
In order for xferstats to parse Apache logfiles, you must use a special LogFile
line: LogFormat "%{%a %b %d %X %Y}t %T %h %b %f %u".
xferstats may or may not support regular Apache logfiles in the future. If this is
something you want, please let me know.
-v, --version
Display what version of
xferstats you're running.
--help
Display a quick summary of the command line options.
--
Use a -- (double hyphen) to indicate that you are finished giving parameters
and will follow with a filename. This is particularly useful when you want to
specify a filename that begins with a + or -
TOGGLE OPTIONS The options in this section can be either forced on with +[option] or forced
off with -[option]. Options in this section, like all other options, will
always override any settings in the xferstats configuration file. Any
arguments to toggle options should only be used when enabling an option.
Supplying an argument when disabling an option will generate an error.
C
When enabled, xferstats will try to weed out corrupt log lines by strictly
checking to make sure that the data is valid. If you --enable-debug_flow at
compile time, xferstats will notify you when corrupt lines are detected.
H
When enabled, xferstats will generate HTML tables and graphs instead
of text tables. In order to actually generate tables instead of broken
graphic icons, you must put the graph JPGs in the directory pointed to by the
GRAPH_PATH configuration command. For more information, see the CONFIGURATION
FILE section.
n
When enabled, xferstats will refrain from adding <HTML>, <BODY>, and <TITLE>
tags, so that the output can be inserted into pages containing custom headers.
s
When enabled, xferstats will output HTML as a single page (filename
"xferstats.html") instead of a page for each section. This option also
disables <HTML> and <BODY> tags, although it does produce a <TITLE> tag unless
the NO_HTML_HEADERS option is also specified.
r
When enabled, xferstats will generate statistics about real user connections
(but not necessarily
only real user connections)
a
When enabled, xferstats will generate statistics about anonymous connections
(but not necessarily
only anonymous connections)
g
When enabled, xferstats will generate statistics about guest connections (but
not necessarily
only guest connections)
i
When enabled, xferstats will generate statistics about inbound (the product of
a PUT) files (but not necessarily
only inbound files)
u
When enabled, xferstats will generate statistics about outbound (the product
of a GET) files (but not necessarily
only outbound files)
h
When enabled, xferstats will generate a report listing traffic by hour
m
When enabled, xferstats will generate a report listing traffic by month
(January, February, etc.)
w
When enabled, xferstats will generate a report listing traffic by day of the
week (Monday, Tuesday, etc.)
M
When enabled, xferstats will generate a report listing traffic by day of the
month (1st, 2nd, 3rd, etc.)
f [number]
When enabled, xferstats will generate a report listing traffic by file. You
can supply an optional argument that will determine how many file records
xferstats displays (0 (zero) disables the limit)
d [number]
When enabled, xferstats will generate a report listing traffic by directory.
You can supply an optional argument that will determine how many directory
records xferstats displays (0 (zero) disables the limit). See the -D and +D
options, as well as the DIR_DEPTH and DIR_FILTER configuration commands for more
information.
t [number]
When enabled, xferstats will generate a report listing traffic by top-level
domain. You can supply an optional argument that will determine how many TLD
records xferstats displays (0 (zero) disables the limit)
O [number]
When enabled, xferstats will generate a report listing traffic by domain. You
can supply an optional argument that will determine how many domain records
xferstats displays (0 (zero) disables the limit)
o [number]
When enabled, xferstats will generate a report listing traffic by host. You
can supply an optional argument that will determine how many host records
xferstats displays (0 (zero) disables the limit)
OTHER OPTIONS
+A
Include all users, generate all reports
+L <number>
Limit the daily report to the last <number> days
-D <number>
Depth of path detail for directories (default 3)
+D <directory>
Generate reports on files located only in
<directory> . For example:
+D /pub will report only on paths under
/pub . Specifying multiple +D options will instruct
xferstats to include data from all specified directories.
+P <path prefix>
Strip the
<prefix> from any pathnames that begin with it. This is the same as specifying a
STRIP_PREFIX configuration file command. Keep in mind that the
DIR_FILTER (or
+D ) is checked before the
STRIP_PREFIX (or
+P ) when using these.
+B <timestamp>
Providing a +B argument instructs xferstats to only generate data on transfers
occuring after that time. The time stamp format is "Jan DD HH:MM:ss
YYYY". One may combine +B and +E to generate a report about a specific window
in time.
+E <timestamp>
Providing a +E argument instructs xferstats to only generate data on transfers
occuring before that time. The time stamp format is "Jan DD HH:MM:ss
YYYY". One may combine +B and +E to generate a report about a specific window
in time.
The default compile-time setting for the
xferlog location is
/var/log/xferlog but your administrator may have changed this. If
xferstats reports an error, try specifying the filename as the last command line
parameter.
CONFIGURATION FILE
The default configuration file path is
/etc/xferstats.cfg although your administrator may have changed this.
Any text after a '#' is ignored, as long as the '#' begins a line or is
prepended by whitespace.
Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive, although some
arguments may be.
LOGFILE
sets the default log file to process. It can be overridden by adding
"<filename>" at the end of the command line options. You may specify as many
LOGFILE lines as you like; they will be processed in the order specified
(although the order processed should not change the output). Example:
LOGFILE /var/log/xferlog
LOGFILE /var/log/xferlog.1
LOGFILE /var/log/xferlog.2
STRICT_CHECK
takes no arguments. When enabled, xferstats will try to weed out corrupt log
lines by strictly checking to make sure that the data is valid. If you
--enable-debug_flow at compile time, xferstats will notify you when corrupt
lines are detected. It can be enabled with "+C" or disabled with "-C" on the
command line.
ANON_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about
anonymous connections (but not necessarily
only anonymous connections). It can be enabled with "+a" or disabled with "-a" on
the command line.
GUEST_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about
guest connections (but not necessarily
only guest connections). This option has no effect when parsing non-wuftpd
logfiles. It can be enabled with "+g" or disabled with "-g" on
the command line.
REAL_TRAFFIC
takes no arguments. When enabled, xferstats will generate statistics about real
user connections (but not necessarily
only real user connections). It can be enabled with "+r" or disabled with "-r" on
the command line.
INBOUND
takes no arguments. When enabled, xferstats will generate statistics about
inbound (the product of a PUT) files (but not necessarily
only inbound files). It can be enabled with "+i" or disabled with "-i" on the command
line.
OUTBOUND
takes no arguments. When enabled, xferstats will generate statistics about
outbound (the product of a GET) files (but not necessarily
only outbound files). It can be enabled with "+u" or disabled with "-u" on the
command line.
HOURLY_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic per hour. It can be enabled with "+h" or disabled with "-h" on the
command line.
DOW_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by day of the week (Monday, Tuesday, etc). It can be enabled with "+w"
or disabled with "-w" on the command line.
DOM_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by day of the month (1st, 2nd, etc). It can be enabled with "+M" or
disabled with "-M" on the command line.
TLD_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by top-level domain (com, edu, net, us, uk, etc). It can be enabled
with "+t <number>" or disabled with "-t" on the command line.
DOMAIN_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by domain (foo.com, blah.edu, etc). It can be enabled with "+O
<number>" or disabled with "-O" on the command line.
HOST_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by host. It can be enabled with "+o" or disabled with "-o" on the
command line.
DIR_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by directory. The directory depth that xferstats will keep track of
can be controlled by the DIR_DEPTH configuration command or with the "-l
<depth>" command line option. It can be enabled with "+d" or disabled with "-d"
on the command line.
FILE_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic by file. It can be enabled with "+f" or disabled with "-f" on the
command line.
MONTHLY_REPORT
takes no arguments. When enabled, xferstats will generate a report listing
traffic per month (January, February, etc). It can be enabled with "+m" or
disabled with "-m" on the command line.
HTML_OUTPUT
takes no arguments. When enabled, xferstats will output all reports into
individual HTML files. Future versions will contain a configuration option to
set the directory in which to place the HTML files. For now they are placed in
the present working# directory. It can be enabled with "+H" or disabled with
"-H" on the command line.
USE_STDIN
takes no arguments. When enabled, xferstats will, by default, read the logfile
from the standard input instead of from a file on disk. It can be enabled with
"+" or disabled with "-" on the command line.
LOG_TYPE
takes one argument, the type of log that will be processed by default. The
argument can be either a number or a case insensitive string and are as follows:
wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"
ncftpd log : "2" or "ncftp" or "ncftpd"
Apache log : "3" or "apache"
Therefore:
LOG_TYPE 1
LOG_TYPE wu-ftp
LOG_TYPE wu-ftpd
LOG_TYPE wuftp
LOG_TYPE wuftpd
are all equivalent.
Future versions will contain support for additional logfile types. It can be
set on the command line with "-T <number>"
DIR_DEPTH
takes one argument, the number of subdirectories deep that xferstats should go
when generating statistics by directory. This configuration command has no
effect when DIR_REPORT has not been enabled. DIR_DEPTH has no upper bound. A
DIR_DEPTH of '0' (zero) tells xferstats to process all subdirectories. The
default value is 3. It can be set on the command line with "-D <number>".
Example:
DIR_DEPTH 3
DIR_FILTER
takes one argument, the directory that xferstats should report on. xferstats
will discard any data that does not have a path that begins with the value set
by DIR_FILTER. This value
is case-sensitive, as it is part of a path name. Specifying multiple statements
will instruct
xferstats to include data from all specified directories. It can be set on the command
line with "+D <path>". Example:
DIR_FILTER /pub/foobar
NUMBER_FILE_STATS
takes one argument, the number of files to report on in the file statistics
section. It must be a number greater than or equal to zero, with zero denoting
no limit. It can be set on the command line with "+f <number>". The default is
50.
NUMBER_FILE_STATS 50
NUMBER_DIR_STATS
takes one argument, the number of directories to report on in the directory
statistics section. It must be a number greater than or equal to zero, with
zero denoting no limit. It can be set on the command line with "+d <number>".
The default is 50.
NUMBER_DAILY_STATS
takes one argument, the number of days to report on in the daily statistics
section. It must be a number greater than or equal to zero, with zero denoting
no limit. It can be set on the command line with "+L <number>". The default
is 50.
Note: the daily statistics section will always be displayed.
NUMBER_TLD_STATS
takes one argument, the number of top-level domains to report on in the TLD
statistics section. It must be a number greater than or equal to zero, with
zero denoting no limit. It can be set on the command line with "+t <number>".
The default is 50.
NUMBER_DOMAIN_STATS
takes one argument, the number of top-level domains to report on in the domain
statistics section. It must be a number greater than or equal to zero, with
zero denoting no limit. It can be set on the command line with "+O <number>".
The default is 50.
NUMBER_HOST_STATS
takes one argument, the number of hosts to report on in the host statistics
section. It must be a number greater than or equal to zero, with zero denoting
no limit. It can be set on the command line with "+o <number>". The default is
50.
MAX_REPORT_SIZE
takes one argument, the maximum size of any one output table. This is
particularly useful to keep xferstats from generating gigantic tables that make
graphical browsers choke. It will split tables to be no larger than# this
value. It must be greater than or equal to ten, or zero, zero denoting no
limit. It can be set on the command line with "+R <number>". The default is
30.
GRAPH_PATH
takes one argument, the location of the JPGs used to draw graphs in the HTML
output. It can be an absolute path (ie, http://foo.bar.com/graphs/) or a
relative path (ie, ../graphs/). You must include a trailing '/' or it
"will not work". This option cannot be set on the command line. The default is nothing (which
will make the browser look for the graphs in the directory with the HTML).
Example:
GRAPH_PATH ../graphs/
NO_HTML_HEADERS
takes no arguments. When enabled, xferstats will refrain from adding <HTML>,
<BODY>, and <TITLE> tags, so that the output can be inserted into pages
containing custom headers. It can be enabled with "+n" or disabled with "-n" on
the command line.
REFRESH
takes one argument, the number of seconds after being displayed that
the browser should refresh it. Unspecified or zero indicates no refresh.
SINGLE_PAGE
takes no arguments. When enabled, xferstats will output HTML as a single page
(entitled "xferstats.html") instead of a page for each section. This option
also disables <HTML> and <BODY> tags, although it does produce a <TITLE> tag
unless the NO_HTML_HEADERS option is also specified. It can be enabled with
"+s" or disabled with "-s" on the command line.
STRIP_PREFIX
takes one argument. When specified, xferstats will strip the given string from
the beginning of any paths. Keep in mind when using this option with
DIR_FILTER that the
DIR_FILTER is applied -before- stripping, so that you must
take this into account when specifying the
DIR_FILTER string. This option cannot be set on the command line. Example:
STRIP_PREFIX /home/ftp
The next 11 entries are all basically the same--they define how the reports are
to be sorted.
0 -- Sort by the number of bytes downloaded
1 -- Sort by the number of files downloaded
2 -- Sort by the "name". This varies by report; for the file, directory, TLD, etc
reports, it sorts them alphabetically; for the file size report it lists the
file sizes in alphabetical order; for the daily report, it lists the days in
order. You get the idea.
3 -- Sort by average throughput, in K/s
FILE_SORT_PREF , DIR_SORT_PREF , DOMAIN_SORT_PREF , TLD_SORT_PREF , HOST_SORT_PREF , DOM_SORT_PREF , DOW_SORT_PREF , HOURLY_SORT_PREF , DAILY_SORT_PREF , SIZE_SORT_PREF , MONTHLY_SORT_PREF
CHUNK_INPUT
takes one argument. When specified, xferstats will parse the logfiles in
chunks of the given size (in number of lines), process them, and continue.
This will keep it from trying to parse the entire log at once, using up a lot
of memory for large logs. Specifying too small a value will cause xferstats
to waste a lot of time switching tasks, while too large a value may slow it
down as your system swaps. Toy with this value a bit and see what works best
for you. If left unspecified, xferstats will continue to parse the logs in
one pass.
FILES
xferlog (generated by wu-ftpd or ncftpd)
/etc/xferstats.cfg contains all of the default settings and many settings which cannot be set from
the command line.
If you installed the RPM version of xferstats, you'll find the JPGs for the
graphs in /usr/share/xferstats/graphs.
BUGS
The JPG graph pieces included are not ideal; sometimes artifacts show up. I
will eventually get around to making new GIFs or something.
Please send bug reports to phil@off.net
If the xferlog that you're using is smaller than 1 megabyte, please send a
gzipped copy, along with the exact parameters you're using.
ACKNOWLEDGEMENTS
I greatly appreciate the suggestions and assistance of Nat Friedman
<ndf@mit.edu>.
Thanks to Mike Shaver <shaver@mozilla.org> and Jamie Zawinski
<jwz@netscape.com> for coughing up gigantic mozilla.org logs to aid my testing.
Thanks to Phil Copeland <bryce@zen.ics.uwe.ac.uk> for getting on my back
constantly to add new stuff, fix nagging bugs, and speed it up. Oh, and for
using it, at least someone does :).
Extreme gratitude to Mark Hagler <hagler@hagler.org>, someone whom I've left
off of this list for too long. Mark was kind enough to Amazon me a copy of
O'Reillys _Pthreads Programming_ which was the catalyst for the threaded version
and really helped get me interested in the project again. While the threaded
version doesn't exist anymore, the book is still terribly useful for other
projects.
xferstats is available from the following locations. The old addresses are valid, but
are deprecated! Please use the new addresses whenever possible.
http://xferstats.off.net/
ftp://xferstats.off.net/pub/xferstats/
AUTHOR
xferstats is Copyright (c) 1997-2000 Phil Schwan <phil@off.net> and is released under
the terms and provisions of the GNU Public License.