mod_redis
The mod_redis
module enables ProFTPD support for caching data in
Redis servers, using the
hiredis client library.
<VirtualHost>
, <Global>
The RedisEngine
directive enables or disables the
mod_redis
module, and thus the configuration of Redis support for
the proftpd
daemon.
<VirtualHost>
, <Global>
The RedisLog
directive is used to specify a log file for
mod_redis
's reporting on a per-server basis. The
file parameter given must be the full path to the file to use for
logging.
Note that this path must not be to a world-writable directory and,
unless AllowLogSymlinks
is explicitly set to on
(generally a bad idea), the path must not be a symbolic link.
<VirtualHost>
, <Global>
The RedisLogOnCommand
directive configures the use of Redis for
logging. Whenever one of the comma-separated list of commands
occurs, mod_redis
will compose a JSON object, using the
LogFormat
named by
format-name as a template for the fields to include in the
JSON object. The JSON object of that event will then be appended to a list
stored in Redis, using format-name as the key name. Multiple
RedisLogOnCommand
directives can be used, for different log formats
for different events.
More on the use of Redis logging, including a table showing how
LogFormat
variables are mapped to JSON object keys can be found
here.
Example:
LogFormat file-transfers "%h %l %u %t \"%r\" %s %b" RedisLogOnCommand APPE,RETR,STOR,STOU file-transfers LogFormat sessions "%{iso8601}" RedisLogOnCommand CONNECT,DISCONNECT sessions
In addition to specific FTP commands, the commands list can specify
"ALL", for logging on all commands. Or it can include the
"CONNECT" and "DISCONNECT" commands, which can be useful for logging the
start and end times of a session. Note that
RedisLogOnCommand
does not currently support the logging
classes that the ExtendedLog
directive supports.
<VirtualHost>
, <Global>
The RedisServer
directive is used to configure the IP address/port
of the Redis server that the mod_redis
module is to use. For
example:
RedisServer 1.2.3.4:6379or, for an IPv6 address, make sure the IPv6 address is enclosed in square brackets:
RedisServer [::ffff:1.2.3.4]:6379
Alternatively, you can configure a Unix domain socket path using e.g.:
RedisServer /var/run/redis.sock
An optional password parameter can be provided, for Redis servers which are password protected.
<VirtualHost>
, <Global>
The RedisTimeouts
directive configures timeouts to be used
when communicating with the Redis server. The connect-millis
parameter specifies a timeout, in milliseconds, to use when first
connecting to the Redis server. The io-millis parameter specifies
a timeout, in milliseconds, to use both when sending commands to Redis, and
when reading responses.
The default is 500 milliseconds for both timeouts:
RedisTimeouts 500 500
mod_redis
module is distributed with ProFTPD. To enable
support and use of the Redis protocol in your proftpd
daemon,
use the --enable-redis
configure option:
$ ./configure --enable-redis ... $ make $ make installThis option causes the
mod_redis
module to be compiled into
proftpd
.
You may also need to tell configure
how to find the
hiredis
header and library files:
$ ./configure --enable-redis \ --with-includes=/path/to/hiredis/include \ --with-libraries=/path/to/hiredis/lib
Configuring Redis for use by other modules, e.g. mod_ban
or mod_tls_redis
:
<IfModule mod_redis.c> RedisEngine on RedisLog /var/log/ftpd/redis.log RedisServer 127.0.0.1:6379 </IfModule>
This example shows the use of Redis logging for all commands:
<IfModule mod_redis.c> RedisEngine on RedisLog /var/log/ftpd/redis.log RedisServer 127.0.0.1:6379 LogFormat redis "%h %l %u %t \"%r\" %s %b" RedisLogOnCommand ALL redis </IfModule>
Redis Logging
When using Redis logging, the following table shows how mod_redis
converts a LogFormat
variable into the key names in the JSON
logging objects:
LogFormat Variable |
Key |
%A |
anon_password |
%a |
remote_ip |
%b |
bytes_sent |
%c |
connection_class |
%D |
dir_path |
%d |
dir_name |
%E |
session_end_reason |
%{name}e |
ENV:name |
%F |
transfer_path |
%f |
file |
%{file-modified} |
file_modified |
%g |
group |
%{gid} |
gid |
%H |
server_ip |
%h |
remote_dns |
%I |
session_bytes_rcvd |
%{iso8601} |
timestamp |
%J |
command_params |
%L |
local_ip |
%l |
identd_user |
%m |
command |
%{microsecs} |
microsecs |
%{millisecs} |
millisecs |
%{note:name} |
NOTE:name |
%O |
session_bytes_sent |
%P |
pid |
%p |
local_port |
%{protocol} |
protocol |
%r |
raw_command |
%S |
response_msg |
%s |
response_code |
%T |
transfer_secs |
%t |
local_time |
%{transfer-failure} |
transfer_failure |
%{transfer-status} |
transfer_status |
%U |
original_user |
%u |
user |
%{uid} |
uid |
%V |
server_dns |
%v |
server_name |
%{version} |
server_version |
%w |
rename_from |
In addition to the standard LogFormat
variables, the
mod_redis
module also adds a "connecting" key for events
generated when a client first connects, and a "disconnecting" key for events
generated when a client disconnects. These keys can be used for determining
the start/finish events for a given session.
Here is an example of the JSON-formatted records generated, using the above example configuration:
{"connecting":true,"timestamp":"2013-08-21 23:08:22,171"} {"command":"USER","timestamp":"2013-08-21 23:08:22,278"} {"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"} {"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"} {"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"} {"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"} {"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}Notice that for a given event, not all of the
LogFormat
variables are filled in. If mod_redis
determines that a given
LogFormat
variable has no value for the logged event, it will
simply omit that variable from the JSON object.
Another thing to notice is that the generated JSON object ignores the textual
delimiters configured by the LogFormat
directive; all that
matters are the LogFormat
variables which appear in the directive.
Question: How can I convert this SQL logging into the equivalent Redis logging?
SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')" SQLLog STOR uploadAnswer: Since the JSON object key names are hardcoded in
mod_redis
, converting the above SQLNamedQuery
into a suitable/matching LogFormat
is the necessary step. Thus
for example it might become:
LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}" RedisLogOnCommand STOR uploadNote that
LogFormat
does not provide a NOW()
function, unlike many SQL databases, thus the %t
variable is
needed to provide/fill in that timestamp.