Translation(s): 中文普通话 - English - Français - Italiano


Rsyslog

Rsyslog is an open source program for transferring log messages over an IP network for UNIX and Unix systems. It implements the core syslog protocol, and extends it with content-based filtering, advanced filtering features, flexible configuration options, and adds features such as the use of TCP, SSL, and RELP for transport. Rsyslog is a direct substitute for syslogd. Rsyslog offers high performance, security features and modular design. Rsyslog can allow us to store log messages in a MySQL, MariaDB, MongoDB or PostgreSQL database that can be configured with dbconfig-common for easy configuration via debconf. The log data can be exploited by a complementary program, such as the LogAnalyzer web interface. The rotation of the log messages is automated.

History of the transition from Sysklogd to Rsyslog

The rsyslogd service was integrated with the Rsyslog package from Debian Lenny (https://wiki.debian.org/DebianLenny) to replace the old syslog: sysklogd. The existing logging rules in syslog.conf can be simply copied to /etc/rsyslog.conf or to the /etc/rsyslog.conf.d folder. Sysklogd which was installed by default is not bad but the package has hardly been maintained in recent years. See the Debian Lenny shared information memo release notes : https://www.debian.org/releases/lenny/i386/release-notes/ch-whats-new#system-changes

Some arguments in favor of Rsyslog in the following discussions :

Installation

Install Rsyslog allows you to store syslog logs in a database. If no database is configured, the LogAnalyzer web interface will be able to use the log data provided by syslog from the disk. If a database is configured, the LogAnalyzer web interface will be able to use the log data provided by the database.

Prerequisites: apache.

Install version 8.39.0 of Rsyslog from official Debian repositories

Install the PGP key in your apt system :

Adiscon repository for v8-stable on Debian (7) Wheezy. Edit your /etc/apt/sources.list and add these lines at the end :

Install the Rsyslog documentation

Documentation installed, by default on Debian, with rsyslogd service :

Install the full Rsyslog documentation in HTML format :

Install rsyslog-mysql to configure the MySQL database that will store the syslog logs

MySQL, MariaDB, MongoDB or PostgreSQL databases are properly supported.

The configuration of rsyslog-mysql is done automatically with dbconfig-common. A MySQL database named Syslog containing two ?SystemEvents and ?SystemEventsProperties tables is created. A rsyslog @ localhost user is added, he has full control over the syslog database. The configuration of the MySQL database connection is available from the /etc/rsyslog.d/mysql.conf file.

Configure the mysql.conf connection file

Edit mysql.conf, the MySQL configuration file. Connect to the syslog database to store syslog logs.

Note: It seems important to use the tab spaces between the entries in the configuration.

# Load the module and save all syslog messages to the database:

# This command filters the alert level of the messages to be stored in the database:

Create syslog database to store syslog data with Rsyslog and MySQL

If the rsyslog-mysql package was not used to create the database, the database can be created manually.

First method

Create the database structure with the official script to store the log messages in MySQL: The definition of the given database schema is available from the createDB.sql file from the Rsyslog 8.39.0 archive. The file path is ./rsyslog-8.39.0/plugins/ommysql/createDB.sql. Start importing the database:

Second method

Connect to MySQL:

If necessary, enter the user's password for the database. Continue with the enter key. Create the database structure manually to store the log messages in MySQL:

Create an rsyslog user to use the syslog database

Log in to MySQL on the command line. The password is empty on the local system, confirm with the enter key to connect. It will secure MySQL with a password for a system in production.

If the rsyslog user does not exist yet, create the rsyslog user to manage the database.

Optimize the syslog database

The MySQL ?SystemEvents table can fill up quickly and become very large. An archiving script placed in a Cron job will avoid unpleasant surprises.

Cron script proposal 1

Cron script proposal 2

Use InnoDB and compressed tables

Index fields that will often be used in queries

Check that the MySQL table is powered with the logs

# Create a log entry with the following command :

# Test if there are records from the terminal with the following command :

# Test if there are records from the MySQL command line interface with the following commands :

# Enter your user password for root or press enter directly if the password is empty. # Select the syslog database on which the request will be launched.

# This command also works and should be faster by selecting less information.

Configure network listening in Rsyslog configuration

# Set up the Rsyslog configuration file.

Default configuration example

#$DebugFile /home/looksaalpha/Bureau/debug.txt
#$DebugLevel 2

#  /etc/rsyslog.conf Configuration file for Rsyslog.
$EscapeControlCharactersOnReceive off

#### MODULES ####
# $ModLoad imuxsock
module(load="imuxsock")

# UDP syslog reception
# module(load="imudp")
# input(type="imudp" port="1514")
# ou
# $Modload imudp
# $UDPServerRun 1514

# TCP syslog reception
# module(load="imtcp")
# input(type="imtcp" port="1514")
# ou
# $ModLoad imtcp
# $InputTCPServerRun 1514

# Kernel connection support and activation of non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

# Default connection rules : /etc/rsyslog.d/50-default.conf
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          /var/log/syslog
syslog.*                        /var/log/rsyslog.log
cron.*                          /var/log/cron.log
kern.*                          /var/log/kern.log
#daemon.*                       /var/log/daemon.log
#lpr.*                          /var/log/lpr.log
#user.*                         /var/log/user.log
#mail.*                         /var/log/mail.log
#mail.info                      /var/log/mail.info
#mail.warn                      /var/log/mail.warn
#mail.err                       /var/log/mail.err
#news.crit                      /var/log/news/news.crit
#news.err                       /var/log/news/news.err
#news.notice                    /var/log/news/news.notice

# Owners of the cron.log file passed to syslog:logadmin

# Add a line to manage other logs by Rsyslog (Apache2, Mysql, ...):
syslog.*                        /var/log/apache2/error.log

#### GLOBAL DIRECTIVES ####
# Seems to be obsolete:
# $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicate messages:
$RepeatedMsgReduction on

$WorkDirectory /var/spool/rsyslog

# Include all config files in /etc/rsyslog.d/
# $IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/mysql.conf

Restart Rsyslog to apply the new configuration

Restart the Rsyslog daemon to apply the new configuration :

# Old orders :
sudo service rsyslog restart
sudo /etc/init.d/rsyslog restart

# New order :
sudo systemctl restart rsyslog

Centralize the logs on a log server with Rsyslog

Completes the previously proposed configuration to use Rsyslog as a centralized log server.

Create a template file where we will create a new custom log format from /etc/rsyslog.d/tmpl.conf I did not test this possibility.

Allow the default Rsyslog UDP 514 port on your firewall. Allow the modified TCP 1514 Rsyslog port on your firewall. The following commands will open this port via UFW:

sudo ufw allow 514/udp
sudo ufw allow 1514/tcp

Restart the UFW service to take into account the changes:

sudo ufw reload

Reload the Rsyslog service using the following command:

systemctl restart rsyslog

In case the log server is behind a router: Consider enabling NAT rules for the ports selected in the configuration. Redirect UDP traffic 514-> 514 to the log server. Redirect TCP traffic 1514-> 1514 to the log server.

Add new clients Rsyslog

Configure the client machines so that they can send their logs to the machine that will act as the log server. Works with syslog or syslog-ng, with different settings for syslog-ng. This feature has not been tested when writing this tutorial.

Step 1: Install the rsyslog package on each client :

apt-get install rsyslog

Step 2: Create a working directory :

mkdir /var/spool/rsyslog

Step 3: Open the Rsyslog configuration file :

nano /etc/rsyslog.conf

Modify the configuration that allows the logs to be sent to the log server : Default location for work files (spool).

$WorkDirectory /var/spool/rsyslog

Start Transfer Rule 1:
$ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone.
$ActionQueueFileName srvrfwd1 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque.
$ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go.
$ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté.
$ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte.
# jouter les lignes suivantes dans la section RULES :
# Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante :
*.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.

Start Transfer Rule 2:
$ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone.
$ActionQueueFileName srvrfwd2 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque.
$ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go.
$ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté.
$ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte.
# jouter les lignes suivantes dans la section RULES :
# Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante :
*.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.

Step 4: Restart the RSyslog Service :

sudo /etc/init.d/rsyslog restart

Record the standard error message (screen) and the system log using the following command :

logger -s " Ceci est un client Rsyslog "

Go to the Rsyslog server under the / var / log / client_logs directory. A new folder named with the host name of your Rsyslog client should be available.

/var/log/client_logs/Client01/

Rsyslog and SSL

SSL encryption for the exchange between Syslog and Rsyslog.

Debug Rsyslogd and get its logs

DebugFile and DebugLevel Method

Open the Rsyslog configuration file :

sudo nano /etc/rsyslog.conf

# Add the following two lines :
$DebugFile /home/USER/Bureau/debug.txt
$DebugLevel 2
# <0|1|2> - Set the debug level:
# 0 is the debug mode disabled.
# 1 on demand debug mode enabled but debug mode disabled.
# 2 is the full debug mode.

# Restart Rsyslog :

sudo service rsyslog restart

Trace method

The trace method can work, but, the first method is more suitable.

In a first console, run the following command:

sudo strace -p $(pgrep rsyslogd) -o fichier.trace

Open a second console, and restart for example Rsyslog :

sudo /etc/init.d/rsyslog restart

See also

Resources used to write this summary

Additional resources