Translation(s): 中文普通话 - English - Français - Italiano

Rsyslog

Rsyslog is an open source program for transferring log messages over an IP network for UNIX and Unix systems. It implements the core syslog protocol, and extends it with content-based filtering, advanced filtering features, flexible configuration options, and adds features such as the use of TCP, SSL, and RELP for transport. Rsyslog is a direct substitute for syslogd. Rsyslog offers high performance, security features and modular design. Rsyslog can allow us to store log messages in a MySQL, MariaDB, MongoDB or PostgreSQL database that can be configured with dbconfig-common for easy configuration via debconf. The log data can be exploited by a complementary program, such as the LogAnalyzer web interface. The rotation of the log messages is automated.

Rsyslog official website : https://www.rsyslog.com
Documentation Version 8 : https://www.rsyslog.com/doc/v8-stable/
General documentation : https://www.rsyslog.com/doc/master/index.html
Configuration documentation : https://www.rsyslog.com/doc/v8-stable/configuration/index.html
Configuration examples : https://www.rsyslog.com/doc/v8-stable/configuration/examples.html

History of the transition from Sysklogd to Rsyslog

The rsyslogd service was integrated with the Rsyslog package from Debian Lenny (https://wiki.debian.org/DebianLenny) to replace the old syslog: sysklogd. The existing logging rules in syslog.conf can be simply copied to /etc/rsyslog.conf or to the /etc/rsyslog.conf.d folder. Sysklogd which was installed by default is not bad but the package has hardly been maintained in recent years. See the Debian Lenny shared information memo release notes : https://www.debian.org/releases/lenny/i386/release-notes/ch-whats-new#system-changes

Some arguments in favor of Rsyslog in the following discussions :

Installation

Install Rsyslog allows you to store syslog logs in a database. If no database is configured, the LogAnalyzer web interface will be able to use the log data provided by syslog from the disk. If a database is configured, the LogAnalyzer web interface will be able to use the log data provided by the database.

Prerequisites: apache.

Install version 8.39.0 of Rsyslog from official Debian repositories

Install the PGP key in your apt system :

 sudo apt-key adv --recv-keys --keyserver keys.gnupg.net AEF0CF8E

Adiscon repository for v8-stable on Debian (7) Wheezy. Edit your /etc/apt/sources.list and add these lines at the end :

 deb http://debian.adiscon.com/v8-stable wheezy/
 deb-src http://debian.adiscon.com/v8-stable wheezy/

 sudo apt update
 sudo apt-get install rsyslog

Install the Rsyslog documentation

Documentation installed, by default on Debian, with rsyslogd service :

```
man rsyslogd
```

Install the full Rsyslog documentation in HTML format :

 sudo apt install rsyslog-doc
 cat /usr/share/doc-base/rsyslog-doc

Document: rsyslog-doc
Title: Rsyslog Documentation
Author: Rainer Gerhards
Abstract: This documentation covers the configuration of rsyslog.
Section: System/Administration

Format: HTML
Index: /usr/share/doc/rsyslog-doc/html/manual.html
Files: /usr/share/doc/rsyslog-doc/html/*.html</code>

Install rsyslog-mysql to configure the MySQL database that will store the syslog logs

MySQL, MariaDB, MongoDB or PostgreSQL databases are properly supported.

```
sudo apt install rsyslog-mysql
```

The configuration of rsyslog-mysql is done automatically with dbconfig-common. A MySQL database named Syslog containing two ?SystemEvents and ?SystemEventsProperties tables is created. A rsyslog @ localhost user is added, he has full control over the syslog database. The configuration of the MySQL database connection is available from the /etc/rsyslog.d/mysql.conf file.

Configure the mysql.conf connection file

Edit mysql.conf, the MySQL configuration file. Connect to the syslog database to store syslog logs.

```
sudo nano /etc/rsyslog.d/mysql.conf
```

Note: It seems important to use the tab spaces between the entries in the configuration.

# Load the module and save all syslog messages to the database:

module (load="ommysql")
*.* action(type="ommysql" server="localhost" db="Syslog" uid="rsyslog" pwd="Password_BDD")

# This command filters the alert level of the messages to be stored in the database:

*.emerg >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.alert >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.crit >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.err >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.warning >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.notice >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.info >localhost,Syslog,Utilisateur_BDD,Password_BDD
*.debug >localhost,Syslog,Utilisateur_BDD,Password_BDD

#? # Syntax to check for a Debian Stretch 9 system:
#?# $ModLoad ommysql
#?# *.* :ommysql:localhost,Syslog,Utilisateur_BDD,Password_BDD

Create syslog database to store syslog data with Rsyslog and MySQL

If the rsyslog-mysql package was not used to create the database, the database can be created manually.

First method

Create the database structure with the official script to store the log messages in MySQL: The definition of the given database schema is available from the createDB.sql file from the Rsyslog 8.39.0 archive. The file path is ./rsyslog-8.39.0/plugins/ommysql/createDB.sql. Start importing the database:

mysql -u rsyslog -D Syslog -p < ./rsyslog-8.39.0/plugins/ommysql/createDB.sql

Second method

Connect to MySQL:

```
sudo bash
mysql -u root -p
```

If necessary, enter the user's password for the database. Continue with the enter key. Create the database structure manually to store the log messages in MySQL:

CREATE DATABASE Syslog;

USE Syslog;

CREATE TABLE SystemEvents

(
ID int unsigned not null auto_increment primary key,
CustomerID bigint,
ReceivedAt datetime NULL,
DeviceReportedTime datetime NULL,
Facility smallint NULL,
Priority smallint NULL,
FromHost varchar(60) NULL,
Message text,
NTSeverity int NULL,
Importance int NULL,
EventSource varchar(60),
EventUser varchar(60) NULL,
EventCategory int NULL,
EventID int NULL,
EventBinaryData text NULL,
MaxAvailable int NULL,
CurrUsage int NULL,
MinUsage int NULL,
MaxUsage int NULL,
InfoUnitID int NULL ,
SysLogTag varchar(60),
EventLogType varchar(60),
GenericFileName VarChar(60),
SystemID int NULL
);

CREATE TABLE SystemEventsProperties

(
ID int unsigned not null auto_increment primary key,
SystemEventID int NULL ,
ParamName varchar(255) NULL ,
ParamValue text NULL
);

Create an rsyslog user to use the syslog database

Log in to MySQL on the command line. The password is empty on the local system, confirm with the enter key to connect. It will secure MySQL with a password for a system in production.

```
mysql -u root -p
```

If the rsyslog user does not exist yet, create the rsyslog user to manage the database.

CREATE USER 'rsyslog'@'localhost' IDENTIFIED BY 'Analyzer';

Give rights to the rsyslog user.

GRANT ALL PRIVILEGES ON Syslog.* TO rsyslog@localhost IDENTIFIED BY 'Analyzer';
FLUSH PRIVILEGES;
exit;

Optimize the syslog database

The MySQL ?SystemEvents table can fill up quickly and become very large. An archiving script placed in a Cron job will avoid unpleasant surprises.

Cron script proposal 1

# Archiving seems to work properly.
# Check the deletion of data in the database.

#!/bin/bash
# Monthly archiving of the MySQL Syslog database logs
RETENTION=180 # Log retention time (in days)
DESTDIR="/var/backups/syslog" # Archive storage directory
ARCHIVE="syslog-$(date '+%Y-%m-%d-%Hh%M').gz"
# MySQL parameters
MYSQL_HOST="localhost"
MYSQL_DB="Syslog"
MYSQL_USER="rsyslog"
MYSQL_PASSWD="motdepasse"
# Archiving
sql="SELECT * FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION"
mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB | gzip > $DESTDIR/$ARCHIVE
# Deleting MySQL records
sql="DELETE FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION"
mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB

Cron script proposal 2

# This script deletes data of more than two days that are stored in the database.
# Text information echo lines were not displayed in the terminal during my tests.

#!/bin/bash
# Purge the logs of the MySQL Syslog database
RETENTION=2 # Log retention time (in days)
# Variable initialization
MYSQL_HOST="localhost"
MYSQL_DB="bddloganalyzer"
MYSQL_USER="Logs"
MYSQL_PASSWD="Analyzer"
TMP_LOG="/tmp/cron.log"

# Count records to delete
sql="SELECT COUNT(*) FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION"
compteur='mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB'
if [ "$compteur" = "0" ]; then
echo "(I) There are no records older than $RETENTION day" >> $TMP_LOG
else
echo "(I) There are $compteur records to delete" >> $TMP_LOG
# Deleting MySQL records
sql="DELETE FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION"
mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB
cr=$?
if [ $cr -gt 0 ]; then
  echo "(E) Error deleting records Return code $cr" >> $TMP_LOG
else
  echo "(I) The operation went well" >> $TMP_LOG
fi
fi

Use InnoDB and compressed tables

# This script has not been tested.

# To use InnoDB and compressed tables :
 {{{
ALTER TABLE SystemEvents
ENGINE=innodb DEFAULT CHARSET=latin1 row_format=COMPRESSED KEY_BLOCK_SIZE=4;

Index fields that will often be used in queries

# This script has not been tested.

# After a few months of operation, queries launched from LogAnalyser can become slower, making the tool hard to use.
# With the MySQL plugin version of rsyslog (v 4.6.4) the indexes are not created in the SystemEvents table at installation.
# When the records are counted in millions in the table it is no longer suitable.
# The solution is to index the fields that will often be used in queries, for example, Facility, Priority, FromHost, SysLogTag and DeviceReportedTime.
ALTER TABLE `SystemEvents` ADD INDEX(`Facility`);
ALTER TABLE `SystemEvents` ADD INDEX(`Priority`);
ALTER TABLE `SystemEvents` ADD INDEX(`FromHost`);
ALTER TABLE `SystemEvents` ADD INDEX(`SysLogTag`);
ALTER TABLE `SystemEvents` ADD INDEX(`DeviceReportedTime`);

Check that the MySQL table is powered with the logs

# Create a log entry with the following command :

```
logger Works!
```

# Test if there are records from the terminal with the following command :

mysql -u rsyslog -p -Bsr -e "SELECT Message FROM SystemEvents LIMIT 0,20" Syslog

# Test if there are records from the MySQL command line interface with the following commands :

```
mysql -u root -p
```

# Enter your user password for root or press enter directly if the password is empty. # Select the syslog database on which the request will be launched.

mysql> use Syslog;
mysql> select * from SystemEvents limit 1;

# This command also works and should be faster by selecting less information.

mysql> select ReceivedAt,Message from SystemEvents;

Configure network listening in Rsyslog configuration

# Set up the Rsyslog configuration file.

sudo nano /etc/rsyslog.conf

# Loading this module seems essential to receive the syslog data locally from the Disk or the MySQL database.
module(load="imuxsock")
# Charger le module pour les anciennes systèmes :
# $ModLoad imuxsock

# The syslog information can be received via UDP or TCP.
# The configuration instructions are slightly different between UDP and TCP.
# In most cases, UDP Syslog should be fully sufficient and efficient.
# If the system buffer for UDP is full during a large message flow, the messages will be lost.
# This is not the case with TCP syslog because the sender and the receiver communicate about the arrival of network packets.
# Some devices, such as routers, can not send TCP Syslog by design. In this case, you still have to use UDP.
# TCP syslog is suitable for environments where log messages should not be lost or which should ensure PCI compliance, for example, banks.

# Prefer the use of the RELP protocol :
# On the (non) reliability of plain TCP syslog, using the RELP protocol would seem to be even more relevant.
# On the (a) reliability of plain TCP syslog : https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html

# Enable remote log reception :
# The UDP syslog is received on port 514 by default.
# The TCP syslog requires a different port because the RPC service often uses this port as well.

# Syntax for Debian. ( To check ! )
# Provides UDP syslog reception :
## $ModLoad imudp
## $UDPServerRun 514
# Provides TCP syslog reception :
## $ModLoad imtcp
## $InputTCPServerRun 1514

# Syntax for Ubuntu 16.04 LTS / Ubuntu 18.04 LTS and Mint Tara 19.
# Provides UDP syslog reception :
module(load="imudp")
input(type="imudp" port="514")
# Provides TCP syslog reception :
module(load="imtcp")
input(type="imtcp" port="1514")

# List of sub networks authorized to connect :
# $AllowedSender UDP, 127.0.0.1, 192.168.1.0/24
# $AllowedSender TCP, 127.0.0.1, 192.168.1.0/24

# Le serveur est maintenant configuré pour enregistrer les logs dans la base de données.
# Il écoute sur le port 514 et 1514 les messages entrant.

Default configuration example

#$DebugFile /home/looksaalpha/Bureau/debug.txt
#$DebugLevel 2

#  /etc/rsyslog.conf Configuration file for Rsyslog.
$EscapeControlCharactersOnReceive off

#### MODULES ####
# $ModLoad imuxsock
module(load="imuxsock")

# UDP syslog reception
# module(load="imudp")
# input(type="imudp" port="1514")
# ou
# $Modload imudp
# $UDPServerRun 1514

# TCP syslog reception
# module(load="imtcp")
# input(type="imtcp" port="1514")
# ou
# $ModLoad imtcp
# $InputTCPServerRun 1514

# Kernel connection support and activation of non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

# Default connection rules : /etc/rsyslog.d/50-default.conf
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          /var/log/syslog
syslog.*                        /var/log/rsyslog.log
cron.*                          /var/log/cron.log
kern.*                          /var/log/kern.log
#daemon.*                       /var/log/daemon.log
#lpr.*                          /var/log/lpr.log
#user.*                         /var/log/user.log
#mail.*                         /var/log/mail.log
#mail.info                      /var/log/mail.info
#mail.warn                      /var/log/mail.warn
#mail.err                       /var/log/mail.err
#news.crit                      /var/log/news/news.crit
#news.err                       /var/log/news/news.err
#news.notice                    /var/log/news/news.notice

# Owners of the cron.log file passed to syslog:logadmin

# Add a line to manage other logs by Rsyslog (Apache2, Mysql, ...):
syslog.*                        /var/log/apache2/error.log

#### GLOBAL DIRECTIVES ####
# Seems to be obsolete:
# $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicate messages:
$RepeatedMsgReduction on

$WorkDirectory /var/spool/rsyslog

# Include all config files in /etc/rsyslog.d/
# $IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/mysql.conf

Restart Rsyslog to apply the new configuration

Restart the Rsyslog daemon to apply the new configuration :

# Old orders :
sudo service rsyslog restart
sudo /etc/init.d/rsyslog restart

# New order :
sudo systemctl restart rsyslog

Centralize the logs on a log server with Rsyslog

Completes the previously proposed configuration to use Rsyslog as a centralized log server.

Create a template file where we will create a new custom log format from /etc/rsyslog.d/tmpl.conf I did not test this possibility.

Allow the default Rsyslog UDP 514 port on your firewall. Allow the modified TCP 1514 Rsyslog port on your firewall. The following commands will open this port via UFW:

sudo ufw allow 514/udp
sudo ufw allow 1514/tcp

Restart the UFW service to take into account the changes:

sudo ufw reload

Reload the Rsyslog service using the following command:

systemctl restart rsyslog

In case the log server is behind a router: Consider enabling NAT rules for the ports selected in the configuration. Redirect UDP traffic 514-> 514 to the log server. Redirect TCP traffic 1514-> 1514 to the log server.

Add new clients Rsyslog

Configure the client machines so that they can send their logs to the machine that will act as the log server. Works with syslog or syslog-ng, with different settings for syslog-ng. This feature has not been tested when writing this tutorial.

Step 1: Install the rsyslog package on each client :

apt-get install rsyslog

Step 2: Create a working directory :

mkdir /var/spool/rsyslog

Step 3: Open the Rsyslog configuration file :

nano /etc/rsyslog.conf

Modify the configuration that allows the logs to be sent to the log server : Default location for work files (spool).

$WorkDirectory /var/spool/rsyslog

Start Transfer Rule 1:
$ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone.
$ActionQueueFileName srvrfwd1 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque.
$ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go.
$ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté.
$ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte.
# jouter les lignes suivantes dans la section RULES :
# Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante :
*.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.

Start Transfer Rule 2:
$ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone.
$ActionQueueFileName srvrfwd2 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque.
$ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go.
$ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté.
$ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte.
# jouter les lignes suivantes dans la section RULES :
# Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante :
*.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.

Step 4: Restart the RSyslog Service :

sudo /etc/init.d/rsyslog restart

Record the standard error message (screen) and the system log using the following command :

logger -s " Ceci est un client Rsyslog "

Go to the Rsyslog server under the / var / log / client_logs directory. A new folder named with the host name of your Rsyslog client should be available.

/var/log/client_logs/Client01/

Rsyslog and SSL

SSL encryption for the exchange between Syslog and Rsyslog.

Encrypting Traffic Syslog with TLS (SSL) [short version] : https://www.rsyslog.com/doc/v8-stable/tutorials/tls.html
Encrypting Traffic Syslog with TLS (SSL) : https://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_summary.html

Debug Rsyslogd and get its logs

DebugFile and DebugLevel Method

Open the Rsyslog configuration file :

sudo nano /etc/rsyslog.conf

# Add the following two lines :
$DebugFile /home/USER/Bureau/debug.txt
$DebugLevel 2
# <0|1|2> - Set the debug level:
# 0 is the debug mode disabled.
# 1 on demand debug mode enabled but debug mode disabled.
# 2 is the full debug mode.

# Restart Rsyslog :

sudo service rsyslog restart

Documentation : Rsyslog Debug Support : https://www.rsyslog.com/doc/v8-stable/troubleshooting/debug.html
Documentation : Debugging Rsyslogd : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-debugging_rsyslog

Trace method

The trace method can work, but, the first method is more suitable.

In a first console, run the following command:

sudo strace -p $(pgrep rsyslogd) -o fichier.trace

Open a second console, and restart for example Rsyslog :

sudo /etc/init.d/rsyslog restart

Rsyslog

Rsyslog

History of the transition from Sysklogd to Rsyslog

Installation

Install version 8.39.0 of Rsyslog from official Debian repositories

Install the Rsyslog documentation

Install rsyslog-mysql to configure the MySQL database that will store the syslog logs

Configure the mysql.conf connection file

Create syslog database to store syslog data with Rsyslog and MySQL

First method

Second method

Create an rsyslog user to use the syslog database

Optimize the syslog database

Cron script proposal 1

Cron script proposal 2

Use InnoDB and compressed tables

Check that the MySQL table is powered with the logs

Configure network listening in Rsyslog configuration

Default configuration example

Restart Rsyslog to apply the new configuration

Centralize the logs on a log server with Rsyslog

Add new clients Rsyslog

Rsyslog and SSL

Debug Rsyslogd and get its logs

DebugFile and DebugLevel Method

Trace method

See also

Resources used to write this summary

Additional resources