Translation(s): 中文普通话 - English - Français - Italiano
Rsyslog
Rsyslog is an open source program for transferring log messages over an IP network for UNIX and Unix systems. It implements the core syslog protocol, and extends it with content-based filtering, advanced filtering features, flexible configuration options, and adds features such as the use of TCP, SSL, and RELP for transport. Rsyslog is a direct substitute for syslogd. Rsyslog offers high performance, security features and modular design. Rsyslog can allow us to store log messages in a MySQL, MariaDB, MongoDB or PostgreSQL database that can be configured with dbconfig-common for easy configuration via debconf. The log data can be exploited by a complementary program, such as the LogAnalyzer web interface. The rotation of the log messages is automated.
Rsyslog official website : https://www.rsyslog.com
Documentation Version 8 : https://www.rsyslog.com/doc/v8-stable/
General documentation : https://www.rsyslog.com/doc/master/index.html
Configuration documentation : https://www.rsyslog.com/doc/v8-stable/configuration/index.html
Configuration examples : https://www.rsyslog.com/doc/v8-stable/configuration/examples.html
History of the transition from Sysklogd to Rsyslog
The rsyslogd service was integrated with the Rsyslog package from Debian Lenny (https://wiki.debian.org/DebianLenny) to replace the old syslog: sysklogd. The existing logging rules in syslog.conf can be simply copied to /etc/rsyslog.conf or to the /etc/rsyslog.conf.d folder. Sysklogd which was installed by default is not bad but the package has hardly been maintained in recent years. See the Debian Lenny shared information memo release notes : https://www.debian.org/releases/lenny/i386/release-notes/ch-whats-new#system-changes
Some arguments in favor of Rsyslog in the following discussions :
https://lists.debian.org/debian-devel/2008/01/thrd3.html#01002
https://lists.debian.org/debian-release/2008/07/msg00117.html
Installation
Install Rsyslog allows you to store syslog logs in a database. If no database is configured, the LogAnalyzer web interface will be able to use the log data provided by syslog from the disk. If a database is configured, the LogAnalyzer web interface will be able to use the log data provided by the database.
Prerequisites: apache.
Install version 8.39.0 of Rsyslog from official Debian repositories
Install the PGP key in your apt system :
sudo apt-key adv --recv-keys --keyserver keys.gnupg.net AEF0CF8E
Adiscon repository for v8-stable on Debian (7) Wheezy. Edit your /etc/apt/sources.list and add these lines at the end :
deb http://debian.adiscon.com/v8-stable wheezy/ deb-src http://debian.adiscon.com/v8-stable wheezy/
sudo apt update sudo apt-get install rsyslog
Install the Rsyslog documentation
Documentation installed, by default on Debian, with rsyslogd service :
man rsyslogd
Install the full Rsyslog documentation in HTML format :
sudo apt install rsyslog-doc cat /usr/share/doc-base/rsyslog-doc
Document: rsyslog-doc Title: Rsyslog Documentation Author: Rainer Gerhards Abstract: This documentation covers the configuration of rsyslog. Section: System/Administration Format: HTML Index: /usr/share/doc/rsyslog-doc/html/manual.html Files: /usr/share/doc/rsyslog-doc/html/*.html</code>
Install rsyslog-mysql to configure the MySQL database that will store the syslog logs
MySQL, MariaDB, MongoDB or PostgreSQL databases are properly supported.
sudo apt install rsyslog-mysql
The configuration of rsyslog-mysql is done automatically with dbconfig-common. A MySQL database named Syslog containing two ?SystemEvents and ?SystemEventsProperties tables is created. A rsyslog @ localhost user is added, he has full control over the syslog database. The configuration of the MySQL database connection is available from the /etc/rsyslog.d/mysql.conf file.
Configure the mysql.conf connection file
Edit mysql.conf, the MySQL configuration file. Connect to the syslog database to store syslog logs.
sudo nano /etc/rsyslog.d/mysql.conf
Note: It seems important to use the tab spaces between the entries in the configuration.
# Load the module and save all syslog messages to the database:
module (load="ommysql") *.* action(type="ommysql" server="localhost" db="Syslog" uid="rsyslog" pwd="Password_BDD")
# This command filters the alert level of the messages to be stored in the database:
*.emerg >localhost,Syslog,Utilisateur_BDD,Password_BDD *.alert >localhost,Syslog,Utilisateur_BDD,Password_BDD *.crit >localhost,Syslog,Utilisateur_BDD,Password_BDD *.err >localhost,Syslog,Utilisateur_BDD,Password_BDD *.warning >localhost,Syslog,Utilisateur_BDD,Password_BDD *.notice >localhost,Syslog,Utilisateur_BDD,Password_BDD *.info >localhost,Syslog,Utilisateur_BDD,Password_BDD *.debug >localhost,Syslog,Utilisateur_BDD,Password_BDD
#? # Syntax to check for a Debian Stretch 9 system: #?# $ModLoad ommysql #?# *.* :ommysql:localhost,Syslog,Utilisateur_BDD,Password_BDD
Create syslog database to store syslog data with Rsyslog and MySQL
If the rsyslog-mysql package was not used to create the database, the database can be created manually.
First method
Create the database structure with the official script to store the log messages in MySQL: The definition of the given database schema is available from the createDB.sql file from the Rsyslog 8.39.0 archive. The file path is ./rsyslog-8.39.0/plugins/ommysql/createDB.sql. Start importing the database:
mysql -u rsyslog -D Syslog -p < ./rsyslog-8.39.0/plugins/ommysql/createDB.sql
Second method
Connect to MySQL:
sudo bash mysql -u root -p
If necessary, enter the user's password for the database. Continue with the enter key. Create the database structure manually to store the log messages in MySQL:
CREATE DATABASE Syslog; USE Syslog; CREATE TABLE SystemEvents ( ID int unsigned not null auto_increment primary key, CustomerID bigint, ReceivedAt datetime NULL, DeviceReportedTime datetime NULL, Facility smallint NULL, Priority smallint NULL, FromHost varchar(60) NULL, Message text, NTSeverity int NULL, Importance int NULL, EventSource varchar(60), EventUser varchar(60) NULL, EventCategory int NULL, EventID int NULL, EventBinaryData text NULL, MaxAvailable int NULL, CurrUsage int NULL, MinUsage int NULL, MaxUsage int NULL, InfoUnitID int NULL , SysLogTag varchar(60), EventLogType varchar(60), GenericFileName VarChar(60), SystemID int NULL ); CREATE TABLE SystemEventsProperties ( ID int unsigned not null auto_increment primary key, SystemEventID int NULL , ParamName varchar(255) NULL , ParamValue text NULL );
Create an rsyslog user to use the syslog database
Log in to MySQL on the command line. The password is empty on the local system, confirm with the enter key to connect. It will secure MySQL with a password for a system in production.
mysql -u root -p
If the rsyslog user does not exist yet, create the rsyslog user to manage the database.
CREATE USER 'rsyslog'@'localhost' IDENTIFIED BY 'Analyzer'; Give rights to the rsyslog user. GRANT ALL PRIVILEGES ON Syslog.* TO rsyslog@localhost IDENTIFIED BY 'Analyzer'; FLUSH PRIVILEGES; exit;
Optimize the syslog database
The MySQL ?SystemEvents table can fill up quickly and become very large. An archiving script placed in a Cron job will avoid unpleasant surprises.
Cron script proposal 1
# Archiving seems to work properly. # Check the deletion of data in the database. #!/bin/bash # Monthly archiving of the MySQL Syslog database logs RETENTION=180 # Log retention time (in days) DESTDIR="/var/backups/syslog" # Archive storage directory ARCHIVE="syslog-$(date '+%Y-%m-%d-%Hh%M').gz" # MySQL parameters MYSQL_HOST="localhost" MYSQL_DB="Syslog" MYSQL_USER="rsyslog" MYSQL_PASSWD="motdepasse" # Archiving sql="SELECT * FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION" mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB | gzip > $DESTDIR/$ARCHIVE # Deleting MySQL records sql="DELETE FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION" mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB
Cron script proposal 2
# This script deletes data of more than two days that are stored in the database. # Text information echo lines were not displayed in the terminal during my tests. #!/bin/bash # Purge the logs of the MySQL Syslog database RETENTION=2 # Log retention time (in days) # Variable initialization MYSQL_HOST="localhost" MYSQL_DB="bddloganalyzer" MYSQL_USER="Logs" MYSQL_PASSWD="Analyzer" TMP_LOG="/tmp/cron.log" # Count records to delete sql="SELECT COUNT(*) FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION" compteur='mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB' if [ "$compteur" = "0" ]; then echo "(I) There are no records older than $RETENTION day" >> $TMP_LOG else echo "(I) There are $compteur records to delete" >> $TMP_LOG # Deleting MySQL records sql="DELETE FROM SystemEvents WHERE DATEDIFF(NOW(), DeviceReportedTime) > $RETENTION" mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWD -e "$sql" -B -s $MYSQL_DB cr=$? if [ $cr -gt 0 ]; then echo "(E) Error deleting records Return code $cr" >> $TMP_LOG else echo "(I) The operation went well" >> $TMP_LOG fi fi
Use InnoDB and compressed tables
# This script has not been tested. # To use InnoDB and compressed tables : {{{ ALTER TABLE SystemEvents ENGINE=innodb DEFAULT CHARSET=latin1 row_format=COMPRESSED KEY_BLOCK_SIZE=4;
Index fields that will often be used in queries
# This script has not been tested. # After a few months of operation, queries launched from LogAnalyser can become slower, making the tool hard to use. # With the MySQL plugin version of rsyslog (v 4.6.4) the indexes are not created in the SystemEvents table at installation. # When the records are counted in millions in the table it is no longer suitable. # The solution is to index the fields that will often be used in queries, for example, Facility, Priority, FromHost, SysLogTag and DeviceReportedTime. ALTER TABLE `SystemEvents` ADD INDEX(`Facility`); ALTER TABLE `SystemEvents` ADD INDEX(`Priority`); ALTER TABLE `SystemEvents` ADD INDEX(`FromHost`); ALTER TABLE `SystemEvents` ADD INDEX(`SysLogTag`); ALTER TABLE `SystemEvents` ADD INDEX(`DeviceReportedTime`);
Check that the MySQL table is powered with the logs
# Create a log entry with the following command :
logger Works!
# Test if there are records from the terminal with the following command :
mysql -u rsyslog -p -Bsr -e "SELECT Message FROM SystemEvents LIMIT 0,20" Syslog
# Test if there are records from the MySQL command line interface with the following commands :
mysql -u root -p
# Enter your user password for root or press enter directly if the password is empty. # Select the syslog database on which the request will be launched.
mysql> use Syslog; mysql> select * from SystemEvents limit 1;
# This command also works and should be faster by selecting less information.
mysql> select ReceivedAt,Message from SystemEvents;
Configure network listening in Rsyslog configuration
# Set up the Rsyslog configuration file.
sudo nano /etc/rsyslog.conf
# Loading this module seems essential to receive the syslog data locally from the Disk or the MySQL database. module(load="imuxsock") # Charger le module pour les anciennes systèmes : # $ModLoad imuxsock # The syslog information can be received via UDP or TCP. # The configuration instructions are slightly different between UDP and TCP. # In most cases, UDP Syslog should be fully sufficient and efficient. # If the system buffer for UDP is full during a large message flow, the messages will be lost. # This is not the case with TCP syslog because the sender and the receiver communicate about the arrival of network packets. # Some devices, such as routers, can not send TCP Syslog by design. In this case, you still have to use UDP. # TCP syslog is suitable for environments where log messages should not be lost or which should ensure PCI compliance, for example, banks. # Prefer the use of the RELP protocol : # On the (non) reliability of plain TCP syslog, using the RELP protocol would seem to be even more relevant. # On the (a) reliability of plain TCP syslog : https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html # Enable remote log reception : # The UDP syslog is received on port 514 by default. # The TCP syslog requires a different port because the RPC service often uses this port as well. # Syntax for Debian. ( To check ! ) # Provides UDP syslog reception : ## $ModLoad imudp ## $UDPServerRun 514 # Provides TCP syslog reception : ## $ModLoad imtcp ## $InputTCPServerRun 1514 # Syntax for Ubuntu 16.04 LTS / Ubuntu 18.04 LTS and Mint Tara 19. # Provides UDP syslog reception : module(load="imudp") input(type="imudp" port="514") # Provides TCP syslog reception : module(load="imtcp") input(type="imtcp" port="1514") # List of sub networks authorized to connect : # $AllowedSender UDP, 127.0.0.1, 192.168.1.0/24 # $AllowedSender TCP, 127.0.0.1, 192.168.1.0/24 # Le serveur est maintenant configuré pour enregistrer les logs dans la base de données. # Il écoute sur le port 514 et 1514 les messages entrant.
Default configuration example
#$DebugFile /home/looksaalpha/Bureau/debug.txt #$DebugLevel 2 # /etc/rsyslog.conf Configuration file for Rsyslog. $EscapeControlCharactersOnReceive off #### MODULES #### # $ModLoad imuxsock module(load="imuxsock") # UDP syslog reception # module(load="imudp") # input(type="imudp" port="1514") # ou # $Modload imudp # $UDPServerRun 1514 # TCP syslog reception # module(load="imtcp") # input(type="imtcp" port="1514") # ou # $ModLoad imtcp # $InputTCPServerRun 1514 # Kernel connection support and activation of non-kernel klog messages module(load="imklog" permitnonkernelfacility="on") # Default connection rules : /etc/rsyslog.d/50-default.conf auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none /var/log/syslog syslog.* /var/log/rsyslog.log cron.* /var/log/cron.log kern.* /var/log/kern.log #daemon.* /var/log/daemon.log #lpr.* /var/log/lpr.log #user.* /var/log/user.log #mail.* /var/log/mail.log #mail.info /var/log/mail.info #mail.warn /var/log/mail.warn #mail.err /var/log/mail.err #news.crit /var/log/news/news.crit #news.err /var/log/news/news.err #news.notice /var/log/news/news.notice # Owners of the cron.log file passed to syslog:logadmin # Add a line to manage other logs by Rsyslog (Apache2, Mysql, ...): syslog.* /var/log/apache2/error.log #### GLOBAL DIRECTIVES #### # Seems to be obsolete: # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Filter duplicate messages: $RepeatedMsgReduction on $WorkDirectory /var/spool/rsyslog # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf $IncludeConfig /etc/rsyslog.d/mysql.conf
Restart Rsyslog to apply the new configuration
Restart the Rsyslog daemon to apply the new configuration :
# Old orders : sudo service rsyslog restart sudo /etc/init.d/rsyslog restart # New order : sudo systemctl restart rsyslog
Centralize the logs on a log server with Rsyslog
Completes the previously proposed configuration to use Rsyslog as a centralized log server.
Create a template file where we will create a new custom log format from /etc/rsyslog.d/tmpl.conf I did not test this possibility.
Allow the default Rsyslog UDP 514 port on your firewall. Allow the modified TCP 1514 Rsyslog port on your firewall. The following commands will open this port via UFW:
sudo ufw allow 514/udp sudo ufw allow 1514/tcp
Restart the UFW service to take into account the changes:
sudo ufw reload
Reload the Rsyslog service using the following command:
systemctl restart rsyslog
In case the log server is behind a router: Consider enabling NAT rules for the ports selected in the configuration. Redirect UDP traffic 514-> 514 to the log server. Redirect TCP traffic 1514-> 1514 to the log server.
Add new clients Rsyslog
Configure the client machines so that they can send their logs to the machine that will act as the log server. Works with syslog or syslog-ng, with different settings for syslog-ng. This feature has not been tested when writing this tutorial.
Step 1: Install the rsyslog package on each client :
apt-get install rsyslog
Step 2: Create a working directory :
mkdir /var/spool/rsyslog
Step 3: Open the Rsyslog configuration file :
nano /etc/rsyslog.conf
Modify the configuration that allows the logs to be sent to the log server : Default location for work files (spool).
$WorkDirectory /var/spool/rsyslog
Start Transfer Rule 1: $ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone. $ActionQueueFileName srvrfwd1 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque. $ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go. $ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté. $ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte. # jouter les lignes suivantes dans la section RULES : # Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante : *.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.
Start Transfer Rule 2: $ActionQueueType LinkedList # Exécuter le traitement de façon asynchrone. $ActionQueueFileName srvrfwd2 # Préfixe de nom unique pour les fichiers spool. Active également le mode disque. $ActionQueueMaxDiskSpace 1g # Limite d'espace de 1 Go. $ActionQueueSaveOnShutdown on # Enregistrer les données sur le disque si Rsyslog est arrêté. $ActionResumeRetryCount -1 # Tentatives infinies en cas d'échec de connexion avec l'hôte. # jouter les lignes suivantes dans la section RULES : # Envoyer tous les messages sur le serveur de journalisation distant avec la commande suivante : *.* @@Cible_IP_serveur_1_LogAnalayzer:514 # Ou 514 est le port d'écoute qui a été défini dans la configuration.
Step 4: Restart the RSyslog Service :
sudo /etc/init.d/rsyslog restart
Record the standard error message (screen) and the system log using the following command :
logger -s " Ceci est un client Rsyslog "
Go to the Rsyslog server under the / var / log / client_logs directory. A new folder named with the host name of your Rsyslog client should be available.
/var/log/client_logs/Client01/
Rsyslog and SSL
SSL encryption for the exchange between Syslog and Rsyslog.
Encrypting Traffic Syslog with TLS (SSL) [short version] : https://www.rsyslog.com/doc/v8-stable/tutorials/tls.html
Encrypting Traffic Syslog with TLS (SSL) : https://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_summary.html
Debug Rsyslogd and get its logs
DebugFile and DebugLevel Method
Open the Rsyslog configuration file :
sudo nano /etc/rsyslog.conf
# Add the following two lines : $DebugFile /home/USER/Bureau/debug.txt $DebugLevel 2 # <0|1|2> - Set the debug level: # 0 is the debug mode disabled. # 1 on demand debug mode enabled but debug mode disabled. # 2 is the full debug mode.
# Restart Rsyslog :
sudo service rsyslog restart
Documentation : Rsyslog Debug Support : https://www.rsyslog.com/doc/v8-stable/troubleshooting/debug.html
Documentation : Debugging Rsyslogd : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-debugging_rsyslog
Trace method
The trace method can work, but, the first method is more suitable.
In a first console, run the following command:
sudo strace -p $(pgrep rsyslogd) -o fichier.trace
Open a second console, and restart for example Rsyslog :
sudo /etc/init.d/rsyslog restart
See also
A summary in French proposed by Visionduweb : https://www.visionduweb.eu/wiki/index.php?title=Adiscon_Rsyslog
Resources used to write this summary
How to Setup LogAnalyzer with Rsyslog On Ubuntu 16.04 LTS / Ubuntu 18.04 LTS : http://yallalabs.com/linux/how-to-setup-loganalyzer-with-rsyslog-on-ubuntu-16-04-lts-ubuntu-18-04-lts/
How to Setup A Centralized Log Server Using Rsyslog on Ubuntu 16.04 LTS : http://yallalabs.com/linux/how-to-setup-a-centralized-log-server-using-rsyslog-on-ubuntu-16-04-lts/
Tutoriel | Consolidation des logs avec Rsylog, Mysql et Loganalyzer : https://journaldunadminlinux.fr/tutoriel-consolidation-des-logs-avec-rsylog-mysql-et-loganalyzer/
Consolidation des logs avec rsyslog, MySQL et ?LogAnalyser : http://tavie.onsenfout.com/2011/07/05/consolidation-des-logs-avec-rsyslog-mysql-et-loganalyser/
How to install Rsyslog with loganalyzer in Ubuntu : http://techies-world.com/how-to-install-rsyslog-with-loganalyzer-in-ubuntu/
How to Setup LogAnalyzer with Rsyslog and MySQL : https://tecadmin.net/setup-loganalyzer-with-rsyslog-and-mysql/
Optimisations ?LogAnalyser : http://tavie.onsenfout.com/2012/03/08/optimisations-loganalyser/
Using the syslog receiver module : https://www.rsyslog.com/using-the-syslog-receiver-module/
Ce contenu de recherche effectué par Visionduweb sur Rsyslog est partagé sur le wiki officiel de Debian : https://wiki.debian.org/Rsyslog -> https://wiki.debian.org/fr/Rsyslog
Vidéo : https://youtu.be/BUEeKG6dgxs
Additional resources
Rsyslog en russe : http://www.k-max.name/linux/rsyslog-na-debian-nastrojka-servera/
Install a Centralized Log Server with Rsyslog in Debian 9 : https://www.howtoforge.com/tutorial/rsyslog-centralized-log-server-in-debian-9/
rsyslogd - syslogd fiable et étendu : https://manpages.debian.org/stretch/rsyslog/rsyslogd.8.en.html
Exemples de configuration : http://wiki.rsyslog.com/index.php/Configuration_Samples Unfortunately, we cannot provide a fixed link at the moment, because the wiki has been disabled for now. We had to disable the wiki to accomodate the changes to european GDPR laws and the current wiki software could not be easily adjusted.