Prometheus

Documentation

This service aims at providing basic monitoring and alerting for services hosted under the DebianDotNet domain. It does *not* aim at replacing ?DSA's internal monitoring services (which are based on Icinga).

It's mostly built from a git repository as a ?Ansible playbook.

The first setup was done by hand. The following packages were installed: 

apt install prometheus prometheus-blackbox-exporter prometheus-alertmanager foot-terminfo git etckeeper unattended-upgrades needrestart ldap-utils

Then a script was written to pull the list of debian.net domains from LDAP, and to shove that in the blackbox exporter configuration. The Prometheus config was tweaked to enable the blackbox exporter ICMP probes. I also had to dpkg-reconfigure the blackbox exporter to give it the extra capabilities needed to run ICMP probes, and lock down to only local host in /etc/default/prometheus-blackbox-exporter.

Most of that configuration except the scrape targets was reimplemented in an Ansible playbook shipped in the above git repository.

Possible improvements would be to maybe tap into the debian.net infra repository for opt-in alert notifications, alongside extra targets configurations, for example node exporters if such hosts are so configured, alongside documentation for admins on how to use this.

Samples are currently kept for a year. Merge requests and feedback welcome in the GitLab project.