Maintenance
Contents
Updating the software
This section explains how to use apt-get upgrade.
Using apt-get is really simply. To update a system you need to execute two commands on the command line as root: apt-get update (which updates the lists of available packages) and apt-get upgrade (which upgrades the packages for which an upgrade is available).
As Debian Edu uses libpam-tmpdir, setting a per user TMP directory, it is a good idea to run apt-get without the TMP and TMPDIR variables set in the LTSP chroot. It is also a good idea to upgrade using the C locale to get known output and sorting order, even though that making a difference is a bug in a package.
LC_ALL=C apt-get update ; LC_ALL=C TMP= TMPDIR= ltsp-chroot apt-get update LC_ALL=C apt-get upgrade -y LC_ALL=C TMP= TMPDIR= ltsp-chroot -p apt-get upgrade -y ltsp-update-kernels # If a new kernel was installed
It is important to run ltsp-update-kernels if a new kernel was installed in the LTSP chroot, to keep the kernel and kernel modules in sync. The kernel is handed out via TFTP when the machine does PXE boot, and the kernel modules are fetched from the LTSP chroot.
Run ltsp-update-image to re-generate the NBD image(s).
It is also a good idea to install cron-apt and apt-listchanges and configure them to send mail to an address you are reading.
cron-apt will notify you once a day via email about any packages that can be upgraded. It does not install these upgrades, but does download them (usually in the night), so you don't have to wait for the download when you do apt-get upgrade.
Automatic installation of updates can be done easily if desired, it just needs the unattended-upgrades package to be configured as described on wiki.debian.org/UnattendedUpgrades. On new installations security updates are enabled by default.
apt-listchanges can send new changelog entries to you via email, or alternativly display them in the terminal when running aptitude or apt-get.
Keep yourself informed about security updates
Running cron-apt as described above is a good way to learn when security updates are available for installed packages. Another way to stay informed about security updates is to subscribe to the Debian security-announce mailinglist, which has the benefit of also telling you what the security update is about. The downside (compared to cron-apt) is that it also includes information about updates for packages which aren't installed.
Backup Management
For backup management point your browser to https://www/slbackup-php. Please note that you need to access this site via SSL, since you have to enter the root password there. If you try to access this site without using SSL it will fail. Note: the site will only work if you temporarily allow ssh root login on the backup server (main server 'tjener' by default).
By default tjener will back up /skole/tjener/home0, /etc/, /root/.svk and LDAP to /skole/backup which is under the LVM. If you only want to have spare copies of things (in case you delete them) this setup should be fine for you.
Be aware that this backup scheme doesn't protect you from failing hard drives.
If you want to back up your data to an external server, a tape device or another hard drive you'll have to modify the existing configuration a bit.
If you want to restore a complete folder, your best option is to use the command-line:
$ sudo rdiff-backup -r <date> \ /skole/backup/tjener/skole/tjener/home0/user \ /skole/tjener/home0/user_<date>
This will leave the content from /skole/tjener/home0/user for <date> in the folder /skole/tjener/home0/user_<date>
If you want to restore a single file, then you should be able to select the file (and the version) from the web interface, and download only that file.
If you want to get rid of older backups, choose "Maintenance" in the menu on the backup page and select the oldest snapshot to keep:
Server Monitoring
Munin
The Munin trend reporting system is available from https://www/munin/. It provides system status measurement graphs on a daily, weekly, monthly and yearly basis, and provides the system administrator with help when looking for bottlenecks and the source of system problems.
The list of machines being monitored using Munin is generated automatically, based on the list of hosts reporting to sitesummary. All hosts with the package munin-node installed are registered for Munin monitoring. It will normally take one day from a machine being installed until Munin monitoring starts, because of the order the cron jobs are executed. To speed up the process, run sitesummary-update-munin as root on the sitesummary server (normally the main server). This will update the /etc/munin/munin.conf file.
The set of measurements being collected is automatically generated on each machine using the munin-node-configure program which probes the plugins available from /usr/share/munin/plugins/ and symlinks the relevant ones to /etc/munin/plugins/.
Information about Munin is available from http://munin-monitoring.org/.
Icinga
Icinga system and service monitoring is available from https://www/icinga/. The set of machines and services being monitored is automatically generated using information collected by the sitesummary system. The machines with the profile Main-server and LTSP-server receive full monitoring, while workstations and thin clients receive simple monitoring. To enable full monitoring on a workstation, install the nagios-nrpe-server package on the workstation.
The username is icingaadmin and the default password is skolelinux. For security reasons, avoid using the same password as root. To change the password you can run the following command as root:
htpasswd /etc/icinga/htpasswd.users icingaadmin
By default Icinga does not send email. This can be changed by replacing notify-by-nothing with host-notify-by-email and notify-by-email in the file /etc/icinga/sitesummary-template-contacts.cfg.
The Icinga configuration file used is /etc/icinga/sitesummary.cfg. The sitesummary cron job generates /var/lib/sitesummary/icinga-generated.cfg with the list of hosts and services to monitor.
Extra Icinga checks can be put in the file /var/lib/sitesummary/icinga-generated.cfg.post to get them included in the generated file.
Information about Icinga is available from https://www.icinga.com/ or in the icinga-doc package.
Common Icinga warnings and how to handle them
Here are instructions on how to handle the most common Icinga warnings.
DISK CRITICAL - free space: /usr 309 MB (5% inode=47%):
The partition (/usr/ in the example) is too full. There are in general two ways to handle this: (1) remove some files or (2) increase the size of the partition. If the partition is /var/, purging the APT cache by calling apt-get clean might remove some files. If there is more room available in the LVM volume group, running the program debian-edu-fsautoresize to extend the partitions might help. To run this program automatically every hour, the host in question can be added to the fsautoresize-hosts netgroup.
APT CRITICAL: 13 packages available for upgrade (13 critical updates).
New package are available for upgrades. The critical ones are normally security fixes. To upgrade, run 'apt-get upgrade && apt-get dist-upgrade' as root in a terminal or log in via ssh to do the same. On LTSP servers, remember to also update the LTSP chroot using ltsp-chroot apt-get update && ltsp-chroot apt-get upgrade.
If you do not want to manually upgrade packages and trust Debian to do a good job with new versions, you can configure unattended-upgrades to automatically upgrade all new packages every night. This will not upgrade the LTSP chroots.
To upgrade the LTSP chroot, one can use ltsp-chroot apt-get update && ltsp-chroot apt-get upgrade. On 64-bit servers, one will have to add -a i386 as an argument to ltsp-chroot. It is a good idea to update the chroot when updating the host system.
WARNING - Reboot required : running kernel = 2.6.32-37.81.0, installed kernel = 2.6.32-38.83.0
The running kernel is older than the newest installed kernel, and a reboot is required to activate the newest installed kernel. This is normally fairly urgent, as new kernels normally show up in Debian Edu to fix security issues.
WARNING: CUPS queue size - 61
The printer queues in CUPS have a lot of jobs pending. This is most likely because of a unavailable printer. Disabled print queues are enabled every hour on hosts that are member of the cups-queue-autoreenable-hosts netgroup, so for such hosts no manual action should be required. The print queues are emptied every night on hosts that are member of the cups-queue-autoflush-hosts netgroup. If a host have a lot of jobs in their queue, consider adding this host to one or both of these netgroups.
Sitesummary
Sitesummary is used to collect information from each computer and submit it to the central server. The information collected is available in /var/lib/sitesummary/entries/. Scripts in /usr/lib/sitesummary/ are available to generate reports.
A simple report from sitesummary without any details is available from https://www/sitesummary/.
Some documentation on sitesummary is available from http://wiki.debian.org/DebianEdu/HowTo/SiteSummary
More information about Debian Edu customisations
More information about Debian Edu customisations useful for system administrators can be found in the Administration Howto chapter and in the Advanced administration Howto chapter