Do not use Edit(GUI) button.

?TableOfContents(4)

Copyright 2007 Osamu Aoki GPL, (Please agree to GPL, GPL2, and any version of GPL which is compatible with DSFG if you update any part of wiki page)

I welcome your contributions to update the wiki pages. You must follow these rules:

System tips

Here, I will describe basic tips to configure and manage system from the console.

Data recording and presentation

The log daemon

Many programs record their activities under the /var/log/ directory.

See @{@thesystemmessage@}@ and @{@thekernelmessage@}@.

Log analyzer

Here are notable log analyzers ("~Gsecurity::log-analyzer" in aptitude).

List of system log analyzers.

1

2

3

package

popcon

size

description

logwatch

-

-

log analyser with nice output written in Perl

fail2ban

-

-

bans IPs that cause multiple authentication errors

analog

-

-

web server log analyzer

awstats

-

-

powerful and featureful web server log analyzer

sarg

-

-

squid analysis report generator

pflogsumm

-

-

Postfix log entry summarizer

syslog-summary

-

-

summarize the contents of a syslog log file

lire

-

-

full-featured log analyzer and report generator

fwlogwatch

-

-

Firewall log analyzer

squidview

-

-

monitors and analyses squid access.log files

visitors

-

-

fast web server log analyzer

swatch

-

-

Log file viewer with regexp matching, highlighting, & hooks

crm114

-

-

The Controllable Regex Mutilator and Spam Filter (CRM114)

icmpinfo

-

-

Interpret ICMP messages

(!) [http://crm114.sourceforge.net/ CRM114] provides language infrastructure to write fuzzy filters with the [http://www.laurikari.net/tre/ TRE regex library]. Its popular use is spam mail filter but it can be used as log analizer.

Recording the shell activities cleanly

The simple use of the script(1) command (see: @{@recordingtheshellactivities@}@) to record the shell activity produces a file with control characters. This can be avoided by using the col(1) command:

$ script
Script started, file is typescript

$ col -bx <typescript >cleanedfile
$ vim cleanedfile

In some odd environment where you lack access to the script command, for example, during the boot process in the initramfs, you can use following instead:

$ sh -i 2>&1 | tee typescript

{i} Some x-terminal-emulator such as gnome-terminal have recording capability. You may wish to extend line buffer for scrollback.

{i} You may use emacs or screen command to perform recording too. emacs shell mode can be started by "M-x shell", and use "C-x C-w" to write the buffer to a file. screen command with "^A H" does the recording.

Customized display of text data

Although pager tools such as more(1) and less(1) (see @{@thepager@}@) and custom tools for highlighting and formatting @{@highlightingandfingplaintextdata@}@ can display text data nicely, general purpose editors (see @{@thetexteditor@}@) are most versatile and customizable.

{i} For vim(1) and its pager mode alias view(1), ":set hls" will enable highlighted search.

Colorizing commands

Colorized commands are handy for inspecting their output.

For colorizing ls(1) command, use alias "ls --color=auto".

For colorizing grep(1), egrep(1), fgrep(1), ..., export environment variable "export GREP_OPTIONS='--color=auto'".

<!> Colorizing should be enabled only when standard output is connected to a terminal (tty). Many commands under such environments under environment where colorized commands are run with "--color=always" option.

Recording the graphic image of an X application

There are few ways to record the graphic image of an X application, including an xterm display.

List of graphic image manipulation tools.

1

2

3

package

popcon

size

command

xbase-clients

25829

-

xwd(1)

gimp

8489

-

GUI menu

imagemagick

5479

-

import(1)

scrot

134

-

scrot(1)

System maintenance tips

System integrity check

Poor system maintenance may expose your system to external exploitation.

For system security check, you should start with:

The hardware identification

For the [http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect PCI]-like devices ([http://en.wikipedia.org/wiki/Accelerated_Graphics_Port AGP], [http://en.wikipedia.org/wiki/PCI_Express PCI-Express], [http://en.wikipedia.org/wiki/PC_Card#CardBus CardBus], [http://en.wikipedia.org/wiki/ExpressCard ?ExpressCard], etc.), lspci(8) command (probably with "-nn" option) is a good start for the hardware identification

Alternatively, you can identify the hardware by reading contents of /proc/bus/pci/devices or browsing directory tree under /sys/bus/pci (see @{@procfsandsysfs@}@).

List of hardware identification tools.

1

2

3

package

popcon

size

description

pciutils

-

-

Linux PCI Utilities, lspci(8)

usbutils

-

-

Linux USB utilities, lsusb(8)

pcmciautils

-

-

PCMCIA utilities for Linux 2.6, pccardctl(8)

scsitools

-

-

Collection of tools for SCSI hardware management, lsscsi(8)

pnputils

-

-

Plug and Play BIOS utilities, lspnp(8)

procinfo

-

-

Displays system information from /proc, lsdev(8)

lshw

-

-

Information about hardware configuration, lshw(1)

discover

-

-

Hardware identification system, discover(8)

The hardware configuration

Although most of the hardware configuration on modern GUI desktop systems such as Gnome and KDE can be managed through accompanying GUI configuration tools, it is a good idea to know some basics methods to configure them.

List of hardware configuration tools.

1

2

3

package

popcon

size

description

hal

-

-

Hardware Abstraction Layer, lshal(1)

console-tools

-

Linux console font and keytable utilities.

x11-xserver-utils

-

X server utilities. xset(1) and xmodmap(1) commands.

acpid

24513

-

Daemon to manage events delivered by the Advanced Configuration and Power Interface (ACPI)

acpi

2563

-

Utilities for ACPI devices

apmd

1222

-

Daemon to manage events delivered by the Advanced Power Management (APM)

powersaved

1038

-

Daemon to manage battery, temperature, ac, cpufreq (?SpeedStep, Powernow!) control and monitor with ACPI and APM supports.

noflushd

95

-

Allow idle hard disks to spin down

sleepd

75

-

Puts a laptop to sleep during inactivity

hdparm

5192

-

Hard disk access optimization. Very effective but dangerous. You must read hdparm(8) first.

smartmontools

3526

-

Control and monitor storage systems using S.M.A.R.T.

setserial

2619

-

Collection of tools for serial port management.

memtest86+

406

-

Collection of tools for memory hardware management.

scsitools

185

-

Collection of tools for SCSI hardware management.

tpconfig

276

-

A program to configure touchpad devices

setcd

82

-

Compact disc drive access optimization.

big-cursor

*121

-

Larger mouse cursors for X

lspowertweak

-

-

Simple front end to powertweak, lspowertweak(8)

Here, ACPI is a newer framework for the power management system than APM.

The time

The following will set system and hardware time to MM/DD hh:mm, CCYY.

# date MMDDhhmmCCYY
# hwclock --utc --systohc
# hwclock --show

Times are normally displayed in the local time on the Debian system but the hardware and system time usually use UTC.

If the hardware (BIOS) time is set to GMT, change the setting to UTC=yes in the /etc/default/rcS.

If you wish to update system time via network, consider to use the NTP service with the packages such as ntp, ntpdate, and chrony. See:

The terminal configuration

There are several components to configure character console and ncurses(3) system features:

If the terminfo entry for xterm doesn't work with a non-Debian xterm, change your terminal type from xterm to one of the feature-limited versions such as xterm-r6 when you log in to a Debian system remotely. See /usr/share/doc/libncurses5/FAQ for more. dumb is the lowest common denominator for terminfo.

Disable the screen saver

For disabling the screen saver, use following commands.

List of commands for disabling the screen saver.

environment

command

The Linux console

setterm -powersave off

The X Window by turning off screensaver

xset s off

The X Window by disabling dpms

xset -dpms

The X Window by GUI configuration of screen saver

xscreensaver-command -prefs

Disable the sound (beep)

One can always unplug the PC speaker. ;-) Removing pcspkr kernel module does this for you.

The following will prevent the readline program used by the bash to beep when encountering "\a" (ASCII=7):

$ echo "set bell-style none">> ~/.inputrc

Storage device hints

You can test disk access speed of harddisk, e.g. /dev/hda, by "hdparm -tT /dev/hda".

You may be able to speed up a harddisk connected with (E)IDE with "hdparm -q -c3 -d1 -u1 -m16 /dev/hda". Please check your hardware and read manpage of hdparam(8) before playing with harddisk configuration because this may be quite dangerous for the data integrity.

You may be able to read badly pressed cdroms on modern high head CD-ROM drive by slowing it down with "setcd -x 2.

Mounting a filesystem with the noatime option is also very effective in speeding up read access to the file. See fstab(5) and mount(8).

Memory usage

The kernel boot message in the /var/log/dmesg contains the total exact size of available memory.

The free(1) and top(1) commands display information on memory resources on the running system.

$ grep '^Memory' /var/log/dmesg
Memory: 990528k/1016784k available (1975k kernel code, 25868k reserved, 931k data, 296k init)
$ free -k
             total       used       free     shared    buffers     cached
Mem:        997184     976928      20256          0     129592     171932
-/+ buffers/cache:     675404     321780
Swap:      4545576          4    4545572

For my MacBook with 1GB=1048576k DRAM (video system steals some of this):

List of memory sizes reported.

report

size

Total size in dmesg

1016784k = 1GB - 31792k

Free in dmesg

990528k

Total under shell

997184k

Free under shell

20256k

Do not worry about the large size of "used" and the small size of "free" in the "Mem:" line, but read the one under them (675404 and 321780 in the example below) and relax.

Invisible disk space consumption

All deleted but open files consumes disk space although they are not visible from normal du(1). They can be listed with their size by:

# lsof -s -X / |grep deleted

Process operation

Time a process

Display time used by the process invoked by the command.

# time some_command >/dev/null
real    0m0.035s       # time on wall clock (elapsed real time)
user    0m0.000s       # time in user mode
sys     0m0.020s       # time in kernel mode

The scheduling priority

A nice value is used to control the scheduling priority for the process.

List of nice values for the scheduling priority.

nice value

scheduling priority

19

lowest priority process (nice)

0

very high priority process for user.

-20

very high priority process for root. (not-nice)

List of tools for the scheduling priority.

2

3

4

command

package

popcon

size

function

nice

coreutils

-

-

run a program with the modified scheduling priority

renice

bsdutils

-

-

modify the scheduling priority of a running processes

# nice  -19 top                                      # very nice
# nice --20 wodim -v -eject speed=2 dev=0,0 disk.img # very fast

Sometimes an extreme nice value does more harm than good to the system. Use this command carefully.

The ps command

The ps(1) command on the Debian support both BSD and SystemV features and helps to identify the process activity statically.

List of ps command styles.

style

typical command

feature

BSD

ps aux

display %CPU %MEM

SystemV

ps -efH

display PPID

For the zombie (defunct) children process, you can kill them by the parent process ID identified in the (PPID) field.

The top command

The top(1) command on the Debian has rich features and helps to identify what process is acting funny dynamically.

List of commands for top.

command key

response

h or ?

To show help.

f

To set/reset display field.

o

To reorder display field.

F

To set sort key field.

k

To kill a process.

r

To renice a process.

q

To quit the top command.

Kill a process

Use the kill(1) command to kill (or send a signal to) a process by the process ID.

Use the killall(1) command to do the same by the process command name.

List of frequently used signals for kill command.

signal value

signal name

function

1

HUP

restart daemon

15

TERM

normal kill

9

KILL

kill hard

Schedule tasks once

Run the at(1) command to schedule a one-time job:

$ echo 'command -args'| at 3:40 monday

Schedule tasks regularly

Use cron(8) to schedule tasks regularly. See crontab(1) and crontab(5).

Run the command "crontab -e" to create or edit a crontab file to set up regularly scheduled events.

Example of a crontab file:

# use /bin/sh to run commands, no matter what /etc/passwd says
SHELL=/bin/sh
# mail any output to paul, no matter whose crontab this is
MAILTO=paul
# Min Hour DayOfMonth Month DayOfWeek command (Day... are OR'ed)
# run at 00:05, every day
5  0  *  * *   $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
# run at 14:15 on the first of every month -- output mailed to paul
15 14 1  * *   $HOME/bin/monthly
# run at 22:00 on weekdays(1-5), annoy Joe. % for newline, last % for cc:
0 22 *   * 1-5 mail -s "It's 10pm" joe%Joe,%%Where are your kids?%.%%
23 */2 1 2 *   echo "run 23 minutes after 0am, 2am, 4am ..., on Feb 1"
5  4 *   * sun echo "run at 04:05 every sunday"
# run at 03:40 on the first Monday of each month
40 3 1-7 * *   [ "$(date +%a)" == "Mon" ] && command -args

Alt-SysRq

Insurance against system malfunction is provided by the kernel compile option "Magic ?SysRq key" (SAK key) which is now the default for the Debian kernel. Pressing Alt-?SysRq followed by one of the following keys does the magic of rescuing control of the system:

List of SAK command keys.

key following Alt-?SysRq

function

r

Unraw restores the keyboard after things like X crashes.

0

Changing the console loglevel to 0 reduces error messages.

k

SAK (system attention key) kills all processes on the current virtual console.

e

Send a SIGTERM to all processes, except for init.

i

Send a SIGKILL to all processes, except for init.

s

Sync all mounted filesystems.

u

Remount all mounted filesystems read-only (umount).

b

Reboot the system without syncing or unmounting.

The combination of "Alt-?SysRq s", "Alt-?SysRq u", and "Alt-?SysRq r" is good for getting out of really bad situations.

See /usr/share/doc/linux-doc-2.6.*/Documentation/sysrq.txt.gz .

<!> The Alt-?SysRq feature may be considered a security risk by allowing users access to root-privileged functions. Placing "echo 0 >/proc/sys/kernel/sysrq" in /etc/rc.local or "kernel.sysrq = 0" in /etc/sysctl.conf will disable the Alt-?SysRq feature.

{i} From SSH terminal etc., you can use the Alt-?SysRq feature by writing to the /proc/sysrq-trigger. For example, "echo s > /proc/sysrq-trigger; echo u > /proc/sysrq-trigger" from the root shell prompt will sync and umount all mounted filesystems.

The kernel

Debian distributes modularized Linux kernel as packages for supported architectures.

Linux kernel 2.6

There are few notable features on Linux kernel 2.6 compared to 2.4.

Kernel headers

Most normal programs don't need kernel headers and in fact may break if you use them directly for compiling. They should be compiled against the headers in /usr/include/linux and /usr/include/asm provided by the libc6-dev package (created from the glibc source package) on the Debian system.

(!) For compiling some kernel-specific programs such as the kernel modules from the external source and the automounter daemon (amd), you must include path to the corresponding kernel headers, e.g. -I/usr/src/linux-particular-version/include/ , to your command line. The module-assistant package helps users to build and install module package(s) easily for one or more custom kernels with the m-a(8) command.

Kernel and module compile

Debian has its own method of compiling the kernel and related modules.

List of key packages to be installed for the kernel recompilation on the Debian system

1

2

3

package

popcon

size

description

build-essential

-

-

essential packages for building Debian packages: make, gcc, ...

bzip2

-

-

compress and decompress utilities for bz2 files

libncurses5-dev

-

-

developer's libraries and docs for ncurses

git-core

-

-

git: distributed revision control system used by the Linux kernel

fakeroot

-

-

provide fakeroot environment for building package as non-root

initramfs-tools

-

-

tool to build an initramfs (Debian specific)

kernel-package

-

-

tool to build Linux kernel packages (Debian specific)

module-assistant

-

-

tool to help build module packages (Debian specific)

devscripts

-

-

helper scripts for a Debian Package maintainer (Debian specific)

linux-tree-2.6.*

-

-

Linux kernel source tree for building Debian kernel images (Debian specific)

If you use initrd in @{@stagecthebootloader@}@, make sure to read the related information in initramfs-tools(8), update-initramfs(8), mkinitramfs(8) and initramfs.conf(5).

/!\ Do not put symlinks to the directories in the source tree (e.g. /usr/src/linux*) from /usr/include/linux and /usr/include/asm when compiling the Linux kernel source. (Some outdated documents suggest this.)

(!) When compiling the latest Linux kernel on the Debian stable system, the use of backported latest tools from the Debian unstable may be needed.

Kernel source compile: Debian standard method

The Debian standard method for compiling kernel source to create a custom kernel package uses make-kpkg(1) command. The official documentation is in (the bottom of) /usr/share/doc/kernel-package/README.gz. See kernel-pkg.conf(5) and kernel-img.conf(5) for customization.

Here is an example for amd64 system:

# aptitude install linux-tree-<version>
$ cd /usr/src
$ tar -xjvf linux-source-<version>.tar.bz2
$ cd linux-source-<version>
$ cp /boot/config-<oldversion> .config
$ make menuconfig
 ...
$ make-kpkg clean
$ fakeroot make-kpkg --append_to_version -amd64 --initrd --revision=rev.01 kernel_image modules_image
$ cd ..
# dpkg -i linux-image*.deb

<!> When you intend to create a non-modularized kernel compiled only for one machine, invoke make-kpkg command without "--initrd" option since initrd is not used. Invocation of "make oldconfig" and "make dep" are not required since "make-kpkg kernel_image" invokes them.

Module source compile: Debian standard method

The Debian standard method for creating and installing a custom module package for a custom kernel package uses module-assistant(8) command and module-source packages. For example, following will build the unionfs kernel module package and installs it.

$ sudo aptitude install module-assistant
...
$ sudo aptitude install unionfs-source unionfs-tools unionfs-utils
$ sudo m-a update
$ sudo m-a prepare
$ sudo m-a auto-install unionfs
...
$ sudo apt-get autoremove

Kernel source compile: classic method

You can still build [http://www.kernel.org/ Linux kernel from the pristine sources] with the classic method. You must take care the details of the system configuration manually.

$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-<version>.tar.bz2
$ tar -xjvf linux-<version>.tar.bz2
$ cd linux-<version>
$ cp /boot/config-<version> .config
$ make menuconfig
 ...
$ make dep; make bzImage
$ make modules
# cp ./arch/x86_64/boot/bzImage /boot/vmlinuz-<version>
# make modules_install
# depmod -a
# update-initramfs -c -k <version>

Non-free hardware drivers

Although most of hardware drivers are available as free software and as a part of the Debian system, you may need to load some non-free external drivers to support some hardwares, such as Winmodem, on your system.

Check pertinent resources:

The chroot

The chroot(8) program is most basic way to run different instances of the GNU/Linux environment on a single system simultaneously without rebooting. I will explain simple [http://en.wikipedia.org/wiki/Chroot chroot] systems in the following as examples.

For serious chroot setup with the detail configuration, please consider to use the specialized schroot package.

Run a different Debian distribution with chroot

A chroot Debian environment can easily be created by the debootstrap or cdebootstrap command.

For example, the following will create a sid chroot on /sid-root while having fast Internet access:

main # debootstrap sid /sid-root http://ftp.debian.org/debian/

main # echo "proc-sid    /sid-root/proc     proc   none 0 0" >> /etc/fstab
main # echo "devpts-sid  /sid-root/dev/pts  devpts defaults 0 0" >> /etc/fstab
main # echo "/etc/passwd /sid-root/etc/passwd ext3 bind,defaults 0 0" >> /etc/fstab
main # echo "/etc/shadow /sid-root/etc/shadow ext3 bind,defaults 0 0" >> /etc/fstab
main # echo "/etc/group  /sid-root/etc/group  ext3 bind,defaults 0 0" >> /etc/fstab
main # echo "/etc/hosts  /sid-root/etc/hosts  ext3 bind,defaults 0 0" >> /etc/fstab
main # mount -a
main # chroot /sid-root /bin/bash
chroot # cd /dev; /sbin/MAKEDEV generic ; cd -
chroot # vi /etc/apt/sources.list

chroot # aptitude update
...
chroot # aptitude install locales
...

Do you want to continue? [Y/n/?] y
chroot # aptitude install mc vim
...
Do you want to continue? [Y/n/?] y
...
chroot # exit
main #

At this point you should have a fully working Debian sid system, where you can play around without fear of affecting your main Debian installation.

This debootstrap trick can also be used to [http://www.debian.org/releases/stable/installmanual install Debian] to a system without using a Debian install disk, but instead from another GNU/Linux distribution.

Setting up login for chroot

Typing "chroot /sid-root /bin/bash" is easy, but it retains all sorts of environment variables that you may not want, and has other issues. A much better approach is to run another login process on a separate virtual terminal where you can log in to the chroot directly.

Since on default Debian systems tty1 to tty6 run Linux consoles and tty7 runs the X Window System, let's set up tty8 for a chrooted console as an example. After creating a chroot system, type from the root shell of the main system:

main # echo "8:23:respawn:/usr/sbin/chroot /sid-root /sbin/getty 38400 tty8"  >> /etc/inittab
main # init q

Setting up X for chroot

You want to run the latest X and GNOME safely in your chroot? That's entirely possible! The following example will make GDM run on virtual terminal vt9.

First install a chroot system. From the root of the main system, copy key configuration files to the chroot system.

main # cp /etc/X11/xorg.conf /sid-root/etc/X11/xorg.conf
main # chroot /sid-root
chroot # cd /dev; /sbin/MAKEDEV generic ; cd -
chroot # aptitude install gdm gnome x-window-system
chroot # vim /etc/gdm/gdm.conf

chroot # /etc/init.d/gdm start

Now you can easily switch back and forth between full X environments in your chroot and your main system just by switching between Linux virtual terminals; e.g. by using Ctrl-Alt-F7 and Ctrl-Alt-F9. Have fun!

Run other distributions with chroot

A chroot environment for another Linux distribution can easily be created. You install a system into separate partitions using the installer of the other distribution. If its root partition is in /dev/hda9:

main # cd / ; mkdir /other-dist
main # mount -t ext3 /dev/hda9 /other-dist
main # chroot /other-dist /bin/bash

Build packages under chroot

There is a more specialized chroot package, pbuilder, which constructs a chroot system and builds a package inside the chroot. It is an ideal system to use to check that a package's build-dependencies are correct, and to be sure that unnecessary and wrong build dependencies will not exist in the resulting package.

Other virtualization tools

There are several system [http://en.wikipedia.org/wiki/Virtualization virtualization] and [http://en.wikipedia.org/wiki/Emulator emulation] related packages in Debian beyond simple [http://en.wikipedia.org/wiki/Chroot chroot].

List of virtualization tools

1

2

3

package

pocon

size

description

schroot

-

-

Specialized tool for executing Debian binary packages in chroot

sbuild

-

-

Tool for building Debian binary packages from Debian sources

pbuilder

-

-

Personal package builder for Debian packages

debootstrap

-

-

Bootstrap a basic Debian system (written in sh)

cdebootstrap

-

-

Bootstrap a Debian system (written in C)

rootstrap

-

-

A tool for building complete Linux filesystem images

user-mode-linux

-

-

User-mode Linux (kernel)

xen-tools

-

-

Tools to manage debian XEN virtual server

bochs

-

-

IA-32 PC emulator

qemu

-

-

Fast generic processor emulator

virtualbox-ose

-

-

A x86 virtualization solution on i386 and amd64

wine

-

-

Windows API Implementation (standard suite)

dosbox

-

-

A x86 emulator with Tandy/Herc/CGA/EGA/VGA/SVGA graphics, sound and DOS

See Wikipedia article [http://en.wikipedia.org/wiki/Comparison_of_virtual_machines] for detail comparison of different virtualization solutions.