Do not use Edit(GUI) button.

?TableOfContents(4)

Copyright 2007, 2008 Osamu Aoki GPL, (Please agree to GPL, GPL2, and any version of GPL which is compatible with DSFG if you update any part of wiki page)

I welcome your contributions to update the wiki pages. You must follow these rules:

Programming

I provide some pointers for people to learn programming on the Debian system enough to trace the packaged source code. Here are notable packages and corresponding documentation packages for programing.

List of packages to help programing.

1

2

3

package

popcon

size

documentation

autoconf

"info autoconf" provided by autoconf-doc

automake

"info automake" provided by automake1.10-doc

bash

"info bash" provided by bash-doc

bison

"info bison" provided by bison-doc

cpp

"info cpp" provided by cpp-doc

ddd

"info ddd" provided by ddd-doc

exuberant-ctags

exuberant-ctags(1)

flex

"info flex" provided by flex-doc

gawk

"info gawk" provided by gawk-doc

gcc

"info gcc" provided by gcc-doc

gdb

"info gdb" provided by gdb-doc

gettext

"info gettext" provided by gettext-doc

gfortran

"info gfortran" provided by gfortran-doc

glade

Help provided via menu

glade-gnome

Help provided via menu

libc6

"info libc" provided by glibc-doc and glibc-doc-reference

make

"info make" provided by make-doc

mawk

"man mawk"

perl

"man 1 perl" and html pages provided by perl-doc and perl-doc-html

python

"man python" and html pages provided by python-doc

tcl8.4

"man tcl" and detail manual pages provided by tcl8.4-doc

tk8.4

"man tk" and detail manual pages provided by tk8.4-doc

ruby

"man ruby" and interactive reference provided by ri

vim

Help(F1) menu provided by vim-doc

susv2

Fetch "[http://www.unix.org/version2/ The Single Unix Specifications v2]"

susv3

Fetch "[http://www.unix.org/version3/ The Single Unix Specifications v3]"

Online references are available by typing man name after installing manpages and manpages-dev packages. Online references for the GNU tools are available by typing info program_name after installing the pertinent documentation packages. You may need to include the contrib and non-free archives in addition to the main archive since some GFDL documentations are not considered to be DSFG compliant.

{i} [http://www.99-bottles-of-beer.net/ Code examples of creating "Song 99 Bottles of Beer"] should give you good idea of practically all the programming languages.

<!> You should install software programs directly compiled from source into /usr/local or /opt to avoid collision with system programs.

/!\ Do not use "test" as the name of an executable test file. "test" is a shell built-in.

The shell script

The [http://en.wikipedia.org/wiki/Shell_script shell script] is a text file with the execution bit set and contains the commands in the following format.

... command lines ...

The first line specifies the shell interpreter which read and execute this file contents.

Reading shell scripts is the best way to understand how a Unix-like system works. Here, I give some pointers and reminders for shell programming. See "Shell Mistakes" (http://www.greenend.org.uk/rjk/2001/04/shell.html) to learn from mistakes.

Unlike shell interactive mode (see: @{@thesimpleshellcommand@}@ and @{@unixliketextprocessing@}@), parameters, conditionals, and loops are used frequently.

POSIX shell compatibility

Many system scripts may be interpreted by any one of [http://en.wikipedia.org/wiki/POSIX POSIX] shells (see @{@listofshellprograms@}@). For this shell bash may be used, but for the sake of speed, dash is recommended. You can switch actual shell program to be used by changing symlink for /bin/sh.

Avoid writing a shell script with "bashisms" or "zshisms" to make it portable among all POSIX shells:

List of bashizms.

Good: POSIX

Avoid: bashism

if [ "$foo" = "$bar" ] ; then ...

if [ "$foo" == "$bar" ] ; then ...

diff -u file.c.orig file.c

diff -u file.c{.orig,}

mkdir /foobar /foobaz

mkdir /foo{bar,baz}

octal format: "\377"

hexadecimal format: "\xff"

The "echo" command must be used with care since its implementation differs among shell built-in commands and external command.

(!) Although "-n option is not really POSIX syntax, it is generally accepted.

{i} Use the "printf" command instead of the "echo" command if you need to embed escape sequences in the output string.

Shell parameters

Special shell parameters are frequently used in the shell script:

List of shell parameters.

shell parameter

value

$0

name of the shell or shell script

$1

first(1) shell argument

$9

ninth(9) shell argument

$#

number of positional parameters

"$*"

"$1 $2 $3 $4 ... "

"$@"

"$1" "$2" "$3" "$4" ...

$?

exit status of the most recent command

$$

PID of this shell script

$!

PID of most recently started background job

Basic parameter expansions to remember:

List of shell parameter expansions.

parameter expression form

If var is set

If var is not set

 ${var:-string} 

 $var 

string

 ${var:+string} 

string

null

 ${var:=string} 

 $var 

string (and run var=string)

 ${var:?string} 

 $var 

(echo string to stderr and exit with error)

Here, the colon ':' in all of these operators is actually optional.

List of key shell parameter substitutions.

parameter substitution form

Result

 ${var%suffix} 

Remove smallest suffix pattern

 ${var%%suffix} 

Remove largest suffix pattern

 ${var#prefix} 

Remove smallest prefix pattern

 ${var##prefix} 

Remove largest prefix pattern

Shell conditionals

Each command returns an exit status which can be used for conditional expressions:

Note that the use here of a 0 value to mean "true" differs from the usual convention in some other areas of computing. Also, `[' is the equivalent of the test command, which evaluates its arguments up to `]' as a conditional expression.

Basic conditional idioms to remember are:

Here trailing "|| true" was needed to ensure this shell script will not exit at this line accidentally when shell is invoked with -e flag.

List of file comparison operators in the conditional expression.

equation

value

-e <file>

True if <file> exists.

-d <file>

True if <file> exists and is a directory.

-f <file>

True if <file> exists and is a regular file.

-w <file>

True if <file> exists and is writable.

-x <file>

True if <file> exists and is executable.

<file1> -nt <file2>

True if <file1> is newer than <file2>. (modification).

<file1> -ot <file2>

True if <file1> is older than <file2>. (modification).

<file1> -ef <file2>

True if they are the same device and inode number.

List of string comparison operators in the conditional expression.

equation

value

-z <str>

True if the length of <str> is zero.

-n <str>

True if the length of <str> is non-zero.

<str1> = <str2>

True if <str1> and <str2> are equal.

<str1> != <str2>

True if <str1> and <str2> are not equal.

<str1> < <str2>

True if <str1> sorts before <str2>. (locale dependent)

<str1> > <str2>

True if <str1> sorts after <str2>. (locale dependent)

Arithmetic integer comparison operators in the conditional expression are -eq, -ne, -lt, -le, -gt, and -ge.

Shell loops

There are several loop idioms to use in POSIX shell:

(!) The C-language like numeric iteration can be realized by using the seq(1) command as the "word" generator.

The shell command-line processing sequence

The shell processes a script as following sequence:

Single quotes within double quotes have no effect.

Executing set -x in the shell or invoking the shell with -x option make the shell to print all of commands executed. This is quite handy for debugging.

Script snippets for looping over each file

When writing script to looping over each file matching *.ext, there are several options. The following examples are written as an part of shell script file.

You must replace each line break with a semicolon (excluding ones escaped by preceding \ which is replaced with ) for the interactive shell prompt.

The above examples are written to ensures proper handling of funny file names such as ones containing spaces.

Utility programs for shell script

In order to make your shell program as portable as possible across Debian system, it is good idea to limit utility programs used within Essential programs listed by "aptitude search ~E" as much as possible.

The coreutils, bsdutils, and debianutils packages contain many useful small utilities.

Shell script dialog

The user interface of a simple shell program can be improved from dull interaction by the "echo" and "read" commands to more interactive one by using one of the so-called dialog program etc.

List of user interface programs.

1

2

3

package

popcon

size

function

xbase-clients

29862

-

The xmessage(1) displays a message or query in a window (X)(etch)

x11-utils

25

-

The xmessage(1) , , (lenny)

whiptail

26659

-

displays user-friendly dialog boxes from shell scripts. (newt)

dialog

4518

-

displays user-friendly dialog boxes from shell scripts. (ncurses)

zenity

3892

-

display graphical dialog boxes from shell scripts. (gtk2.0)

xdialog

210

-

X11 replacement for the text utility dialog. (gtk1.2)

gtkdialog

28

-

GUI-creation command-line utility based on GTK+ library. (gtk2.0+glade2)

ssft

14

-

Shell Scripts Frontend Tool. (wrapper for zenity, kdialog, and dialog with gettext)

gettext

6671

-

The gettext.sh for translate message

Shell script example with zenity

Here is a simple script which creates ISO image with RS02 data supplemented by dvdisaster(1):

# gmkrs02 : Copyright (C) 2007 Osamu Aoki <osamu@debian.org>, Public Domain
#set -x
error_exit()
{
  echo "$1" >&2
  exit 1
}
# Initialize variables
DATA_ISO="$HOME/Desktop/iso-$$.img"
LABEL=$(date +%Y%m%d-%H%M%S-%Z)
if [ $# != 0 ] && [ -d "$1" ]; then
  DATA_SRC="$1"
else
  # Select directory for creating ISO image from folder on desktop
  DATA_SRC=$(zenity --file-selection --directory  \
    --title="Select the directory tree root to create ISO image") \
    || error_exit "Exit on directory selection"
fi
# Check size of archive
xterm -T "Check size $DATA_SRC" -e du -s $DATA_SRC/*
SIZE=$(($(du -s $DATA_SRC | awk '{print $1}')/1024))
if [ $SIZE -le 520 ] ; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for CD backup:\\n $SIZE MB"
elif [ $SIZE -le 3500 ]; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for DVD backup :\\n $SIZE MB"
else
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is too big to backup : $SIZE MB"
  error_exit "The data size is too big to backup :\\n $SIZE MB"
fi
# only xterm is sure to have working -e option
# Create raw ISO image
rm -f "$DATA_ISO" || true
xterm -T "genisoimage $DATA_ISO" \
  -e genisoimage -r -J -V "$LABEL" -o "$DATA_ISO" "$DATA_SRC"
# Create RS02 supplemental redundancy
xterm -T "dvdisaster $DATA_ISO" -e  dvdisaster -i "$DATA_ISO" -mRS02 -c
zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
  --text="ISO/RS02 data ($SIZE MB) \\n created at: $DATA_ISO"
# EOF

You may wish to create launcher on the desktop with command set something like "/usr/local/bin/gmkrs02 %d".

Make

[http://en.wikipedia.org/wiki/Make_(software) Make] is a utility to maintain groups of programs. Upon execution of make(1), make read the rule file, Makefile, and updates a target if it depends on prerequisite files that have been modified since the target was last modified, or if the target does not exist. The execution of these updates may occur concurrently.

The rule file syntax is :

target: [ prerequisites ... ]
 [TAB]  command1
 [TAB]  -command2 # ignore errors
 [TAB]  @command3 # suppress echoing

Here " [TAB] " is a TAB code. Each line is interpreted by the shell after make variable substitution. Use \ at the end of a line to continue the script. Use "$$" to enter "$" for environment values for a shell script.

Implicit rules for the target and prerequisites can be written, for example, as:

%.o: %.c header.h

Here, the target contains the character "%" (exactly one of them). The "%" can match any nonempty substring in the actual target filenames. The prerequisites likewise use "%" to show how their names relate to the actual target name.

List of make automatic variables.

automatic variable

value

$@

target

$<

first prerequisite

$?

all newer prerequisites

$^

all prerequisites

$*

% matched stem in the target pattern

List of make variable expansions.

variable expansion

effects

foo1 := bar

One-time expansion

foo2  = bar

Recursive expansion

foo3 += bar

Append

Run make -p -f/dev/null to see automatic internal rules.

C

You can set up proper environment to compile programs written in the [http://en.wikipedia.org/wiki/C_(programming_language) C programming language] by:

# aptitude install glibc-doc manpages-dev libc6-dev gcc build-essential

The libc6-dev package, i.e., GNU C Library, provides [http://en.wikipedia.org/wiki/C_standard_library C standard library] which is collection of header files and library routines used by the C programming language.

References for C:

Simple C program (gcc)

A simple example to compile example.c with a library libm into an executable run_example: Debian New Maintainers' Guide

$ cat > example.c << EOF
#include <stdio.h>
#include <math.h>
#include <string.h>

int main(int argc, char **argv, char **envp){
        double x;
        char y[11];
        x=sqrt(argc+7.5);
        strncpy(y, argv[0], 10); /* prevent buffer overflow */
        y[10] = '\0'; /* fill to make sure string ends with '\0' */
        printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]);
        return 0;
}
EOF
$ gcc -Wall -g -o run_example example.c -lm
$ ./run_example
        1, 2.915, ./run_exam,     (null)
$ ./run_example 1234567890qwerty
        2, 3.082, ./run_exam, 1234567890qwerty

Here, -lm is needed to link library libm for sqrt(). The actual library is in /lib/ with filename libm.so.6, which is a symlink to libm-2.1.3.so.

Look at the last parameter in the output text. There are more than 10 characters even though %10s is specified.

The use of pointer memory operation functions without boundary checks, such as sprintf and strcpy, is deprecated to prevent buffer overflow exploits that leverage the above overrun effects. Instead, use snprintf and strncpy.

Debugging with gdb

In order to be a good Debian user, you must be able to produce meaningful bug report using [http://en.wikipedia.org/wiki/Debugger debugger]. The fist step is to install gdb:

# aptitude install gdb gdb-doc build-essential devscripts

Good tutorial of gdb is provided by "info gdb" or found [http://www.unknownroad.com/rtfm/gdbtut/gdbtoc.html elsewhere on the web].

Basic gdb execution

Here is a simple example of using gdb(1) on a program compiled with the -g option to produce debugging information.

$ gdb program
(gdb) b 1                # set break point at line 1
(gdb) run args           # run program with args
(gdb) next               # next line
...
(gdb) step               # step forward
...
(gdb) p parm             # print parm
...
(gdb) p parm=12          # set value to 12
...
(gdb) quit

{i} Many gdb commands can be abbreviated. Tab expansion works as in the shell.

Debugging the Debian package

Since all installed binaries should be stripped on the Debian system by default, most debugging symbols are removed in the normal package. In order to debug Debian packages with make gdb, corresponding *-dbg packages need to be installed (e.g. libc6-dbg in the case of libc6).

If the package to be debugged does not provide *-dbg package, you need to install rebuild packages as follows:

$ mkdir /path/new ; cd /path/new
$ sudo aptitude update
$ sudo aptitude dist-upgrade
$ sudo aptitude install fakeroot devscripts build-essential
$ sudo apt-get build-dep source_package_name
$ apt-get source package_name
$ cd package_name*
 ... fix bugs if needed
$ dch -i
 ... bump version
$ export DEB_BUILD_OPTIONS=nostrip,noopt
$ debuild
$ cd ..
$ sudo debi package_name*.changes 

{i} Use of non-official different package version such as one appended with ~pre1 is good idea when recompiling locally.

You need to check the build scripts of the package and ensure to use CFLAGS=-g -Wall for compiling binaries.

Obtaining backtrace

When you encounter program crash, reporting bug report with cut-and-pasted backtrace information is a good idea.

The backtrace can be obtained by the following steps:

In case of program freeze, you can crash the program by pressing Ctrl-C in the terminal running gdb to obtain gdb prompt.

{i} Often, you will see a backtrace where one or more of the top lines is in malloc() or g_malloc(). When this happens, chances are your backtrace isn't very useful. The easiest way to find some useful information is to set the environment variable MALLOC_CHECK_ to a value of 2. You can do this while running gdb by doing this:

 $ MALLOC_CHECK_=2 gdb hello

Advanced gdb commands

List of advanced gdb commands

objective

commands

To get a backtrace for all threads for multi-threaded program.

(gdb) thread apply all bt

To get parameters came on the stack of function calls.

(gdb) bt full

To get a backtrace and parameters as the combination of the preceding options.

(gdb) thread apply all bt full

To get them for top 10 calls to cut off irrelevant output.

(gdb) thread apply all bt full 10

To write log of gdb output to a file (the default is gdb.txt).

(gdb) set logging on

Debugging X Errors

If the Gnome program has received an X error; i.e. you see a message of the form:

The program 'preview1' received an X Window System error.

then you can try running the program with "--sync", and break on the "gdk_x_error" function in order to obtain a backtrace.

Check dependency on libraries

Use ldd to find out a program's dependency on libraries:

$ ldd /bin/ls
        librt.so.1 => /lib/librt.so.1 (0x4001e000)
        libc.so.6 => /lib/libc.so.6 (0x40030000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x40153000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

For ls to work in a chrooted environment, the above libraries must be available in your chrooted environment.

The following commands will also be useful:

Debugging with memory leak detection tools

There are several memory leak detection tools available in Debian.

List of memory leak detection tools

1

2

3

package

popcon

size

description

libc6-dev

-

-

mtrace(1): malloc debugging functionality in glibc

valgrind

-

-

memory debugger and profiler

kmtrace

-

-

KDE memory leak tracer using glibc's "mtrace(1)"

alleyoop

-

-

Gnome front-end to the Valgrind memory checker

electric-fence

-

-

malloc(3) debugger

ccmalloc

-

-

memory profiler/debugger

leaktracer

-

-

memory-leak tracer for C++ programs

libdmalloc5

-

-

debug memory allocation library

mpatrolc2

-

-

library for debugging memory allocations

Disassemble binary

You can disassemble binary code with objdump(1). For example:

$  objdump -m i386 -b binary -D /usr/lib/grub/x86_64-pc/stage1

(!) gdb(1) may be used to disassemble code interactively.

Flex -- a better Lex

[http://en.wikipedia.org/wiki/Flex_lexical_analyser Flex] is a a [http://en.wikipedia.org/wiki/Lex_programming_tool Lex]-compatible fast [http://en.wikipedia.org/wiki/Lexical_analysis lexical analyzer] generator.

Tutorial for flex(1) can be found in "info flex".

You need to provide your own main() and yywrap(), or your program.l should look like this to compile without a library (yywrap is a macro; %option main turns on %option noyywrap implicitly):

%option main
%%
.|\n    ECHO ;
%%

Alternatively, you may compile with the -lfl linker option at the end of your cc command line (like AT&T-Lex with -ll). No %option is needed in this case.

Bison -- a better Yacc

Several packages provide a [http://en.wikipedia.org/wiki/Yacc Yacc]-compatible lookahead [http://en.wikipedia.org/wiki/LR_parser LR parser] or [http://en.wikipedia.org/wiki/LALR_parser LALR parser] generator in Debian:

List of Yacc-compatible LALR parser generators

1

2

3

package

popcon

size

description

bison

-

-

[http://en.wikipedia.org/wiki/GNU_bison GNU LALR parser generator]

byacc

-

-

The Berkeley LALR parser generator

btyacc

-

-

Backtracking parser generator based on byacc

Tutorial for bison(1) can be found in "info bison".

You need to provide your own main() and yyerror(). main() calls yyparse() which calls yylex(), usually created with Flex.

%%

%%

Autoconf

[http://en.wikipedia.org/wiki/Autoconf Autoconf] is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of Unix-like systems using the entire GNU build system.

autoconf(1) produces the configuration script configure. configure automatically creates a customized Makefile using the Makefile.in template.

Compile and install a program

/!\ Do not overwrite system files with your compiled programs when installing them.

Debian does not touch files in /usr/local/ or /opt. So if you compile a program from source, install it into /usr/local/ so it will not interfere with Debian.

$ cd src
$ ./configure --prefix=/usr/local
$ make
$ make install # this puts the files in the system

Uninstall program

If you still have the source and if it uses autoconf/automake and if you can remember how you configured it:

$ ./configure ''all-of-the-options-you-gave-it''
# make uninstall

Alternatively, if you are absolutely sure that the install process puts files only under /usr/local/ and there is nothing important there, you can erase all its contents by:

# find /usr/local -type f -print0 | xargs -0 rm -f

If you are not sure where files are installed, you should consider using checkinstall, which provides a clean path for the uninstall.

Perl short script madness

Although any [http://en.wikipedia.org/wiki/AWK AWK] scripts can be automatically rewritten in [http://en.wikipedia.org/wiki/Perl Perl] using a2p(1), one-liner AWK scripts are best converted to one-liner perl scripts manually. For example

awk '($2=="1957") { print $3 }' |

is equivalent to any one of the following lines:

perl -ne '@f=split; if ($f[1] eq "1957") { print "$f[2]\n"}' |

perl -ne 'if ((@f=split)[1] eq "1957") { print "$f[2]\n"}' |

perl -ne '@f=split; print $f[2] if ( $f[1]==1957 )' |

perl -lane 'print $F[2] if $F[1] eq "1957"' |

perl -lane 'print$F[2]if$F[1]eq+1957' |

The last one is a riddle. It took advantage of the Perl features that

See perlrun(1) for the command-line options. For more crazy Perl scripts, [http://perlgolf.sourceforge.net Perl Golf] may be interesting.

Web

Basic interactive dynamic web pages can be made as follows:

For security reasons it is better not to hand craft new hacks for parsing CGI parameters. There are established modules for them in Perl and Python. [http://www.php.net/ PHP] comes with these functionalities. When client data storage is needed, cookies are used. When client side data processing is needed, javascript is frequently used.

For more, see [http://hoohoo.ncsa.uiuc.edu/cgi/ The Common Gateway Interface], [http://www.apache.org/ The Apache Software Foundation], and [http://www.mozilla.org/js/ ?JavaScript].

Searching "CGI tutorial" on Google by typing encoded URL [http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial] directly to the browser address is a good way to see the CGI script in action on the Google server.

Static code analysis tools

There are [http://en.wikipedia.org/wiki/Lint_programming_tool lint] like tools for [http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis static code analysis]:

List of tools for static code analysis

1

2

3

package

popcon

size

description

splint

-

-

A tool for statically checking C programs for bugs

flawfinder

-

-

A tool to examine C/C++ source code and looks for security weaknesses

perl

-

-

The perl package has internal code static checker: B::Lint(3perl)

pylint

-

-

A python code static checker

jlint

-

-

A Java program checker

weblint-perl

-

-

A syntax and minimal style checker for HTML

linklint

-

-

A fast link checker and web site maintenance tool

libxml2-utils

-

-

The libxml2-utils package provides xmllint(1) command to validate XML files

The source code translation

There are program to convert source codes:

List of source code translation tools.

1

2

3

package

popcon

size

keyword

function

perl

37451

-

AWK->PERL

The a2p converter from AWK to PERL.

f2c

123

-

FORTRAN->C

The converter from A FORTRAN 77 to C/C++.

protoize

21

-

ANSI C

Create/remove ANSI prototypes from C code.

intel2gas

8

-

intel->gas

The converter from NASM (intel format) to the GNU Assembler (GAS).

Making Debian package

If you want to make a Debian package, read:

There are packages such as dh-make, dh-make-perl, etc., which help packaging.