Do not use Edit(GUI) button.

?TableOfContents(4)

Copyright 2007, 2008 Osamu Aoki GPL, (Please agree to GPL, GPL2, and any version of GPL which is compatible with DSFG if you update any part of wiki page)

I welcome your contributions to update the wiki pages. You must follow these rules:

• Do not use Edit(GUI) button of MoinMoin.

• You can update anytime for:
  • grammar errors
  • spelling errors
  • moved URL location
  • package name transition adjustment (emacs23 etc.)
  • clearly broken script.
• Before updating real contents:

  • Read "[http://wiki.debian.org/DebianReference/Test Guide for contributing to Debian Reference]".

Data conversion

Standard based tools are in very good shape but support for proprietary data formats are limited.

Text data conversion tools

Following packages for the text data conversion caught my eyes:

Basics of encoding

The default text data format on the Debian system uses the [http://en.wikipedia.org/wiki/UTF-8 UTF-8] encoding of the Unicode character for "LANG=xx_YY.UTF-8" (see: @{@langvariable@}@ and @{@thelocale@}@). For LANG=C", the [http://en.wikipedia.org/wiki/ASCII ASCII] character set is used instead.

The [http://en.wikipedia.org/wiki/UTF-8 UTF-8] encoding system is a multibyte code sequence and uses code points smartly. ASCII data, which consist only with 7-bit range codes, are always UTF-8 data.

For character sets which fit in single byte such as [http://en.wikipedia.org/wiki/ASCII ASCII] and [http://en.wikipedia.org/wiki/ISO/IEC_8859 ISO-8859] character sets, the [http://en.wikipedia.org/wiki/Character_encoding character encoding] means almost the same thing as the character set.

For character sets with many characters such as [http://en.wikipedia.org/wiki/JIS_X_0213 JIS X 0213] for Japanese or [http://en.wikipedia.org/wiki/Universal_Character_Set Universal Character Set (UCS, Unicode, ISO-10646-1)] for practically all languages, there are many encoding schemes to fit them into the sequence of the byte data, such as [http://en.wikipedia.org/wiki/EUC EUC] and [http://en.wikipedia.org/wiki/ISO/IEC_2022 ISO/IEC 2022 (also known as JIS X 0202)] for Japanese, or [http://en.wikipedia.org/wiki/UTF-8 UTF-8] and [http://en.wikipedia.org/wiki/UTF-32/UCS-4 UTF-32/UCS-4] for Unicode. So there are clear differentiation between the character set and the character encoding.

The [http://en.wikipedia.org/wiki/Code_page code page] is used as the synonym to the character encoding tables for the vender specific ones.

The vender specific old non-UTF-8 encoding systems tend to have minor but annoying diferences on some charactrs such as graphic ones for many countries. The deplyment of the UTF-8 system by the modern OSs practically solved these conflicting encoding issues.

To convert a text file with iconv

The iconv command converts the encoding of characters:

$ iconv -f encoding1 -t encoding2 input.txt >output.txt

The encoding name part may be normalized internally to achieve cross platform compatibility by removing all - and by converting all characters into lower case. It is also referred as character code set. The supported encodings can be checked by the "iconv -l" command.

Since this iconv command is provided as a part of the libc6 package, it is always available.

The recode command may be used too and offers more than the combined functionality of the iconv, fromdos, todos, frommac, and tomac commands. For more, see pertinent description in the info recode.

Please note most encoding systems share the same code with ASCII for the 7 bit characters. But there are some exceptions. If you are converting old Japanese C programs and URLs data from the casually-called shift-JIS encoding format to UTF-8 format, use "CP932" as the encoding name instead of "shift-JIS" to get the expected results: 0x5C -> "\" and 0x7E -> "~" . Otherwise, these are converted to wrong characters.

To convert file names with iconv

Here is an example script to convert encoding of the file name from ones created under older OS to modern UTF-8 ones for the simple case.

ENCDN=iso-8859-1
for x in *;
 do
 mv "$x" $(echo "$x" | iconv -f $ENCDN -t utf-8)
done

The "$ENCDN" variable should be set by the encoding values from @{@listofencodingvauesandtheirusage@}@ .

For more complicated case, please mount disk drive containing such file names with proper encoding as the mount(8) option and copy entire disk to another disk drive mounted as UTF-8 with "cp -a" command.

EOL conversion

The text file format, specifically the end-of-line (EOL) code, is dependent on the platform:

The EOL format conversion programs, fromdos(1), todos(1), frommac(1), and tomac(1), are quite handy. The recode command is also useful.

The use of "sed -e '/\r$/!s/$/\r/'" instead of "todos" is better when you want to unify the EOL style to the MSDOS style from the mixed MSDOS and Unix style. (e.g., after merging 2 MSDOS style files with diff3.) This is because "todos" adds CR to all lines.

Some data on the Debian system, such as the wiki page data for the python-moinmoin, use MSDOS style CR-LF as the EOL code. So the above rule is just general rule.

Most editors (eg. vim, emacs, gedit, ...) can handle files in MSDOS style EOL transparently.

TAB conversion

You can expand the tab code in the text to the multiple spaces in vim using the ":retab" command.

There are few popular specialized programs to convert the tab codes:

The indent(1) from the indent package completely reformats whitespaces in the C program.

Editors with auto-conversion

Intelligent modern editors such as the vim program are quite smart and copes well with any encoding systems and any file formats. You should use these editors under the UTF-8 locale in the UTF-8 capable console for the best compatibility.

An old western European Unix text file, "u-file.txt", stored in the latin1 encoding can be edited simply with vim as:

$ vim u-file.txt

This is possible since the auto detection mechanism of the file encoding in vim assumes the UTF-8 encoding first and, if it fails, assumes it to be latin1.

An old Polish Unix text file, "pu-file.txt", stored in the latin2 encoding can be edited with vim as:

$ vim '+e ++enc=latin2 pu-file.txt'

An old Japanese unix text file, "ju-file.txt", stored in the eucJP encoding can be edited with vim as:

$ vim '+e ++enc=eucJP ju-file.txt'

An old Japanese MS-Windows text file, "jw-file.txt", stored in the so called shift-JIS encoding (more precisely: CP932) can be edited with vim as:

$ vim '+e ++enc=CP932 ++ff=dos jw-file.txt'

When a file is opened with "++enc" and "++ff" options, the "w" in the Vim command line stores it in the original format and overwrite the original file. You can also specify the saving format and the file name in the Vim command line, e.g., "w ++enc=utf8 new.txt".

Please refer to the mbyte.txt "multi-byte text support" in vim on-line help.

The emacs family of programs can perform the equivalent functions.

Plain text extraction

Following will read a web page into a text file. This is very useful when copying configurations off the Web or applying basic Unix text tools such as grep on the web page.

$ lynx -dump http://www.remote-site.com/help-info.html >textfile

Similarly, you can extract plain text data from other formats using followings:

Highlighting and formatting plain text data

XML data

[http://en.wikipedia.org/wiki/XML The Extensible Markup Language (XML)] is a markup language for documents containing structured information.

[http://xml.com/ XML.COM] has good introductory information:

• [http://www.xml.com/pub/a/98/10/guide0.html "What is XML?"]

• [http://xml.com/pub/a/2000/08/holman/index.html "What Is XSLT?"]

• [http://xml.com/pub/a/2002/03/20/xsl-fo.html "What Is XSL-FO?"]

• [http://xml.com/pub/a/2000/09/xlink/index.html "What Is XLink?"]

Basic hints for XML

XML text looks somewhat like HTML. It enables us to manage multiple formats of output for a document. One easy XML system is docbook-xsl, which is used here.

Each XML file starts with standard XML declaration:

<?xml version="1.0" encoding="UTF-8"?>

The basic syntax for one XML element is marked up as:

<name attribute="value">content</name>

XML element with empty content is marked up in the short form as:

<name attribute="value"/>

The "attribute="value"" in the above examples are optional.

The comment section in XML is marked up as:

<!-- comment -->

Other than adding markups, XML requires minor conversion to the content using predefined entities for the following character:

"<" or "&" can not be used in attributes or elements.

When SGML style user defined entities, e.g. "&some-tag:", are used, the first definition wins over others. The entity definition is expressed in "<!ENTITY some-tag "entity value">".

As long as the XML markup are done consistently with certain set of the tag name (either some data as content or attribute value), conversion to another XML is trivial task using XSLT.

XML processing

There are many tools available to process XML files such as [http://en.wikipedia.org/wiki/Extensible_Stylesheet_Language the Extensible Stylesheet Language (XSL)].

Basically, once you create well formed XML file, you can convert it to any format using Extensible Stylesheet Language for Transformation (XSLT).

Although the Extensible Stylesheet Language for Formatting Object (XSL-FO) is supposed to be solution for formatting, FOP program is not in the Debian main (yet?). So the LaTeX code is usually generated from XML using XSLT and the LaTeX system is used to create printable file such as DVI, ?PostScript, and PDF.

Since XML is subset of [http://en.wikipedia.org/wiki/SGML Standard Generalized Markup Language (SGML)], it can be processed by the extensive tools available for SGML, such as [http://en.wikipedia.org/wiki/Document_Style_Semantics_and_Specification_Language Document Style Semantics and Specification Language (DSSSL)].

The XML data extraction

You can extract HTML or XML data from other formats using followings:

For non-XML HTML files, you can convert them to XHTML which is an instance of well formed XML and can be processed by XML tools.

Once proper XML is generated, you can use XSLT technology to extract data based on the mark-up context etc.

Printable data

The Ghostscript

The core of printable data manipulation is the the Ghostscript ?PostScript interpreter. CUPS uses the Ghostscript as its backend.

The latest upstream Ghostscript from Artifex was re-licensed from AFPL to GPL and merged all the latest ESP version changes such as CUPS related ones at 8.60 release as unified release.

Merge two PS or PDF files

You can merge two PS or PDF files using the gs(1) command of the Ghostscript.

$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pswrite -sOutputFile=bla.ps -f foo1.ps foo2.ps
$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=bla.pdf -f foo1.pdf foo2.pdf

The [http://en.wikipedia.org/wiki/Portable_Document_Format Portable Document Format (PDF)], which is widely used cross-platform printable data format, is essentially the compressed PS format with few additional features and extensions.

Printable data utilities

The following packages for the printable data utilities caught my eyes:

Printing with CUPS

Both lp and lpr commands offered by [http://en.wikipedia.org/wiki/Common_Unix_Printing_System Common Unix Printing System (CUPS)] provides options for customized printing the printable data.

For printing 3 copies of a file collated:

$ lp -n 3 -o Collate=True filename

, or

$ lpr -#3 -o Collate=True filename

You can further customize printer operation by using printer option such as "-o number-up=2", "-o page-set=even", "-o page-set=odd", "-o scaling=200", "-o natural-scaling=200", etc., documented at [http://localhost:631/help/options.html Command-Line Printing and Options].

Type setting

The Unix [http://en.wikipedia.org/wiki/Troff troff] originally developed by AT&T can be used for simple type setting. It is usually used to create manpages.

[http://en.wikipedia.org/wiki/TeX TeX] created by Donald Knuth is very powerful type setting tool and is the de facto standard . [http://en.wikipedia.org/wiki/LaTeX LaTeX] originally written by Leslie Lamport enables a high-level access to the power of TeX.

roff typesetting

Traditionally, roff is the main Unix text processing system.

See roff(7), groff(7), groff(1), grotty(1), troff(1), groff_mdoc(7), groff_man(7), groff_ms(7), groff_me(7), groff_mm(7), and info groff.

A good tutorial on -me macros exists. If you have groff (1.18 or newer), find /usr/share/doc/groff/meintro.me.gz and do the following:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
     groff -Tascii -me - | less -R

The following will make a completely plain text file:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
    GROFF_NO_SGR=1 groff -Tascii -me - | col -b -x > meintro.txt

For printing, use ?PostScript output.

$ groff -Tps meintro.txt | lpr
$ groff -Tps meintro.txt | mpage -2 | lpr

TeX/LaTeX

Preparation:

# aptitude install texlive

References for LaTeX:

• The teTeX HOWTO: The Linux-teTeX Local Guide (http://www.tldp.org/HOWTO/TeTeX-HOWTO.html)

• tex(1)

• latex(1)

• "The ?TeXbook", by Donald E. Knuth, (Addison-Wesley)

• LaTeX - A Document Preparation System, by Leslie Lamport, (Addison-Wesley)

• The LaTeX Companion, by Goossens, Mittelbach, Samarin, (Addison-Wesley)

This is the most powerful typesetting environment. Many SGML processors use this as their back end text processor. Lyx provided by lyx, lyx-xforms, or lyx-qt and GNU ?TeXmacs provided by texmacs package offers nice WYSIWYG editing environment for LaTeX while many use Emacs and Vim as the choice for the source editor.

There are many online resources available:

• The TEX Live Guide - TEX Live 2007 (/usr/share/doc/texlive-doc-base/english/texlive-en/live.html) (texlive-doc-base package)

• A Simple Guide to Latex/Lyx (http://www.stat.rice.edu/~helpdesk/howto/lyxguide.html)

• Word Processing Using LaTeX (http://www-h.eng.cam.ac.uk/help/tpl/textprocessing/latex_basic/latex_basic.html)

• Local User Guide to teTeX/LaTeX (http://supportweb.cs.bham.ac.uk/documentation/LaTeX/lguide/local-guide/local-guide.html)

When documents become bigger, sometimes TeX may cause errors. You must increase pool size in /etc/texmf/texmf.cnf (or more appropriately edit /etc/texmf/texmf.d/95NonPath and run update-texmf) to fix this.

The TeX source of "The ?TeXbook" is available at ftp://ftp.dante.de/pub/tex/systems/knuth/tex/texbook.tex . This file contains most of the required macros. I heard that you can process this document with tex after commenting lines 7 to 10 and adding "\input manmac \proofmodefalse". It's strongly recommended to buy this book (and all other books from Donald E. Knuth) instead of using the online version but the source is a great example of TeX input!

Pretty print a manual page

The following will print a manual page into a ?PostScript file/printer.

$ man -Tps some_manpage | lpr
$ man -Tps some_manpage | mpage -2 | lpr

Creating a manual page

Although writing manpage in plain troff is possible, there are few helper packages to create the manpage.

The mail data conversion

The following packages for the mail data conversion caught my eyes:

The [http://en.wikipedia.org/wiki/Internet_Message_Access_Protocol Internet Message Access Protocol] version 4 (IMAP4) server (see: @{@popdimapeserver@}@) may be used to move mails out from the proprietary mail system if the mail client software can be configured to use IMAP4 server too.

Mail data basics

Mail (SMTP) data should be limited to 7 bit. So binary data and 8 bit text data are encoded into 7 bit format with the [http://en.wikipedia.org/wiki/MIME Multipurpose Internet Mail Extensions (MIME)] and the selection of the charset (see: @{@basicsofencoding@}@).

The standard mail storage format is mbox formatted according to [http://tools.ietf.org/html/rfc2822 RFC822], [http://tools.ietf.org/html/rfc2822 RFC2822]. See man 5 mbox (provided by the mutt package).

For European languages, "Content-Transfer-Encoding: quoted-printable" with the ISO-8859-1 charset is usually used since there are no much 8 bit characters. If the text is in UTF-8, "Content-Transfer-Encoding: quoted-printable" is also used since it is mostly 7 bit data.

For Japanese, traditionally "Content-Type: text/plain; charset=ISO-2022-JP" should be used to keep text in 7 bits. But mails from older Microsoft systems may use in Shift-JIS without proper declaration. For Japanese, if the text is in UTF-8, it contains many 8 bit data and is encoded into 7 bit data by [http://en.wikipedia.org/wiki/Base64 Base64]. The situation of other Asian languages is similar.

If your non-Unix mail data is accessible by a non-Debian client software which can talk to the IMAP4 server, you may be able to move them out by running your own IMAP4 server (see: @{@popdimapeserver@}@).

If you use other mail storage formats, moving them to mbox format is the good first step. The versatile client program such as mutt may be handy for this.

You can split mailbox contents to each message using procmail(1) and formail(1).

Each mail message can be unpacked using the munpack(1) command from the mpack package (or other specialized tools) to obtain the MIME encoded contents.

Graphic data tools

The following packages for the graphic data conversion, editing, and organization tools caught my eyes:

Search more image tools using @{@theaptituderegexformula@}@: "~Gworks-with::image".

Although GUI programs such as gimp are very powerful, command line tools such as imagemagik are quite useful for automating image manipulation with the script.

The de facto image file format of the digital camera is the [http://en.wikipedia.org/wiki/Exchangeable_image_file_format Exchangeable Image File Format] (EXIF) which is the JPEG image file format with additional metadata tags. It can hold information such as date, time, and camera settings.

[http://en.wikipedia.org/wiki/Lempel-Ziv-Welch The Lempel-Ziv-Welch (LZW) lossless data compression] patent has been expired. [http://en.wikipedia.org/wiki/Graphics_Interchange_FormatnThe Graphics Interchange Format (GIF)] utilities which use the LZW compression method are now freely available on the Debian system.

Any digital camera or scanner with removable recording media will work with Linux through [http://en.wikipedia.org/wiki/USB_mass_storage_device_class USB Mass Storage] readers.

Miscellaneous data conversion

There are many other programs for converting data. Following packages caught my eyes ("~Guse::converting" in aptitude):

You can also extract data from RPM format with:

$ rpm2cpio file.src.rpm | cpio --extract