Differences between revisions 28 and 29
Revision 28 as of 2009-05-04 12:02:14
Size: 19686
Comment: added thanks, storage in /var/lib, alternative upgrade commands
Revision 29 as of 2009-05-04 12:04:09
Size: 19701
Comment:
Deletions are marked like this. Additions are marked like this.
Line 396: Line 396:
 * To Damyan Ivanov  * To Damyan Ivanov (first review)

Translation(s): none

(!) ?Discussion


This page contains a proposal for a way to handle upgrade where user's data and package maintainer data are merged with minimal interaction with user. The package approx will be used as an example.

Configuration upgrade by package

Status

This page is a proposal. It is not (yet?) accepted by Debian project. Stay tuned.

Introduction

For a casual user (like you mother-in-law or your neighbour), the questions asked during package upgrades can be intimidating. Casual users barely know that some files exist outside of their home directory and are completely lost when looking at the /etc directory. Unfortunately, during package upgrade, user is faced with cryptic questions whether to keep his configuration, use the maintainers configuration or look at a diff. The casual user will seldom know what to do in this case.

This situation can be made worse if user's configuration was edited with a tool (e.g. cups configuration is edited through a web browser): the user has no knowledge of the syntax of the configuration file.

This proposal aims to provide a way for Debian package to minimize the number of questions raised to the user when upgrading packages. If questions are necessary, an interface will be provided to the user so that manual edition of configuration files will not be necessary.

Upgrades with Config::Model

Config-Model provides a framework for editing and validating the content of any configuration file or data. With a configuration model (expressed in a data structure), Config-Model provides a user interface and a tool to validate configuration.

The configuration model contains the following specifications :

  • The structure of the configuration model. This structure is a tree that can be flat (for simple files like approx.conf) or quite deep (for complex configuration like xorg.conf). The configuration tree nodes are instances of configuration class and the tree leaves contain the configuration values.

  • The constraints of the leaf values (i.e. integer, enum type, min values, max values ...)

  • the default values where the default value must be written in the configuration file.

  • the built-in default value where the value need not be written in the configuration file. This default value is built in the application.

This notion of default value versus built-in default value is important to distinguish:

  1. the values that were customized by the user (or sysadmin)
  2. the value that are recommended by the package maintainer (and specified as default in the model)

  3. the value that are built in the application.

During a package upgrade we want to propagate only the customized values from the old configuration file to the upgraded configuration file. The customized values are different from the built_in values (specified by upstream in the application) and different from the default values (specified by Debian maintainer in the configuration model).

So the idea is:

  • In old-prerm script, run config-edit to dump custom values (using old configuration model since we are at old-prerm phase) and store the result ( where ? ). This step does not modify user's configuration file (we may want to back it up though)

  • Upgrade the packages files as usual (including upgrade of configuration model)
  • In postinst, run config-edit to load user's customized data using new configuration model (but don't load data from user's configuration file) and write back the merged data in configuration file (i.e. merged values from user's data and data from new configuration model).

  • If the last step fails, we can either launch config-edit in interactive mode (using Debconf as a user interface ?) or fall-back to ucf.

The same principles can be applied to downgrade with some limitations (see below for more details)

Well, that was the theory. Let's experiment with Approx configuration file and work out the details.

Approx example

configuration analysis

From approx.conf(5) man page, we find that approx has a very simple configuration. Approx will need one configuration class that will have:

  • 10 configuration parameters, either boolean, integer or string. All of them have default built in approx.
  • a hash containing distribution names and their URL. These will be modeled with a hash of uniline values

The syntax of approx.conf is fairly simple but there's no Perl module available to read and write it. A dedicated parser/writer will be required.

Approx configuration model

To create the Approx configuration model, you can use the GUI provided by libconfig-model-itself-perl.

In a empty directory, run config-edit-model -model Approx and fill the fields as required. This model creation process is shown in this flash movie. In order to reduce the length of the movie, only 3 out of 10 configuration items were created in this movie. (Please do not mind the error message that pops in the middle of the movie, this bug has been fixed since ;-) ).

In the end you will get some kind of a translation of approx.conf(5) man page in this model (reformated and expunged of help text for brevity):

[
 {
  'name' => 'Approx',
  'element' 
  => [
      'max_rate'     , { type => 'leaf', value_type => 'uniline',                      },
      'max_redirects', { type => 'leaf', value_type => 'integer', built_in => '5'      },
      'user'         , { type => 'leaf', value_type => 'uniline', built_in => 'approx' },
      'group'        , { type => 'leaf', value_type => 'uniline', built_in => 'approx' },
      'syslog'       , { type => 'leaf', value_type => 'uniline', built_in => 'daemon' },
      'pdiffs'       , { type => 'leaf', value_type => 'boolean', built_in => '1'      },
      'offline'      , { type => 'leaf', value_type => 'boolean', built_in => '0'      },
      'max_wait'     , { type => 'leaf', value_type => 'integer', built_in => '10'     },
      'verbose'      , { type => 'leaf', value_type => 'boolean', built_in => '0'      },
      'debug'        , { type => 'leaf', value_type => 'boolean', built_in => '0'      },
      'distributions', { type => 'hash', index_type => 'string' ,
                         cargo => { value_type => 'uniline', type => 'leaf',},
                       }
      ]
 }
] ;

You can view the actual model on Approx model SVN

/!\ Any Debian specific requirement will have to be implemented in the model and not in the template configuration file provided to the user (e.g. not in /etc/approx/approx.conf)

Parsing and writing approx configuration file

First, how to read or write the configuration file must declared in Approx model using the autoread feature. For more detail see this movie. The reader writer class must be declared in Approx model:

   read_config => [
                   {
                    backend    => 'custom',
                    class      => 'Config::Model::Approx',
                    config_dir => '/etc/approx',
                    file       => 'approx.conf',
                   }
                 ],

This indicates that a custom will be used to read /etc/approx/approx.conf. The class Config::Model::Approx must provide a read method. By default, the same setup is used to write back approx.conf with the write method.

The read method is specified in Config::Model::Approx. Note that comments are discarded:

sub read {
    my %args = @_ ;

    foreach ($args{io_handle}->getlines) {
        chomp;
        s/#.*//;
        s/\s+/=/; # translate file in string loadable by C::M::Loader
        next unless $_; # skip empty lines
        my $load = s/^\$// ? $_ : "distributions:".$_;
        $args{object}->load($load) ;
    }

    return 1;
}

Likewise, the write method is provided in Config::Model::Approx. Note that some documentation is extracted from Approx model and used in the comments of the generated file. So any user reviewing the approx.conf file will not be lost:

sub write {
    my %args = @_ ;

    my $node = $args{object} ;
    my $ioh  = $args{io_handle} ;

    $ioh->print("# This file was written by Config::Model with Approx model\n");
    $ioh->print("# You may modify the content of this file. Configuration \n");
    $ioh->print("# modification will be preserved. Modification in the comments\n");
    $ioh->print("# will be discarded\n\n");

    foreach my $elt ($node->get_element_name) {
        next if $elt eq 'distributions';

        # write some documentation in comments
        $ioh->print("# $elt:", $node->get_help(summary => $elt));
        my $built_in = $node->fetch_element($elt) -> fetch('built_in') ;
        $ioh->print(" ($built_in)") if defined $built_in;
        $ioh->print("\n") ;

        # write value
        my $v = $node->grab_value($elt) ;
        $ioh->printf("\$%-10s %s\n",$elt,$v) if defined $v ;
        $ioh->print("\n") ;
    }

    my $h = $node->fetch_element('distributions') ;
    $ioh->print("# ", $node->get_help(summary => 'distributions'),"\n");
    foreach my $dname ($h->get_all_indexes) {
        $ioh->printf("%-10s %s\n",$dname,
                     $node->grab_value("distributions:$dname")
                    ) ;
    }
    return 1;
}

Commands to manage upgrade

Let's start with this approx.conf file:

debian          http://ftp.fr.debian.org/debian
$max_wait       12
$user           approx
$group          approx

Note that $user and $group are set to the application default values.

Only customized values are dumped with -dump option:

$ config-edit-approx -dump
max_wait=12
distributions:debian=http://ftp.fr.debian.org/debian -

In old-prerm script, customized values can be saved as:

config-edit -model Approx -dump /var/lib/config-model/upgrade/etc/approx/approx.cds 

or ( /!\ still being discussed on debian-perl list)

config-edit -model Approx -save_for_upgrade

/!\ Actual config-edit command may change

and reloaded in postinst script with

   config-edit -model Approx -load /etc/approx/approx.cds -ui none

or ( /!\ still being discussed on debian-perl list)

config-edit -model Approx -upgrade

/!\ FIXME: how to use a helper script (dh_ like) that will help in setting up the instructions mentioned above in the package scripts.

After upgrade, approx.conf has:

# This file was written by Config::Model with Approx model
# You may modify the content of this file. Configuration
# modification will be preserved. Modification in the comments
# will be discarded

# max_rate:maximum download rate from remote repositories

# max_redirects:maximum number of HTTP redirections (5)

# user:user that owns the files in the approx cache (approx)

# group:group that owns the files in the approx cache (approx)

# syslog:syslog(3) facility to use when logging (daemon)

# pdiffs:support IndexFile diffs (1)

# offline:use cached files when offline (0)

# max_wait:max wait for concurrent file download (10)
$max_wait   12

# verbose: (0)

# debug: (0)

# remote repositories
debian     http://ftp.fr.debian.org/debian

Integrate upgrade command in package scripts

The modification to approx package are:

  • addition of Approx model file (debian/config-model/Approx.pm)

  • addition of Approx reader/writer (debian/config-model/model/Approx.pl)

  • addition of debian/approx.prerm script:

cds_file=/etc/approx/approx.cds
cfg_file=/etc/approx/approx.conf

case "$1" in
    remove)
       if [ -e $cds_file ] ; then
           rm -rf $cds_file ;
       fi
       ;;
    *)
       if [ -e $cfg_file ] ; then
           config-edit -model Approx -dump /etc/approx/approx.cds || true
       fi
       ;;
esac

  • modification of debian/approx.postinst with addition of:

if [ -e /etc/approx/approx.cds ] ; then
    config-edit -model Approx -load /etc/approx/approx.cds -ui none \
        && rm -f /etc/approx/approx.cds
fi
  • addition of the following lines in debian/approx.install:

debian/config-model/Approx.pm usr/share/perl5/Config/Model
debian/config-model/model/Approx.pl usr/share/perl5/Config/Model/models

The cherry on the pie

With the addition proposed above, your user will also get a GUI to edit approx.conf when running config-edit -model Approx:

gui.png

Simple upgrade cases

package installation

During installation of approx package, any existing approx.conf is not touched by Config::Model.

package upgrade

  • during upgrade of old approx package (without config-model patch) with new approx package (with patch), approx.conf is not touched by Config::Model.

  • when upgrading from a patched approx to a patched approx:
    • all customized values of approx.conf are dumped in approx.cds by old-prerm
    • new-postinst loads approx.cds using new configuration model and writes back approx.conf

package downgrade

Mostly the same scenario as upgrade apply except:

  • when downgrading from a patched approx to a plain approx:
    • all customized values of approx.conf are dumped in approx.cds by old-prerm
    • new-postinst leaves approx.cds alone

package purge

Any leftover approx.cds is removed by prerm script

More complex upgrade cases

So far, we've reviewed the simple upgrade cases where no configuration changes were brought by package changes.

But, Config::Model can also handle more complex cases where:

  • new parameter are introduced or old parameters are removed by the application
  • new parameter with mandatory values are added by the application (require user interaction)
  • Debian packager changes a default value
  • parameter names are changed
  • parameter allowed value are changed

{*} Reader, feel free to add your upgrade case {*}

These cases are not required for Approx model, but they will be for other packages.

These will be explained in another wiki page (to be written)

Limitations

User comments

When upgrading, the original layout and the user comments are not preserved. approx.conf is written back in a canonical order specifed by the model. I.e. all parameters are written back using the parameter order of Approx model. Most casual users will not mind as they are reluctant to manually edit file in /etc. But advanced users may frown upon the loss of their comments.

That said, there are several possibility to address this problem:

  • Comments can be preserved only if Augeas is used with Config::Model. For this, a lens must be created for approx.conf. This is another layer of abstraction which is not really easy to grasp, thus more work for package maintainers if the lens is not provided by Augeas.

  • Some information regarding Approx parameters are written as comments by Config::Model. Thus the produced configuration file have some comments to provide the summary or description information stored in Approx model. But, without Augeas, Config::Model cannot preserve modifications done to comments1.

  • Config::Model will be enhanced to offer annotation through the GUI. This annotation will serve the same purpose as comments in the file but would be stored elsewhere.
  • Should we offer to skip upgrade part for users that want to keep the comments ? (if they want to keep the comments, they are probably able to manually upgrade their configuration)

I want to try

To try this upgrade proposal, you will have to:

  • apt-get install libconfig-model-perl (>= 0.635) from Sid/unstable

  • apt-get build-dep approx

  • apt-get source approx

  • apply upgrade-mininal patch in approx source directory (path -p1 < upgrade-minimal.patch)

  • build the package (dpkg-buildpackage -uc -us)

  • install the new package (this first installation will not perform any upgrade)
  • re-install the new package (this second installation will trigger the upgrade done by Config::Model)

If you want to modify the model provided by the upgrade-minimal patch, use the following command:

$ config-model-edit -dir debian/config-model/model/ -model Approx

Open questions

  • What if Approx model is provided in a separate package (i.e. not in approx but in libconfig-model-approx-perl) ?
  • How to handle errors in user's approx.conf file ?
    • Should bad data be discarded (possible with -force option of config-edit)

    • Should we skip data migration (as is done currently)
  • How to handle missing data ? (this scenario is not relevant with current Approx model, but may become relevant if mandatory values are added to Approx model). In this case, should Config::Model be run in approx.config script and connect to Debconf to ask question to user ? (Could use C::M::?WizardHelper to scan configuration tree and ask question in case of error or missing mandatory values)

Thanks

  • To Damyan Ivanov (first review)
  • To Jonas Smedegaard for the constructive exchange and the helper scripts and /var/lib/ storage ideas
  • The Gregor Hermann for sponsoring all the libconfig-model-* packages



CategoryDebianDevelopment

  1. Preserving comments is very hard because associated comments with configuration data is guesswork (1)