Differences between revisions 52 and 53
Revision 52 as of 2011-02-26 16:39:53
Size: 19703
Comment: Replaced backend creation with just-ask-the-team instructions
Revision 53 as of 2011-02-26 17:13:46
Size: 19824
Comment: Use approx example with a minimal model targeted for upgrades
Deletions are marked like this. Additions are marked like this.
Line 32: Line 32:
The configuration model contains the following specifications : A typical configuration model contains the following specifications :
Line 35: Line 35:
 * The [[http://search.cpan.org/dist/Config-Model/lib/Config/Model/Value.pm#Value_model_declaration|constraints]]  of the leaf values (i.e. integer, enum type, min values, max values ...)  * The [[http://search.cpan.org/dist/Config-Model/lib/Config/Model/Value.pm#Value_model_declaration|constraints]] of the leaf values (i.e. integer, enum type, min values, max values ...)
Line 38: Line 38:
 * Help text displayed with interactive configuration editor.
Line 43: Line 45:

When creating a model just for upgrades, a model does not need to feature:
 * constraints. Because the configuration file to be upgraded is supposed to be correct
 * help text, because the upgrade is done in non-interactive mode
 * built-in default values because they are not used during upgrades
Line 70: Line 77:
FIXME: replace with example that shows minimal model creation
Line 75: Line 80:
 * 10 configuration parameters, either boolean, integer or string. All of them have default built in approx.  * a bunch configuration parameters, either boolean, integer or string. All of them have default built in approx.
Line 79: Line 84:

=== Upgrade scenario ===

For the sake of the example, let's assume that ''offline'' parameter was named ''nointernet'' in previous vefsions of Approx. This is of course not true.. Apologies to Eric Cooper for such a ludicrous idea.

So the upgrade model must feature:
 * instructions to migrate ''nointernet'' content to ''offline'' parameter
 * instructions to handle leaf parameter
 * instructions to handle distributions as we must avoid collisions between distribution names and approx parameter names (the syntax of approx.conf was changed with version 2.9.0 for the same reason)
Line 86: Line 100:
This model creation process is shown in this [[http://config-model.sourceforge.net/approx/approx-model-creation.html|flash movie]]. In order to reduce the length of the movie, only 3 out of 10 configuration items were created in this movie. (Please do not mind the error message that pops in the middle of the movie, this bug has been fixed since ;-) ).

In the end you will get some kind of a translation of `approx.conf(5)` man page in this model (reformated and expunged of help text for brevity):

You will have to:
 * Create the ''Approx'' configuration class
 * handle the old hypothetical ''nointernet'' parameter:
   * create a ''leaf'' parameter with value_type boolean
   * use a ''deprecated'' status as this parameter is to be migrated
 * handle the new ''offline'' parameter:
   * create a ''leaf'' parameter with value_type boolean
   * specify migration instruction where the old parameter is specified in a variable ('- nointernet') and the formula specifies how is used the old value. In this case, it's a simple substitution.
 * handle the distribution parameter (a hash of unilines)
 * specify what to do with unknown parameters. The ''accept'' instruction will match any other parameter (with the '.*' regexp) and use them as uniline values

In the end you will get :
Line 96: Line 120:
      'max_rate' , { type => 'leaf', value_type => 'uniline', },
      'max_redirects', { type => 'leaf', value_type => 'integer', upstream_default => '5' },
      'user' , { type => 'leaf', value_type => 'uniline', upstream_default => 'approx' },
      'group' , { type => 'leaf', value_type => 'uniline', upstream_default => 'approx' },
      'syslog' , { type => 'leaf', value_type => 'uniline', upstream_default => 'daemon' },
      'pdiffs' , { type => 'leaf', value_type => 'boolean', upstream_default => '1' },
      'offline' , { type => 'leaf', value_type => 'boolean', upstream_default => '0' },
      'max_wait' , { type => 'leaf', value_type => 'integer', upstream_default => '10' },
      'verbose' , { type => 'leaf', value_type => 'boolean', upstream_default => '0' },
      'debug' , { type => 'leaf', value_type => 'boolean', upstream_default => '0' },
      nointernet => { type => 'leaf', value_type => 'boolean', status => 'deprecated' },
      offline => { type => 'leaf', value_type => 'boolean',
                      migrate_from => {
                                        variables => { old => '- nointernet' } ,
                                        formula => '$old',
                                      }
                    },
Line 109: Line 130:
      ]       ],
  'accept' => [
      '.*' => { type => 'leaf', value_type => 'uniline', },
  ],
Line 114: Line 138:
You can view the actual model on [[http://config-model.hg.sourceforge.net/hgweb/config-model/config-model/file/tip/config-model-approx/lib/Config/Model/models/Approx.pl| Approx model on Mercurial]] You can view the actual model on [[http://config-model.hg.sourceforge.net/hgweb/config-model/config-model/file/tip/config-model-approx/lib/Config/Model/models/Approx.pl| Approx model on Mercurial]]. THis model is more complex as all parameters and help text are provided.
Line 154: Line 178:
# max_rate:maximum download rate from remote repositories

# max_redirects:maximum number of HTTP redirections (5)

# user:user that owns the files in the approx cache (approx)

# group:group that owns the files in the approx cache (approx)

# syslog:syslog(3) facility to use when logging (daemon)

# pdiffs:support IndexFile diffs (1)

# offline:use cached files when offline (0)

# max_wait:max wait for concurrent file download (10)
Line 171: Line 180:
# verbose: (0)

# debug: (0)

# remote repositories
Line 200: Line 204:

=== The cherry on the pie ===

With the addition proposed above, your user will also get a GUI to edit `approx.conf` when running `config-edit -model Approx`:

{{attachment:gui.png}}

Translation(s): none

(!) ?Discussion


This page contains a proposal for a way to handle upgrade where user's data and package maintainer data are merged with minimal interaction with user. The package approx will be used as an example.

Configuration upgrade by package

Status

This page is a proposal. It is not (yet?) accepted by Debian project. Stay tuned.

Introduction

For a casual user (like you mother-in-law or your neighbour), the questions asked during package upgrades can be intimidating. Casual users barely know that some files exist outside of their home directory and are completely lost when looking at the /etc directory. Unfortunately, during package upgrade, user is faced with cryptic questions whether to keep his configuration, use the maintainers configuration or look at a diff. The casual user will seldom know what to do in this case.

This situation can be made worse if user's configuration was edited with a tool (e.g. cups configuration is edited through a web browser): the user has no knowledge of the syntax of the configuration file.

This proposal aims to provide a way for Debian package to minimize the number of questions raised to the user when upgrading packages. If questions are necessary, an interface will be provided to the user. Hopefully, manual edition of configuration files will not be necessary.

Upgrades with Config::Model

Config-Model provides a framework for editing and validating the content of any configuration file or data. With a configuration model (expressed in a data structure), Config-Model provides a user interface and a tool to validate configuration.

A typical configuration model contains the following specifications :

  • The structure of the configuration model. This structure is a tree that can be flat (for simple files like approx.conf) or quite deep (for complex configuration like xorg.conf). The configuration tree nodes are instances of configuration class and the tree leaves contain the configuration values.

  • The constraints of the leaf values (i.e. integer, enum type, min values, max values ...)

  • the default values where the default value must be written in the configuration file.

  • the built-in default value where the value need not be written in the configuration file. This default value is built in the application.

  • Help text displayed with interactive configuration editor.

This notion of default value versus built-in default value is important to distinguish:

  1. the values that were customized by the user (or sysadmin)
  2. the value that are recommended by the package maintainer (and specified as default in the model)

  3. the value that are provided upstream (specified as upstream_default)

When creating a model just for upgrades, a model does not need to feature:

  • constraints. Because the configuration file to be upgraded is supposed to be correct
  • help text, because the upgrade is done in non-interactive mode
  • built-in default values because they are not used during upgrades

During a package upgrade we want to:

  • propagate values from the old configuration file to new configuration file.
  • discard obsolete data (if any)
  • introduce new mandatory values (if any)
  • migrate (or translate) data in old format to new format

So the idea is:

  • Upgrade the packages files as usual (including upgrade of configuration model)
  • In postinst, run config-edit to load old configuration file using new configuration model and write back the updated data in new configuration file (i.e. merged values from user's data and data from new configuration model).

  • If the last step fails, launch config-edit in interactive mode (using Debconf as a user interface ?) or fall-back to ucf (any other alternative ?)

The same principles can be applied to downgrade with some limitations (see below for more details)

If you want to apply config-model based upgrades to your favorite application

You will have to:

  • Create a minimal model that describe:
    • items that need special treatment during upgrades
    • how to handle other parameters
  • Find a backend that support the syntax of the configuration file of your application. If none exists, ask on config-model-users at lists.sourceforge.net, we'll find a solution that won't require you to learn Perl.

Well, that was the theory. Let's experiment with Approx configuration file and work out the details.

Approx example

configuration analysis

From approx.conf(5) man page, we find that approx has a very simple configuration. Approx will need one configuration class that will have:

  • a bunch configuration parameters, either boolean, integer or string. All of them have default built in approx.
  • a hash containing distribution names and their URL. These will be modeled with a hash of uniline values

The syntax of approx.conf is fairly simple but there's no Perl module available to read and write it. A dedicated parser/writer will be required.

Upgrade scenario

For the sake of the example, let's assume that offline parameter was named nointernet in previous vefsions of Approx. This is of course not true.. Apologies to Eric Cooper for such a ludicrous idea.

So the upgrade model must feature:

  • instructions to migrate nointernet content to offline parameter

  • instructions to handle leaf parameter
  • instructions to handle distributions as we must avoid collisions between distribution names and approx parameter names (the syntax of approx.conf was changed with version 2.9.0 for the same reason)

Approx configuration model

To create the Approx configuration model, you can use the GUI provided by libconfig-model-itself-perl.

In a empty directory, run config-edit-model -model Approx and fill the fields as required.

You will have to:

  • Create the Approx configuration class

  • handle the old hypothetical nointernet parameter:

    • create a leaf parameter with value_type boolean

    • use a deprecated status as this parameter is to be migrated

  • handle the new offline parameter:

    • create a leaf parameter with value_type boolean

    • specify migration instruction where the old parameter is specified in a variable ('- nointernet') and the formula specifies how is used the old value. In this case, it's a simple substitution.
  • handle the distribution parameter (a hash of unilines)
  • specify what to do with unknown parameters. The accept instruction will match any other parameter (with the '.*' regexp) and use them as uniline values

In the end you will get :

[
 {
  'name' => 'Approx',
  'element' 
  => [
      nointernet => { type => 'leaf', value_type => 'boolean', status => 'deprecated' },
      offline    => { type => 'leaf', value_type => 'boolean', 
                      migrate_from => {
                                        variables => { old => '- nointernet' } ,
                                        formula => '$old', 
                                      }
                    },
      'distributions', { type => 'hash', index_type => 'string' ,
                         cargo => { value_type => 'uniline', type => 'leaf',},
                       }
      ],
  'accept' => [
      '.*'  => { type => 'leaf', value_type => 'uniline',                      },
  ],    
 }
] ;

You can view the actual model on Approx model on Mercurial. THis model is more complex as all parameters and help text are provided.

/!\ Any Debian specific requirement (e.g. default value) will have to be implemented in the model and not in the template configuration file provided to the user (e.g. not in /etc/approx/approx.conf)

Config::Model commands to manage configuration upgrade

{i} These commands are not specific to Debian and can be applied to other distros or OS.

Let's start with a simple approx.conf file:

debian          http://ftp.fr.debian.org/debian
$max_wait       12

Simple upgrade case

In most cases, simply running config-edit will be enough to upgrade a configuration files. Even if the configuration model is changed from package version N to version N+1, the upgrade will work most to the time. config-edit will exit on error if some data to upgrade are not recognized.

Upgrading the configuration is done by postinst script with

   config-edit -model Approx -ui none -save

config-edit will:

  • parse the old configuration file
  • validate all values (possibly discarding or migrating obsolete parameters) FIXME add link to details
  • add new values declared in N+1 model (if any)
  • write back the configuration file (-save option)

/!\ Without Augeas and a dedicated lens for approx, original comments will be lost during an upgrade

After upgrade, approx.conf has:

# This file was written by Config::Model with Approx model
# You may modify the content of this file. Configuration
# modification will be preserved. Modification in the comments
# will be discarded

$max_wait   12

debian     http://ftp.fr.debian.org/debian

More complex upgrade

See PackageConfigComplexUpgrade

Integrate simple upgrade command in package scripts

To get configuration upgrade on package upgrade, approx package needs to be modified this way:

  • addition of Approx model file (debian/config-model/Approx.pm)

  • addition of Approx reader/writer (debian/config-model/model/Approx.pl)

  • modification of debian/approx.postinst with addition of:

    config-edit -model Approx -ui none -save 
  • addition of the following lines in debian/approx.install:

debian/config-model/Approx.pm usr/share/perl5/Config/Model
debian/config-model/model/Approx.pl usr/share/perl5/Config/Model/models

/!\ FIXME: Define what can be done if config-edit fails. Should we abort of retry with -force option which discard invalid data ? Should user be queried with debconf ou should -force option always be used ?

Simple upgrade cases

package installation or upgrade

During installation of approx package, existing approx.conf (application version N) are upgraded by Config::Model using configuration model from version N+1.

package downgrade

Mostly the same scenario as upgrade apply. The configuration data for N+1 are digested by Config::Model using configuration model from N.

/!\ If N+1 did obsolete or migrate parameters, some informations will be lost.

dh_config_model_upgrade

The approx package modification described above can also be done with a new dh helper script: dh_config_model_upgrade

The synopsis is

 dh_config_model_upgrade [ -p pkg ]
 [ debhelper options ] [ -model_name xx ] [ -model_package xx [ -model_version yy ] ]

The complete doc is written in pod and can be seen at the end of the source code in svn

{i} This probably can be better integrated in CDBS. Suggestions are welcome

{i} dh_config_model_upgrade is a work in progress that could use feedback from experienced Debian developers

Limitations

User comments

When upgrading, the original layout and the user comments are not preserved. approx.conf is written back in a canonical order specified by the model. I.e. all parameters are written back using the parameter order of Approx model. Most casual users will not mind as they are reluctant to manually edit file in /etc. But advanced users may frown upon the loss of their comments.

That said, there are several possibility to address this problem:

  • Comments can be preserved only if Augeas is used with Config::Model. For this, a lens must be created for approx.conf. This is another layer of abstraction which is not really easy to grasp, thus more work for package maintainers if the lens is not provided by Augeas.

  • Some information regarding Approx parameters are written as comments by Config::Model. Thus the produced configuration file have some comments to provide the summary or description information stored in Approx model. But, without Augeas, Config::Model cannot preserve modifications done to comments1.

  • Config::Model will be enhanced to offer annotation through the GUI. This annotation will serve the same purpose as comments in the file but would be stored elsewhere.
  • Should we offer to skip upgrade part for users that want to keep the comments ? (if they want to keep the comments, they are probably able to manually upgrade their configuration)

Examples

Approx

This simple patch will use the brand new lib-config-model-approx-perl to provide the upgrade capacity.

The patch features:

  • A new dependency on libconfig-model-approx-perl in approx control file
  • A call to config-edit in approx postinst

To try this example, you will have to:

  • apt-get install libconfig-model-perl (>= 0.637) from Sid/unstable or Karmic

  • apt-get build-dep approx

  • apt-get source approx

  • apply approx-simple.patch in approx source directory (path -p1 < approx-simple.patch)

  • build the package (dpkg-buildpackage -uc -us)

  • install the new package (this installation will trigger the upgrade done by Config::Model)

The author hereby humbly admits that he could not figure out how to use dh_config_model_upgrade in the cdbs scripts used by Approx :\

As an exercise, one can also provide the configuration model within approx package. Thus the upgrade capacity may not depend on a dedicated model package.

The patch will add:

  • Approx configuration model
  • Approx.pm to read/write approx.conf
  • A new dependency on libconfig-model-perl in approx control file
  • A conflict with libconfig-model-approx-perl (since the patch will deliver the same model in the same location)

To try this example, you will have to:

  • apply approx-add-model.patch in approx source directory (path -p1 < approx-add-model.patch) after the approx-simple patch.

  • build the package (dpkg-buildpackage -uc -us)

  • install the new package (this installation will trigger the upgrade done by Config::Model)

If you want to modify Approx configuration model provided by the approx-add-model patch, use the following command:

$ config-model-edit -dir debian/config-model/models -model Approx

OpenSsh

Let's modify ?OpenSsh packages to use dh_config_model_upgrade script. Unlike Approx, the configuration model is already provided in Debian, so there's no need to include it in a patch.

The modifications are: * Update rules file so that build openssh-server package will call dh_config_model_upgrade * Some small modification to make openssh package work with dh_*

Let's view the call to dh_config_model_upgrade:

        dh_config_model_upgrade --model_name Sshd -p openssh-server \
           --model_package --model_version 1.206

This command will modify postinst using Sshd configuration model provided by package libconfig-model-openssh-perl (with version >> 1.206).

That's it. There's no need to do more. See dh-cm-upgrade.patch patch

/!\ dh_config_model_upgrade cannot be called for openssh-client, because package upgrade cannot upgrade user's configuration (except when they explicitly volunteer - see ikiwiki-mass-rebuild(8) in the ikiwiki package for an example).

Other packages based on dh

Packages using tiny rules should use the --with config_model option. E.g:

%:
         dh --with config_model $@

and provide a debian/<pkg>.config-model file.

See this example for open-ssh server package:

$ cat debian/openssh-server.config-model
model_name: Sshd
model_package: lib-config-model-openssh-perl
model_version: 1.206

Open questions

  • How to handle errors in user's approx.conf file ?
    • Should bad data be discarded (possible with -force option of config-edit)

    • Should we skip data migration (as is done currently)
  • How to handle missing data ? (this scenario is not relevant with current Approx model, but may become relevant if mandatory values are added to Approx model). In this case, should Config::Model be run in approx.config script and connect to Debconf to ask question to user ?

Working with Debconf

Warning: work in progress

Main ideas:

  • Run config-edit -ui debconf -saveindebconf in pre-inst

  • Run config-edit -ui debconf -loadfromdebconf -save in postinst

  • Translate data from model into debconf template. Debconf template will be used by
    • C::M::Debconf::Wizard and Config::Moldel::WizardHelper to scan configuration tree and ask question in case of error or missing mandatory values (or may be different level of scanning to match debconf's priority. See Config::Model::DebConfUI (still coding...))

Fortunately, the semantic of Config::Model value_type mostly matches debconf's type:

  • Config::Model

    Debconf

    uniline

    string

    integer

    string

    number

    string

    string

    string (problem with multi-line ?)

    boolean

    boolean

    enum

    select

    checklist

    multiselect

    help stored in model

    note

    $@ from eval{die ...} ;-)

    error

    title

    not necessary

Thanks

  • To Damyan Ivanov (first review)
  • To Jonas Smedegaard for the constructive exchange and the helper scripts and /var/lib/ storage ideas
  • The Gregor Hermann for sponsoring all the libconfig-model-* packages



CategoryDebianDevelopment

  1. Preserving comments is very hard because associating comments with configuration data is guesswork (1)