!MySQL and !MariaDB currently both use /var/lib/mysql to store their data, and both are packaged in Debian unstable. When !MariaDB originally forked from !MySQL it used a compatible data format. As the variants have diverged, incompatibilities have arisen, leading to a complex matrix of crossgrades that may or may not be automatically possible.

As a user can switch variants by simply installing the other variant's server package, this can cause problems when the maintainer scripts cannot automatically and losslessly convert the data.

At the London !MySQL packaging sprint in January 2018, we resolved to fix this as follows.

1. Box abstraction specification:

Conceptually each variant and major version stores its data in a "box". Every variant and major version combination may have zero or one (but not more) boxes. The (variant, major_version) pair associated with a box is its label. A box may be active or inactive. Only one box may be active at any one time. A box maps to a filesystem directory, but you must use the abstraction API to determine where it is. The abstraction provides the following methods:

    Query what boxes exist -> list of (variant, major_version) pairs.

    Query which is the active box -> None or (variant, major_version) pair.

    Create a box for a new (variant, major_version) pair. This will fail if it already exists.

    Return the filesystem directory associated with a particular (variant, major_version) pair.

    Copy a box to a new box. Parameters: old and new (variant, major_version) pairs. This may fail if the box contains symlinks or filesystem mounts.

    In-place convert a box to a different (variant, major_version) pair. This has the effect of changing the box's label, and can be done on an active box.

    Change the active box. This may fail if the implementation uses a constant filesystem path for the active path, or if active and inactive boxes are stored on different underlying filesystems. In this case changing the box requires renames, and may fail if either the active or inactive box contains filesystem mounts or symlinks.


Other details in this specification:

    mysql-files and mysql-keyring are out of the scope of boxes

    Box operations must only be carried out when all mysql and variant daemons are stopped, and it is therefore safe to rename, copy and move around data directories.


2. Initial filesystem mapping implementation:

    The active box will always be <datadir> where <datadir> is whatever is configured in /etc/mysql or /var/lib/mysql if not defined. <datadir> must be supplied by the variant maintainer script to the common box handling code in mysql-common. The box filesystem location can be assumed by variant maintainer script implementations. Unless customised by a sysadmin, /var/lib/mysql (or alternative datadir) is a directory and not a symlink.

    Inactive boxes will always be /var/lib/<variant>-<major_version>/

    A box label shall be in the form of a file "<variant>-<version>.flag".

    The old "debian-<version>.flag" files shall remain on upgrade and continue to be created until 2022 so that old packages already shipped can also handle them.

    If the directory name and the flag inside don't match, we believe the label flag inside the directory.


3. No longer required.

4. Initial variant maintscript implementation (closes: #853008)

    Goal: inform box implementation in mysql-common about everything going on, allowing variants to interoperate better. No functional changes should be evident to users, except it will be less buggy when handling multiple variants.

    On configure:

    List existing boxes and determine active box.

    If a box exists matching this variant and version, activate it if it is not active already. If activation fails, enter frozen mode and exit.

    If a matching box does not exist:

    If at least one box exists for this variant but older version, pick the latest one out of this set, activate it and rename it to the current version after confirming with the user because this is not reversible.. If activation fails, enter frozen mode and exit.

    If no previous version box for this variant is found, then rename current active box if a format conversion is possible (eg. MySQL 5.5 -> MariaDB 10.1) after confirming with the user because this is not reversible. If this is not possible, then enter frozen mode and exit.

    Proceed as before, performing a schema upgrade if necessary.

    On postrm (purge):

    Delete the correct box (determining its filesystem location using the box abstraction API since at the time of purge the running postrm variant may not be active any more) if this is what the user wants.

    No need to deactivate the box. Doing so risks problems with symlinks and filesystem mounts.


5. Variant maintscript enhancements to permit automatic data migration between variants where automated methods exist upstream (Closes: #841345, LP: #1490071)

    Make available various options when a new box is required depending on what is possible:

    Create a new empty box

    Copy and format-convert an existing box of a different variant/version. Unavailable if not enough space or format migration is not possible.

    In-place format-convert an existing box of a different variant/version if the user explicitly agrees (cannot go backwards). Unavailable if a format migration is not possible.

    Delete other boxes if the user explictly agrees. For example, this may help to free up space for the copy and format-convert option to become available.

    Do not prompt the user on distribution release upgrades. These should remain automatic in-place format conversions. Otherwise the disk will steadily fill over many release upgrades. For consistency, this means that distribution packages should always do in-place format conversions when upgrading major version but remaining on the same variant. However, packages not in the distribution archive (eg. PPAs and upstream apt repositories) may choose to prompt and/or always copy-and-convert.


6. Remove the requirement for the active box to be /var/lib/mysql or custom datadir.

    The permitted assumption for variant maintscripts to assume that the active box is in a particular place is removed. Variant maintscripts must always query the abstraction API to determine the location of a box, and must use the answer given.

    We stop using /var/lib/mysql for storage and could make /var/lib/mysql simply a symlink to the currently active box.


7. Possible future extensions

    Permit multiple boxes, e.g. mysql_8.0_a/mysql-8.0.flag and mysql_8.0_b/mysql-8.0.flag 

    Allow inactive boxes to be anywhere on the filesystem (which in this spec are currently fixed to /var/lib/<variant>_<version>).

    Eliminate the concept of an active box, deprecating completely the /var/lib/mysql filesystem location which will no longer exist.