Abandoned

As of July 2014, there will be no Emdebian Grip stable release based on Jessie or any subsequent Debian release.

The reasons are simple:

0: A lack of devices which lack expandable storage

1: Too much work to prepare the updates and make the releases.

The rest of the content of this page is retained for historical purposes but can be removed in due course.

Integrating Emdebian Grip into Debian

Emdebian has long described itself as an "official sub project of Debian" but this status has not actually been granted by Debian yet. The discrepancy recently led to discussions with the Debian Project Leader and on the debian-embedded mailing list about integrating Emdebian into Debian so that the status of Emdebian can be decided properly.

It rapidly became clear that Emdebian could integrate the Emdebian Grip flavour more closely with Debian and that such integration was desirable both within Emdebian and within Debian.

At DebConf11, discussions took place with representatives of the ftp team, wanna-buld team and release team around the mechanisms by which Emdebian Grip could be integrated directly into the main Debian archive and Debian release process.

Process

Emdebian has a role account on a buildd managed by DSA which runs the emgrip script instead of using schroot. This buildd has a read-only copy of ProjectB which provides information on all package movements through the normal Debian buildd system, including source uploads, other buildd uploads, binary NMU's and security updates.
The buildd uses rsync to get the list of packages in which Emdebian is interested and this list would be modifiable via bug reports against buildd.emdebian.org. The list is constructed from the Packages files of the Emdebian unstable-grip suite.
The buildd would use [edos-debcheck] for dependency resolution to bring in extra packages when an existing package changes dependencies. (This is necessary for the release team migration process to work correctly and does not need to happen immediately, as long as the dependencies are resolved before the package tries to migrate. The expectation is that the package list would be altered automatically in time for the next buildd operation. However, this may involve some level of "back-tracking" to bring in a package which was last built for Debian some period previously.)
The ftp team have agreed to take care of removing old dependencies or packages removed from Debian using the same mechanisms as are used for the current Debian archive.
The ftp team have also agreed to push the Emdebian Grip packages to all existing primary Debian mirrors which do not explicitly ask not to take them. Currently, this is about 25Gb of data.
The release team have agreed to work with the Emdebian team to synchronise all releases, including stable updates. The Emdebian Grip processes would need to ensure that the release process is not delayed by missing packages or other discrepancies.
The release team have agreed to run the testing migrations of Emdebian Grip packages alongside their Debian counterparts, possibly using a second britney process.
Emdebian has agreed to drop the dev, debug, java and doc components from Grip unstable and Wheezy in order to make this process work - all packages will revert to the "main" component and Emdebian would gain support for contrib and non-free (although no packages currently exist for those components in Emdebian Grip).
The ftp team require that the Emdebian packages are separated from the Debian packages by two mechanisms, the current em1 version suffix and a distinctive suite name. Emdebian will use unstable-grip, alongside sid-grip and similarly for testing-grip, wheezy-grip, stable-grip etc.
Still to be decided is how to manage existing packages but the suggestion from Emdebian would be that the ftp team synchronise the new suites with the existing Emdebian packages for the existing releases (Lenny and Squeeze) and that www.emdebian.org continues to provide those releases under the existing suites. Similarly, the existing packages on ftp.uk.debian.org/emdebian/grip will be retained for Lenny and Squeeze but will eventually migrate to being symlinks to the new suite names in ftp.uk.debian.org/debian in time for the Wheezy release and ongoing for unstable. Packages in the new suites for Lenny and Squeeze would only use the "main" component, with the old components remaining available via the current Emdebian Grip repository and mirror as symlinks. i.e. Emdebian will retain the links only until Wheezy becomes old-stable.
At this stage, Emdebian does not expect to need support for experimental.
stable-proposed-updates and testing-proposed-updates could work exactly as in Debian, instead of the previous best-effort support available via the Emdebian team.
The list of architectures supported by Emdebian Grip and the names of those architectures do not change. As armhf is now an official Debian port, Emdebian will gain the same support for armhf.
Unofficial ports cannot be handled via this integrated method and the Emdebian Grip support for sh4 and powerpcspe will continue to be processed using the existing scripts from the emdebian-grip-server package but this package would then drop the buggy support for testing migrations. If an unofficial port is adopted by Debian, that architecture will migrate to using the wanna-build method for Emdebian Grip under the control of the ftp team and release team.
packages.debian.org and packages.qa.debian.org will be able to list the new suites and the versions available within those suites as well as a comparison of the file list and package sizes.
Emdebian will continue to forcibly truncate the long description of packages as part of the emgrip process, subject to other changes in Debian. Other processing of Emdebian Grip packages will remain, including the removal of manpages, translations and documentation, the compression of the copyright file and removal of changelogs. The fate of Emdebian TDebs is undecided as these are not compatible with Debian but Debian TDeb support is in development.
emgrip itself has been modified to support setting the new suite names in the changelog entry which adds the em1 suffix.
Existing lintian profile support can be used to check the packages.
Documentation will be required for maintainers who may not currently know that their package has already been released as part of Emdebian Grip 1.0 (Lenny) and Emdebian Grip 2.0 (Squeeze). This documentation will need to cover the changes made within the package and how to deal with bug reports which refer to the Emdebian Grip version. (Reportbug already has support for directing bug reports which claim a Version: with the em1 suffix to the buildd.emdebian.org pseudo-package in the BTS but this has not been extensively tested.) Maintainers need to be clear that the Emdebian Grip package is binary compatible with the Debian version of the same package without the suffix. Emdebian has not come across packages which change behaviour as a result of the conversion to an Emdebian Grip package.
Maintainers also need to be clear that Emdebian has two main flavours and these changes only affect the binary-compatible Grip flavour currently. Just describing these packages as "Emdebian" may lead to confusion in bug reports. This is why the proposed suite names include "grip" rather than "emdebian".
For Emdebian, once development of Emdebian Crush restarts after the MultiArch transition, consideration will need to be given to how best to handle the necessary changes. Emdebian Crush development is currently stalled but will involve functional changes in selected packages which break binary compatibility with Debian and Emdebian Grip in order to achieve reductions in the dependency chains of those packages. e.g. Crush will support dropping LDAP dependencies from packages like curl and similar changes to cairo, openssh and dpkg as well as a fully functional rebuild of busybox to replace coreutils. Other packages which do not require changes will be drawn from Emdebian Grip. It is currently proposed that Emdebian Crush adopts new source package names and new binary package names for the modified packages and that all consequent problems of dependency resolution and meta packages is left entirely to Emdebian Crush to sort out and must not affect Emdebian Grip or Debian at this stage.

Mechanisms

An emdebian role account on blavet.debian.org will provide the interface to the read-only postgres projectb database and this is queried to obtain data about movements within the Debian archive managed by dak. That data is then to be processed by the emgrip script. A new .changes file is created and is uploaded - initially to emdebian.org and later to ftp-master.

Emdebian scripts

SVN: http://www.emdebian.org/trac/browser/current/host/trunk/debian-grip/trunk

On blavet, the emdebian-blavet.pl script prepares the data.

The emdebian-buildd.pl script downloads the packages for processing and passes them through emgrip, then generates a suitable .changes file.

This is not a normal buildd process, so generating the .changes file needs a little careful handling:

Grip processing is architecture-agnostic so generates a .changes file for any architecture from any architecture. This means that dpkg-genchanges is run under dpkg-architecture -a${targetArch} -c to ensure that only the relevant architecture is specified in the .changes file itself. i.e. we pretend to be native.
As there is no actual build, debian/files does not exist so it is generated.
Not all binary packages from a particular source package are relevant to Grip, so the incoming data omits certain binary packages - this is handled when the debian/files content is prepared.
The current Emdebian Sources.gz file is checked to see if there is a soutce package movement which needs to be handled. Source uploads precede any binary uploads.
The real control and changelog files are extracted from the .dsc. This means that the .changes file Changes: section matches the original upload.
The suite gets changed to have -grip appended.
Current test scripts upload to www.emdebian.org. The change to ftp-master can only happen once the rest of the packages are synchronised to ftp-master.

See DebianGripScripts for details of how the scripts are to be used.

Current process

Using the processes above and the DebianGripScripts:

blavet checks the postgres database tables for recent changes
blavet reads the Debian binary files for packages which have changed and runs emgrip over each relevant file.
blavet uploads the gripped packages to emdebian.org
emdebian.org pushes the updates to the configured emdebian mirrors.

CategoryEmdebian