Differences between revisions 1 and 507 (spanning 506 versions)
Revision 1 as of 2010-10-02 15:29:21
Size: 1799
Comment: Issues to package sage
Revision 507 as of 2018-10-10 13:30:11
Size: 9041
Editor: ?TobiasHansen
Comment: Shorten text about failing tests.
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Some people within [[DebianScience]] would like to package again Sage for Debian. An old version of Sage got packaged long time ago by Tim Abbott, but, because of lack of maintainance, it became totally unusable and was removed from Debian. Some people within [[DebianScience]] have again packaged [[https://packages.debian.org/source/sid/sagemath|Sage for Debian]]. This is one of the most complex packages in Debian and will need continual maintenance - we are always looking for new contributors! If you want to help out, read below to learn how to build the package from source, then pick one of the [[#Open_tasks|open tasks]] and get going. You can contact us via the [[https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath|mailing list]] or in #debian-science on irc.oftc.net.
Line 3: Line 3:
Here is the current list of open issues:
 * flintqs / SIMPQS issue. FlintQS should probably be removed as a SPKG in Sage, but it may require work (qsieve). Bill Hart may be able to comment further (William Stein).
 * opencdk / gnutls. William Stein thinks it is only needed to "provide a secure SSL mode for the Sage notebook". So we can probably use the Debian version of gnutls, but that will require assessment.
 * Sage does not yet support scipy 0.8 because it depends on numpy-1.4 and "Sage is not going there yet". Strangely, Debian sid currently has scipy 0.7.2 (good), but includes numpy 1.4.1 (ut-oh). So that will probably create issues.
 * This pickles patch is still not integrated into Python upstream, but it is needed for Sage: http://bugs.python.org/issue7689.
 * This python2.6 issue probably affects sid: http://bugs.python.org/issue7491 which breaks stuff, witness: http://github.com/cschwan/sage-on-gentoo/issues#issue/1
 * pexpect issue (Debian pkg: python-pexpect): Debian ships version 2.3, but Sage requires version 2.0 (though one of the necessary patches is already in 2.4). Plotting in the notebook is likely to break with newer versions that are in Debian; and >2.0 is reported to be slow.
 * ecl is OK unless compiled with Unicode support (http://github.com/cschwan/sage-on-gentoo/issues/closed#issue/2). Debian may do that, so it could be an issue.
 * Unpackaged dependencies:
  * ratpoints
  * cliquer
 * Dependencies that need to be updated:
  * libflint
  * libfplll0
  * gfan
  * lcalc
  * m4ri
<<TableOfContents()>>

= Build from source =

Our packages in the Debian official archives should be buildable using the [[https://www.debian.org/doc/manuals/maint-guide/build.en.html|normal Debian build processes]]. This is for users that wish to build from source, as well as build bots.

For maintainers, our development workflow is to build from [[https://salsa.debian.org/science-team/sagemath|our git repository]]. Important elements are the build dependencies of Sage, patches for Sage and a script called "pruner" that uses autoconf to determine if the necessary packages are installed.

||{{{git clone --recursive https://salsa.debian.org/science-team/sagemath.git && cd sagemath}}}|| Recursive git clone to also get sage which is included as a git submodule.||

To work with this repository, you must start with '''Debian unstable''' (sid) with [[DebianExperimental|experimental]] sources enabled. It is nice - optional, but highly recommended - to do this in an schroot, which you can set up using the instructions in [[sbuild]], after which you'll probably need to apply the following workarounds: <<FootNote(If your build fails with "Permission denied", try adding the following line to `/etc/schroot/sbuild/fstab`: __`tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0`__ - more details [[https://falstaff.agner.ch/2013/10/29/setting-up-schroot-for-python-multiprocessing/|here]])>>. <<FootNote(If your build fails with "out of pty devices", log into the chroot using __`sudo sbuild-shell source:${chroot_name}`__ then run __`rm /dev/ptmx && mknod -m 666 /dev/ptmx c 5 2`__ - more details in DebianBug:817236.)>>. Next, see `debian/README.source` section "Local system patches", for any workarounds you might have to apply. Then:

||{{{debian/rules build-dep-maint}}}|| Install packages that are required before installing the build dependencies. ||

To build the package, run:

||{{{debian/rules build-dep}}}|| Install build dependencies; you may need to run `apt-get -f install` afterwards. If this fails, install `aptitude` then try it again. ||
||{{{debian/rules reset}}}|| Apply patches and force-reset the upstream sage submodule. ||
||{{{debian/rules prune}}} || Run the pruner script. Check that the output matches the one below. ||
||{{{debian/rules build}}} || Build sage and run the test suite. ||
||{{{fakeroot debian/rules binary}}} || Build the Debian packages. ||

The output of the pruner tells you in the end which dependencies were not found on the machine ("__`XXX will not come from Debian`__").
Since all dependencies should come from Debian, such a line normally indicates that something went wrong. The pruner should only output __`XXX system packages will be used`__.
Otherwise, if you are sure that you want to try this, you could go on nevertheless. Sage will then try to download and build the missing dependencies in the build step, before Sage itself is built.

See `debian/README.source` for further details, such as how to debug the failing doctests. We allow a cetain number of tests to fail, it's always good to lower the number of failing tests.<<FootNote(It's perhaps "not worth it" for us to lower it to 0, since the final 20 or so are not "hard" errors but are uninteresting differences like different but equivalent symbolic mathematical expressions or slightly different random number algorithms; these are generally caused by different versions of dependencies in Debian. "Fixing" them would simply involve patching Debian's Sage to expect different values. This is good for documentation purposes but you may also decide that your time is better spent doing other, more important, tasks.)>>

Once you are happy with the full build process, you can try a test release build:

||{{{debian/rules release-experimental}}} || Build using [[sbuild]], with experimental sources enabled. ||

= Open tasks =

The status pages ([[http://people.debian.org/~thansen/debian-sage-status.html|master]], [[http://people.debian.org/~thansen/debian-sage-dev-status.html|develop]]) give an overview about package versions in Sage and Debian. Current high-priority tasks are:

 1. Deal with RC bugs in sagemath and its dependencies (see [[https://tracker.debian.org/teams/debian-sagemath/|tracker]]).
 1. Get the number of failed doctests down (see [[http://people.debian.org/~thansen/sage-test-status.html|unstable]], [[http://people.debian.org/~thansen/sage-test-status-experimental.html|experimental]]).
 1. Upstream patches that are suitable for upstream (see [[https://salsa.debian.org/science-team/sagemath/blob/experimental/debian/patches/series|series]]).
 1. Package new sagemath versions and dependencies in experimental, upload everything to unstable together.

See `debian/TODO` for more tasks and more details.


= Updating Sage in Debian =

Every version of Sage is tested upstream with a specific version of each dependency (see our status pages). It is not always possible to provide the same versions of all dependencies in Debian. It is however advisable to follow the versions as closely as possible, especially for specialized math packages, which tend to have less stable interfaces than general purpose libraries.

This means that each update of the package '''sagemath''' comes together with a bunch of updates of other packages. As some of these updates could break the previous sagemath version in unstable, many of these updates should first be uploaded to Debian experimental. The new version of Sage will then be uploaded to experimental as well.

Once the new Sage version has been tested in experimental and is considered ready for unstable, the batch of package updates for this version is migrated from Debian experimental to unstable. This should be coordinated and not take too long, to avoid having a broken '''sagemath''' package in unstable for a long time.

== Testing migration ==

In order to have a working '''sagemath''' package in Debian testing at all times, the whole batch of package updates for a new Sage version needs to migrate to testing together. To this end, the dependencies should declare
{{{
Breaks: sagemath (<< x.y~)
}}}
(where x.y is the new Sage version)
for at least one binary package on which the binary package '''sagemath''' depends.
An exception to this are libraries with a soname jump (if sagemath depends only on the library package). In this case, the new version of the library may migrate to testing before sagemath, because the old library package will be kept around until it is not used anymore.


= Links =

 * [[http://people.debian.org/~thansen/debian-sage-status.html|Sage in Debian status page (master branch)]]
 * [[http://people.debian.org/~thansen/debian-sage-dev-status.html|Sage in Debian status page (develop branch)]]
 * [[http://people.debian.org/~thansen/debian-sage-7.4-status.html|Sage 7.4 in Debian status page]]
 * [[http://people.debian.org/~thansen/sage-test-status.html|Overview over failed doctests in Debian unstable]]
 * [[http://people.debian.org/~thansen/sage-test-status-experimental.html|Overview over failed doctests in Debian experimental]]
 * [[https://salsa.debian.org/science-team/sagemath|Our sagemath git repo]]
 * [[https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath|Our mailing list]]
 * [[https://tracker.debian.org/teams/debian-sagemath/|Our team on tracker.debian.org]]

Sage out of Debian:

 * [[http://fedoraproject.org/wiki/SIGs/SciTech/SAGE|Sage in Fedora]] ([[https://apps.fedoraproject.org/packages/sagemath/sources|patches]])
 * [[https://wiki.archlinux.org/index.php/SageMath|Sage in Arch Linux]] ([[https://git.archlinux.org/svntogit/community.git/tree/trunk?h=packages/sagemath|patches]])
 * [[https://github.com/cschwan/sage-on-gentoo/tree/master/sci-mathematics/sage/files|Sage-on-Gentoo patches]]
 * [[https://trac.sagemath.org/query?summary=~upgrade&desc=1&order=status|List of planned package upgrades in the SageMath Trac]]

Historical information:

 * [[https://lists.debian.org/debian-science/2015/02/msg00024.html|E-mail]] announcing the pruner script.
 * [[http://wiki.debian.org/JulienPuydt|Graph of dependencies between Sage 5.2 packages]]
 * Sage [[http://snapshot.debian.org/package/sagemath/|3.0.5]] got packaged long time ago by Tim Abbott, but, because of lack of maintainance, it became totally unusable and was [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573538|removed]] from Debian.

Some people within DebianScience have again packaged Sage for Debian. This is one of the most complex packages in Debian and will need continual maintenance - we are always looking for new contributors! If you want to help out, read below to learn how to build the package from source, then pick one of the open tasks and get going. You can contact us via the mailing list or in #debian-science on irc.oftc.net.

Build from source

Our packages in the Debian official archives should be buildable using the normal Debian build processes. This is for users that wish to build from source, as well as build bots.

For maintainers, our development workflow is to build from our git repository. Important elements are the build dependencies of Sage, patches for Sage and a script called "pruner" that uses autoconf to determine if the necessary packages are installed.

git clone --recursive https://salsa.debian.org/science-team/sagemath.git && cd sagemath

Recursive git clone to also get sage which is included as a git submodule.

To work with this repository, you must start with Debian unstable (sid) with experimental sources enabled. It is nice - optional, but highly recommended - to do this in an schroot, which you can set up using the instructions in sbuild, after which you'll probably need to apply the following workarounds: 1. 2. Next, see debian/README.source section "Local system patches", for any workarounds you might have to apply. Then:

debian/rules build-dep-maint

Install packages that are required before installing the build dependencies.

To build the package, run:

debian/rules build-dep

Install build dependencies; you may need to run apt-get -f install afterwards. If this fails, install aptitude then try it again.

debian/rules reset

Apply patches and force-reset the upstream sage submodule.

debian/rules prune

Run the pruner script. Check that the output matches the one below.

debian/rules build

Build sage and run the test suite.

fakeroot debian/rules binary

Build the Debian packages.

The output of the pruner tells you in the end which dependencies were not found on the machine ("XXX will not come from Debian"). Since all dependencies should come from Debian, such a line normally indicates that something went wrong. The pruner should only output XXX system packages will be used. Otherwise, if you are sure that you want to try this, you could go on nevertheless. Sage will then try to download and build the missing dependencies in the build step, before Sage itself is built.

See debian/README.source for further details, such as how to debug the failing doctests. We allow a cetain number of tests to fail, it's always good to lower the number of failing tests.3

Once you are happy with the full build process, you can try a test release build:

debian/rules release-experimental

Build using sbuild, with experimental sources enabled.

Open tasks

The status pages (master, develop) give an overview about package versions in Sage and Debian. Current high-priority tasks are:

  1. Deal with RC bugs in sagemath and its dependencies (see tracker).

  2. Get the number of failed doctests down (see unstable, experimental).

  3. Upstream patches that are suitable for upstream (see series).

  4. Package new sagemath versions and dependencies in experimental, upload everything to unstable together.

See debian/TODO for more tasks and more details.

Updating Sage in Debian

Every version of Sage is tested upstream with a specific version of each dependency (see our status pages). It is not always possible to provide the same versions of all dependencies in Debian. It is however advisable to follow the versions as closely as possible, especially for specialized math packages, which tend to have less stable interfaces than general purpose libraries.

This means that each update of the package sagemath comes together with a bunch of updates of other packages. As some of these updates could break the previous sagemath version in unstable, many of these updates should first be uploaded to Debian experimental. The new version of Sage will then be uploaded to experimental as well.

Once the new Sage version has been tested in experimental and is considered ready for unstable, the batch of package updates for this version is migrated from Debian experimental to unstable. This should be coordinated and not take too long, to avoid having a broken sagemath package in unstable for a long time.

Testing migration

In order to have a working sagemath package in Debian testing at all times, the whole batch of package updates for a new Sage version needs to migrate to testing together. To this end, the dependencies should declare

Breaks: sagemath (<< x.y~)

(where x.y is the new Sage version) for at least one binary package on which the binary package sagemath depends. An exception to this are libraries with a soname jump (if sagemath depends only on the library package). In this case, the new version of the library may migrate to testing before sagemath, because the old library package will be kept around until it is not used anymore.

Links

Sage out of Debian:

Historical information:

  1. If your build fails with "Permission denied", try adding the following line to /etc/schroot/sbuild/fstab: tmpfs  /dev/shm  tmpfs  rw,nosuid,nodev  0  0 - more details here (1)

  2. If your build fails with "out of pty devices", log into the chroot using sudo sbuild-shell source:${chroot_name} then run rm /dev/ptmx && mknod -m 666 /dev/ptmx c 5 2 - more details in 817236. (2)

  3. It's perhaps "not worth it" for us to lower it to 0, since the final 20 or so are not "hard" errors but are uninteresting differences like different but equivalent symbolic mathematical expressions or slightly different random number algorithms; these are generally caused by different versions of dependencies in Debian. "Fixing" them would simply involve patching Debian's Sage to expect different values. This is good for documentation purposes but you may also decide that your time is better spent doing other, more important, tasks. (3)