|
Size: 11172
Comment: add a sudo
|
Size: 9727
Comment: Add note about sphinx 1.8 patch to buster roadmap.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Some people within [[DebianScience]] would like to package again Sage for Debian. An old version of Sage got packaged long time ago by Tim Abbott, but, because of lack of maintainance, it became totally unusable and was removed from Debian. The status pages ([[http://people.debian.org/~thansen/debian-sage-status.html|master]], [[http://people.debian.org/~thansen/debian-sage-dev-status.html|develop]]) give an overview about package versions in Sage and Debian. If you want to help out, pick one of the open tasks or issues below and get going. Or contact us via the [[https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath|mailing list]] or in #debian-science on irc.oftc.net. | Some people within [[DebianScience]] have again packaged [[https://packages.debian.org/source/sid/sagemath|Sage for Debian]]. This is one of the most complex packages in Debian and will need continual maintenance - we are always looking for new contributors! If you want to help out, read below to learn how to build the package from source, then pick one of the [[#Open_tasks|open tasks]] and get going. You can contact us via the [[https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath|mailing list]] or in #debian-science on irc.oftc.net. |
| Line 3: | Line 3: |
| = Building Sage with Debian packages = | <<TableOfContents()>> |
| Line 5: | Line 5: |
| We have a git repository ([[https://anonscm.debian.org/cgit/debian-science/packages/sagemath.git|sagemath]]) to document the steps we currently do to experiment with building Sage against Debian packages. Important elements are the build dependencies of Sage, patches for Sage and a script called "pruner" that uses automake to determine if the necessary packages are installed (see also [[https://lists.debian.org/debian-science/2015/02/msg00024.html|this mail]]). | = Build from source = |
| Line 7: | Line 7: |
| Start with Debian unstable with [[DebianExperimental|experimental]] sources enabled. It is nice to do this in a chroot with [[Schroot|schroot]] (>=1.6.10-2 to avoid the [[https://falstaff.agner.ch/2013/10/29/setting-up-schroot-for-python-multiprocessing/|workaround to enable Python multiprocessing]]). |
Our packages in the Debian official archives should be buildable using the [[https://www.debian.org/doc/manuals/maint-guide/build.en.html|normal Debian build processes]]. This is for users that wish to build from source, as well as build bots. |
| Line 10: | Line 9: |
| thansen: I still need the workaround. If Python multiprocessing does not work in a chroot, and the build fails with "Permission denied", add the following line to '''/etc/schroot/default/fstab''': {{{ tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 }}} |
For maintainers, our development workflow is to build from [[https://salsa.debian.org/science-team/sagemath|our git repository]]. Important elements are the build dependencies of Sage, patches for Sage and a script called "pruner" that uses autoconf to determine if the necessary packages are installed. |
| Line 15: | Line 11: |
| To set up your system for working with our repo: | ||{{{git clone --recursive https://salsa.debian.org/science-team/sagemath.git && cd sagemath}}}|| Recursive git clone to also get sage which is included as a git submodule.|| |
| Line 17: | Line 13: |
| ||{{{sudo apt-get install git quilt devscripts python-sphinx}}}|| Install packages that are required before installing the build dependencies. || ||{{{for d in deb deb-src; do \}}}<<BR>>{{{ echo "$d https://debian-science.alioth.debian.org/apt sid-sage/"; done \}}}<<BR>>{{{ | sudo tee /etc/apt/sources.list.d/deb-sci-sage.list}}}|| Add our unstable-sage repo for [[https://debian-science.alioth.debian.org/apt/sid-sage/|patched dependencies]]. This is optional; if you prefer you can build these yourself from the instructions in the table below. || ||{{{gpg --recv-keys 0xD1188AF4A1398A5FCC3F576654B3F967097676A0}}}|| Get the repo key. || ||{{{gpg --export 0xD1188AF4A1398A5FCC3F576654B3F967097676A0 | sudo apt-key add -}}}|| Install the repo key. || ||{{{printf "Package: *\nPin: release n=sid-sage\nPin-Priority: 100\n" \}}}<<BR>>{{{ | sudo tee /etc/apt/preferences.d/deb-sci-sage}}}<<BR>>{{{sudo apt-get update}}}|| Pin our repo at a low priority so it doesn't mess with the rest of your system. OTOH, if you really really want this repo's packages, to make sure you get the latest and greatest efforts of the Debian Sage packaging team above all else (including maybe our patched dependency packages messing up your system) then you can set the Pin-Priority to a high number like 800.|| ||{{{git clone --recursive https://anonscm.debian.org/git/debian-science/packages/sagemath.git}}}|| Recursive git clone to also get sage which is included as a git submodule.|| |
To work with this repository, you must start with '''Debian unstable''' (sid) with [[DebianExperimental|experimental]] sources enabled. It is nice - optional, but highly recommended - to do this in an schroot, which you can set up using the instructions in [[sbuild]], after which you'll probably need to apply the following workarounds: <<FootNote(If your build fails with "Permission denied", try adding the following line to `/etc/schroot/sbuild/fstab`: __`tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0`__ - more details [[https://falstaff.agner.ch/2013/10/29/setting-up-schroot-for-python-multiprocessing/|here]])>>. <<FootNote(If your build fails with "out of pty devices", log into the chroot using __`sudo sbuild-shell source:${chroot_name}`__ then run __`rm /dev/ptmx && mknod -m 666 /dev/ptmx c 5 2`__ - more details in DebianBug:817236.)>>. Next, see `debian/README.source` section "Local system patches", for any workarounds you might have to apply. Then: ||{{{debian/rules build-dep-maint}}}|| Install packages that are required before installing the build dependencies. || |
| Line 26: | Line 19: |
| ||{{{debian/rules reset}}}|| Apply patches and force-reset the upstream sage submodule. Note: also removes ~/.sage/local and ~/.local/share/sagemath || ||{{{debian/rules build-dep}}}|| Install build dependencies. (Run 'apt-get -f install' afterwards.) || |
||{{{debian/rules build-dep}}}|| Install build dependencies; you may need to run `apt-get -f install` afterwards. If this fails, install `aptitude` then try it again. || ||{{{debian/rules reset}}}|| Apply patches and force-reset the upstream sage submodule. || |
| Line 30: | Line 23: |
| ||{{{fakeroot debian/rules binary}}} || Build the Debian packages. || | |
| Line 31: | Line 25: |
| The output of the pruner tells you in the end which dependencies were not found on the machine. These will be downloaded and built in the build step, before Sage itself is built. Currently (October 2016) the pruner should output something like the following. {{{ ipywidgets will not come from debian! jmol will not come from debian! notebook will not come from debian! planarity will not come from debian! sagenb_export will not come from debian! widgetsnbextension will not come from debian! 144 system packages will be used }}} |
The output of the pruner tells you in the end which dependencies were not found on the machine ("__`XXX will not come from Debian`__"). Since all dependencies should come from Debian, such a line normally indicates that something went wrong. The pruner should only output __`XXX system packages will be used`__. Otherwise, if you are sure that you want to try this, you could go on nevertheless. Sage will then try to download and build the missing dependencies in the build step, before Sage itself is built. |
| Line 42: | Line 29: |
| See debian/README.Debian for any workarounds you might have to apply, such as patching local system files. | See `debian/README.source` for further details, such as how to debug the failing doctests. We allow a cetain number of tests to fail, it's always good to lower the number of failing tests.<<FootNote(It's perhaps "not worth it" for us to lower it to 0, since the final 20 or so are not "hard" errors but are uninteresting differences like different but equivalent symbolic mathematical expressions or slightly different random number algorithms; these are generally caused by different versions of dependencies in Debian. "Fixing" them would simply involve patching Debian's Sage to expect different values. This is good for documentation purposes but you may also decide that your time is better spent doing other, more important, tasks.)>> Once you are happy with the full build process, you can try a test release build: ||{{{debian/rules release-experimental}}} || Build using [[sbuild]], with experimental sources enabled. || |
| Line 46: | Line 37: |
| == Packages in our development APT repository: == Up-to-date information about the packages in the repo can be found on the status pages. ([[http://people.debian.org/~thansen/debian-sage-status.html|master branch]], [[http://people.debian.org/~thansen/debian-sage-dev-status.html|develop branch]]) |
The status pages ([[http://people.debian.org/~thansen/debian-sage-status.html|master]], [[http://people.debian.org/~thansen/debian-sage-dev-status.html|develop]]) give an overview about package versions in Sage and Debian. Current high-priority tasks are: |
| Line 49: | Line 39: |
| ||<tablestyle="border:1pt solid" width=139pt> '''Package''' ||<width=80pt> '''Bug / Link''' || '''Reason for being in the repository''' || || [[https://anonscm.debian.org/cgit/debian-science/packages/polybori.git|brial/polybori]] || [[https://packages.qa.debian.org/p/polybori.html|PTS]] || polybori has been renamed to [[https://github.com/BRiAl/BRiAl|brial]], package is in NEW. || || [[https://anonscm.debian.org/cgit/debian-science/packages/fplll.git|fplll]] || TBD || Sage 7.3 uses old development versions of libfplll. Let's wait for Sage 7.4 which should work with the fplll from unstable. || || [[https://anonscm.debian.org/cgit/debian-science/packages/gap.git|gap]] || || Sage 7.3 uses former version 4.8.3 of GAP. || || [[https://anonscm.debian.org/cgit/debian-science/packages/maxima-sage.git|maxima-sage]] || DebianBug:779804 || Sage uses ECL as Common Lisp compiler, Maxima in Debian uses GCL and should be patched to support both. || || [[https://packages.qa.debian.org/p/python-numpy.html|numpy]] || TBD || Debian maintainer needs to apply [[https://github.com/numpy/numpy/pull/6659|this proposed upstream patch]] - basically accepted but needs unit tests || || [[https://anonscm.debian.org/cgit/debian-science/packages/pari-sage.git|pari-sage]] || DebianBug:835027 || Sage uses PARI 2.8 || || [[https://anonscm.debian.org/git/python-modules/packages/cysignals.git|cysignals]] || [[https:/packages.qa.debian.org/c/cysignals.html|PTS]] || Sage uses CySIgnals with PARI (>=2.8) support (cf [[https://sources.debian.net/src/cysignals/1.1.1%2Bds-2/debian/README.source/|d/README.source]]).|| || [[https://anonscm.debian.org/cgit/debian-science/packages/eclib.git|eclib]] || || Must be re-built against our version of PARI. || || [[https://anonscm.debian.org/cgit/debian-science/packages/lcalc.git|lcalc]] || || Must be re-built against our version of PARI || || [[https://anonscm.debian.org/cgit/debian-science/packages/sagenb.git|sagenb]] || || The current sagenb package ships a bunch of javascript libraries. It should use at least these Debian packages instead: [[https://packages.qa.debian.org/m/mathjax.html|mathjax]], [[https://packages.qa.debian.org/t/tinymce.html|tinymce]], [[https://packages.qa.debian.org/j/jquery.html|jquery]], [[https://packages.qa.debian.org/j/jqueryui.html|jqueryui]], [[https://packages.qa.debian.org/c/codemirror-js.html|codemirror]] || ||nbsphinx|| DebianBug:838806 || jupyter-sphinx-theme Build-Depends on nbsphinx; in NEW queue.|| ||jupyter-sphinx-theme || DebianBug:838798 || ipywidgets Build-Depends on jupyter-sphinx-theme; an ITP has been submitted (DebianBug:838798).|| |
1. Deal with RC bugs in sagemath and its dependencies (see [[https://tracker.debian.org/teams/debian-sagemath/|tracker]]). 1. Get the number of failed doctests down (see [[http://people.debian.org/~thansen/sage-test-status.html|unstable]], [[http://people.debian.org/~thansen/sage-test-status-experimental.html|experimental]]). 1. Upstream patches that are suitable for upstream (see [[https://salsa.debian.org/science-team/sagemath/blob/experimental/debian/patches/series|series]]). 1. Package new sagemath versions and dependencies in experimental, upload everything to unstable together. |
| Line 63: | Line 44: |
| == Other packages: == | See `debian/TODO` for more tasks and more details. |
| Line 65: | Line 46: |
| ||<tablestyle="border:1pt solid" width=139pt> '''Package''' ||<width=80pt> '''Bug / Link''' || '''Assigned''' ||'''Comments''' || ||<-4 style="border-top:1pt solid; border-bottom:1pt solid"> '''Need to be packaged:''' || ||ipywidgets || DebianBug:838684 || || || ||jupiter-notebook || DebianBug:801366 || || || ||[[https://anonscm.debian.org/cgit/debian-science/packages/sagenb-export.git/|sagenb_export]] || DebianBug:839782|| || Build-Depends on jupyter-notebook || ||widgetsnbextension || DebianBug:838683 || || || ||<-4 style="border-top:1pt solid; border-bottom:1pt solid"> '''Need to be packaged for Sagemath 7.4:''' || || fpylll || DebianBug:841005 || || || || thebe || || || [[https://github.com/oreillymedia/thebe]], lots of [[https://github.com/oreillymedia/thebe/blob/master/bower.json|javascript dependencies]] || ||<-4 style="border-top:1pt solid; border-bottom:1pt solid"> '''Packages that need a new maintainer and an upload to Debian:''' || ||[[https://packages.qa.debian.org/j/jmol.html|jmol]] || || || There is a request for help, see DebianBug:719330 || |
== Buster roadmap == |
| Line 77: | Line 48: |
| = Uploading patched packages = | The buster freeze [[https://release.debian.org/buster/freeze_policy.html|is approaching]]. We can still upload last improvements to the package if we upload at least 10 days before the full freeze on March 12. Please test the sagemath package! |
| Line 79: | Line 50: |
| We've set up a mini-dinstall APT repo to upload our patched Debian packages to, so that we can build sage against these. This is a temporary measure to make everything work more quickly, and these patched packages are unlikely to be added to Debian officially. | Things that should be looked at: |
| Line 81: | Line 52: |
| To upload, you must be a member of the debian-science group on alioth. Then, you should add this to your ~/.dput.cf: | * Fix build on mipsel and mips64el: DebianBug:920147 * Apply improvements to the sphinx 1.8 patch if they emerge [[https://trac.sagemath.org/ticket/26451|upstream]]. * Fix `sage/matrix/matrix_double_dense.pyx # Testing error: no result` in `sage/matrix/matrix_double_dense.pyx` when atlas is installed. * Fix numpy deprecation warnings from matplotlib: DebianBug:918819 |
| Line 83: | Line 57: |
| = Updating Sage in Debian = Every version of Sage is tested upstream with a specific version of each dependency (see our status pages). It is not always possible to provide the same versions of all dependencies in Debian. It is however advisable to follow the versions as closely as possible, especially for specialized math packages, which tend to have less stable interfaces than general purpose libraries. This means that each update of the package '''sagemath''' comes together with a bunch of updates of other packages. As some of these updates could break the previous sagemath version in unstable, many of these updates should first be uploaded to Debian experimental. The new version of Sage will then be uploaded to experimental as well. Once the new Sage version has been tested in experimental and is considered ready for unstable, the batch of package updates for this version is migrated from Debian experimental to unstable. This should be coordinated and not take too long, to avoid having a broken '''sagemath''' package in unstable for a long time. == Testing migration == In order to have a working '''sagemath''' package in Debian testing at all times, the whole batch of package updates for a new Sage version needs to migrate to testing together. To this end, the dependencies should declare |
|
| Line 84: | Line 69: |
| [deb-sci-sage] method = scp fqdn = alioth.debian.org incoming = /home/groups/debian-science/htdocs/apt/mini-dinstall/incoming/ post_upload_command = ssh alioth.debian.org /home/groups/debian-science/mini-dinstall |
Breaks: sagemath (<< x.y~) |
| Line 90: | Line 71: |
| (where x.y is the new Sage version) for at least one binary package on which the binary package '''sagemath''' depends. An exception to this are libraries with a soname jump (if sagemath depends only on the library package). In this case, the new version of the library may migrate to testing before sagemath, because the old library package will be kept around until it is not used anymore. |
|
| Line 91: | Line 75: |
| `mini-dinstall` is pretty forgiving and you should be able to overwrite any existing version in this repo or even downgrade versions. However, you must set the distribution in debian/changelog to `sid-sage` or `unstable-sage`, otherwise your package will get rejected. Alternatively, do the build without changing debian/changelog, then afterwards edit the .changes file to say `Distribution: sid-sage`, then debsign it and upload. Please include the .orig tarball in your upload, which helps our status pages report more accurately. To do this, you can give `--changes-option=-sa` to `dpkg-buildpackage`, or if you forget to do this then you can run `changestool $XXX.changes includeallsources` after the build, then debsign it and upload. To remove a package: * {{{ssh}}} to {{{alioth.debian.org}}} * Delete the packages from {{{/home/groups/debian-science/htdocs/apt/sid-sage}}} * Run {{{/home/groups/debian-science/mini-dinstall -b --no-db}}} |
|
| Line 104: | Line 80: |
| * [[https://debian-science.alioth.debian.org/apt/sid-sage/|Development APT repository]] * [[https://anonscm.debian.org/cgit/debian-science/packages/sagemath.git|Our sagemath git repo]] |
* [[http://people.debian.org/~thansen/debian-sage-7.4-status.html|Sage 7.4 in Debian status page]] * [[http://people.debian.org/~thansen/sage-test-status.html|Overview over failed doctests in Debian unstable]] * [[http://people.debian.org/~thansen/sage-test-status-experimental.html|Overview over failed doctests in Debian experimental]] * [[https://salsa.debian.org/science-team/sagemath|Our sagemath git repo]] |
| Line 107: | Line 85: |
| * [[https://tracker.debian.org/teams/debian-sagemath/|Our team on tracker.debian.org]] Sage out of Debian: * [[http://fedoraproject.org/wiki/SIGs/SciTech/SAGE|Sage in Fedora]] ([[https://apps.fedoraproject.org/packages/sagemath/sources|patches]]) * [[https://wiki.archlinux.org/index.php/SageMath|Sage in Arch Linux]] ([[https://git.archlinux.org/svntogit/community.git/tree/trunk?h=packages/sagemath|patches]]) * [[https://github.com/cschwan/sage-on-gentoo/tree/master/sci-mathematics/sage/files|Sage-on-Gentoo patches]] * [[https://trac.sagemath.org/query?summary=~upgrade&desc=1&order=status|List of planned package upgrades in the SageMath Trac]] Historical information: * [[https://lists.debian.org/debian-science/2015/02/msg00024.html|E-mail]] announcing the pruner script. |
|
| Line 108: | Line 98: |
| * [[http://fedoraproject.org/wiki/SIGs/SciTech/SAGE|Sage in Fedora]] * [[https://wiki.archlinux.org/index.php/SageMath|Sage in Arch Linux]] ([[https://git.archlinux.org/svntogit/community.git/tree/trunk?h=packages/sagemath|patches]]) * [[https://trac.sagemath.org/query?summary=~upgrade&desc=1&order=status|List of planned package upgrades in the SageMath Trac]] |
* Sage [[http://snapshot.debian.org/package/sagemath/|3.0.5]] got packaged long time ago by Tim Abbott, but, because of lack of maintainance, it became totally unusable and was [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=573538|removed]] from Debian. |
Some people within DebianScience have again packaged Sage for Debian. This is one of the most complex packages in Debian and will need continual maintenance - we are always looking for new contributors! If you want to help out, read below to learn how to build the package from source, then pick one of the open tasks and get going. You can contact us via the mailing list or in #debian-science on irc.oftc.net.
Contents
Build from source
Our packages in the Debian official archives should be buildable using the normal Debian build processes. This is for users that wish to build from source, as well as build bots.
For maintainers, our development workflow is to build from our git repository. Important elements are the build dependencies of Sage, patches for Sage and a script called "pruner" that uses autoconf to determine if the necessary packages are installed.
git clone --recursive https://salsa.debian.org/science-team/sagemath.git && cd sagemath |
Recursive git clone to also get sage which is included as a git submodule. |
To work with this repository, you must start with Debian unstable (sid) with experimental sources enabled. It is nice - optional, but highly recommended - to do this in an schroot, which you can set up using the instructions in sbuild, after which you'll probably need to apply the following workarounds: 1. 2. Next, see debian/README.source section "Local system patches", for any workarounds you might have to apply. Then:
debian/rules build-dep-maint |
Install packages that are required before installing the build dependencies. |
To build the package, run:
debian/rules build-dep |
Install build dependencies; you may need to run apt-get -f install afterwards. If this fails, install aptitude then try it again. |
debian/rules reset |
Apply patches and force-reset the upstream sage submodule. |
debian/rules prune |
Run the pruner script. Check that the output matches the one below. |
debian/rules build |
Build sage and run the test suite. |
fakeroot debian/rules binary |
Build the Debian packages. |
The output of the pruner tells you in the end which dependencies were not found on the machine ("XXX will not come from Debian"). Since all dependencies should come from Debian, such a line normally indicates that something went wrong. The pruner should only output XXX system packages will be used. Otherwise, if you are sure that you want to try this, you could go on nevertheless. Sage will then try to download and build the missing dependencies in the build step, before Sage itself is built.
See debian/README.source for further details, such as how to debug the failing doctests. We allow a cetain number of tests to fail, it's always good to lower the number of failing tests.3
Once you are happy with the full build process, you can try a test release build:
debian/rules release-experimental |
Build using sbuild, with experimental sources enabled. |
Open tasks
The status pages (master, develop) give an overview about package versions in Sage and Debian. Current high-priority tasks are:
Deal with RC bugs in sagemath and its dependencies (see tracker).
Get the number of failed doctests down (see unstable, experimental).
Upstream patches that are suitable for upstream (see series).
- Package new sagemath versions and dependencies in experimental, upload everything to unstable together.
See debian/TODO for more tasks and more details.
Buster roadmap
The buster freeze is approaching. We can still upload last improvements to the package if we upload at least 10 days before the full freeze on March 12. Please test the sagemath package!
Things that should be looked at:
Fix build on mipsel and mips64el: 920147
Apply improvements to the sphinx 1.8 patch if they emerge upstream.
Fix sage/matrix/matrix_double_dense.pyx # Testing error: no result in sage/matrix/matrix_double_dense.pyx when atlas is installed.
Fix numpy deprecation warnings from matplotlib: 918819
Updating Sage in Debian
Every version of Sage is tested upstream with a specific version of each dependency (see our status pages). It is not always possible to provide the same versions of all dependencies in Debian. It is however advisable to follow the versions as closely as possible, especially for specialized math packages, which tend to have less stable interfaces than general purpose libraries.
This means that each update of the package sagemath comes together with a bunch of updates of other packages. As some of these updates could break the previous sagemath version in unstable, many of these updates should first be uploaded to Debian experimental. The new version of Sage will then be uploaded to experimental as well.
Once the new Sage version has been tested in experimental and is considered ready for unstable, the batch of package updates for this version is migrated from Debian experimental to unstable. This should be coordinated and not take too long, to avoid having a broken sagemath package in unstable for a long time.
Testing migration
In order to have a working sagemath package in Debian testing at all times, the whole batch of package updates for a new Sage version needs to migrate to testing together. To this end, the dependencies should declare
Breaks: sagemath (<< x.y~)
(where x.y is the new Sage version) for at least one binary package on which the binary package sagemath depends. An exception to this are libraries with a soname jump (if sagemath depends only on the library package). In this case, the new version of the library may migrate to testing before sagemath, because the old library package will be kept around until it is not used anymore.
Links
Sage out of Debian:
Historical information:
E-mail announcing the pruner script.
Sage 3.0.5 got packaged long time ago by Tim Abbott, but, because of lack of maintainance, it became totally unusable and was removed from Debian.
If your build fails with "Permission denied", try adding the following line to /etc/schroot/sbuild/fstab: tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 - more details here (1)
If your build fails with "out of pty devices", log into the chroot using sudo sbuild-shell source:${chroot_name} then run rm /dev/ptmx && mknod -m 666 /dev/ptmx c 5 2 - more details in 817236. (2)
It's perhaps "not worth it" for us to lower it to 0, since the final 20 or so are not "hard" errors but are uninteresting differences like different but equivalent symbolic mathematical expressions or slightly different random number algorithms; these are generally caused by different versions of dependencies in Debian. "Fixing" them would simply involve patching Debian's Sage to expect different values. This is good for documentation purposes but you may also decide that your time is better spent doing other, more important, tasks. (3)