This is the homepage outlining the progress being made on packaging Tensorflow for debian. Any information gleamed from various threads in the debian-ai mailing list will be made available here.
The Salsa repo is here: https://salsa.debian.org/deeplearning-team/tensorflow The package is classic quilt (3.0) source, so patches outside the debian dir should be managed in quilt. There are upstream, pristine-tar and master branches, which I believe is the conventional git-buildpackage layout.
TensorFlow is an important package for Debian due to it's importance as one of the top 2 Machine Learning frameworks alongside PyTorch. Packaging work is ongoing since the build system bazel was uploaded in Oct 2020. This work is being done under the Debian Deep Learning team banner, co-ordinated on the debian-ai mailing list.
Because bazel is new (and has some ideas antithetical to debian packaging) there is no debhelper support so packaging is relatively hard work, and includes developing workflows and patterns for debian builds of bazel-using packages.
Before bazel was packaged in debian it was very difficult to build TensorFlow. Mo Zhou <firstname.lastname@example.org> worked around it by using an alternate build system and succeeded in uploading a package to experimental [https://tracker.debian.org/news/1015398/accepted-tensorflow-1101dfsg-a2-source-into-experimental/|experimental]] However it was removed the following year due to the maintenance burden for a fast-moving package being too high. The Debian-Med team along with Google managed to package Bazel as bazel-bootstrap in October 2020, enabling a more conventional packaging effort to proceed.
Wookey <email@example.com> has recently (1st half of 2020) been doing work on getting it in a state suitable for upload.
Version 2.3.1 is being packaged initially. We will move to 2.4 as soon as 2.3.1 is basically working.
The current packaging produces three packages: libtensorflow-cc2 (C++ library) libtensorflow-framework2 (framework library) libtensorflow-dev (headers, pkgconfig files)
The c-library version does not yet build because it depends on unpackaged things: google-storageapi and various aws-* packages The python bindings and tensorflowlite libraries have not yet been packaged but are high priorities.
Conversations are taking place under the firstname.lastname@example.org mailing list and it is best to keep TensorFlow related discussions there if possible.
Things still to do:
Salsa ci Continuous integration (https://salsa.debian.org/salsa-ci-team/pipeline)
googleapis packaging (https://lists.debian.org/debian-ai/2021/04/msg00004.html) Repo at https://salsa.debian.org/deeplearning-team/googleapis
- static builds (are these actually useful?)
- debug builds
- fix rpaths properly (in bazel/upstream)
- Cleaner clean builds
- Better mechanism for defaulting to system-libraries
- Get bazel to install source files (e.g headers)
- Generate library .so symlinks properly
- ensure builds will use accelerators where possible (vector units, GPUs, NPUs)
- is bazel cache/output in /tmp insecure?
- get non-empty debug packages (Currently the 'dbg' build fails, and the 'opt' build has no symbols for dh_dwz to filter out)
- System libraries
bazel is from the new breed of build systems that love to download 'latest everything' off the net, and cache binaries across builds for speed. This is nice for developers but the exact opposite of a debian package: self-contained, reproducible, offline-buildable, using system libraries. Tensorflow has a TF_SYSTEM_LIBS mechanism which can list a set of libraries to use the system versions of, but this is somewhat laborious and arse-backwards: a sane build system should default to system paths and libraries, and have exceptions for local copies, not the other way round. bazel has support for cc_import as well as cc_shared (https://docs.bazel.build/versions/master/be/c-cpp.html#cc_import). I'm pretty sure there are better ways to do this than the current stuff, which we could upstream for sane 'distro builds'.
- 'make install' The bazel design is sandboxed so it deliberately cannot do a 'make install' type operation outside its build tree. However it does track the files for dependencies and can install into a tarball (and maybe local file tree?). We should probably devise a generic mechanism to move such output into debian/tmp from whence normal debhelper operations can occur. This particularly affects the -dev package which currently collects all the headers using rsync runes in debian/rules from the source tree
- rpaths tensorflow builds with rpaths set by default, and I have failed to turn them off so they are currently hacked out with chrpath in debian/rules. Could be better.
- mock_repos The build sets up override 'mock repo' workspaces for skylib, cc and java. I'm not sure why?
- Bazel has 'bazel clean --expunge' but it doesn't actually clean the cache. Why not? Currently we add the lumphammer of rm -rf /tmp/.cache/bazel, but I think that cleans _all_ bazel builds on this system, not just the package being built. Can we do better?
- bazel does everything under an output dir. Currently set to /tmp/.cache/bazel. Do we need to worry about insecurities in /tmp being world-writeable? A comment says that using the more obvious $(CURDIR)/.cache/bazel produces recursive sysmlinks. Needs checking.
Current open questions arising from mailing list conversations:
> > However then this question of ABIs gets sidetracked by something I
> > noticed whilst looking at the symbols situation: The symbols file for
> > libtensorflow_cc2 is 24MB (that's really quite fat) Is it worth
> > putting that in the package? I'm not sure anyone is going to actually
> > 'maintain' it beyond autogenerating a new one each version. Symbols
> > files work OK for C but are bloated and awkward for C++. Even so 24MB
> > seems huge.
> C++ symbols are known to be hard to track. As currently we don't
> expect many reverse dependencies of libtensorflow, maybe we should
> not track it manually, at least for now.
Any differing opinions in the crowd?
Commands to attempt build
# Commands to attempt to build packages locally if useful: DEBEMAIL="email@example.com" >DEBFULLNAME="Firstname Lastname" export DEBEMAIL DEBFULLNAME git clone https://salsa.debian.org/deeplearning-team/tensorflow.git salsa wget http://wookware.org/files/tensorflow_2.3.1.orig.tar.xz cd salsa # Install the build-depends of tensorflow mk-build-deps --install --tool 'apt-get --yes --no-install-recommends' --remove ./debian/control rm -rf tensorflow-build-deps_*.buildinfo rm -rf tensorflow-build-deps_*.changes debuild -sa -b -uc -us --jobs=8 || true
Adding Patches to Source
TensorFlow packaging uses Quilt for patches rather than Git. A very useful tutorial on its usage can be found here: https://raphaelhertzog.com/2012/08/08/how-to-use-quilt-to-manage-patches-in-debian-packages/
Building Tensorflow Lite
Tensorflow Lite has not yet been packaged, figuring out how to add it as a target for Bazel is the issue, there is however the possibility of building it with CMake from v2.4 onwards:
Packages depending on Tensorflow