|
Size: 9196
Comment: Discuss more options, and be a bit more nuanced
|
Size: 6973
Comment: revert to simpler and firmer version
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| There are difficulties when the package source code contains autogenerated files (typically, provided by upstream). Obviously, the source code of those files must be supplied. But that is not sufficient. If autogenerated files are even present, there is a risk that we use those autogenerated files, rather than building things from source. This is highly undesirable, because it means that the files we ship in our .debs have not been generated using the source code we think corresponds to them! Sometimes, what we think of as the source code can't even be built, or builds something noticeably different. These situations are breaches of our principles and of the DFSG. | Auto-generated prebuilt files should be removed from the upstream VCS and source tarballs. Upstream might want to only add prebuilt files to binary packages or bundle the prebuilt files into a single but separate source tarball. Once upstream has fixed the issue, the Debian package can then be updated to the fixed version. If upstream refuse to remove the prebuilt files, then Debian should either repack the upstream tarball using Files-Excluded (if there is a DFSG or size issue) or remove the files in `debian/rules clean` and very early in `debian/rules build`, so that they are always built from source and there is no chance of the prebuilt versions being used by the build process. |
| Line 5: | Line 5: |
| Therefore, usually, '''we should always build our packages from the actual source, and not use any intermediate files. And we should take steps to ''make sure'' that upstream-provided prebuilt files aren't used.''' In many cases auto-generated prebuilt files should be removed from the upstream VCS and source tarballs. Upstream might want to only add prebuilt files to binary packages or bundle the prebuilt files into a single but separate source tarball. If upstream agree, and remove the autogenerated files, the Debian package can then be updated to the fixed version. However, from upstream's point of view, it can be useful to provide autogenerated files (committed in the VCS and in the tarball) if downstreams (including downstream users) might not have the necessary tooling. (Indeed, autoconf's `./configure` is an example, since to build that one needs the right version of autotools installed). If upstream wish to retain prebuilt files, then in Debian we should take steps to make sure that any remaining pre-built files are not used in the Debian package build. Options include: * Repack the upstream tarball using Files-Excluded (if there is a DFSG or size issue) A downside of this approach is that it is quite inconvenient and can complicate exchangeing changes with upstream. * Remove the files in `debian/rules clean` and very early in `debian/rules build`, so that they are always built from source and there is no chance of the prebuilt versions being used by the build process. A downside of this approach is that it means that `debian/rules clean` in a git working tree leaves the git tree dirty (since it has deleted files); this could be mitigated by having a developer-only option to skip it; and anyway, many packages have this problem. This is the approach taken by `dh_autoreconf` for autotools. * Linkfarm in `debian/rules`: you can make a link farm containing only actual source files, and run the package's upstream build system in that. Whether this is convenient can depend on the package build system. A downside of this approach is that it isn't well supported by all of Debian's tooling and it can sometimes confuse editors, IDEs, etc. There may be some exceptions where a lower level of assurance is OK: |
There may be some exceptions to this: |
| Line 25: | Line 15: |
The Debian Free Software Guidelines (DFSG) state in item 2 that for Debian packages, The program must include source code and the FTP-masters interpret this as requiring source for all packages.
Auto-generated prebuilt files should be removed from the upstream VCS and source tarballs. Upstream might want to only add prebuilt files to binary packages or bundle the prebuilt files into a single but separate source tarball. Once upstream has fixed the issue, the Debian package can then be updated to the fixed version. If upstream refuse to remove the prebuilt files, then Debian should either repack the upstream tarball using Files-Excluded (if there is a DFSG or size issue) or remove the files in debian/rules clean and very early in debian/rules build, so that they are always built from source and there is no chance of the prebuilt versions being used by the build process.
There may be some exceptions to this:
- autotools files should probably be removed from the upstream VCS but kept in the upstream tarball since autotools is meant to be used this way, but the Debian package should rebuild the generated autotools files using autoreconf. Modern debhelper compat versions will do this automatically.
If the prebuilt files take multiple weeks to build or require specialised hardware to build, then it is acceptable for Debian to use the prebuilt files, as long as they are reproducibly buildable within Debian main using only the declared build dependencies, they only occur in a separate upstream tarball and the debian/rules has a target to remove and force rebuild of the prebuilt files.
Even if you can show that the prebuilt files can be reproduced from their original source just with packages from Debian main, it is still strongly recommended to regenerate them from source during the build. The advantages of doing so are:
- re-test on every rebuild that the prebuilt files actually can be reproduced with what is currently in the build environment in Debian
- the version of the tools used to generate the files in Debian may contain important changes such as security fixes, reproducibility improvements, internationalization improvements, and added features (for example automatically enabling cross-build support for autoconf)
- document the required steps to build the project from source
There is not yet any manual tracking of packages that embed prebuilt files (including unused ones).
No wiki pages mention lists of prebuilt files yet.
There are several tools for detecting prebuilt files:
Lintian detects prebuilt binaries, doxygen documentation, Flash objects, Flash projects, Java objects, JavaScript objects, Windows documentation, pandoc documentation, Python objects, Silverlight objects, sphinx documentation, WASM binaries, Windows binaries and Parse::Yapp parsers.
suspicious-source from devscripts detects all files that are not from a list of MIME types of source files
deblob can detect some types of prebuilt files.
linux-libre deblob detects blobs embedded in source files
scancode-toolkit detects strings indicative of generated files
licensecheck detects via Regexp::Pattern::License strings indicative of generated files
check-all-the-things has a few tests for detecting prebuilt files.
The Debian Sources website collects hashes and ctags of all Debian source code and allows searching for specific hashes and ctags, which may be useful for detecting specific prebuilt files in multiple source packages.
If you have a prebuilt file with some unique string you can likely find other copies using the Debian code search site or external code search engines such as searchcode and GitHub.
If a prebuilt file has a fairly unique name or extension, you can often find copies of that file by searching the contents of Debian source packages using apt-file:
apt-file search -I dsc somefile apt-file search -I dsc -x '\.o$'
Some Debian folks keep track of prebuilt files they found via usertags: