Differences between revisions 12 and 13
Revision 12 as of 2014-05-03 21:01:21
Size: 8639
Editor: JakubWilk
Comment: Fix typos.
Revision 13 as of 2014-05-03 23:17:55
Size: 8695
Editor: GuillemJover
Comment: Update for new dpkg features, reflow and reword some of the entries
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
A: This means another instance of dpkg or a frontend is running, so the correct solutions are to: let the other process end its execution, find the frontend and quit it, or kill any such frozen processes. A: This means another instance of dpkg or a frontend is running, so the correct solutions are to:

 *
let the other process end its execution,
 *
find the frontend and quit it, or
 *
kill any such frozen processes.
Line 19: Line 23:
 * Otherwise, you might consider using the dpkg --force-unsafe-io option, which avoids those fsync(2)s on filesystem files, but still performs them on the dpkg database. This option might improve performance at the cost of losing data, '''use with care'''.
 * If this is still not good enough, you might consider using DebianPkg:eatmydata, which will get rid of any fsync(2) at all. You might want to have backups around, though.
 * Use the dpkg --force-unsafe-io option, which avoids those fsync(2)s on filesystem files, but still performs them on the dpkg database. This option might improve performance at the cost of losing data, '''use with care'''.
 * If --force-unsafe-io is still not good enough, you might consider using (in addition or instead) DebianPkg:eatmydata, which will get rid of any fsync(2) at all. You might want to have backups around, though.
Line 79: Line 83:
A: No. As dpkg does not currently track file metadata, it does not know if a symlink or directory was switched by a previous package or by the sysadmin. As part of the dpkg credo, preserving human configuration is of utmost importance,
and this kind of change has always been considered as so.
A: No. As dpkg does not currently track file metadata, it does not know if a symlink or directory was switched by a previous package or by the sysadmin. As part of the dpkg credo, preserving human configuration is of utmost importance, and this kind of change has always been considered as so.
Line 82: Line 85:
As of now, this should be handled by the package maintainer scripts, failing to do so should be considered a serious bug. As of now, this should be handled by the package maintainer scripts (possibly with aid from dpkg-maintscript-helper), failing to do so should be considered a serious bug.
Line 94: Line 97:
  1. Ship an initial file (possibly empty) on the .deb that gets modified on the filesystem later on, but this will make the file hashes not match with stuff like debsums or the upcoming «dpkg --verify». While this might be the most convenient option, it's also possibly the most wrong one. This '''should''' never be used for volatile files such as logs or caches or the like.   1. Ship an initial file (possibly empty) on the .deb that gets modified on the filesystem later on, but this will make the file hashes not match with stuff like the new «dpkg --verify» or debsums. While this might be the most convenient option, it's also possibly the most wrong one. This '''should''' never be used for volatile files such as logs or caches or the like.
Line 104: Line 107:
A: Adding more compression formats support to dpkg-source, has some benefits
as it avoids having to repack the original source tarball, and in this way
checksums or upstream signatures are preserved, it might also improve
compression ratio compared to previous supported compressions.
A: Adding more compression formats support to dpkg-source, has some benefits as it avoids having to repack the original source tarball, and in this way checksums or upstream signatures are preserved, it might also improve compression ratio compared to previous supported compressions.
Line 109: Line 109:
But then the costs of adding support for an additional compression
format are quite high, to the point that those benefits seem small
compared to them. The costs are (at least): increase in the build-essential
set; the new compression becomes part of the source format forever, as it
can never be taken back as old Debian packages or 3rd-party packages in
the wild might be using that format; it's another (uncommon) package
required on non-Debian-based systems to be able to handle those source packages.
But then the costs of adding support for an additional compression format are quite high, to the point that those benefits seem small compared to them. The costs are (at least): increase in the build-essential set; the new compression becomes part of the source format forever, as it can never be taken back as old Debian packages or 3rd-party packages in the wild might be using that format; it's another (uncommon) package required on non-Debian-based systems to be able to handle those source packages.
Line 117: Line 111:
With the above in mind, the new compression format should really be very
widely used to distribute upstream source and it should improve compression
ratio compared to previously supported formats to start considering adding it.
With the above in mind, the new compression format should really be very widely used to distribute upstream source and it should improve compression ratio compared to previously supported formats to start considering adding it.
Line 123: Line 115:
A: Certainly. But before considering adding any flag to the default, the
following things would need to be done:
A: Certainly. But before considering adding any flag to the default, the following things would need to be considered:

User Questions

Q: What can be done when the dpkg lock is held?

A: This means another instance of dpkg or a frontend is running, so the correct solutions are to:

  • let the other process end its execution,
  • find the frontend and quit it, or
  • kill any such frozen processes.

Removing the dpkg lock is not a correct solution, as dpkg uses region locking on an existing file. The presence of the file does not mean that the lock is currently being held, and removing it can most probably cause dpkg database corruption.

Q: Why is dpkg so slow when using new filesystems such as btrfs or ext4?

A: To guarantee that the filesystem data is always consistent and safe, dpkg performs fsync(2)s on its database and files unpacked from packages. Newer filesystems (like btrfs or ext4) that implement delayed allocation do require those fsync(2)s as they trade data safety for performance, and expect programs to performs those fsync(2)s, but at the same time they have shown very poor performance on the behavior they require from all applications, or they might end up producing zero-length files on system crashes or abrupt shutdowns.

Here's a list of available solutions and workarounds you might consider using:

  • For ext4, use instead the "nodelalloc" mount option, which should fix both the performance degradation and the data safety issues, and not for just dpkg, but for any application in the system.
  • Switch to another filesystem entirely that does not force choosing between acceptable performance and data safety.
  • Use the dpkg --force-unsafe-io option, which avoids those fsync(2)s on filesystem files, but still performs them on the dpkg database. This option might improve performance at the cost of losing data, use with care.

  • If --force-unsafe-io is still not good enough, you might consider using (in addition or instead) eatmydata, which will get rid of any fsync(2) at all. You might want to have backups around, though.

(See 430958, 567089, 578635, 584254, 588254, 588339, 595927, 600075, 605009, 635993 and https://bugzilla.kernel.org/show_bug.cgi?id=15910 for all the gory details.)

Q: Why does ''dpkg --set-selections'' not record selections for unknown packages?

A: Due to multiarch, the dpkg command-line interface dealing with package name input needed to change. To avoid the possibility for dpkg to be unable to refer to packages in its own database, selections for unknown packages are not recorded any longer. So the available database needs to be up-to-date before any selection is set, this also gives the benefit of warning on unknown packages so that the user knows what will be missing. There are several ways to update the available database, depending on the package manager frontend being used, any single one of them will be enough:

  • apt-based (apt, aptitude, synaptic, etc):

    • (from apt, with dpkg 1.17.7 and later)

          apt-cache dumpavail | dpkg --merge-avail

      (from apt, with dpkg 1.17.6 and earlier)

          avail=`mktemp`
          apt-cache dumpavail >"$avail"
          dpkg --merge-avail "$avail"
          rm "$avail"

      (from apt)

          /usr/lib/dpkg/methods/apt/update /var/lib/dpkg/

      (from dctrl-tools)

          sync-available

      (from dselect, using the apt access method)

          dselect update
  • cupt:

    • (from cupt, with dpkg 1.17.7 and later)

          cupt show -a '*' | egrep -v '^(Status|[^:]*\s.*:)' | dpkg --merge-avail

      (from cupt, with dpkg 1.17.6 and earlier)

          avail=`mktemp`
          cupt show -a '*' | egrep -v '^(Status|[^:]*\s.*:)' > "$avail"
          dpkg --merge-avail "$avail"
          rm "$avail"
  • dselect

    • (from dselect, using any of the builtin access methods)

          dselect update

Behavior Questions

A: No. As dpkg does not currently track file metadata, it does not know if a symlink or directory was switched by a previous package or by the sysadmin. As part of the dpkg credo, preserving human configuration is of utmost importance, and this kind of change has always been considered as so.

As of now, this should be handled by the package maintainer scripts (possibly with aid from dpkg-maintscript-helper), failing to do so should be considered a serious bug.

Q: Can dpkg be told to avoid invoking a harmful prerm from an installed package on upgrade?

A: No. There's currently no way to tell dpkg not to execute such prerm maintainer scripts, but there is an extreme workaround available.

Only if the prerm is doing something really harmful, not just for simple failures or actions that can be easily reverted by the new postinst, then the only available way is to make a new, independent package that replaces the broken package's prerm, and then make the new version of the broken package pre-depend on it. This requires modifying the dpkg database, and as such is strongly discouraged, and should only be used as a last resort measure. Proposed by Joey Hess.

Q: Can dpkg handle volatile files?

A: Partially. dpkg does not have full native support (yet) for volatile files (those that might change during the life of a specific installed version), but there are some workarounds that can be used for now:

  1. Ship an initial file (possibly empty) on the .deb that gets modified on the filesystem later on, but this will make the file hashes not match with stuff like the new «dpkg --verify» or debsums. While this might be the most convenient option, it's also possibly the most wrong one. This should never be used for volatile files such as logs or caches or the like.

  2. Do not ship the file in the .deb, and generate it at installation time, run-time or whenever, but the maintainer scripts will need to deal with the removal manually. This is the more correct but slightly more cumbersome option.
  3. If the volatile file changes are due to external forces, like updates from a website or similar, then it might be best to automatically generate a new package every time these files change, and ship them normally in a package from a repository.

In the future, dpkg might acquire the ability to register external files, and one kind of such files could be changing files, so dpkg would take care of removing them, and would not store file hashes for these, etc.

Feature Request Questions

Q: Can we add support for new compressors for .dsc packages?

A: Adding more compression formats support to dpkg-source, has some benefits as it avoids having to repack the original source tarball, and in this way checksums or upstream signatures are preserved, it might also improve compression ratio compared to previous supported compressions.

But then the costs of adding support for an additional compression format are quite high, to the point that those benefits seem small compared to them. The costs are (at least): increase in the build-essential set; the new compression becomes part of the source format forever, as it can never be taken back as old Debian packages or 3rd-party packages in the wild might be using that format; it's another (uncommon) package required on non-Debian-based systems to be able to handle those source packages.

With the above in mind, the new compression format should really be very widely used to distribute upstream source and it should improve compression ratio compared to previously supported formats to start considering adding it.

Q: Can we add support for new default build flags to dpkg-buildflags?

A: Certainly. But before considering adding any flag to the default, the following things would need to be considered:

  • The flag should not involve any warning about style issues (those are subjective and dependant on upstream coding standards).
  • A test rebuild of the whole archive w/ and w/o the flags, and a comparison to see how much difference is there in the amount built.
  • For non-warning flags, a comparison of the build logs to see the memory and build time difference if these might seem relevant (sbuild should provide those).
  • Once these are done, and if it still seems worth it, a discussion started in debian-devel proposing the change, asking if there's other known issues, concerns, etc.