This page details a proposed new APT method for communicating between APT and the DebTorrent program. More information on APT methods can be found in the libapt-pkg-doc package.

The Original State of Communication

Originally, the DebTorrent program makes use of the HTTP retrieval method for communicating with APT. It implements almost a complete proxy for downloading files from HTTP mirrors. The only exceptions are, since it considers Packages files to be torrents, it notes when they are requested and starts the corresponding torrent running. Also, when DebTorrent receives a request for a package file (which it identifies by extension), it finds the appropriate torrent that contains that file and begins to download it using the DebTorrent protocol (i.e. not using HTTP). Once the download is complete, it passes the file on to APT as if it had been downloaded directly from the HTTP mirror.

The major problems with this method are:

To solve the first problem of slow startup of downloads, multiple packages need to be downloaded at once from the same torrent, without waiting for one to finish before starting another. This could be as simple as telling APT to pipeline multiple requests to DebTorrent, which would alleviate some of the problem.

The second problem of ordered downloads requires a new transport method to solve.

The last problem is trickier, as APT will only be aware of when downloads begin and end. Pipelined downloads returned in any order will help though, as there may be more activity of files starting and stopping so that the user will not notice so much. However, in BitTorrent downloads usually occur in such a way that all the files complete at near the same time at the end of the download. So, even with pipelined downloads, it may appear to the user that nothing is happening (at which point they may abort the download), until finally all will suddenly come in at the end.

HTTP/1.1 Pipelining

The HTTP protocol already has functionality built into it to allow for pipelining multiple requests over the same connection. The current DebTorrent APT request listener implements HTTP/1.1, so the current http method of APT can be used to pipeline multiple requests. The APT http method has a configuration parameter (Acquire::http::Pipeline-Depth) for controlling the maximum number of outstanding requests that can be sent on a single connection (defaults to 5), however there is a maximum of 10 set in the code for this value. The HTTP protocol also specifies that the files returned by the pipelined connection must be in the same order as they were requested, since there is no way to identify the file by the HTTP server response, and APT's http method conforms to this.

The advantages of this are:

The disadvantages are:

A New APT Method: apt-transport-debtorrent

Instead of using APT's http method, a new debtorrent method can by APT to request downloads of files (and possibly other information, see the proposal below) from DebTorrent. This method is indicated in the sources.list file by "debtorrent://...". This method is based on APT's http method, and will use a protocol similar to HTTP to communicate with DebTorrent, which will allow it to be accessed easily from other machines on the network. The debtorrent method will send all requests from APT to the DebTorrent program immediately, without waiting for any requests to complete. This "ultra-pipelining" will allow all the downloads to occur in parallel. The debtorrent method will also expect the downloaded files to be returned in an arbitrary order, which will then be passed to the APT program (which supports an arbitrary ordering of returned packages).

This new method has been created as a separate package apt-transport-debtorrent, and so does not require any major changes to the APT code. However, one minor change to the APT code is still needed, as the current APT code will only pass a maximum of 10 requests to a method at a time. In order for this method to improve on the HTTP/1.1 Pipelining solution, this limit will be raised to 1000 (this change has already been made to the APT code, and should be in the next release after 0.7.6).

The advantages are:

The disadvantages are:

Proposal: Status Updates

The last problem, that of no status updates in APT while the download is underway, has not yet been solved. Using the debtorrent APT method described above does solve some of the problems with this lack of updates (since files are constantly being downloaded), but APT's indication of the download completion is inaccurate and there is no indication of how long the download will take to complete. As an example, here is some data on the inaccuracies reported by APT from a dist-upgrade using the debtorrent APT method, compared with the accurate value taken from the DebTorrent status page.

APT Reports

Actual Completion

10%

40%

20%

70%

30%

85%

40%

90%

50%

93%

60%

96%

70%

97.5%

80%

98%

90%

99%

These inaccuracies occur because pieces of a multiple piece file have been downloaded, but APT does not yet know about them since the file is not yet complete.

To fix this, status updates need to be passed from the DebTorrent client to the debtorrent method. These updates will use a new status code message within the debtorrent APT method, which is not defined by the HTTP protocol (a 1xx informational message). This message will be passed at any point between messages containing downloaded files from the DebTorrent client to the debtorrent APT method. There are 2 possibilities for it's contents and use.

Updates when pieces download

This status message will be sent whenever a new piece of a multi-piece file is downloaded. It will contain the name of the file, and the size of the downloaded piece (but not the piece data itself). If the file was only a single piece, or if the piece is the final one needed to complete a multi-piece file, this message will not be sent, as instead the complete file will be sent.

The debtorrent APT method will use this message to fill the target files with empty data to indicate to the APT program that the file is partially downloaded. The file will be filled with all 0's, until the file itself is returned from the DebTorrent client, at which point the file will be rewritten with the actual data.

The advantages are:

The disadvantages are:

Updates of amount of download completed

This status message will be sent periodically from the DebTorrent client to the debtorrent APT method. It will contain the total number of bytes that have been downloaded from all the requests sent on this connection from APT. This will need to be tracked somehow

The debtorrent APT method will use this information to send a download completion update to the main APT program using a new informational message added to the APT method communication protocol. The APT program will use this value instead of checking the size of the target files of the download to determine how much of the download is complete, and how fast it is progressing.

The advantages are:

The disadvantages are: