This page details a proposed new APT method for communicating between APT and the DebTorrent program. More information on APT methods can be found in the [http://packages.debian.org/libapt-pkg-doc libapt-pkg-doc package].

The Current State of Communication

Currently, the DebTorrent program makes use of the HTTP retrieval method for communicating with APT. It implements almost a complete proxy for downloading files from HTTP mirrors. The only exceptions are, since it considers Packages files to be torrents, it notes when they are requested and starts the corresponding torrent running. Also, when DebTorrent receives a request for a package file (which it identifies by extension), it finds the appropriate torrent that contains that file and begins to download it using the DebTorrent protocol (i.e. not using HTTP). Once the download is complete, it passes the file on to APT as if it had been downloaded directly from the HTTP mirror.

The major problems with this method are:

To solve the first problem of slow startup of downloads, multiple packages need to be downloaded at once from the same torrent, without waiting for one to finish before starting another. This could be as simple as telling APT to pipeline multiple requests to DebTorrent, which would alleviate some of the problem.

The second problem is trickier, as APT will only be aware of when downloads begin and end. Pipelined downloads may help though, as there may be more activity of files starting and stopping so that the user will not notice so much. However, in BitTorrent downloads usually occur in such a way that all the files complete at near the same time at the end of the download. So, even with pipelined downloads, it may appear to the user that nothing is happening (at which point they may abort the download), until finally all will suddenly come in at the end.

The Proposed Method

Instead of using APT's http method, a new method will be developed to be used by APT to request downloads of files, as well as other information, from DebTorrent. This method will be indicated in the sources.list file by "debtorrent://...". This method will be based on the current http method, and will use the HTTP protocol to communicate with DebTorrent, which will allow it to be accessed from other machines on the network.

To solve the first problem, the debtorrent method will send all requests from APT to the DebTorrent program immediately, without waiting for current requests to complete. This "ultra-pipelining" will open a lot of connections to DebTorrent, so a limit may need to be set on the maximum number of requests to have open at a time, but it will be quite high (maybe 100).

To solve the second problem, a special HTTP connection will be opened between the debtorrent method and the DebTorrent program. This connection will communicate the status of ongoing downloads from the DebTorrent program. The status will be communicated using a so-far-undetermined format. Two candidates are XML-RPC, and bencoded dictionaries.

The [http://www.xmlrpc.com/spec XML-RPC parameter format] is a good candidate for the communication format, as it is simple, well understood, operates over HTTP, and contains all the functionality needed for the status information. There is also a library available in the Python standard library for [http://docs.python.org/lib/node658.html creating and parsing XML-RPC strings from Python data].

Although the XML-RPC method does have the advantage of readability, it also involves a lot of overhead, both in transmitted bytes and parsing time. A more efficient method, which is also more familiar to BitTorrent users, would be to use a [http://en.wikipedia.org/wiki/Bencode Bencoded dictionary] instead of XML. Since all torrent information is stored bencoded, BitTorrent clients have all the functionality needed to bencode/bdecode all Python variable types. The disadvantage of this method is that bencoded data is not as human-readable as XML, and functionality would have to be added to the debtorrent APT method to encode/decode it (though this is probably true of XML as well).

APT Changes Required

The method described above could be added as a separate package (for an example, see the [http://packages.debian.org/apt-transport-https apt-transport-https package]), and so does not require any changes to the APT code. However, some changes would be needed to get the updated status information from the debtorrent method. The current method-messaging protocol would need to be expanded to include a new message (or maybe a few) for retrieving this information.