Differences between revisions 34 and 36 (spanning 2 versions)
Revision 34 as of 2015-11-17 09:42:29
Size: 15820
Comment: shared libraries within the Android SDK packages
Revision 36 as of 2016-02-08 08:00:17
Size: 17884
Comment: note about source update procedure
Deletions are marked like this. Additions are marked like this.
Line 38: Line 38:


= Updating the source packages =

The packages in this team are structured somewhat unusually because we are trying to keep the source packages as close as possible to the upstream source organization while still working in a Debian way. Google builds the Android OS and SDK as one giant thing, something like 10 gigs of source code. But the code is broken up into many different git repos that are coordinated using the Android team's tool called `repo`. Also, there are lots of shared libraries used between the various Android SDK tools, but since everything is always built together, those shared libraries are unversioned.

All this means that it is essential that any Android SDK package is only built against the exact same version of all its `Build-Depends` and only uses the exact same version of an Android SDK package as a `Depends`. To achieve that, we use the substvars variable in dependency declarations: `(>= ${source:Version})`.

Additionally, because of this and the circular dependencies, it is important to upload updates in the correct order. Some packages also have to be uploaded using a multi-stage method. Here is the order that was used for 6.0.0+r26:

* android-platform-external-libunwind
* android-platform-external-libselinux
* android-platform-frameworks-native
* android-platform-build_stage1
* android-platform-libcore
* android-platform-system-core_stage1
* android-platform-system-extras
* android-platform-libnativehelper
* android-platform-frameworks-base
* android-platform-dalvik
* android-platform-build
* android-platform-system-core
Line 145: Line 167:
= Original android-tools package =

This team started with the [[https://packages.debian.org/src:android-tools|android-tools]] package, which is a manually assembled collection of source code needed to build `adb`, `fastboot`, etc. This approach is not maintainable in the long run, especially as we work to add the whole Android SDK. So the ''android-tools'' package is deprecated, and will be replaced by the collection of packages described above.

This page is a gathering place for information about the android-tools packaging team, which is focused on packaging the Android development tools for Debian. There are also some packages which help run Debian in a chroot on Android. The goal of this team is to get as much of the Android SDK and development tools into Debian as possible. There are many advantages to having the SDK and tools in Debian, rather than relying only on the Google distributions:

  • easy install and update channel that all Debian users already know
  • automatic trustworthy downloads, no need to verify hash sums
  • eliminate need for insecure wrapper scripts, like ./gradlew
  • trivial install for specific tools, like adb, fastboot, etc.

To read more about the rationale behind this work, see this blog post: https://guardianproject.info/2015/04/30/getting-android-tools-into-debian/

To communicate with this team, join our low traffic mailing list, android-tools-devel@lists.alioth.debian.org and on the IRC channel #debian-mobile (webchat).

Package naming scheme

The naming scheme for android-tools packages is as follows:

Source package structure

The structure of each source package is documented in the README.source of each source package for this team:

Updating the source packages

The packages in this team are structured somewhat unusually because we are trying to keep the source packages as close as possible to the upstream source organization while still working in a Debian way. Google builds the Android OS and SDK as one giant thing, something like 10 gigs of source code. But the code is broken up into many different git repos that are coordinated using the Android team's tool called repo. Also, there are lots of shared libraries used between the various Android SDK tools, but since everything is always built together, those shared libraries are unversioned.

All this means that it is essential that any Android SDK package is only built against the exact same version of all its Build-Depends and only uses the exact same version of an Android SDK package as a Depends. To achieve that, we use the substvars variable in dependency declarations: (>= ${source:Version}).

Additionally, because of this and the circular dependencies, it is important to upload updates in the correct order. Some packages also have to be uploaded using a multi-stage method. Here is the order that was used for 6.0.0+r26:

* android-platform-external-libunwind * android-platform-external-libselinux * android-platform-frameworks-native * android-platform-build_stage1 * android-platform-libcore * android-platform-system-core_stage1 * android-platform-system-extras * android-platform-libnativehelper * android-platform-frameworks-base * android-platform-dalvik * android-platform-build * android-platform-system-core

Upstream repository of tools in Android SDK

Tools in Android SDK come from various repositories. Here is a list of tools consisting the entire Android SDK, with each tool followed with the corresponding upstream repository name.

SDK Tools

SDK Platform-tools

SDK Build-tools

Other Tools

  • Android Plugin for Gradle: Not sure
  • repo: tools/repo

Android's upstream version names

There are many naming schemes for versions in Android, and none are used everywhere. For the Android OS itself, there are three schemes: version names, like Gingerbread or Kitkat; version numbers, like 2.3.7 or 4.4.2; and, "API level" numbers, like android-10 or android-19. None of these line up with each other. For example, the Jelly Bean name spans 4.1 through 4.3.1 and android-16 through android-18. The version numbers and API level also do not line up, with android-14 covering 4.0.1 - 4.0.2, android-15 covering 4.0.3 - 4.0.4, but android-16 covers all of 4.1.x.

The madness doesn't end there. Next up we have SDK version numbers, which are like 20, or 23.0.2. Part of the SDK includes the build-tools package, which has its own version numbers, which are like 18.1.1 and 20.0.0, but these do not line up with the SDK version numbers. The platform-tools seems to have its own versioning scheme also, but it is not really exposed. Then there are the NDK release numbers, which are like r8b or r10, and they also do not line up with anything else.

In the git repo, there are a mishmash of tags and branches that do not represent all of these various versions. There does seem to be consistent tagging of the OS releases, you can see those listed under Source Code Tags and Builds. There are a few branches that seem to line up to the SDK version numbers, like tools_r21 and tools_r22.2, but there is not a complete set of those branches.

example versions

Android SDK Tools seems to have it's own versioning scheme, since the major version is ahead of the SDK major version:

  • v24.2.0
  • v24.1.0
  • v23.0.0

Android SDK Build-tools seems to follow the SDK major version, but have its own minor and micro versions:

  • v22.0.1
  • v21.1.2
  • v21.0.0

Android SDK Platform-tools seems to follow the SDK major version only.

  • v22
  • v21
  • v20

You can find the official Google package names and versions in the index XML files that the android tool downloads when doing updates:

Android SDK version declarations

However, the exact version number of Android SDK toolsets can be found in a source.properties under each directories. Here are the source locations of each version declaration:

Which matches the official release notes.

Note that according to the source code the major version number of Platform-tools and Build-tools is simply the API Level, which is defined here. But for SDK Tools it follows its own version pattern and does not relate to the API Level.

Some tools tracks their own version and do not follow the SDK version:

  • adb
  • dx

Original android-tools package

This team started with the android-tools package, which is a manually assembled collection of source code needed to build adb, fastboot, etc. This approach is not maintainable in the long run, especially as we work to add the whole Android SDK. So the android-tools package is deprecated, and will be replaced by the collection of packages described above.

Why is this so complicated?

Packaging all of these Android SDK Tools is so complicated because the source code is organized into semi-arbitrary "projects" that are split up between many different git repos. Those git repos are then all checked out at the same time using Google's crazy repo tool. Then the entire Android OS and SDK are built using a single, unified build system that requires something like 8 gigs of source code be downloaded.

shared libraries within the Android SDK packages

The Android SDK has a number of private libraries that are shared between many different utilities that are part of the Android SDK. Upstream statically links them into each executable. That didn't seem very Debian-ish, so instead they are structured as private shared libraries. Nothing should link to these android-lib* shared libraries that is not part of the Android source. This private shared library arrangement allows for security patches in the android shared code to be applied without having to rebuild everything.

All of the android-* packages must be updated to the latest version at the same time. It is more unpredictable to have android-* packages at different upstream versions since no one is running that configuration, and upstream does not do anything to support that (e.g. no versioned ABI, etc). Therefore, the ABI is guaranteed to be compatible since they'll all be built together. In practice, this means that the process of updating to the latest upstream version has to start from the most core packages, then progress to the ones that depend on it.

A single source package might make some things easier, but it would be about 8-12 gigs in size. That would make it very difficult for many people to work with that source package.

Needs doing

There is lots left to do before someone can do Android development using only official Debian packages. The best way to get started is to join the email list and ask there. We're also in IRC at #debian-mobile. Here are some general topics that need work:

  • figure out packaging scheme for various android "platforms", i.e. /opt/android-sdk/platforms/android-* and /opt/android-ndk/platforms/android-*

  • figure out how to represent Android NDK gcc and clang "toolchains" in Debian, i.e. /opt/android-sdk/toolchains

  • build "android support" jars
  • help Google integrate Android's QEMU into upstream (since Google has started to do this, I think this is the best approach to getting the Android emulator into Debian)

fdroidserver

fdroidserver is the tool suite for managing FDroid app repos and making release builds. It uses the Android SDK and NDK to make the builds and work with APKs. There are three notable milestones for fdroidserver packaging needs:

  1. everything to create and manage app repos
  2. everything to make pure Java builds
  3. everything to make native builds

The first milestone will be done once the aapt package is accepted into Debian. It is part of the android-platform-frameworks-base package that is waiting in the NEW queue.

low-hanging fruit

Here are a couple of easy tasks to do that will further this effort along:

  • get answer from Google on how the SDK versions are marked in the source git repos (i.e. what tag/branch is SDK v23.0.2? And what tag/branch is build-tools v20.0.0?)

    • Is https://source.android.com/source/code-lines.html#terms-and-caveats any helpful? Among other things, it says:

      • A release corresponds to a formal version of the Android platform, such as 1.5, 2.1, and so on. Generally speaking, a release of the platform corresponds to the version in the SdkVersion field of AndroidManifest.xml files and defined within frameworks/base/api in the source tree.

    • seems to vaguely contribute to understanding, but no concrete answers for which commit ID to build for SDK release v23.0.2. -- HansChristophSteiner 2014-10-07 19:51:57

  • android-platform-system-extras package, replacing android-tools-fsutils package.

  • build adb as part of android-platform-system-core package, replacing the android-tools-adb package.

  • build fastboot as part of android-platform-system-core package, replacing the android-tools-fastboot package.

  • update all source packages to the latest SDK release (v23.0.2, provided you can find the right tag or branch)

See Also