Differences between revisions 68 and 91 (spanning 23 versions)
Revision 68 as of 2017-03-13 13:55:43
Size: 23535
Comment: Building apps with these packages
Revision 91 as of 2021-11-24 03:26:47
Size: 28258
Editor: PaulWise
Comment: update link
Deletions are marked like this. Additions are marked like this.
Line 12: Line 12:
To communicate with this team, join our low traffic mailing list, [[https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/android-tools-devel|android-tools-devel@lists.alioth.debian.org]] and on the IRC channel [[irc://irc.debian.org/debian-mobile|#debian-mobile]] ([[https://webchat.oftc.net/?channels=debian-mobile|webchat]]). To communicate with this team, join our low traffic mailing list, [[https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/android-tools-devel|android-tools-devel@lists.alioth.debian.org]] and on the IRC channel [[irc://irc.debian.org/debian-android-tools|#debian-android-tools]] ([[https://webchat.oftc.net/?channels=debian-android-tools|webchat]]). You can also join the IRC Channel through [[https://matrix.to/#/#_oftc_#debian-android-tools:matrix.org|Matrix]]
Line 15: Line 15:
The binaries for the Android SDK downloadable from Google have a proprietary license but the source code is free software so Debian is packaging it. Not all Android SDK packages be installed from Debian, some never will be in Debian because they are too specific to Android. Sylvain Beucler's [[http://android-rebuilds.beuc.net/|libre Android rebuilds]] and/or Google's non-free binaries can also be used with the Debian Android SDK. The binaries for the Android SDK downloadable from Google have a proprietary license but the source code is free software so Debian is packaging it. Not all Android SDK packages can be installed from Debian, some never will be in Debian because they are too specific to Android. Sylvain Beucler's [[https://gitlab.com/android-rebuilds|libre Android rebuilds]] and/or Google's non-free binaries can also be used with the Debian Android SDK.
Line 21: Line 21:
Here are the steps for building Android apps using Debian's Android SDK on Stretch. {{{#!wiki debian
If you're just starting out with building apps, we suggest that you first read the [[AndroidTools/IntroBuildingApps|Introduction to build packages with Debian's Android SDK]].
}}}

If you are already familiar with how to use these tools, you might want to look these brief instructions below. Here are the steps for building Android apps using Debian's Android SDK on Stretch.
Line 44: Line 48:
Here's an example project:
https://gitlab.com/Matrixcoffee/hello-world-debian-android
Line 50: Line 55:
 * main IRC chat room: [[irc://irc.debian.org/debian-mobile|#debian-mobile]] ([[https://webchat.oftc.net/?channels=debian-mobile|webchat]]).  * main chat room: `#debian-android-tools` [[irc://irc.debian.org/debian-android-tools|OFTC IRC]] [[https://matrix.to/#/#_oftc_#debian-android-tools:matrix.org|[matrix]]] ([[https://webchat.oftc.net/?channels=debian-android-tools|webchat]]).
Line 56: Line 61:
= Joining the Android Tools Team =

We want make it as easy as possible for anyone to get involved in the Android Tools Team, and contribute in ways that they think are important. It is important to note that some of the packages here are quite complicated, and the Android SDK is a very large project. That means that changes must be discussed with the team before they can be committed and pushed. That discussion can happen in anywhere, including merge requests.

So here is the Android Tools Team policy for granting membership to the team, based on the [[https://salsa.debian.org/help/user/permissions|GitLab user roles]]:

 * any request to become a member is granted at least "Reporter" level
 * any Debian Developer or previous contributor is granted at least "Developer" level
 * requests from anyone else must be posted to https://salsa.debian.org/android-tools-team/admin/issues and are approved by "lazy consensus"
 * specific requests for "Master" level must also be posted to https://salsa.debian.org/android-tools-team/admin/issues and are approved by "lazy consensus"

This is based on "Lazy consensus": if no one objects to a request within one week or so, then any Master/Owner is free to grant the request. So silence is considered approval, but not automatic approval. An existing Master/Owner must actually grant the request after the waiting period.

=== standalone vs. interdependent packages ===

Mostly, our procedures are about which packages are standalone and can be freely uploaded, versus which packages are interwoven and must be uploaded in coordinated batches (e.g. the Android SDK). ''repo'' or ''enjarify'' is a standalone package, for example. The interwoven ones basically all follow the Android SDK git repo naming scheme for the source package names:

 * ''android-platform-*''
 * ''android-framework-*''
 * ''android-sdk-meta''


Line 71: Line 99:
 * http://anonscm.debian.org/cgit/android-tools/android-permissions.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-platform-build.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-platform-frameworks-base.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-platform-frameworks-native.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-platform-system-core.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-platform-system-extras.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/androidsdk-tool.git/tree/debian/README.source
 * http://anonscm.debian.org/cgit/android-tools/android-tools.git/tree/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-permissions/tree/master/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-platform-build/tree/master/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-platform-frameworks-base/tree/master/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-platform-frameworks-native/tree/master/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-platform-system-core/tree/master/debian/README.source
 * https://salsa.debian.org/android-tools-team/android-platform-system-extras/tree/master/debian/README.source
Line 91: Line 117:
   * android-platform-external-boringssl
Line 92: Line 119:
   * android-platform-external-libselinux    * android-platform-external-libselinux (should be switched to android-platform-external-selinux from 8.0 release)
Line 103: Line 130:
   * android-platform-art
Line 109: Line 137:
   * android-platform-art
Line 131: Line 160:
 * apkanalyzer:
   *git clone [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
   *source code [[https://android.googlesource.com/platform/tools/base/+/studio-master-dev/apkparser/analyzer/ | studio-master-dev/apkparser/analyzer/]]
 *avdmanager:
   *git clone [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
   *source code [[https://android.googlesource.com/platform/tools/base/+/studio-master-dev/sdklib/src/main/java/com/android/sdklib/internal/avd/ | sdklib/internal/avd/]]
  * D8/R8:
   *git clone [[https://android.googlesource.com/platform/external/r8 | platform/external/r8]]
   *source code [[https://android.googlesource.com/platform/external/r8/+/refs/heads/master/src/main/java/com/android/tools/r8 | android/tools/r8]]
Line 133: Line 171:
 * jobb: [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
 * lint: [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
 * jobb:
   *git clone [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
   *source code [[https://android.googlesource.com/platform/tools/base/+/studio-master-dev/jobb/ | studio-master-dev/jobb/]]
 * hidl-gen:
   *git clone [[https://android.googlesource.com/platform/system/tools/hidl | platform/system/tools/hidl]]
   *source code [[https://android.googlesource.com/platform/system/tools/hidl/+/refs/heads/master| refs/heads/master]]
 * lint:
   *git clone [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
   *source code [[https://android.googlesource.com/platform/tools/base/+/studio-master-dev/lint/ | studio-master-dev/lint/]]
Line 136: Line 181:
 * monkeyrunner: [[https://android.googlesource.com/platform/tools/swt | platform/tools/swt]] (deprecated)  * monkeyrunner:
   *git clone
[[https://android.googlesource.com/platform/tools/swt | platform/tools/swt]] (deprecated)
   *source code [[https://android.googlesource.com/platform/tools/swt/+/master/monkeyrunner/| master/monkeyrunner/]] (deprecated)
Line 139: Line 186:
 * sdkmanager:
   * git clone [[https://android.googlesource.com/platform/tools/base | platform/tools/base]]
   *source code [[https://android.googlesource.com/platform/tools/base/+/studio-master-dev/sdklib/src/main/java/com/android/sdklib/tool/sdkmanager/ | sdklib/tool/sdkmanager/]]
Line 157: Line 207:
 * dexdump: [[https://android.googlesource.com/platform/dalvik | platform/dalvik]]  * dexdump: [[https://android.googlesource.com/platform/art | platform/art]]
Line 212: Line 262:
 * https://dl.google.com/android/repository/repository-11.xml
 * https://dl.google.com/android/repository/repository-12.xml
 * https://dl.google.com/android/repository/repository2-1.xml
Line 257: Line 310:
There is lots left to do before someone can do Android development using only official Debian packages. The best way to get started is to [[https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/android-tools-devel|join the email list]] and ask there. We're also in IRC at [[https://webchat.oftc.net/?channels=debian-mobile|#debian-mobile]]. Here are some general topics that need work: There is lots left to do before someone can do Android development using only official Debian packages. The best way to get started is to [[https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/android-tools-devel|join the email list]] and ask there. We're also in IRC at [[https://webchat.oftc.net/?channels=debian-android-tools|#debian-android-tools]]. Here are some general topics that need work:
Line 286: Line 339:
 * [[http://anonscm.debian.org/gitweb/?p=android-tools/android-platform-system-extras.git|android-platform-system-extras]] package, replacing `android-tools-fsutils` package.
 * update all source packages to the latest SDK release (v23.0.2, provided you can find the right tag or branch)
 * [[https://android.googlesource.com/toolchain/jack|Jack]]
 * --([[http://anonscm.debian.org/gitweb/?p=android-tools/android-platform-system-extras.git|android-platform-system-extras]] package, replacing `android-tools-fsutils` package.)--
 * --(update all source packages to the latest SDK release (v23.0.2, provided you can find the right tag or branch))--
 * --([[https://android.googlesource.com/toolchain/jack|Jack]])-- Deprecated
Line 296: Line 349:
= Knowledge dump =
For lack of a better name, this basically contains some of the things we have learnt and some of the general ways in which we work with android packages.
Line 304: Line 358:

----

CategoryPackaging

This page is a gathering place for information about the android-tools packaging team, which is focused on packaging the Android development tools for Debian. There are also some packages which help run Debian in a chroot on Android. The goal of this team is to get as much of the Android SDK and development tools into Debian as possible. There are many advantages to having the SDK and tools in Debian, rather than relying only on the Google distributions:

  • easy install and update channel that all Debian users already know
  • automatic trustworthy downloads, no need to verify hash sums
  • eliminate need for insecure wrapper scripts, like ./gradlew
  • trivial install for specific tools, like adb, fastboot, etc.

To read more about the rationale behind this work, see this blog post: https://guardianproject.info/2015/04/30/getting-android-tools-into-debian/

To communicate with this team, join our low traffic mailing list, android-tools-devel@lists.alioth.debian.org and on the IRC channel #debian-android-tools (webchat). You can also join the IRC Channel through Matrix

The binaries for the Android SDK downloadable from Google have a proprietary license but the source code is free software so Debian is packaging it. Not all Android SDK packages can be installed from Debian, some never will be in Debian because they are too specific to Android. Sylvain Beucler's libre Android rebuilds and/or Google's non-free binaries can also be used with the Debian Android SDK.

Building apps with these packages

If you're just starting out with building apps, we suggest that you first read the Introduction to build packages with Debian's Android SDK.

If you are already familiar with how to use these tools, you might want to look these brief instructions below. Here are the steps for building Android apps using Debian's Android SDK on Stretch.

  1. sudo apt install android-sdk android-sdk-platform-23
  2. export ANDROID_HOME=/usr/lib/android-sdk
  3. In build.gradle, change compileSdkVersion to 23 and buildToolsVersion to 24.0.0

  4. run gradle build

The Gradle Android Plugin is also packaged. Using the Debian package instead of the one from online Maven repositories requires a little configuration before running Gradle. In the buildscript {} block:

  • add maven { url 'file:///usr/share/maven-repo }` to repositories

  • use compile 'com.android.tools.build:gradle:debian' to load the plugin

Currently there is only the target platform of API Level 23 packaged, so only apps targeted at android-23 can be built with only Debian packages. We will add more API platform packages via backports afterwards. Only Build-Tools 24.0.0 is available, so in order to use the SDK, build scripts need to be modified. Beware that the Lint in this version of Gradle Android Plugin is still problematic, so running the :lint tasks might not work. They can be turned off with lintOptions.abortOnError in build.gradle. Google binaries can be combined with the Debian packages, for example to use a different version of the platform or build-tools.

In stretch-backports (and soon testing), the Gradle Android Plugin is patched to work with Debian's Android SDK. It detects what versions of API Levels and Build-Tools are available and you no longer need to modify the build scripts. In order to build apps, do the following:

  1. sudo apt install android-sdk android-sdk-platform-23 android-sdk-helper
  2. export ANDROID_HOME=/usr/lib/android-sdk
  3. gradle build --init-script /usr/share/android-sdk-helper/init.gradle

Thus, init.gradle forces Gradle to use the Gradle Android Plugin in Debian and the plugin will do the rest.

Here's an example project: https://gitlab.com/Matrixcoffee/hello-world-debian-android

Communication Channels

There are a number of ways that people working on packaging Android Tools communicate. Here is a list:

Joining the Android Tools Team

We want make it as easy as possible for anyone to get involved in the Android Tools Team, and contribute in ways that they think are important. It is important to note that some of the packages here are quite complicated, and the Android SDK is a very large project. That means that changes must be discussed with the team before they can be committed and pushed. That discussion can happen in anywhere, including merge requests.

So here is the Android Tools Team policy for granting membership to the team, based on the GitLab user roles:

This is based on "Lazy consensus": if no one objects to a request within one week or so, then any Master/Owner is free to grant the request. So silence is considered approval, but not automatic approval. An existing Master/Owner must actually grant the request after the waiting period.

standalone vs. interdependent packages

Mostly, our procedures are about which packages are standalone and can be freely uploaded, versus which packages are interwoven and must be uploaded in coordinated batches (e.g. the Android SDK). repo or enjarify is a standalone package, for example. The interwoven ones basically all follow the Android SDK git repo naming scheme for the source package names:

  • android-platform-*

  • android-framework-*

  • android-sdk-meta

Package naming scheme

The naming scheme for android-tools packages is as follows:

Source package structure

The structure of each source package is documented in the README.source of each source package for this team:

Updating the source packages

The packages in this team are structured somewhat unusually because we are trying to keep the source packages as close as possible to the upstream source organization while still working in a Debian way. Google builds the Android OS and SDK as one giant thing, something like 10 gigs of source code. But the code is broken up into many different git repos that are coordinated using the Android team's tool called repo. Also, there are lots of shared libraries used between the various Android SDK tools, but since everything is always built together, those shared libraries are unversioned.

All this means that it is essential that any Android SDK package is only built against the exact same version of all its Build-Depends and only uses the exact same version of an Android SDK package as a Depends. To achieve that, we use the substvars variable in dependency declarations: (>= ${source:Version}).

Additionally, because of this and the circular dependencies, it is important to upload updates in the correct order. Some packages also have to be uploaded using a multi-stage method. Here is the estimated upload order:

  • Stage 1
    • android-framework-$apilevel
    • android-platform-external-boringssl
    • android-platform-external-jsilver (Usually no update needed)
    • android-platform-external-libselinux (should be switched to android-platform-external-selinux from 8.0 release)
    • android-platform-external-libunwind
    • android-platform-frameworks-data-binding
    • android-platform-frameworks-native
    • android-platform-libcore
    • android-platform-tools-analytics-library
  • Stage 2
    • android-platform-external-doclava
    • android-platform-system-core~stage1
    • android-platform-tools-base
  • Stage 3
    • android-platform-art
    • android-platform-development
    • android-platform-frameworks-base
    • android-platform-libnativehelper
    • android-platform-system-extras
    • android-platform-system-tools-aidl
  • Stage 4
    • android-platform-art
    • android-platform-build
    • android-platform-dalvik
    • android-platform-system-core
  • android-platform-dalvik (Needs android.jar of the latest API Level)

Each packages belonging to the same stage are unrelated to each other and can be uploaded in any order.

This order only represents one possible way of updating all of them to a new major version. In practice, you can delay or advance any packages as long as the build-dependencies satisfy.

Upstream repository of tools in Android SDK

Tools in Android SDK come from various repositories. Here is a list of tools consisting the entire Android SDK, with each tool followed with the corresponding upstream repository name.

SDK Tools

SDK Platform-tools

SDK Build-tools

Reverse engineering tools

dexdump apktool gplaycli androguard enjarify dmtracedump

Other Tools

Android's upstream version names

There are many naming schemes for versions in Android, and none are used everywhere. For the Android OS itself, there are three schemes: version names, like Gingerbread or Kitkat; version numbers, like 2.3.7 or 4.4.2; and, "API level" numbers, like android-10 or android-19. None of these line up with each other. For example, the Jelly Bean name spans 4.1 through 4.3.1 and android-16 through android-18. The version numbers and API level also do not line up, with android-14 covering 4.0.1 - 4.0.2, android-15 covering 4.0.3 - 4.0.4, but android-16 covers all of 4.1.x.

The madness doesn't end there. Next up we have SDK version numbers, which are like 20, or 23.0.2. Part of the SDK includes the build-tools package, which has its own version numbers, which are like 18.1.1 and 20.0.0, but these do not line up with the SDK version numbers. The platform-tools seems to have its own versioning scheme also, but it is not really exposed. Then there are the NDK release numbers, which are like r8b or r10, and they also do not line up with anything else.

In the git repo, there are a mishmash of tags and branches that do not represent all of these various versions. There does seem to be consistent tagging of the OS releases, you can see those listed under Source Code Tags and Builds. There are a few branches that seem to line up to the SDK version numbers, like tools_r21 and tools_r22.2, but there is not a complete set of those branches.

example versions

Android SDK Tools seems to have it's own versioning scheme, since the major version is ahead of the SDK major version:

  • v24.2.0
  • v24.1.0
  • v23.0.0

Android SDK Build-tools seems to follow the SDK major version, but have its own minor and micro versions:

  • v22.0.1
  • v21.1.2
  • v21.0.0

Android SDK Platform-tools seems to follow the SDK major version only.

  • v22
  • v21
  • v20

You can find the official Google package names and versions in the index XML files that the android tool downloads when doing updates:

Android SDK version declarations

However, the exact version number of Android SDK toolsets can be found in a source.properties under each directories. Here are the source locations of each version declaration:

Which matches the official release notes.

Note that according to the source code the major version number of Platform-tools and Build-tools is simply the API Level, which is defined here. But for SDK Tools it follows its own version pattern and does not relate to the API Level.

Some tools tracks their own version and do not follow the SDK version:

  • adb
  • dx

Original android-tools package

This team started with the android-tools package, which is a manually assembled collection of source code needed to build adb, fastboot, etc. This approach is not maintainable in the long run, especially as we work to add the whole Android SDK. So the android-tools package is deprecated, and will be replaced by the collection of packages described above.

Why is this so complicated?

Packaging all of these Android SDK Tools is so complicated because the source code is organized into semi-arbitrary "projects" that are split up between many different git repos. Those git repos are then all checked out at the same time using Google's crazy repo tool. Then the entire Android OS and SDK are built using a single, unified build system that requires something like 8 gigs of source code be downloaded. On top of that, one of the more confusing aspects of this whole project is that the same source code base is used to build the OS and the SDK. Those two can start with the same source code files, and end up with very different resulting binaries.

shared libraries within the Android SDK packages

The Android SDK has a number of private libraries that are shared between many different utilities that are part of the Android SDK. Upstream statically links them into each executable. That didn't seem very Debian-ish, so instead they are structured as private shared libraries. Nothing should link to these android-lib* shared libraries that is not part of the Android source. This private shared library arrangement allows for security patches in the android shared code to be applied without having to rebuild everything.

All of the android-* packages must be updated to the latest version at the same time. It is more unpredictable to have android-* packages at different upstream versions since no one is running that configuration, and upstream does not do anything to support that (e.g. no versioned ABI, etc). Therefore, the ABI is guaranteed to be compatible since they'll all be built together. In practice, this means that the process of updating to the latest upstream version has to start from the most core packages, then progress to the ones that depend on it.

A single source package might make some things easier, but it would be about 8-12 gigs in size. That would make it very difficult for many people to work with that source package.

Needs doing

There is lots left to do before someone can do Android development using only official Debian packages. The best way to get started is to join the email list and ask there. We're also in IRC at #debian-android-tools. Here are some general topics that need work:

  • build an app using only the Debian packages for gradle, gradle plugins, build tools and platform tools (i.e. apt-get install android-sdk then gradle build), then file bug reports on anything that isn't working

  • figure out packaging scheme for various android "platforms", i.e. /opt/android-sdk/platforms/android-* and /opt/android-ndk/platforms/android-*

  • build "android support" jars
  • figure out how to represent Android NDK clang "toolchains" in Debian, i.e. /opt/android-sdk/toolchains

  • help Google integrate Android's QEMU into upstream (since Google has started to do this, I think this is the best approach to getting the Android emulator into Debian)

fdroidserver

fdroidserver is the tool suite for managing FDroid app repos and making release builds. It uses the Android SDK and NDK to make the builds and work with APKs. There are three notable milestones for fdroidserver packaging needs:

  1. everything to create and manage app repos DONE!

  2. everything to make pure Java builds
  3. everything to make native builds

low-hanging fruit

Here are a couple of easy tasks to do that will further this effort along:

  • Continuous integration tests
  • build Android apps using apt install android-sdk on stretch and newer

  • package these existing SDK packages for other distros (Arch, Fedora, NixOS, Homebrew, ?MacPorts, Cygwin, etc)

  • get answer from Google on how the SDK versions are marked in the source git repos (i.e. what tag/branch is SDK v23.0.2? And what tag/branch is build-tools v20.0.0?)

    • Is https://source.android.com/source/code-lines.html#terms-and-caveats any helpful? Among other things, it says:

      • A release corresponds to a formal version of the Android platform, such as 1.5, 2.1, and so on. Generally speaking, a release of the platform corresponds to the version in the SdkVersion field of AndroidManifest.xml files and defined within frameworks/base/api in the source tree.

    • seems to vaguely contribute to understanding, but no concrete answers for which commit ID to build for SDK release v23.0.2. -- HansChristophSteiner 2014-10-07 19:51:57

  • android-platform-system-extras package, replacing android-tools-fsutils package.

  • update all source packages to the latest SDK release (v23.0.2, provided you can find the right tag or branch)

  • Jack Deprecated

  • Android Support Library

  • The rest of an "Android platform" ("android.jar" is done).

  • android-platform-dalvik needs update to include "shrinkedAndroid.jar" generated with "android.jar"

  • android-platform-tools-swt and probably though unlikely android-platform-tools-buildsrc

  • android-platform-frameworks-compile-libbcc for bcc_compat, llvm-rs-cc, etc.

  • "renderscript" directory under "build-tools"

Knowledge dump

For lack of a better name, this basically contains some of the things we have learnt and some of the general ways in which we work with android packages.

See Also


CategoryPackaging