The CommonsBlog

Inside Code Transparency: The Verification Process

A week ago, I looked at the contents of the JWT file created by the code transparency process. Today, let’s peek at how that gets verified.

`bundletool` Commands

Last week, when I showed you a bundletool command to add code transparency, I used a command that used a Java keystore directly. That does not seem to be an option for the verification step. For that (or for adding code transparency), you need an actual certificate file. You can obtain one from your keystore using keytool:

keytool -export \
  -alias WhateverAliasYouUsed \
  -keystore /path/to/your/keystore.jks \
  -rfc \
  -file /path/to/your/exported.cert

You can then use the check-transparency command to verify the contents of… something. The --mode option indicates what the “something” is. --mode=bundle says that you are verifying an App Bundle, such as one created by you or your CI server:

bundletool check-transparency \
  --mode=bundle \
  --bundle=/path/to/your/AppBundleWithCT.aab \
  --transparency-key-certificate=/path/to/your/exported.cert

If you leave off the --transparency-key-certificate option, bundletool will print the SHA-256 fingerprint of the certificate:

No APK present. APK signature was not checked.
Code transparency signature is valid. SHA-256 fingerprint of the code transparency key certificate (must be compared with the developer's public key manually): 25 98 AA 59 62 BA 4C C0 7B 40 74 F4 19 09 02 A0 2A CD F1 1B 1F 42 84 92 93 23 8B 6F 87 E5 42 B4
Code transparency verified: code related file contents match the code transparency file.

This should match the one you get from keytool:

keytool -list \
  -alias WhateverAliasYouUsed \
  -keystore /path/to/your/keystore.jks

Alternatively, you can have bundletool verify the code transparency for an installed app, via --mode=connected_device:

bundletool check-transparency \
  --mode=connected_device \
  --package-name=com.commonsware.scrap

As before, if you include --transparency-key-certificate, bundletool will check against it; otherwise it will print the SHA-256 fingerprint.

`bundletool` Implementation

Much of the code for code transparency support in bundletool resides in the com.android.tools.build.bundletool.transparency package.

The core “driver” of the verification resides in a set of static methods on ApkTransparencyCheckUtils. This code works off of a list of filesystem paths to the APKs to check. Where those APKs come from depends on your --mode. Of particular note, for --mode=connected_device, bundletool uses adb shell commands to copy the APKs to a temporary directory for analysis – the verification is not performed in situ on the device.

The code uses this JSON Web Toolkit library, which seems to be actively maintained, which is nice.

Unfortunately, the code for bundletool seems to be fairly monolithic. It does not appear to be organized as a library with a first-class API that also happens to have a CLI — it looks like it is just a CLI. And, since bundletool historically has only been needed for development machines and CI servers, in many places it seems to assume that environment. Getting verification logic that can run on-device will require reverse-engineering a spec from the implementation and creating a separate library, unless Google has interest in a significant reworking of bundletool.

—Jul 18, 2021

Random Musings on the Android 12 Beta 3

Android 12 Beta 3 is out! And, as one would expect from a late beta, not much has changed.

The good news is that we can use 31 for compileSdkVersion and targetSdkVersion. That means that we should have reached API stability.

Of the stuff that was announced:

App Search: you saw it here first!
The permission group lookup APIs are nice, but I seem to recall Google getting rather testy about apps trying to determine the relationship between permissions and groups. I wonder what changed… 🤔

Beyond that:

We lost some things from previous editions of Android 12, such as setShowForegroundImmediately() on Notification.Builder.
getAllNetworks() on ConnectivityManager is deprecated. There is an alternative involving registerNetworkCallback() and the awkwardly-named setIncludeOtherUidNetworks().
REQUEST_COMPANION_START_FOREGROUND_SERVICES_FROM_BACKGROUND is why we need auto-complete.

Strangely, TranslationManager and the rest of android.view.translation remain unannounced. They showed up in Beta 1, and Google hasn’t said anything about them AFAICT. More stuff to 🤔

This should wrap up the 2021 edition of the “Random Musings” posts — if we have reached API stability, there should be nothing more for me to muse about.

—Jul 15, 2021

"Elements of Android Jetpack" Version 2.1 Released

Subscribers now have access to an update to Elements of Android Jetpack, known as Version 2.1, in PDF, EPUB, and MOBI/Kindle formats, in addition to the online reader. Just log into your Warescription page and download away, or set up an account and subscribe!

OK, this took a lot longer to be released than I had expected, in part because Android Studio 4.2 took a lot longer to be released than I had expected.

There are a lot of changes in this update:

There is a new chapter, focusing on app widgets, those interactive home screen elements that apps can contribute
There is also a new chapter on using library modules, plus a new section on creating library modules
The chapter on dependency inversion was moved up one in the chapter sequence, and the chapter on Room was updated to use Koin’s DI implementation for the Kotlin sample
The chapter on Jetpack Navigation now also covers the Kotlin DSL
The various uses of startActivityForResult() were replaced by registerForActivityResult() and ActivityResultContracts
Everything was updated for Android Studio 4.2.2, the current shipping version

And, in addition to all of that, there are the usual suite of bug fixes, to the prose and to the sample code.

There should be one more update in 2021, after Android Studio 2020.3.1 Arctic Fox ships in stable form. That is in a beta right now, so it is likely to be at least a month or two before the stable release.

—Jul 12, 2021

Inside Code Transparency: The JWT File

I am starting to spend a bit of time poking around the implementation of code transparency, with an eye towards filling in some of the gaps that I wrote about in my initial thoughts post.

This time, let’s add code transparency to an App Bundle and see what that really means.

Adding Code Transparency

The first thing that you will need is a suitable Java keystore. This should not be the one that you use for any purpose other than code transparency. It also needs to have a 3072-bit key size (or higher, presumably). This means that the Android Studio keystore UI will not work, and you will need to create the keystore the old-fashioned way: using keytool.

You will also need an up-to-date copy of bundletool.

And, of course, you will need an App Bundle for your app.

From there, you can use the add-transparency command to add code transparency to a copy of the App Bundle:

bundletool add-transparency \
  --bundle=/path/to/your/AppBundle.aab \
  --output=/path/to/your/AppBundleWithCT.aab \
  --ks=/path/to/your/keystore.jks \
  --ks-key-alias=WhateverAliasYouUsed

In a nutshell:

--bundle points to the App Bundle that you created (e.g., from Studio)
--output points to where you want bundletool to write the augmented App Bundle
--ks points to your keystore
--ks-key-alias is the alias of the key inside that keystore that you wish to use

You will be prompted for the keystore password at the command line, or there are ways to use a --ks-pass command-line option to supply it.

This may take several seconds or longer, depending on the size of your App Bundle, the power of your machine running bundletool, the current phase of the moon, etc.

Examining the Augmented App Bundle

App Bundle .aab files are really ZIP archives, so you can examine them using your favorite ZIP utility. In an App Bundle with code transparency, you will find a file at:

/BUNDLE-METADATA/com.android.tools.build.bundletool/code_transparency_signed.jwt

The BUNDLE-METADATA/ directory “is what it says on the tin”: it is metadata about the contents of the App Bundle. Akin to JAR metadata contents, the contents of BUNDLE-METADATA/ appear to be namespaced by use, so com.android.tools.build.bundletool will contain metadata related to bundletool. code_transparency_signed.jwt is the actual code transparency file.

Decoding the JWT

That file is a JSON Web Token (JWT). It will be a very long encoded string. For example, a new project from Android Studio 4.2.2 resulted in a 3460-character code transparency JWT, looking a bit like:

eyJhbGciOiJSUzI1NiIsIng1YyI6WyJNSUlFMVRDQ0FyMmdBd0lCQWdJRVMwekdQVEFOQ...

(with a few thousand additional characters in place of the ...)

So, we need to decode it by one means or another. JWT is used by lots of systems, and so you may already have some tools for decoding its contents. Otherwise, this Web site offers online decoding, and Linux developers can add a bash function to decode at the command line.

(macOS and Windows developers: I’m sure you have something good to use too!)

Note that these tools merely decode the JWT, allowing us to see what is inside of them. They do not validate that the JWT has not been modified — that is a separate step.

Examining the JWT

That scrap project — based on the “Empty Activity” template FWIW — gives us the following JWT payload, after decoding:

{
  "codeRelatedFile": [
    {
      "path": "base/dex/classes.dex",
      "sha256": "c8a57ffe798c896f1b2c5f33862cbde817bb233c291217e3511245e1e9b91c82"
    },
    {
      "path": "base/dex/classes2.dex",
      "sha256": "192a3e51f14682fb41b91b99e916b068e818ad598d7d5659ea9c7e26c201de15"
    },
    {
      "path": "base/dex/classes3.dex",
      "sha256": "08cae05a2180b2249079bbabedaf3a8ac20bef1e4a3d4f8ea319ea4c2f42a396"
    }
  ]
}

There is no specification for this payload, something that we will need to rectify at some point. But, inside the codeRelatedFile array, we have individual JSON objects, each having:

path: a relative path, from the base of the .aab ZIP contents, of a “code-related file”, such as a DEX file
sha256: the SHA-256 hash of the contents of the identified file

DEX files get this shorthand JSON syntax. Native libraries have a couple of additional properties:

    {
      "path": "base/lib/x86_64/libflipper.so",
      "type": "NATIVE_LIBRARY",
      "apkPath": "lib/x86_64/libflipper.so",
      "sha256": "3f4d0a1a03b7825cb5350453c579d39f9f3369f885b8906dfdf1748510186664"
    },

As it turns out, there is a protobuf .proto file in the bundletool project that appears to describe this JSON structure. NATIVE_LIBRARY is an enum value, where DEX is the other value (and presumably is the default value if nothing is provided). apkPath is documented as “Path to file in the APK”; it is not quite clear to me why this is needed for native libraries but not DEX files.

I still have not yet torn into the bundletool implementation, but the verification process probably is something like:

Confirm that the JWT is signed by the expected signing key (with that chore largely being up to us)
Iterate over the JWT payload entries, find the corresponding DEX or .so file in the APKs installed for this app, and validate the SHA-256 hashes
Iterate over the DEX and .so files of the APKs installed for this app and confirm that everything there was represented in the JWT (so there has not been a code insertion attack)

I will continue blogging about code transparency in the coming weeks and months, as I try to make sense of how we can cover what Google has not: actually using this to ensure that our apps are not being manipulated.

—Jul 11, 2021

Initial Thoughts on Code Transparency

Over nine months since I broached the uncomfortable questions about app signing, we have the official response. It is simultaneously more than I would have expected and less than what we need.

These materials have only been in my hand for a few hours, and I expect I’ll write more about the situation next week. But, let’s see what we got.

First, there is this FAQ entry from “The future of Android App Bundles is here”, the post where Google firmly declared that the App Bundle requirement is coming in a month:

When distributing apps on Google Play, how do I ensure my app is delivered to users the way I intend?

At any time, you can download and inspect artifacts from the Play Store, from the app bundle explorer in the Play Console, and via the Play Developer API to verify your app. In addition, code transparency for app bundles is a new, optional feature that can be used to inspect that code running on a device matches the code that was originally built and signed by the developer.

The first sentence is typical subterfuge. At best, all downloading those artifacts do is tell us the state of those specific artifacts. Google is perfectly capable of delivering different artifacts to different people. It is not significantly different than is serving different Web pages to different people, which Google has been doing since its inception. So, they can give unmodified artifacts to the developer and give tampered artifacts to other parties.

In terms of the second sentence, the passive tense in “can be used” is doing a lot of heavy lifting. As it turns out, “can” is somewhat theoretical at this time.

More of the details are in the “Code transparency for app bundles” page in the developer documentation.

(Google seems indecisive over whether or not App Bundle is a proper noun — I will give it The Capital Treatment here for consistency with past posts on this subject)

Code transparency creates a roster of SHA256 hashes for each DEX file and each .so file that is part of the App Bundle. If that file ships to a user as part of an app, somebody could use it to confirm that the DEX files and .so files in the app match their hashes. And, the code transparency file itself is signed, using a signing key private to the developer, so in principle we can determine if the code transparency file itself has been modified (e.g., to reflect hashes of tampered files rather than the original files).

That sounds good, and to an extent, it is.

However, all Google needs to do is remove the code transparency file from the apps that they deliver to users. After all:

They have full signing authority, so they can remove whatever they want
There is no legal or contractual requirement for them to ship this file
There is nothing in the operating system that is looking for this file, as they readily admit:

Important: The Android OS does not verify code transparency files at install time, and continues to rely on the APK signing schemes for verification of any installed APKs.

If there is no code transparency file, there is no code transparency.

Also, verifying that the code transparency file itself is the original implies that we can validate that its signature is intact and was from the developer’s signing key. There is no current infrastructure for this, as Google also admits:

Important: To verify that the signature comes from the original developer, the printed fingerprint from one of the following methods must be compared with the public key communicated by the developer through a trusted channel. For example, the developer can host their public certificate on a secure website that is known to belong to them. To ensure no other changes have been made to the app, checking the APK signature is also necessary and recommended.

In addition, the code transparency file conveniently omits some things:

It does not include the manifest (or resources, assets, etc.)
~~It does not seem to preclude there being *additional* DEX or `.so` files, beyond the ones in the code transparency file~~

(UPDATE 2021-07-02: The documentation has since been revised to clarify that the code transparency file is meant to be a complete snapshot of the DEX and .so files — if an app contains code that is not in the code transparency file, that should fail the transparency check)

~~There has been an app-resigning attack available since Android 8.0 that neatly fits those two omissions.~~

The upshot is that we, the developer ecosystem, will need to build a secure, reliable way for developers to advertise, for any given versionCode of an applicationId, that a code transparency file was included in the App Bundle, along with the certificate (or certificate hash) that was used to sign that code transparency file. Google, at least at present, is washing their hands of that mess. Without this, we have no way of knowing, for any given app, whether a missing code transparency file is a problem and whether an existing code transparency file is valid.

Then, we, the developer ecosystem, will need to create practical tooling around this stuff. What Google has given us is new commands for bundletool. As a reference implementation, that’s great. As a practical matter, that is very limited. For example, an anti-malware app running on Android itself is not going to be using bundletool directly via a shell command, if that’s even realistic. We are going to need a specification for the code transparency file, along with implementations that can be used in more flexible ways (e.g., a library). It is unclear to what extent Google will be interested in any of that.

In addition, we, the developer ecosystem, will need to experiment with extending this system to include other developer-chosen content, such as the manifest, and deal with the “what if there’s other stuff here than what’s listed?” issue.

Plus, we, the developer ecosystem, will need to get tools in the hands of users to actually validate the code transparency. It does us no good to have code transparency files if nothing checks for them and verifies that the app was not modified by Google. This will likely need to be some mix of existing security-related apps (e.g., the aforementioned anti-malware apps) and dedicated code transparency audit apps. Google is unlikely to help here, and frankly, we should view any such help as being suspect (“the fox guarding the henhouse”).

Also:

None of this affects APK-based uses of Play App Signing — official code transparency is only for App Bundles
None of this affects other distribution channels that employ equivalents to Play App Signing… such as the Amazon AppStore for Android, coming to a Windows 11 installation near you soon

We, the developer ecosystem, are going to need to figure out how to address those scenarios as well.

Worst of all, these are just the things that I have thought of. I am certain to be missing some problems or skipping over some scenarios, and those might add to the pile of work to be done.

Google’s implementation of code transparency is more than I expected, insofar as I really did not expect Google to do much of anything. I have no idea how robust the core elements of their solution are, and we will only find that out over time. But, the fact that this even exists is a positive step, incomplete as it may be.

That incompleteness is why this is less than what we need.

And all of this is just to avoid allowing developers to continue distributing APKs the way that they have for over a decade, if they so choose.

In essence, Google has slashed our car tires, and then has generously offered to pay for a lift home. While that is a nice gesture, it does not address the problem with the car, and it would have been nicer if Google had not slashed the tires in the first place.

Again, I expect to write more about this in the coming weeks and months. Suffice it to say: there’s a lot of work ahead for those of us concerned about the problem. If you are concerned about the problem, enough to perhaps help with that work, please reach out!

—Jun 29, 2021

Older Posts