Initial Responses to Uncomfortable Questions

A couple of months ago, I asked some uncomfortable questions about App Signing. Those were sparked by Google’s declaration that all new apps have to be distributed as App Bundles, which in turn require App Signing. App Signing introduces a big problem: Google can modify apps before they sign them. And while Google on its own may not necessarily have a reason do make any such modifications, other parties do and may try to force Google to alter apps.

My post appears to have gotten a fair bit of attention and, for some, it has touched a nerve. And now, with a target timeframe of August 2021 for that App Bundle requirement… we have some responses to the questions!

Recently, Chet Haase, Wojtek Kalicinski, Iurii Makhno, and Ben Weiss held a Q&A session on App Bundles as part of the MAD Skills presentation series. They took questions from Twitter and YouTube comments, in addition to a few warmup questions. And several of the questions touched upon the concerns that I outlined in that original blog post and that others have echoed in the days and weeks that followed.

First, I am very grateful that Google took the time to respond! And, the entire Q&A discussion was very well done, despite 2020’s limitations (e.g., Chet’s video stream froze at one point… though it’s possible that it was just a very impressive impersonation of a frozen stream, given Chet’s comedy skills).

The biggest downside of these responses coming in the form of a YouTube video is that videos do not always get transcribed and indexed in text form. So, I took the time to write down the questions and answers of relevance to my original post — though, again, the entire discussion is worth watching.

For each one, I provide:

• A link to the particular offset into the video
• The question
• The panelist delivering the answer, and that answer
• Some comments of my own for several of them

Note that in some cases, I had difficulty understanding what was being said, or the questions were somewhat garbled. Where you see [stuff in brackets], that is my translation of what was said to try to make the material readable. Where you see {stuff in braces}, that is a placeholder where I could not even venture a guess as to what was said. My apologies for the gaps!

15:35: Is it possible to verify that an APK on the Play Store matches its open source code repository? And, is it possible to do that with app bundles as well?

Iurii:

Currently, I am not aware of any solution to [sign only DEX files] that are available right now, so but on the other hand, I can say that bundletool manipulates the DEX files for only one specific reason. As you probably know, Android platform below 5.0 doesn’t support multiple DEX files natively for your application. So, in case your app needs to support such devices, and doesn’t fit into one DEX file, and requires multidex support, bundletool in this case will merge DEX files to {garbled} and actually make your application compatible with legacy multidex. Other than that, bundletool doesn’t touch your DEX files at all. You can check it via checking source code of bundletool. Also, you can install your APKs on a device, take it from it, and check in Android Studio that your DEX files are not changed at all.

Personally, my concern is not whether bundletool is modifying anything. We could find that out easily enough. However, bundletool output can be modified by Google on their servers before distributing the APKs based on that output — that is the concern.

While developers have means of inspecting a set of APKs, they do not have the means of inspecting all sets of APKs. Google serves billions of custom Web pages per day. Serving custom APKs — some altered, some not — is not a very big hurdle for Google to overcome.

This next question is one that I posted via Twitter!

17:47: Will you extend App Bundles to allow for developer-signed artifacts and no App Signing?

Wojtek:

So, I talked briefly about the requirement next year for new apps to use app bundles, and one thing that comes with that is that by extension we will require Play App Signing. So developers will need to either generate the App Signing key on Play or upload their own key to Play… because that’s a prerequisite for app bundles. We’ve heard from developers that some of them just don’t want to do it. They don’t want to have keys managed by Play. And currently that’s not possible if you want to use app bundles. But, we’ve heard that feedback, and… I can’t talk about anything right now, we don’t have anything to announce, but we are looking into how we could alleviate some of these concerns. It doesn’t necessarily have to be allowing to keep your own key while uploading bundles. We’re looking into different options. We just don’t have a solution to announce right now. But, we still have around a year until the requirement, so I’m really hopeful that we’ll have an answer for developers for this.

The fact that Google is thinking about these concerns, and discussing them publicly, is a wonderful development!

19:41: Can you address the statements that App Bundles can be decompiled by the Play Store to add malicious code?

Iurii:

I think this question really relates to something that I… answered previously. I already described that DEX files are not touched, and a way how developers can ensure it right now.

Google modifies uploaded content all the time. Google serves modified content all the time. Google has more than sufficient engineering capability to extend this to modifying APKs. The only thing that slowed them down, historically, was the signing process, and App Signing removes that impediment. That is why I am concerned about mandatory App Signing for new apps.

28:35: [Is there a way] we can keep our existing key (without giving [it] to you) and key still take benefit of the app bundle?

Wojtek:

I briefly talked about this before. We are looking into options… not right now, and I don’t have anything to announce, but… we’re hearing this feedback and we are looking into it. But I don’t know what the end result will be.

37:28: The APKs are not created on the store, runtime protectors will now see them as being tampered with. Any plans to support this going forward?

All members of the panel passed on this question. They may not have recognized the name “runtime protector”, but I assume that they are familiar with DexGuard and other tools that try to help defend apps against reverse engineering, kernel debuggers, and other forms of attack.

38:19: Is there a possibility maybe in the future… to be able to generate the APKs by a tool, maybe something like bundletool, and upload those APKs… {Chet froze}

Wojtek:

So, again, this is a question that I think I’ve already answered. Probably not as it’s described in the question, as this would make the publishing process even more difficult for developers, and we actually want to make it simpler and safer. However, again, we’ve heard this feedback, and we will be looking into options how to make this possible, however probably not in the way that was described here.

I was surprised about the specific comment regarding that it would “make the publishing process even more difficult for developers”. ZIPping the signed APK set that bundletool creates would seem to be a trivial addition to bundletool. Uploading a ZIP of APKs would be as easy as uploading an AAB file… in part because an AAB file is also a ZIP file.

Perhaps the concern is that the ZIP-of-APKs might be larger than the corresponding AAB, and there will be some set of developers where that incremental upload bandwidth might be a problem. But, I expect that most developers interested in opting out of App Signing, for the occasional app update, can deal with a few extra MB of bandwidth.

My sincere hope, though, is that this was just a case where the question arrived in real time. Answering on-the-fly questions is difficult. That is one of the reasons why I would not ask this sort of question that way — the only reason why I raised my own was because I could do it well in advance (via Twitter) and Google could decide whether to include it and how to respond.

So, on the whole, I considered the set of responses to be positive. They offered no specifics, but I would not expect them to do so in that sort of presentation anyway.

Perhaps my biggest concern is that I do not know who Google is working with in terms of figuring out the impacts of their mandatory App Signing decision. They are delivering apps to billions of people; decisions along these lines desperately need the input of civil society groups (EFF and Access Now in the US, plus their counterparts around the world).

And while I am absolutely no substitute for such groups… if anyone at Google would like to bounce some ideas around, just let me know.

Otherwise, now we wait for details of what (if anything) will be done, prior to the August 2021 App Bundle requirement.