The CommonsBlog

Android Studio 3.0: What "Injecting Monitoring Logic" Means

Michael Comella published a nice bit of analysis last week, showing the ramifications of this innocuous sentence in the documentation for the Android Studio 3.0 profiler:

To show you advanced profiling data, Android Studio must inject monitoring logic into your compiled app.

When you run your app from Android Studio, in addition to deploying the APK, Android Studio also pushes three files to the root of your app’s portion of internal storage:

  • perfa.jar
  • perfd

These will be pushed over if:

  • You run the app from the Run option in the IDE (versus installing the app by other means)

  • You run the app on an Android 8.0+ device (and possibly on older ones as well, if you opt into that)

  • Have ever had the Android Profiler tool window open in Android Studio since you started up this copy of the IDE

It would appear that these three files contain code that gets injected into your app’s process as part of profiling it, in line with the quoted sentence.

Tactically, this can screw up some tests. Mr. Comella found out about the problem when testing Firefox Focus, Mozilla’s privacy-centric browser for Android. One of the tests involves ensuring that no extra stuff gets left behind after a browsing session, and the test tripped up over these unrecognized files.

(as an aside, testing to ensure that the files in internal storage are what you expect is a really cool test)

From a security standpoint, on the one hand, there does not appear to be an immediate problem. In my testing, these files are not packaged into the APK itself, but instead get pushed over separately. Hence, these files should not make it into your production environment. Still, I really hope that whatever mechanism Android Studio uses to inject this stuff is adequately secured.

My only complaint is that we cannot turn this off, short of closing up Android Studio, re-opening it, and making sure not to touch the Android Profiler tool. If the Profiler needs this injected stuff to work, that’s fine, but there still should be better developer-level control over the code injection. With luck, this will get addressed sometime.

So, my hat’s off to Mr. Comella for documenting this, and to Nathan Freitas for raising awareness further.

Dec 04, 2017

Android's Architecture Components Version 0.6 Released

Subscribers now have access to an update to Android’s Architecture Components, known as Version 0.6, in PDF, EPUB, and MOBI/Kindle formats. Just log into your Warescription page and download away, or set up an account and subscribe!

This is yet another small update, with two new chapters:

  • An overview of the repository pattern

  • Configuring SQLite beyond what Room directly supports

It also updates the samples to use 1.0.0 of the main Architecture Components, and to use 3.0.1 of the Android Gradle Plugin. Plus, the samples were updated for Java 8 support, including lambda expressions.

And, as usual, it fixes a variety of errata, and makes other small improvements.

Nov 27, 2017

AccessibilityServices, Play Store Bans, and API Design

TL;DR: When designing an API, keep features with privacy and security ramifications separate from other features, such that both you and developers using the API have a very clear sense of what is and is not sensitive.

Many of you may have heard about Google’s threats to remove apps from the Play Store that have an AccessibilityService that is not primarily for accessibility. Many outlets reported on this, including XDA Developers and Ars Technica. According to the purported emails, Google is stating that if your app has requested the BIND_ACCESSIBILITY_SERVICE permission:

you must explain to users how your app is using the android.permission.BIND_ACCESSIBILITY_SERVICE to help users with disabilities use Android devices and apps

Later, the email states that you should remove the app if “the permission is being used for something other than helping users with disabilities use Android devices and apps”.

On the one hand, I completely understand the rationale for instituting this sort of ban. Accessibility APIs have been a privacy and security problem for some time, even if many people are only recently noticing it. It is easier for Google to whitelist various apps that have legitimate need to use those APIs than it is to root out all malicious uses.

However, whenever you see bath water being thrown out, look for an associated baby.

In Android 8.0, Google added fingerprint gesture support. While relatively few devices have hardware support for this, it effectively turns the fingerprint sensor into a tiny trackpad. At present, you can detect swipes in the four cardinal directions (up, down, left, right), and you can take some action based upon this via a registered callback.

However, they put fingerprint gesture support in AccessibilityService. Hence, only apps that request the BIND_ACCESSIBILITY_SERVICE permission can find out about fingerprint gestures.

It is unclear why Google thinks that fingerprint gestures are primarily for users who need assistive technologies. After all, trackpads are fairly commonplace elsewhere, used by people with a wide range of abilities. Google even supports trackpads with Android apps on Chromebooks, without requiring the app to have an AccessibilityService.

Moreover, by putting fingerprint gestures into the accessibility APIs, not only would app developers need to fuss with an AccessibilityService, and not only would enabling that service trigger all sorts of privacy and security warnings to the user, but with the new Play Store policy, any app seeking fingerprint gestures needs to be for users with disabilities, period.

While the ban is aiming to defend users against unnecessary privacy and security challenges, there is nothing in today’s fingerprint gesture API that is sensitive, and it’s unclear what they could add in the future that would be sensitive. Even if the gestures somehow get integrated with actual fingerprint scans, there’s a separate API for fingerprint scans, not tied into accessibility.

So we get a new general-purpose feature, and courtesy of an engineering choice (where the API resides) and a policy choice (ban on unnecessary use of accessibility APIs), most developers cannot use that feature.

This brings me back to the TL;DR from the outset. Keep APIs with privacy and security ramifications separate from other APIs. Make it very obvious to all parties what portions of your API may put users at risk. Conversely, don’t attach normal APIs onto an API set that is sensitive just because it’s easier or, in the absence of those privacy and security ramifications, it might make sense. Bright lines help everybody understand what is and is not safe. And, in the future, should bans like this one be required, it is simpler to only affect those misusing the sensitive APIs if the sensitive APIs are clearly separated out from the remaining APIs.

In this case, given that accessibility APIs are going to have privacy and security ramifications indefinitely, IMHO the better approach would have been to have fingerprint gestures be somewhere else. For example, they could be just simple MotionEvent sources, not significantly different than how trackpads work elsewhere. Even if there are technical reasons for keeping fingerprint gestures out of the regular input framework, there could be a FingerprintGestureManager system service or some such that provided this ability, independent of the accessibility APIs. It’s not out of the question that this could be addressed in the timeframe of Android P (Peppermint? Pistachio? Pomegranate?), should Google choose to do so. Only time will tell if we can put this baby back into some bath water.

After all, you have to be very careful with some babies.

Nov 20, 2017

The Storage Situation: Removable Storage

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. This post is an updated edition of the original post on removable storage.

There is a lot of confusion regarding Android’s storage model. That confusion got a lot worse with Android 4.4’s changes to that storage model, and it has not really improved much since. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the third post in a three-part series covering this storage model, to help clarify what is going on. Monday, we looked at internal storage. Yesterday, we looked at external storage. Today, we will look at removable storage, the source of an unfortunate amount of angst.

What Your Users Think “Removable Storage” Means

Many of your users will have a device that has some sort of removable media. Often times this is a micro SD card. Some tablets or docks have a full SD card slot. Plus, USB mass storage is possible via USB On-The-Go and USB Type C connectors (not to mention devices or docks with a full USB host port).

Your users will think that they can work with removable storage much like they can with a desktop or notebook.

Unfortunately, your users are largely mistaken, and are even more mistaken with Android 4.4+. That’s because Google’s approach towards removable storage is… unconventional.

What Google Thinks “Removable Storage” Means

In the beginning, as was noted yesterday, external storage was often in the form of a removable micro SD card. At that time, many developers got in the habit of thinking that external storage == removable storage.

However, as Android 3.0 and higher started rolling out en masse, developers soon started to realize two things:

  1. External storage != removable storage on most of those devices

  2. There’s nothing in the Android SDK for removable storage

Wait, Wut?

That’s right: until Android 4.4, there was no official support for removable media in Android. Quoting Dianne Hackborn:

…keep in mind: until Android 4.4, the official Android platform has not supported SD cards at all except for two special cases: the old school storage layout where external storage is an SD card (which is still supported by the platform today), and a small feature added to Android 3.0 where it would scan additional SD cards and add them to the media provider and give apps read-only access to their files (which is also still supported in the platform today).

Android 4.4 is the first release of the platform that has actually allowed applications to use SD cards for storage. Any access to them prior to that was through private, unsupported APIs. We now have a quite rich API in the platform that allows applications to make use of SD cards in a supported way, in better ways than they have been able to before: they can make free use of their app-specific storage area without requiring any permissions in the app, and can access any other files on the SD card as long as they go through the file picker, again without needing any special permissions.

But… But… But… What About All These Apps That Use Removable Media?

They fall into three buckets:

  1. Some are just relying on MediaStore indexing. So, for example, a video player can find out about videos on all available media by querying the MediaStore, and if the device manufacturer hasn’t broken the MediaStore indexing of removable media, the player will be able to play back videos on removable media.

  2. Some are apps that ship with the hardware. The hardware manufacturer knows the device and what the rules of the game are for that device. The hardware manufacturer is also far less concerned about cross-device compatibility, as their apps aren’t (usually) shipping on the Play Store. Hence, a hardware manufacturer has carte blanche to work with removable media.

  3. Some are apps written by developers who decided to go past the boundaries of the Android SDK. There are various recipes online for examining various Linux system files (and file-like substances) to determine what “mount points” exist, and from there apply some heuristics to determine what represents removable media. While reliability across devices could easily be an issue, beyond that, these techniques worked… until Android 4.4, when everything changed.

What Happened in Android 4.4: The Good News

Two things happened in the timeframe of Android 4.4 that affect removable media.

On the plus side, we gained some measure of official Android SDK support for removable media. Specifically getExternalFilesDirs() and getExternalCacheDirs() (note the plural form) will not only return directories that we can use on “real” external storage, but also will return directories that we can use on any available and supported removable media. Our apps do not need any specific permissions to use any of those directories.

Also, the Storage Access Framework gives device manufacturers some options for exposing removable media to our apps in a more controlled fashion. Quoting Jeff Sharkey:

However, apps can create/write files outside of their package-specific directories on secondary external storage devices by using the new CREATE_DOCUMENT intent, which involves the user in the file creation process.

What Happened in Android 4.4: The Angst-Riddled News

Since Android 4.2, there has been a request from Google for device manufacturers to lock down removable media. Generally, this was ignored.

For Android 4.4+, Google amended the Compatibility Test Suite (CTS) that device manufacturers must comply with in order to ship a device containing Google’s proprietary apps (e.g., Play Store, Maps, Gmail; otherwise known as “GMS”). Quoting Dave Smith:

However, new tests were added in CTS for 4.4 that validate whether or not secondary storage has the proper read-only permissions in non app-specific directories, presumably because of the new APIs to finally expose those paths to application developers. As soon as CTS includes these rules, OEMs have to support them to keep shipping devices with GMS (Google Play, etc.) on-board.

As a result, apps can read files on removable media using the various undocumented and unsupported tricks for finding removable media. However, apps cannot write to or otherwise modify such removable storage. Note that device manufacturers themselves may have ways of dealing with this, but ordinary app developers do not.

The android-platform Google Group has been the home of a rather epic discussion thread regarding Google’s decision here, including Dianne Hackborn’s entry linked to earlier in this post.

Despite the hand-wringing, I do not expect Google to backpedal on this issue. For every user that complains about how Android 4.4+ makes their micro SD card useless, there’s a pundit complaining about how Android lets any app run amok and read stuff on storage, raising privacy and security issues. In fact, I would not be the least bit surprised if read access to removable media is blocked in the future, beyond an app’s own directory on that media.

Where Can I Learn More?

Two of the better resources for this topic are ones I linked to earlier in this post:

OK, So What Are We Supposed to Do?

Ideally, use the Storage Access Framework:

  • ACTION_OPEN_DOCUMENT (akin to a legacy system file-open dialog)
  • ACTION_OPEN_DOCUMENT_TREE (akin to a legacy system directory-open dialog)
  • ACTION_CREATE_DOCUMENT (akin to a legacy system save-as dialog)

On Android 7.0+, you can also use StorageVolume to request access to specific directory trees on removable storage.

All of these give you a Uri back, representing a document or a document tree. You would use those with things like DocumentFile and ContentResolver to work with the content (or tree of content).

The big advantage of this approach is that your code is now storage-agnostic. The same code will work with:

  • External storage
  • Removable storage
  • Google Drive
  • Samba/CIFS network file servers (if the user has Google’s app for that installed)
  • Anything else exposed by a third-party documents provider

While your code will be somewhat different than when using, it’s not that different:

  • You still use InputStream and OutputStream to read and write content, but you get them from ContentResolver (e.g., openInputStream()), rather than from FileInputStream and FileOutputStream

  • You still can traverse document trees, but you do so using DocumentFile, not File

The biggest difference is that you should no longer assume that there is a meaningful filename, using DocumentFile to get a “display name” to show the user.

Certain types of apps, such as media players, might be better served using the MediaStore and ignoring the filesystem entirely. Not only can you query the MediaStore to find out the available media of relevance, but you can create a MediaStore Uri that has the MediaStore stream the content to you, even for content that you might not be able to access.

But, That Doesn’t Fit My UI

From what I’ve seen, the #1 source of complaints about the “lost” removable storage access comes from developers of file managers. Their argument is that they need to be able to list all sources of files and traverse them.

That’s just not how Android works. With some tweaks to the UX, you can get something that is far more Android-friendly.

For example, rather than considering your app to be the “source of truth” for document trees, let the user do that, in the form of a “bookmark” system. Your app would come with a pre-configured bookmark for external storage. If the user taps an action bar item (or whatever) to add a new bookmark, you use ACTION_OPEN_DOCUMENT_TREE and let them choose a tree from whatever they want. You take persistable permissions on the Uri that you get back, then save the Uri as part of your app data (e.g., in a database). You then use DocumentFile for traversing the document tree for presentation in your UI.

With this approach, you fit better with Android, by using the Storage Access Framework. Along the way, you offer more functionality to the users, as now your “file manager” can manage content from a variety of places, including things that are not local files. Similar approaches can be used for other types of apps, switching from a “file” metaphor to a “content” metaphor and leveraging the Storage Access Framework.

Nov 15, 2017

The Storage Situation: External Storage

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. This post is an updated edition of the original post on external storage.

There is a lot of confusion regarding Android’s storage model. That confusion got a lot worse with Android 4.4’s changes to that storage model, and it has not really improved much since. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the second post in a three-part series covering this storage model, to help clarify what is going on. Yesterday, we looked at internal storage. Today, we will look at external storage.

What Your Users Think “External Storage” Means

Many Android device users will have no idea what “external storage” means. There is nothing in the device UI that will necessarily use that term. At best, your users will think that an SD card is external storage. That’s not quite right. Except on Android 4.4+, in which case it is only partially right.

What Google Thinks “External Storage” Means

The Android SDK documentation has this to say in terms of a definition of “external storage”:

Every Android-compatible device supports a shared “external storage” that you can use to save files. This can be a removable storage media (such as an SD card) or an internal (non-removable) storage. Files saved to the external storage are world-readable and can be modified by the user when they enable USB mass storage to transfer files on a computer.

Much of what has been written in the documentation and elsewhere about external storage was for the pre-Android 4.4 world. In those halcyon days of yesteryear, there was a single volume known as “external storage”, and it was effectively defined as “the stuff that shows up when the user plugs their device into a computer using a USB cable”. Even that wasn’t completely accurate, as some manufacturers would also allow access to their devices’ removable media via the USB cable as well. And Android 4.4 added yet more wrinkles in terms of removable media… which is why removable media gets its own blog post tomorrow.

For the purposes of this blog post – and to line up with what most other written material will refer to – “external storage” is defined as the directory tree returned by Environment.getExternalStorageDirectory().

The Many Places Where External Storage Is Stored

External storage, like a tumbleweed, goes where the wind blows it.

In Android 1.x and most 2.x devices, external storage was generally some form of removable media, typically a micro SD card. More importantly, for all Android 1.x and 2.x devices, external storage was a separate partition with a separate filesystem. While a few Android 2.3 devices elected to use on-board flash for external storage, that was still a separate partition from the partition that held internal storage. As a result, we wound up with devices that might advertise having several GB worth of storage, but that storage tended to be mostly for external storage, as the partitions could not be resized by ordinary users.

Android 3.0 changed this around, allowing internal and external storage to each be on the same partition, just in separate directory trees. This provided a lot more flexibility for users, as now there was no artificial hard distinction between space for internal storage and space for external storage. Device manufacturers still could elect to have external storage be a separate partition, or even be on removable media, but typically they did not.

The Many Paths Under Which External Storage is Stored

For us as developers, the actual path under which this magical “external storage” was accessed varied over the years, from /sdcard, to locations under /storage, to eventually the current /mnt/shell/emulated/0. And just as secondary users of an Android 4.2+ tablet get their own internal storage, they get their own external storage, with its own root directory.

Hence, as I mentioned yesterday:


Use various methods to get the base directory to work from… though this too has gotten more complicated over the years.

In the beginning, everyone used Environment.getExternalStorageDirectory(), which pointed to the root of external storage. This led to external storage being just a big basket of random content.

Later, Google offered more organization:

  • getExternalFilesDir() and getExternalCacheDir() on Context, pointing to an application-specific directory on external storage, one that would be deleted when the app is uninstalled

  • Environment.getExternalStoragePublicDirectory(), for centralized places to store well-known file types, like photos and movies

Note that the Context methods have plural forms on Android 4.4+ (getExternalFilesDirs() and getExternalCacheDirs()), which ties into removable media, which we will get into more tomorrow.

External Storage and Permissions

Just as the location – physically and logically – of external storage keeps changing, so does our ability to work with it.

Originally, apps could do whatever they wanted.

Android 1.5 added the WRITE_EXTERNAL_STORAGE permission, which apps had to hold to be able to write files to external storage. That way, the user would be informed at install time that the app intended to modify external storage contents. Any app could still read from external storage, though.

Android 4.4 started enforcing a READ_EXTERNAL_STORAGE permission, so you cannot even read from external storage if you do not hold some permission. Note that WRITE_EXTERNAL_STORAGE implies READ_EXTERNAL_STORAGE, so you only need one of those, not both. And, for getExternalFilesDir() and getExternalCacheDir(), you do not need either of those permissions – you can read and write in those directories with impunity. Android now has an android:maxSdkVersion attribute on <uses-permission>, specifically so that you can drop WRITE_EXTERNAL_STORAGE if you no longer need it, because you are only working with getExternalFilesDir() and getExternalCacheDir().

Poking Around External Storage

As a developer, assuming that you can find where external storage really resides for your version of Android, you have unfettered access to it from the Device File Explorer in Android Studio, for both emulators and production devices.

You can also use a USB cable, much like your users will use. However, bear in mind that what is presented to the USB interface is not what is on external storage… but, instead, is what has been indexed on external storage in the MediaStore. Hence, unless you take steps to ensure that new files that you create get indexed, they may not be immediately visible.

Under the covers, Android is using the Media Transfer Protocol for its USB communications. This enables a lot more flexibility than does Mass Storage Mode (a.k.a., what thumb drives use) that Android used originally. However, some MTP clients may cache directory listings, so even after you get your file indexed by MediaStore, an already-attached client may need to be refreshed.

How Do I Secure My Files on External Storage?

In general, you don’t. The user has complete access to external storage, and other apps can get to external storage if they hold the right permissions.

One thing that you can do is use something like Facebook’s Conceal. This encrypts your files on external storage, but uses a generated encryption key kept on internal storage. From a security standpoint, the net effect is to make external storage closer to internal storage in terms of read access. Note, though, that Conceal cannot prevent other apps, or the user, from deleting your files on external storage, or perhaps trying to write to them and corrupting the files as a result.

Nov 14, 2017

Older Posts