The Death of External Storage: Correcting a Mistake

UPDATE 2019-06-08: With Q Beta 4 out, the story is now much simpler. I am leaving the original post here for historical reasons.

My post from last Monday had a significant mistake. I wrote:

You cannot write to the standard directories available from Environment.getExternalStoragePublicDirectory(). So, you cannot write to DIRECTORY_DOWNLOADS or DIRECTORY_MUSIC or places like that.

That is incorrect — you can write to these locations… sort of.

The reason for my mistake and the reason for the “sort of” qualification help to illustrate what is going on with these external storage changes.

You can write to those directories, but the directories may not already exist for your app… even if they appear for the user.

If you have been working in Android from the early days, you may recall a time when emulators (and occasionally devices) would not ship with the common external storage directories already created. So, if you wanted to use Environment.getExternalStoragePublicDirectory(), you would need to call mkdirs() on that File to ensure that the directory existed before using it.

However, that missing-directory problem largely cleared up years ago. In my test app, I foolishly did not bother trying to create the directories. After all, I could see the directories when browsing the device. So, when I was getting FileNotFoundException errors, I assumed it was a permissions thing.

Instead, even though I could see the directory, my app could not. Once I updated my app to create the directory, those locations worked as expected… more or less.

The “more or less” part is that, like most of external storage, what the app creates is invisible to the user, whether via the Storage Access Framework or via a USB cable.

So, it appears that each app has its own independent external storage, which Google refers to as the “isolated storage sandbox” in the documentation. Two apps can write the same file to the same path with different contents, because while each app thinks that it is writing to the one true external storage, instead it is writing to its own external storage sandbox, which is independent of the external storage sandbox of other apps.

This explains the problems with the Storage Access Framework and the USB cable. Effectively, the the Storage Access Framework and the USB cable show what I will call “the user’s sandbox”, which is not the same sandbox as that of any particular app.

The primary exception to this are the Context-supplied locations, like getExternalFilesDir() and getExternalCacheDir(). There, the user’s sandbox includes the per-app sandbox for those directories. While I doubt that it is implemented this way, you can think of this as a symlink: the user’s sandbox gets symlinks to each app’s sandbox’s entries in Android/data/, so the user can see what the app sees within those areas.

From a privacy and security standpoint, this is a slick solution.

From a usability standpoint, it’s a problem… if Google follows through on its threat and says that all apps, regardless of targetSdkVersion, will be subject to these rules on Android R. Many legacy apps put their files on external storage outside of the Context-supplied areas and expect users to be able to get to those files. Not only does scoped storage not support this, if my analysis is correct, scoped storage can’t support this without some substantial change to allow the user to say “show me external storage from the perspective of App X”. Since we cannot do that even with adb shell run-as, it may not be technically possible. Even if it is, from a UX standpoint, it is unclear how this would be represented to the user (e.g., one USB volume per app?).

UPDATE 2019-04-29: pzychotix pointed out that there is a way for the user to view these sandboxes: go into Android/sandbox/ from the user’s external storage root. That provides a list of app-specific directories by application ID, and those directories are the app’s own external storage roots. This works, and for developers and other power users is reasonable. I suspect that the average user will not know about this and may have difficulty identifying the desired app in the sea of application IDs.

A regular targetSdkVersion solution may be required for usability. In other words, only apps targeting 29 Q or higher would get this behavior, with legacy apps writing directly to the user’s sandbox. Since the Play Store requires an ever-increasing targetSdkVersion, most actively-maintained apps will get to 29 Q in the not-too-distant future. Yet the apps that are not being maintained, but still provide value, would still work as they have.

Regardless of the circumstances, I apologize for my mistake. I should have been more careful in my earlier tests.