The Storage Situation: Removable Storage

UPDATE 2019-10-11: This is the older edition of this blog post — I suggest that you click through and read the newer one.

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. This post is an updated edition of the original post on removable storage.

There is a lot of confusion regarding Android’s storage model. That confusion got a lot worse with Android 4.4’s changes to that storage model, and it has not really improved much since. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the third post in a three-part series covering this storage model, to help clarify what is going on. Monday, we looked at internal storage. Yesterday, we looked at external storage. Today, we will look at removable storage, the source of an unfortunate amount of angst.

What Your Users Think “Removable Storage” Means

Many of your users will have a device that has some sort of removable media. Often times this is a micro SD card. Some tablets or docks have a full SD card slot. Plus, USB mass storage is possible via USB On-The-Go and USB Type C connectors (not to mention devices or docks with a full USB host port).

Your users will think that they can work with removable storage much like they can with a desktop or notebook.

Unfortunately, your users are largely mistaken, and are even more mistaken with Android 4.4+. That’s because Google’s approach towards removable storage is… unconventional.

What Google Thinks “Removable Storage” Means

In the beginning, as was noted yesterday, external storage was often in the form of a removable micro SD card. At that time, many developers got in the habit of thinking that external storage == removable storage.

However, as Android 3.0 and higher started rolling out en masse, developers soon started to realize two things:

  1. External storage != removable storage on most of those devices

  2. There’s nothing in the Android SDK for removable storage

Wait, Wut?

That’s right: until Android 4.4, there was no official support for removable media in Android. Quoting Dianne Hackborn:

…keep in mind: until Android 4.4, the official Android platform has not supported SD cards at all except for two special cases: the old school storage layout where external storage is an SD card (which is still supported by the platform today), and a small feature added to Android 3.0 where it would scan additional SD cards and add them to the media provider and give apps read-only access to their files (which is also still supported in the platform today).

Android 4.4 is the first release of the platform that has actually allowed applications to use SD cards for storage. Any access to them prior to that was through private, unsupported APIs. We now have a quite rich API in the platform that allows applications to make use of SD cards in a supported way, in better ways than they have been able to before: they can make free use of their app-specific storage area without requiring any permissions in the app, and can access any other files on the SD card as long as they go through the file picker, again without needing any special permissions.

But… But… But… What About All These Apps That Use Removable Media?

They fall into three buckets:

  1. Some are just relying on MediaStore indexing. So, for example, a video player can find out about videos on all available media by querying the MediaStore, and if the device manufacturer hasn’t broken the MediaStore indexing of removable media, the player will be able to play back videos on removable media.

  2. Some are apps that ship with the hardware. The hardware manufacturer knows the device and what the rules of the game are for that device. The hardware manufacturer is also far less concerned about cross-device compatibility, as their apps aren’t (usually) shipping on the Play Store. Hence, a hardware manufacturer has carte blanche to work with removable media.

  3. Some are apps written by developers who decided to go past the boundaries of the Android SDK. There are various recipes online for examining various Linux system files (and file-like substances) to determine what “mount points” exist, and from there apply some heuristics to determine what represents removable media. While reliability across devices could easily be an issue, beyond that, these techniques worked… until Android 4.4, when everything changed.

What Happened in Android 4.4: The Good News

Two things happened in the timeframe of Android 4.4 that affect removable media.

On the plus side, we gained some measure of official Android SDK support for removable media. Specifically getExternalFilesDirs() and getExternalCacheDirs() (note the plural form) will not only return directories that we can use on “real” external storage, but also will return directories that we can use on any available and supported removable media. Our apps do not need any specific permissions to use any of those directories.

Also, the Storage Access Framework gives device manufacturers some options for exposing removable media to our apps in a more controlled fashion. Quoting Jeff Sharkey:

However, apps can create/write files outside of their package-specific directories on secondary external storage devices by using the new CREATE_DOCUMENT intent, which involves the user in the file creation process.

What Happened in Android 4.4: The Angst-Riddled News

Since Android 4.2, there has been a request from Google for device manufacturers to lock down removable media. Generally, this was ignored.

For Android 4.4+, Google amended the Compatibility Test Suite (CTS) that device manufacturers must comply with in order to ship a device containing Google’s proprietary apps (e.g., Play Store, Maps, Gmail; otherwise known as “GMS”). Quoting Dave Smith:

However, new tests were added in CTS for 4.4 that validate whether or not secondary storage has the proper read-only permissions in non app-specific directories, presumably because of the new APIs to finally expose those paths to application developers. As soon as CTS includes these rules, OEMs have to support them to keep shipping devices with GMS (Google Play, etc.) on-board.

As a result, apps can read files on removable media using the various undocumented and unsupported tricks for finding removable media. However, apps cannot write to or otherwise modify such removable storage. Note that device manufacturers themselves may have ways of dealing with this, but ordinary app developers do not.

The android-platform Google Group has been the home of a rather epic discussion thread regarding Google’s decision here, including Dianne Hackborn’s entry linked to earlier in this post.

Despite the hand-wringing, I do not expect Google to backpedal on this issue. For every user that complains about how Android 4.4+ makes their micro SD card useless, there’s a pundit complaining about how Android lets any app run amok and read stuff on storage, raising privacy and security issues. In fact, I would not be the least bit surprised if read access to removable media is blocked in the future, beyond an app’s own directory on that media.

Where Can I Learn More?

Two of the better resources for this topic are ones I linked to earlier in this post:

OK, So What Are We Supposed to Do?

Ideally, use the Storage Access Framework:

  • ACTION_OPEN_DOCUMENT (akin to a legacy system file-open dialog)
  • ACTION_OPEN_DOCUMENT_TREE (akin to a legacy system directory-open dialog)
  • ACTION_CREATE_DOCUMENT (akin to a legacy system save-as dialog)

On Android 7.0+, you can also use StorageVolume to request access to specific directory trees on removable storage.

All of these give you a Uri back, representing a document or a document tree. You would use those with things like DocumentFile and ContentResolver to work with the content (or tree of content).

The big advantage of this approach is that your code is now storage-agnostic. The same code will work with:

  • External storage
  • Removable storage
  • Google Drive
  • Samba/CIFS network file servers (if the user has Google’s app for that installed)
  • Anything else exposed by a third-party documents provider

While your code will be somewhat different than when using java.io.File, it’s not that different:

  • You still use InputStream and OutputStream to read and write content, but you get them from ContentResolver (e.g., openInputStream()), rather than from FileInputStream and FileOutputStream

  • You still can traverse document trees, but you do so using DocumentFile, not File

The biggest difference is that you should no longer assume that there is a meaningful filename, using DocumentFile to get a “display name” to show the user.

Certain types of apps, such as media players, might be better served using the MediaStore and ignoring the filesystem entirely. Not only can you query the MediaStore to find out the available media of relevance, but you can create a MediaStore Uri that has the MediaStore stream the content to you, even for content that you might not be able to access.

But, That Doesn’t Fit My UI

From what I’ve seen, the #1 source of complaints about the “lost” removable storage access comes from developers of file managers. Their argument is that they need to be able to list all sources of files and traverse them.

That’s just not how Android works. With some tweaks to the UX, you can get something that is far more Android-friendly.

For example, rather than considering your app to be the “source of truth” for document trees, let the user do that, in the form of a “bookmark” system. Your app would come with a pre-configured bookmark for external storage. If the user taps an action bar item (or whatever) to add a new bookmark, you use ACTION_OPEN_DOCUMENT_TREE and let them choose a tree from whatever they want. You take persistable permissions on the Uri that you get back, then save the Uri as part of your app data (e.g., in a database). You then use DocumentFile for traversing the document tree for presentation in your UI.

With this approach, you fit better with Android, by using the Storage Access Framework. Along the way, you offer more functionality to the users, as now your “file manager” can manage content from a variety of places, including things that are not local files. Similar approaches can be used for other types of apps, switching from a “file” metaphor to a “content” metaphor and leveraging the Storage Access Framework.

 

Meta


Creative Commons License