The Death of External Storage: I Can Haz File?

Under the fairly innocuous title of “scoped storage”, Google has announced that external storage, as Android developers have used it, effectively is dead, for Android Q and onwards.

I am going to spend some time on this over the course of the week, as this change needs some attention. Yesterday, I described what has changed and what you are supposed to use. Today, I will cover why “what you are supposed to use” will not always work and what to do about it.

In an ideal world, any stream that we get from an arbitrary Uri would meet all our needs. Unfortunately, that is not the case.

Take, for example, OkHttp. It is one of the most popular libraries for Android. Among other things, it allows you to do HTTP POST and PUT operations, including supplying bulk data for upload. However, its primary class for that — RequestBody — only supports byte[], File, String, and the Okio ByteString classes as sources of what should be uploaded. It does not support Uri, as OkHttp is not an Android library. And it does not support InputStream, as the content may need to be re-read multiple times, depending on server responses and OkHttp configuration.

Which means that, on Android Q, uploading a file using OkHttp may be a problem, if you do not have a file in the first place.

For small content, you could try to read it all into memory and then use the resulting byte[]/String/ByteString with OkHttp. That will not work with larger content, though.

Libraries with file-only APIs are going to be troublesome. That is why I suggested nearly six years ago that libraries try to focus on streams, not files. But there are other scenarios where you may need a file:

  • AFAIK, the NDK knows nothing about Uri, and even using an InputStream may be difficult

  • Some framework classes, such as SQLiteDatabase, need files, because they need random access

  • Some media APIs may work with some streams (those backed directly by files on the filesystem) but not others (those needing the provider to do work on the content, such as decrypting it)

So, what can you do?

One possibility is to ask the user to put the file in one of the locations on external or removable storage where you do have filesystem access, such as getExternalFilesDir(). This makes it easy on you, but it is not very user-friendly. These locations are awkward: Android/data/one.package.name.among.hundreds/files is a pain to navigate to. Plus, since other user-installed apps have no access to your app’s corner of external storage, the user will be limited to whatever system apps have access or breaking out the USB cable and doing it from their desktop or notebook.

Another possibility is to make a copy of the data to some file that you control, perhaps on internal storage. You use the Uri that you have to get an InputStream, then copy the bytes from there to some FileOutputStream on a file that you control. Then, you use the file with whatever needs it. This eliminates the user headache, but now you are working off of a copy, with all the problems that entails:

  • Changes made to your copy are not reflected in the original

  • Making the copy may take a while and take up lots of space

  • You need to get rid of the copy at some point, and knowing when to do that may be difficult to determine

  • And so on

In some cases, you might be able to work around the problem by using something other than a stream. ContentResolver also offers APIs to let you get a ParcelFileDescriptor or AssetFileDescriptor, and those in turn let you get to java.io.FileDescriptor objects. Perhaps the API that you are using will accept one of those instead of a File. Just bear in mind that not every Uri that you get will be able to supply a file descriptor, as not every Uri is backed by a file on the filesystem. You will need to “gracefully degrade” when you cannot get a file descriptor, at the very least explaining to the user that your app is incompatible with wherever that Uri came from.

The best long-term answer is to get more APIs rewritten to avoid needing files. In some cases, that will mean pushing down this sort of work-off-a-cached-copy into the library, though at least it may know better when to get rid of the copy. Or, the library could offer some way for you to provide fresh streams when needed.

Going back to the OkHttp example, it is at least conceivable that there could be a `RequestBody.create()` variant that took a `StreamSupplier` parameter instead of a `File`. `StreamSupplier` would be an interface with a single abstract method (e.g., `openInputStream()`) that OkHttp could call as many times as needed, whenever it would need to start reading the bytes from the top. You, as the user of OkHttp, would simply create a `StreamSupplier` that turned around and called `openInputStream()` on a `ContentResolver`. In principle, this should address OkHttp's need for starting over. I have not looked at the guts of OkHttp to see whether something like this could work, and it would not surprise me if there are strong technical reasons why this would be impractical for them. It's also a hack. But, tough times call for tough measures, and we may need libraries to take this sort of approach to deal with the `Uri`-centric world we are in.

UPDATE 2019-03-27: Going back to the OkHttp example, you need to create a custom RequestBody that knows how to get an InputStream from a ContentResolver on-demand and pour the data through the BufferedSink supplied to writeTo(). cketti pointed this out to me on Twitter and provided a sample implementation.

One way or another, we’re going to have to do something.

Tomorrow, I will have a bit of info on the other side of the problem: how to avoid or detect when a legacy app tries handing you a file, one that you may not be able to access.

 

Meta


Creative Commons License