The CommonsBlog

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. I updated them in 2017 to reflect changes in Android… but Android keeps changing.

This post is an updated edition of the 2017 post on removable storage.

Working with files on the filesystem in Android is seriously confusing. It was confusing in 2008, and it has only gotten more confusing as the years rolled on. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the third post in a series covering this storage model, to help clarify what is going on. Earlier, we looked at internal storage and external storage. Today, we will look at removable storage, the source of an unfortunate amount of angst.

What Your Users Think “Removable Storage” Means

Many of your users will have a device that has some sort of removable media. Often times this is a micro SD card. Some tablets or docks have a full SD card slot. Plus, USB mass storage is possible via USB On-The-Go and USB Type C connectors (not to mention devices or docks with a full USB host port).

Your users will think that they can work with removable storage much like they can with a desktop or notebook.

Unfortunately, your users are largely mistaken, and are even more mistaken with Android 4.4+. That’s because Google’s approach towards removable storage is… unconventional.

What Google Thinks “Removable Storage” Means

In the beginning, external storage was often in the form of a removable micro SD card. At that time, many developers got in the habit of thinking that external storage == removable storage.

However, as Android 3.0 and higher started rolling out en masse, developers soon started to realize two things:

1. External storage != removable storage on most of those devices

2. There’s nothing in the Android SDK for removable storage

Wait, Wut?

That’s right: until Android 4.4, there was no official support for removable media in Android. Quoting Dianne Hackborn:

…keep in mind: until Android 4.4, the official Android platform has not supported SD cards at all except for two special cases: the old school storage layout where external storage is an SD card (which is still supported by the platform today), and a small feature added to Android 3.0 where it would scan additional SD cards and add them to the media provider and give apps read-only access to their files (which is also still supported in the platform today).

Android 4.4 is the first release of the platform that has actually allowed applications to use SD cards for storage. Any access to them prior to that was through private, unsupported APIs. We now have a quite rich API in the platform that allows applications to make use of SD cards in a supported way, in better ways than they have been able to before: they can make free use of their app-specific storage area without requiring any permissions in the app, and can access any other files on the SD card as long as they go through the file picker, again without needing any special permissions.

But… But… But… What About All These Apps That Used Removable Media?

They fall into three buckets:

1. Some are just relying on MediaStore indexing. So, for example, a video player can find out about videos on all available media by querying the MediaStore, and if the device manufacturer hasn’t broken the MediaStore indexing of removable media, the player will be able to play back videos on removable media. This strategy is broken on Android 10 and higher.

2. Some are apps that ship with the hardware. The hardware manufacturer knows the device and what the rules of the game are for that device. The hardware manufacturer is also far less concerned about cross-device compatibility, as their apps aren’t (usually) shipping on the Play Store. Hence, a hardware manufacturer has carte blanche to work with removable media.

3. Some are apps written by developers who decided to go past the boundaries of the Android SDK. There are various recipes online for examining various Linux system files (and file-like substances) to determine what “mount points” exist, and from there apply some heuristics to determine what represents removable media. While reliability across devices could easily be an issue, beyond that, these techniques at least sorta worked… until Android 4.4, when everything changed.

What Happened in Android 4.4

Starting with Android 4.2, there was a request from Google for device manufacturers to lock down removable media. Generally, this was ignored.

For Android 4.4+, Google amended the Compatibility Test Suite (CTS) that device manufacturers must comply with in order to ship a device containing Google’s proprietary apps (e.g., Play Store, Maps, Gmail; otherwise known as “GMS”). Quoting Dave Smith:

However, new tests were added in CTS for 4.4 that validate whether or not secondary storage has the proper read-only permissions in non app-specific directories, presumably because of the new APIs to finally expose those paths to application developers. As soon as CTS includes these rules, OEMs have to support them to keep shipping devices with GMS (Google Play, etc.) on-board.

As a result, apps were able to read files on removable media using the various undocumented and unsupported tricks for finding removable media. However, apps cannot write to or otherwise modify such removable storage. Note that device manufacturers themselves may have ways of dealing with this, but ordinary app developers do not.

However, we were given getExternalFilesDirs() and getExternalCacheDirs() in Android 4.4. Android 5.0 added getExternalMediaDirs(). These methods give us locations on external and removable storage into which our apps can write. We do not need any permissions to use them — not even the runtime permissions from Android 6.0. However, the filesystem locations are a bit user-hostile and the content that we place in those directories will be removed when the app is uninstalled.

What Happened in Android 10

Removable storage, like external storage, was largely locked down. We still have access to the locations from the aforementioned methods. However, any other “back door” approaches that developers used were completely shut down.

This blog post has much more detail on what Android 10 did to external storage, and all that also applies to removable storage.

Should I Hardcode Paths in My App?

No.

Other Posts in This Series

This series includes:

• A post on internal storage
• A post on external storage
• This post on removable storage

—Oct 11, 2019

Every weekday, as part of maintaining AndroidX Tech, I round up all of the new and updated artifacts that are available on Google’s Maven repo (those that begin with androidx., anyway).

These showed up this morning:

• androidx.compose:compose-compiler:0.1.0-dev01
• androidx.compose:compose-runtime:0.1.0-dev01
• androidx.ui:ui-core:0.1.0-dev01
• androidx.ui:ui-layout:0.1.0-dev01
• androidx.ui:ui-framework:0.1.0-dev01
• androidx.ui:ui-animation:0.1.0-dev01
• androidx.ui:ui-animation-core:0.1.0-dev01
• androidx.ui:ui-android-text:0.1.0-dev01
• androidx.ui:ui-material:0.1.0-dev01
• androidx.ui:ui-tooling:0.1.0-dev01
• androidx.ui:ui-platform:0.1.0-dev01
• androidx.ui:ui-vector:0.1.0-dev01
• androidx.ui:ui-foundation:0.1.0-dev01
• androidx.ui:ui-text:0.1.0-dev01
• androidx.ui:ui-test:0.1.0-dev01

In other words, Jetpack Compose has arrived, in an early pre-release form.

Though, as it turns out, not really. The androidx.compose artifacts appear to be empty, other than a META-INF/MANIFEST.MF file in each.

Ian Lake did not seem to mention Compose in his release tweetstream. That, and their empty contents, means that is possible that these androidx.compose artifacts were accidentally put into the Maven repo.

A quick skim of the androidx.ui artifacts shows that they do have code. However, they depend upon androidx.compose:compose-runtime, and since that is empty, it is unclear if the androidx.ui artifacts will work either.

So, I do not know what to make of these artifacts. I and others will experiment with them, and if we can get them working somehow, we’ll be writing about it.

—Oct 10, 2019

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. I updated them in 2017 to reflect changes in Android… but Android keeps changing.

This post is an updated edition of the 2017 post on external storage.

Working with files on the filesystem in Android is seriously confusing. It was confusing in 2008, and it has only gotten more confusing as the years rolled on. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the second post in a series covering this storage model, to help clarify what is going on. Yesterday, we looked at internal storage. Today, we will look at external storage.

What Your Users Think “External Storage” Means

Many Android device users will have no idea what “external storage” means. There is nothing in the device UI that will necessarily use that term. At best, your users will think that an SD card is external storage. That’s not quite right. Except on Android 4.4+, in which case it is only partially right.

Did I mention that this stuff is confusing?

What Google Thinks “External Storage” Means

The Android SDK documentation has this to say in terms of a definition of “external storage”:

Every Android device supports a shared “external storage” space that you can use to save files. This space is called external because it’s not guaranteed to be accessible—it is a storage space that users can mount to a computer as an external storage device, and it might even be physically removable (such as an SD card).

In the halcyon days of yesteryear, there was a single volume known as “external storage”, and it was effectively defined as “the stuff that shows up when the user plugs their device into a computer using a USB cable”. Even that wasn’t completely accurate, as some manufacturers would also allow access to their devices’ removable media via the USB cable as well. And Android 4.4 added yet more wrinkles in terms of removable media… which is why removable media gets its own blog post tomorrow. Android 10 further “upset the apple cart” with all of this.

For the purposes of this blog post – and to line up with what most other written material will refer to – “external storage” is defined as the directory tree returned by Environment.getExternalStorageDirectory().

The Many Paths Under Which External Storage is Stored

For us as developers, the actual path under which this magical “external storage” was accessed varied over the years, from /sdcard, to locations under /storage, to /mnt/shell/emulated/0, then back to locations under /storage (e.g., /storage/emulated/0). And just as secondary users of an Android 4.2+ tablet get their own internal storage, they get their own external storage, with its own root directory.

Hence, as I mentioned previously:

NEVER HARDCODE PATHS

Use various methods to get the base directory to work from… though this too has gotten more complicated over the years.

The Many APIs for Finding External Storage Locations

In the beginning, everyone used Environment.getExternalStorageDirectory(), which pointed to the root of external storage. This led to external storage being just a big basket of random content.

Later, Google offered more organization:

• getExternalFilesDir() and getExternalCacheDir() on Context, pointing to an application-specific directory on external storage, one that would be deleted when the app is uninstalled

• Environment.getExternalStoragePublicDirectory(), for centralized places to store well-known file types, like photos and movies

Note that the Context methods have plural forms on Android 4.4+ (getExternalFilesDirs() and getExternalCacheDirs()), which ties into removable media, which we will get into more tomorrow.

And note that on Android 10, the methods on Environment are officially deprecated and are no longer accessible by default — more in this in a bit.

External Storage and Permissions

Just as the location – physically and logically – of external storage keeps changing, so does our ability to work with it.

Originally, apps could do whatever they wanted.

Android 1.5 added the WRITE_EXTERNAL_STORAGE permission, which apps had to hold to be able to write files to external storage. That way, the user would be informed at install time that the app intended to modify external storage contents. Any app could still read from external storage, though.

Android 4.4 started enforcing a READ_EXTERNAL_STORAGE permission, so you cannot even read from external storage if you do not hold some permission. Note that WRITE_EXTERNAL_STORAGE implies READ_EXTERNAL_STORAGE, so you only need one of those, not both. And, for getExternalFilesDir() and getExternalCacheDir(), you do not need either of those permissions – you can read and write in those directories with impunity. Android now has an android:maxSdkVersion attribute on <uses-permission>, specifically so that you can drop WRITE_EXTERNAL_STORAGE if you no longer need it, because you are only working with getExternalFilesDir() and getExternalCacheDir().

Android 6.0 made developers have to start asking for READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE have a via runtime permissions (e.g., checkSelfPermission(), requestPermissions()). That is because these permissions have a protectionLevel of dangerous.

What Android 10 Did

By and large, external storage is no longer accessible by default.

You still have access to getExternalFilesDir(), getExternalCacheDir(), and getExternalMediaDir() methods on Context and their associated filesystem paths. You do not need any permissions to work in those directories. However, they are in an inconvenient location for the user (Android/data/.../files/, where ... is your application ID). Plus, the content in those directories is removed when your app is uninstalled.

While there are ways of working around this limitation for Android 10, those workarounds are slated to be removed with 2020’s Android release.

This blog post has much more detail on what Android 10 did to external storage.

Poking Around External Storage

As a developer, assuming that you can find where external storage really resides for your version of Android, you have unfettered access to it from the Device File Explorer in Android Studio, for both emulators and production devices.

You can also use a USB cable, much like your users will use. However, bear in mind that what is presented to the USB interface is not what is on external storage… but, instead, is what has been indexed on external storage in the MediaStore. Hence, unless you take steps to ensure that new files that you create get indexed, they may not be immediately visible.

Under the covers, Android is using the Media Transfer Protocol for its USB communications. This enables a lot more flexibility than does Mass Storage Mode (a.k.a., what thumb drives use) that Android used originally. However, some MTP clients may cache directory listings, so even after you get your file indexed by MediaStore, an already-attached client may need to be refreshed.

How Do I Secure My Files on External Storage?

In general, you don’t. The user has complete access to external storage. Also, other apps can get to external storage if they hold the right permissions or the user chooses to grant them access to content (e.g., via ACTION_OPEN_DOCUMENT from the Storage Access Framework).

One thing that you can do is use something like Facebook’s Conceal. This encrypts your files on external storage, but uses a generated encryption key kept on internal storage. From a security standpoint, the net effect is to make external storage closer to internal storage in terms of read access. Note, though, that Conceal cannot prevent other apps, or the user, from deleting your files on external storage, or perhaps trying to write to them and corrupting the files as a result.

Should I Hardcode Paths in My App?

No.

Other Posts in This Series

This series includes:

• A post on internal storage
• This post on external storage
• A post on removable storage

—Oct 08, 2019

Back in 2014, I wrote a series of blog posts to try to clear up confusion around where you can read and write files in Android. I updated them in 2017 to reflect changes in Android… but Android keeps changing.

This post is an updated edition of the 2017 post on internal storage.

Working with files on the filesystem in Android is seriously confusing. It was confusing in 2008, and it has only gotten more confusing as the years rolled on. There are countless Stack Overflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the first post in a series covering this storage model, to help clarify what is going on. Today, we will look at “internal storage”… including its various definitions.

What Your Users Think “Internal Storage” Means

Some of your users will not recognize “internal” as a qualifier on “storage”. At best, they will think that “internal storage” represents all of the on-board flash. For example, users may still see “internal storage” in places like an Explorer window in Windows, when their device is connected via USB.

What Google Thinks “Internal Storage” Means

Alas, that is not what the Android SDK thinks “internal storage” means, which is where some of the confusion lies.

If you read the Android documentation on internal storage, they never actually bother to define the term.

In truth, the Android SDK’s definition of “internal storage” is a specific directory, unique to your app, where your app can place files. Those files are read-write for your app, but no other apps will have access to those files.

(exception: users running file managers with superuser privileges on rooted devices can access anything)

There are a handful of methods on Context that give you access to particular locations on internal storage, including:

• getCacheDir()
• getDir()
• getDatabasePath()
• getFilesDir()
• openFileInput()
• openFileOutput()

Other methods will rely upon these, such as openOrCreateDatabase(). Other classes also will rely upon these, such as SQLiteOpenHelper and SharedPreferences.

Where Internal Storage Is Stored… Sometimes

If you look around various blog posts, StackOverflow answers, and books that came out in 2012 or earlier, you will be told that your app’s “internal storage” resides at:

/data/data/your.application.package.name

(where your.application.package.name is replaced by your application ID, as is declared in the package attribute in the manifest or modified via Gradle)

Inside of there will be some directories automatically created by Android as you use some of those Context methods. For example, getFilesDir() returns a File object that points to a files/ directory inside of your app’s internal storage.

Where Internal Storage Is Stored… The Rest of the Time

However, this is not always where your app’s internal storage resides. If there is one rule for developers that you should take away from this blog post series, it is:

NEVER HARDCODE PATHS

Every now and then, I will see developers do something like this:

File f=new File("/data/data/their.app.package.name/files/foo.txt");

This is not a good idea.

First, it is more typing than using getFilesDir():

File f=new File(getFilesDir(), "foo.txt");

More importantly, internal storage is not always at the same place. Notably, we have the notion of separate user profiles, starting in Android 4.2 for tablets and Android 5.0 for phones. Each user gets his or her own “internal storage” area. While the aforementioned directory is still used for the primary user, that is not guaranteed, and it will not be used for secondary accounts.

Poking Around Internal Storage

The Device File Explorer tool in Android Studio 3.0+ can browse all of internal storage on an emulator, plus internal storage of debuggable apps on production devices. This should handle most of your scenarios:

From the command line, you can use the run-as option with adb at the command line. For example, to download a database from the primary user’s internal storage to your development machine, you might use:

adb shell 'run-as your.application.package.name cp /data/data/your.application.package.name/databases/dbname.db /sdcard'

Note that:

• You will need to change the destination to wherever on your device external storage is mapped to (shown here as /sdcard/, which will not work on all devices)

• You may need to use cat instead of cp on older devices

Once the file is on external storage, you can use adb pull to download it to your development machine, or access it by other conventional means (e.g., via mounting the device as a drive on your development machine).

Limitations of Internal Storage

On ancient Android 1.x and 2.x devices, internal storage was usually on a dedicated filesystem partition, and that partition was usually rather tiny. The HTC Dream (a.k.a., T-Mobile G1), the original Android device, had a whopping 70MB of internal storage for use by all apps.

(And, no, that’s not a typo. We measured storage in megabytes back then. We also had onions on our belts, as that was the style at the time.)

Nowadays, internal storage usually is substantially larger: 32GB, 64GB, 128GB, and so on.

Frequently-Asked Questions

Here are some FAQs about what the SDK refers to as internal storage:

Should I Make Files on Internal Storage World-Readable or World-Writable?

No. That is no longer an option on modern versions of Android, and it was never a particularly good idea.

Use FileProvider and serve that content via that ContentProvider implementation. Then, you at least have the option of using Android’s permission system to manage access to those files, versus having any app on the system be able to monkey with those files.

Well, How About android:sharedUserId?

I also counsel against this.

android:sharedUserId is an attribute you can place in your manifest that indicates a logical user ID to be used for your app. Any other app that is installed that is signed by the same signing key and requests the same android:sharedUserId will use the same Linux user from a security standpoint. The net effect is that those two apps will be able to work with each other’s files with impunity, as those files are all owned by the same Linux user.

This attribute is really designed for pre-installed apps, such as some software suite pre-loaded by the device manufacturer, carrier, or ROM mod maintainer. In particular, once you ship your app, you cannot reliably change your android:sharedUserId value without locking your user out of any existing files… as Android does not change the ownership of existing files when it changes the Linux user account that your app runs as.

There are various risks in having multiple processes work simultaneously with files. Some subsystems, like SQLite, have built-in logic to deal with this. But if you are doing your own file access yourself (e.g., via File and Java I/O), then you have to somehow deal with simultaneous access, which can get tricky.

You also have to deal with what happens when one app is uninstalled, taking away files that another app was using. In a hub-and-spoke model, like an app and a suite of plugins, perhaps this is not that risky. In other models, where apps are closer to being peers, you cannot afford to have your app’s data vanish because the user elected to uninstall some separate app.

Finally, you do not know what the future might bring. Right now, you might view your set of apps as a tightly-coupled suite. Somebody who acquires those apps, or acquires your firm, might wish to go a different route. Using data sharing options that are more loosely coupled, like a ContentProvider, gives you greater flexibility than does assuming that your apps should co-mingle their files. In an ideal world, your app should treat other apps as a fairly-reliable but not always available resource, just like you treat your own Web service.

Besides, android:sharedUserId was deprecated in Android 10 and is slated to be removed from the OS in a future release.

How Do I Prevent Users of Rooted Devices From Accessing My Files on Internal Storage?

Simple: don’t put those files on internal storage. Users of rooted devices can access whatever they want on the device, so the only way to prevent them from accessing your data is to not have it on the device.

Some developers will attempt to encrypt their files, using a hard-coded password, to prevent rooted device users from using those files. This does erect a “speed bump”, but it is a small one. All it takes is one person with interest to reverse-engineer your app, determine how to decrypt those files, then write up a blog post or discussion board entry on how to do it.

On the whole, there are relatively few people with rooted devices – I estimate it at well under 1%. IMHO, you will have better success by focusing your engineering work on writing a better app, rather than spending that time trying to block rooted device users.

Did Any of Those Android 10 Changes Affect Internal Storage?

No. You can work with internal storage on Android 10 the same as you have in other recent Android versions.

Should I Hardcode Paths in My App?

No.

Other Posts in This Series

This series includes:

• This post on internal storage
• A post on external storage
• A post on removable storage

—Oct 06, 2019

The documentation for view binding states:

Note: View binding is available in Android Studio 3.6 Canary 11+.

That’s not strictly accurate.

Things like view binding are not part of Android Studio. If they were, we could not use view binding in projects that are built from the command line or from a CI server.

The more accurate description is:

Note: View binding is available in Android Studio 3.6 Canary 11+, once you upgrade to 3.6.0-alpha11 or higher of the Android Gradle Plugin

For a new project created in Android Studio 3.6, you will get the corresponding Android Gradle Plugin line with a version matching that of Android Studio. But, if you open an existing project, you may have an older Android Gradle Plugin version in your Gradle scripts.

The real requirements for using view binding are:

• Use 3.6.0-alpha11 or higher of the Android Gradle Plugin:

buildscript {
    ext.kotlin_version = '1.3.50'
    repositories {
        google()
        jcenter()
        
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:3.6.0-alpha12'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    }
}

• Use Gradle 5.6.1 or higher, as that is required by this version of the Android Gradle Plugin

• Use Android Studio 3.6 Canary 11+, as Android Studio 3.5 (or older) will not open a project using 3.6.0-alpha11 or higher of the Android Gradle Plugin

For solo-developer projects, making these changes is easy. For team projects, everybody will need to switch to Android Studio 3.6 Canary 11+, which may or may not be practical, depending on your team’s stance on pre-release tools.

—Sep 22, 2019