How to Screw Over Your Beta Users Without Really Trying

An addendum to this week's post about releasing buggy apps: a story of how everything can go wrong.

Trello Android has a beta program filled with lovely beta testers who willingly volunteer to try our less-than-perfect builds.

One way to screw over your beta users is to send them buggy code. That's an obvious problem, though, and basically the reason betas exist.

There was an unexpected, surprising way we screwed over our beta users recently I'd like to share.

We had a beta build going for a while, but needed to push a hotfix for our current production build. When you push a new build to production, the Play Store automatically graduates all beta users to the production version of the app. That meant that all our beta users were given the hotfix release.

However, we'd forgotten that our beta users were using a newer version of the database. As such, when they installed the hotfix, the SQLiteOpenHelper called onDowngrade(), which we don't handle at all (we just crash)!

In other words, this is our timeline:

• December: Everyone is on 4.7.0
• January: Production has 4.7.0, beta users get 4.8.0
• February: Production gets 4.7.1, and beta users are downgraded to 4.7.1

Luckily, we quickly caught on to what was happening (monitor those crash reports after any release!) and only a handful of people hit the issue before we rolled out another beta build (so beta users could continue using 4.8.0 safely).

I've heard rumors that you can avoid the automatic graduation of beta users to production builds by using wildly high build numbers for the betas, but I haven't confirmed that myself.

Anyways, the moral of the story is that you need to be careful with which builds your beta users receive.