Trello Android's Git Branching Strategy
Many years ago, Trello Android used a fairly simple git branching strategy - or so we thought at the time.
We would develop entirely off main using pull requests. main was intended to be releasable at all times, though we would occasionally create a release/x.y.z branch, if we felt some set of features needed extra testing time.
After being acquired by Atlassian, we ran into a couple problems with our process.
First, Atlassian is a publicly traded company, and thus it must comply with the Sarbanes-Oxley Act (SOX). For us, that meant we had to make release branches SOX-compliant, adhering to additional rules and constraints. Creating a new SOX-compliant branch had a lot of overhead and deleting one was even more heinous. Thus, release branches were now a gigantic PITA to create and delete.
Also, our old strategy had a propensity to create nightmare situations when fixing release bugs. Suppose we were preparing release/1.3.0, but it turns out release/1.2.0 had a severe bug. We’d fix that on hotfix/1.2.1, oh but how do we make sure that bug fix makes it into main and release/1.3.0, fuck it let’s just cherry-pick everything. Let’s hope the hotfix doesn’t require any further hotfixes or things get really ugly.
(I tried to make a chart demonstrating the above but it was such a clusterfuck I’ve spared you all from it.)
In other words, the “simple” solution, when faced with unfortunately common circumstances, became quite complex and lacked a well-defined strategy. This problem was compounded by a growing team; the amount of communication required on where to merge code grows exponentially with the number of developers.
With all this in mind, we set out to find a better branching strategy.
Defining The Right Tool for the Job
I don’t believe in a one-size-fits-all solution for basically anything and that’s especially true for code management strategies. Your context, goals, and constraints will guide you to the correct tool for the job.
For example, GitHub Flow is great! The only shared branch is main, you merge features via pull request, and bam - it’s deployed immediately. It’s fantastically simple! But… GitHub Flow relies on continuous deployment and Android apps can’t do that.
Thus, for our first step, we made a list of factors to take into consideration.
Here are the problems we were trying to fix:
- Creating a SOX-compliant branch is expensive.
- We need a well-defined strategy for all situations.
- Our strategy should scale for more than a handful of developers.
Then we listed out the cultural preferences of the team:
- We review code before merging.
- We avoid long-running feature branches (instead using feature flags).
- We want to be able to test/fix a release on its own branch before shipping.
Finally, a couple facts about our environment:
- We release three builds regularly: internal, beta, and production.
- We only release one version of the software at a time to production; we never need to maintain old legacy releases.
- Android doesn’t support continuous deployment.
Our Pick: Three-Flow
We ended up basing our branching strategy off of Three-Flow. The basic idea of three-flow is that you only have three long-running, stable branches - one for development, release candidates, and releases.
Our three branches are main, candidate, and release. main is for ongoing development. When you want to start a new beta, you merge main into candidate. Once that build is stable enough, you merge candidate into release.
If a bug is found in candidate, then you fix it in candidate and merge it back down to main. Likewise, if a severe bug is found in release and you need to do a hotfix, you fix it in release, then merge that fix down to candidate and then main.
(Notice how there are never any merges directly from main to release or vice versa; the benefit of always going through candidate means simplicity and consistency that is IMO worth the extra work.)
Whenever you want to merge code to any of the three branches (main, candidate, release), you create your own feature branch (e.g. dlew/feature) and open a pull request on the target branch.
We use feature flags to avoid shipping pre-release code to users. A half-baked feature can still be merged into candidate and release, just in a disabled state.
Three-Flow Advantages
Three-flow fixed our issues:
- Stable branches means never having to create or delete a SOX-compliant branch ever again.
- Each stable branch is releasable to either our internal, beta, or production tracks.
- We have a playbook for developing features, resolving beta issues, and deploying hotfixes.
- It’s a simple strategy that makes it easy to know where to merge code (main for dev, candidate for beta, release for hotfixes).
It’s also compatible with our existing preferences:
- We can still use pull requests to review code.
- It explicitly embraces feature flags and shuns long-running feature branches.
- It gives us time to bake a release candidate before pushing to production.
As a bonus, three-flow also doesn’t add extra cruft that we don’t require. By cutting out support for things like continuous deployment and legacy releases, it keeps the complexity down.
Three-Flow Modifications
The core of three-flow about having three stable branches, but how you manage those three branches can differ from team to team. Thus, it’s worth noting that we diverged from the original article in two ways:
- Short-lived feature branches - We are big fans of code reviews, thus opening pull requests on branches is a key part of our flow. Remote branches are a prerequisite for PRs, but the original article considers any sort of remote branch anathema. They propose just rebasing all commits onto master - but then how do you do code reviews?
Now, we do not use long-running feature branches - we agree that these are nightmares waiting to happen and that feature flags are superior. - Merging between branches - The article uses force pushes for the release branch. While it makes your git trees look prettier, we don’t like rewriting git history (and SOX-compliance also prevents us from rewriting history anyways).
There is a certain elegance to always merging between the three branches; it’s always the same process for going from one to the other, and it makes code history entirely traceable. Plus, using merges means you can just use pull requests to merge between branches - no having to memorize more git commands.
Why Other Options Didn’t Work
We ruled out other popular git branching strategies for various reasons:
- Trunk Based Development (also known as Stable Mainline or Cactus) - This strategy was basically what we were doing before.
- GitFlow - It’s significantly more complex than other branching strategies and requires creating many release branches.
- OneFlow - Simpler than GitFlow, but still requires creating many release branches.
- GitHub Flow - It requires continuous delivery in order to work.
Should You Use It?
I’m not proposing that everyone go out and use three-flow today. It’s great for us given our constraints, but it may not work for everyone. In particular, there are at least two cases where you should definitely not use it:
- If you can do continuous deployment, use GitHub Flow. It’s simpler!
- If you have to support multiple releases at once (legacy versions + a current version), three-flow simply won’t work.
Barring those cases, I would consider giving three-flow a chance. We’ve been using it for over two years with no problems.