Syncing Changes

Going offline meant using our database for all immediate changes to the client. But obviously we later want to sync those changes with the server.

The basic idea behind our sync is to store changes made offline as a series of deltas that are later uploaded to the server. In this post I’m going to examine how exactly we construct and upload those deltas.

Delta Structure

First, each new change to a particular model is stored as an entry. These are broad ideas like “create board”, “update list”, or “delete card.” Each entry has some metadata associated with it, such as when the change was created, the current state of the change (e.g. pending, uploading, failed, cancelled), or how many times we’ve attempted to sync the change.

For each model that’s changed, there can be multiple fields that are modified. We call these changes deltas; each delta represents a single field being modified in some way. These are rather simple; they just point to a field and store what value existed before and after the change.

Together, these two tables are enough to represent any changes we need to sync with Trello.

Calculating Deltas

On Android we wrote an annotation-based delta calculator. First, you annotate fields @DeltaField:

public class Card {
    @DeltaField(ModelField.NAME)
    private String name;

    @DeltaField(ModelField.DESC)
    private String description;
}

Then you feed the calculator two models and we can compute the difference between them. While in rare corner cases (necessitated by some ill-thought-out data structures) we still had to calculate deltas by hand, for the most part delta calculation is automatic.

It turned out that calculating deltas wasn’t just useful for syncing: Anytime we needed to know how different two objects were we could use the delta calculator. For example, we keep track of how many conflicts occur when syncing data, and the delta calculator was great for computing this information.

Uploading Deltas

Once the app has connectivity again, we fire up a background syncer which uploads all our deltas.

We always execute deltas in the order received. That way you know you couldn’t end up with any data hierarchy issues. For example, a user might create a card then edit the description of that card. If we tried to upload the “edit description” change first, the server would be severely confused because it doesn’t even know the card exists.

For each delta, we convert it into an HTTP request that goes out to the server. Due to convenient standardization in the Trello API, in many cases these requests could be formed in a generalized fashion just by filling in the blanks:

> POST https://trello.com/1/{model}/{id}?field=value,field2=value2

It wasn’t all perfect, though. In some cases, we’d have to use custom code to translate deltas into HTTP requests. That only happened when we couldn’t directly map a field name to what the API was expecting or when we had to access the child of a model (e.g., checkitems must go through /1/checklist/{id}/checkitem/{id}).

Once we’d formed the HTTP request, we’d execute it. If the execution worked, great! Move on to the next delta. If the upload didn’t work… well, then we’ve got a problem. That’ll be what I go over in the next post: our extensive failure handling.

This article was originally posted on the Trello engineering blog and has been reproduced here for posterity.