Detailed Lecture Notes
Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau
Spanner is a wide-area distributed transaction system that supports strictly serializable transactions. Spanner was implemented and designed by Google.
Generally, strictly serializable transactions are extremely burdensome and slow to implement. However, Google uses a bevy of techniques to avoid taking too much of a hit to performance. Spanner is a case-study of how a system can exhibit extremely strong consistency guarantees while still being performant and relatively high-throughput.
Spanner contains several ideas we have seen in previous sections:
As with other systems, Spanner started as an internal service to google. Spanner is a database that supports transactions, and these transactions are strictly serializable (or externally consistent as the paper refers to it).
A main goal for Spanner was also to be highly scalable. Today, Spanner holds 15 exabytes of data and supports 5 billion transactions per second. Spanner backs all major google applications today such as gmail, docs, etc.
Spanner‘s requirements:
Unlike Dynamo, Spanner supports Sql-like queries.
The database consists of multiple tables and each table must have a primary key.
This primary key allows Spanner to easily partition the table (which is critical for sharding).
Effectively, because Spanner uses this primary key, we can still imagine data as a key-value store, with the key being the primary key and the value being the rest of the row.
Spanner shards its datastore based on the primary key. Each shard is also replicated (by default 3 replicas exist for each shard). Specifically, each shard is replicated globally across datacenters spread around the world.
Source: Google
This global distribution has two benefits:
Replicas across zones run a paxos variant algorithm for distributed consensus. This means for each shard there will be one paxos leader. Certain operations will go through the paxos leaders, which enforces ordering.

Aside: In spanner, compute and storage within a zone are decoupled. Sometimes this pattern is also called disaggregation in research. Why might one want to decouple compute and storage in this fashion? Primarily, this disaggregation makes it easier to scale each layer independently. If we need more storage, we can easily add more storage machines and not necessarily dedicate cost to needlessly adding more compute. Furthermore, we have the ability to utilize heterogenous hardware– having specialized machines with many cores, or large amounts of memory, or large amounts of storage.
To understand how operations occur in Spanner we must understand there are different plans for different types of transactions. Considering different types of transactions with different plans allows Spanner to optimize performance without sacrificing strong consistency guarantees.
Specifically we must consider:
Some transactions will only impact data that exists in a single shard. The story for this type of transaction is simpler due to the fact that only a single paxos group is responsible for the entire transaction (we will not have a distributed commit decision across paxos groups).
Thus, our plan for single-shard read/write transactions will be to go through the current paxos leader for the shard in question.
For our transactions to be serializable, they must be isolated. Thus, we must implement some concurrency control mechanism to enforce isolation when transactions may run concurrently.
Spanner elects to use 2PL to enforce isolation between transactions.
The plan for R/W transactions on a single shard:
Effectively, this plan is very similar to typical distributed consensus as in Raft, we use the leader to enforce a strict order of locks and operations and we use distributed consensus protocol to ensure that it is replicated on the follower servers.
Spanner makes one modification to 2PC to avoid potential deadlock. The method Spanner uses is “Wound-Wait”. This mechanism ensures that transactions will make progress.
Wound wait works as follows:
What if transactions contain read/writes of data across shards? Our plan must become a bit more complicated because now operations occur across multiple paxos groups! This means there is no single leader for all operations within the transaction.
To guarantee our transactions are atomic we must implement a distributed commit protocol across the paxos groups. (With a single shard, the abort/commit decision is atomically made by the paxos leader).
Our plan for isolation will be the same– we will use 2PL. To acquire a read or write lock for a data object, we will contact the shard leader for that data object.

To run 2PC across shards we will select some paxos group to act as the transaction coordinator. All operations that the TC performs will be replicated across the paxos group.
Now, a R/W txn across shards can occur as follows:
Notice, the key downside for 2PC was indefinite blocking while holding locks. Still, in Spanner’s scheme participants hold locks in the prepare phase. But Spanner argues that this downside to 2PC is not significant for them. Why?
The key difference between Spanner’s implementation of 2PC and the implementation we saw in the prior section is that Spanner replicates the TC and all participants on a paxos group.
Thus, when does indefinite blocking occur? Only if a majority of the paxos participants, across datacenters, fails. This is far less likely to occur than any one server failing.
Thus, Spanner circumvents indefinite blocking as long as the paxos group is available. This argument was controversial at the time given 2PC’s bad reputation.
Remember the global distribution aspect of shard replicas. So then we consider, is this operation of replicating across paxos fast? No, generally, performance of these transactions is a bit costly.
But, let’s remember the workload of Spanner. The most common operation is a read. Thus, in our plan for read-only transactions, can we avoid replicating the operation across the paxos group, avoid locks, and forgo 2PC?
Our first plan is as follows: can we simply query a local replica (local in this case meaning physically close, i.e. a client in Utah, queries the salt lake datacenter replica) for a read.
However, let’s consider is this serializable?
Let’s consider the following example:
T3 is a read-only transaction. It simply sent Rx to a local replica and received 3, the result after T1 completes. However, T2 then runs, and updates y. Then T3 performs Ry and receives a 1. Is this serializable? No. There is no order of T1, T2,T3 which would result in Rx3,Ry1 being a valid read result.
Thus a question arises, how can we enforce serializability while still having high throughput for read-only transactions?
Spanner uses snapshot isolation to enforce serializability.
When we perform a R/W transaction, we will also note the timestamp of that read/write transaction (assigned on transaction end). We will then maintain multiple versions of each data item, associated with each timestamp.
Then, each read transaction will read the value of a data item relative to the timestamp of the transaction. A read transaction is assigned this timestamp on transaction start.
Consider the following example with two multi-versioned datastores. If we read relative to TS 15, we return that x is 30, as the write of X is 10 is the most recent value, but before the timestamp.

Notice: that this approach of snapshot isolation enforces serializability with any unique timestamps. However, if we want real-time guarantees of strict serializability we will need our timestamps to also correlate to real-time. We will explore how to assign real-time timestamps in the following sections.
Consider the following three transactions now. T1 is assigned a timestamp of 10 at transaction end. T2 is assigned a timestamp of 20 at transaction 20. T3 is assigned a timestamp of 15 at transaction start.
Say these timestamps correctly correspond to real-time. Is the below execution strictly serializable?
Notice, yes it is. The order of transactions appears to be T1,T3,T2. It is fine to order T3 and T2 in this fashion as the transactions overlap. Strict serializability requires non-overlapping transactions to occur in real-time order.
Let’s review, what is strict serializability?
Linearizability guarantees that the operations in our system have a global total order that obeys real time. However, linearizability makes no guarantees about sets of operations, or transactions.
Serializability guarantees that transactions appear to have been executed in some serial order. Serializability does not make a guarantee about real time.
Strict Serializability implies both serializability and linearizability. Strict serializability guarantees that transactions appear to have been executed in a serial order that respects real time. In particular, if a transaction starts in real time after another transaction’s commit time will see the effects of the earlier transaction. Linearizability can be viewed as a special case of strict serializability where each transaction is a single operation.
Attempt 1: Achieving Strict Serializability
How might we achieve strict serializability and assign timestamps that correspond to real time? A naive approach could be to designate some central authority to assign these timestamps. However, this would be prohibitively costly and bottleneck every transaction through this timestamp authority.
Instead, Spanner uses a clever construction: its TrueTime API.
Spanner’s true time timestamp incorporates an interval of uncertainty into the timestamp.
Upon calling TrueTime.now() a machine will receive a TrueTime timestamp of the form [earliest, latest].
The real time at that moment is guaranteed (or at least with such extremely high probability, that we can guarantee) to fall within this interval. In practice, this interval can be fairly small (7ms in 2012).
The foundation of TrueTime is built on top of google’s atomic clocks. Then, across servers that have these atomic clocks and/or GPS antennas, effectively we run a variant of NTP to synchronize clocks. This protocol limits the error of TrueTime to be within this small, millisecond bound.
Given two events, e1 and e2, each with their own moment of execution in real time t1, t2. If t1<t2.
The following properties hold for the two TT timestamps [earliest1, latest1] and [earliest2,latest2]
earliest1 <= T1 <= latest1earliest2 <= T2 <= latest2And together: earliest1 <= t1 < t2 <= lastest2.
How does spanner then use TrueTime to assign timestamps to transactions?
For each incoming transaction, the coordinator assigns it a timestamp:
TS = TrueTime.Now().latest
If this is a read-write transaction: this time represents the commit timestamp of the transaction — this is when results become visible to clients. The timestamp is assigned at the transaction commit.
If this is a read-only transaction: this time represents the start of the transaction — any writes prior to that timestamp will be visible to the transaction but later writes will be ignored. The timestamp is assigned at transaction start.
To enforce strict serializability, given this interval of uncertainty, can we be sure that when we commit and make writes visible to reads, we are doing so in the order that respects real-time?
It would be not acceptable in Spanner for a read, which occurs later in real-time, with this uncertainty window to grab a timestamp which causes it to not see a recent (but earlier in real-time) committed write.
To ensure that we do, Spanner adds an additional rule.
The commit wait rule says: after we assign the r/w transaction’s timestamp, we must wait until TrueTime.Now().earliest > TS.
In other words, we must wait until the current moment is certain to not overlap with our commit timestamp before making our writes visible.
This ensures that reads that occur after our commit point should also see that now is after the transaction. Other read transactions can not see data of the transaction until after the commit timestamp.
This commit wait rule ensures that r/w transactions are strictly serializable.
In the following example, without commit wait, it is possible that S1 assigns a timestamp 12 to T1. Then, a transaction T2 at S2 is assigned a timestamp of 8 (due to the out-of-sync clocks). Even though S2 commit point was later in real-time (treal=10 compared to 7), we have an issue as the timestamps order S2 earlier! This breaks strict serializability!
Now, consider that S1 must follow the commit-wait rule. We delay the commit point in S1, in real time so the above does not occur.
In the above example S1 delays committing and responding back to the client until TS interval no longer possibly overlaps with the time.Now() interval. Thus, in the above example, S1 commits after S2, which means that it is OK that S2’s timestamp of 8 would cause T1’s write to not be visible, because in real time, T1 commits after T2.
Now consider, in wanting to make reads fast, we allowed local replicas to respond to reads, isolated through snapshot isolation.
However, remember that this replica is a member of a paxos group. It is certainly possible that some followers may be isolated and have stale data. This is only a problem for r/o transactions, as other transaction types occur through the paxos leader.
How can we then ensure that r/o transactions in Spanner are still strictly serializable? This is implemented via safe time.
Safe time is the timestamp of the most up-to-date write operation that this follower has heard about.
The safetime per server is continually updated with paxos commits:
In other words, when a replica hears about a later write timestamp, it updates its safetime to that timestamp. Each replica can only respond to right requests for timestamps before this safetime. This property ensures that r/o transactions never are stale. However, it may cause some small delay. The authors of spanner argue that because writes occur frequently, this is not an issue.
Spanner is an excellent feat of distributed systems engineering acting as Google’s globally-distributed database, powering services like GMail, Maps, Search, Photos, Ads.
Spanner holds ~15 Exabytes of data spread in datacenters all over the world and supports ~5 billion transactions per second.
Spanner puts into practice a wide range of techniques we have discussed previously:
It also suggests a novel and clever construction for real time guarantees with its TrueTime api for transaction ordering.