Detailed Lecture Notes
Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau
Up until this point, we have informally discussed one form of consistency– strong consistency.
We described strong consistency informally, and as several ideas:
Now, let’s expand our understanding of consistency and consider the following model.
Consider the distributed data store with two replicas. This datastore is responsible for selling Taylor Swift tickets. Consider that Alice queries the datastore for the remaining number of tickets. She receives a response of 1. She then reserves the ticket. She messages Bob, her friend, that she just reserved the last ticket.
A little while later, Bob comes along, and then requests the datastore himself for the number of tickets left. He, however, talks to a different underlying replica, gets a stale result, and sees that the number of tickets remaining is still one, despite Alice’s reservation.
This example illustrates a possible execution of a system with a weak consistency model. On a single machine, we could not have the operations read tickets_left = 1, write tickets_left = 0, read tickets_left = 1 occur!
The strongest consistency model is linearizability.
Informally, we can consider linearizability to mean:
| An execution is linearizable if there exists a global total order of operations in the system and that order obeys real time. Global Total Order: An order of all the operations/results within an execution which one could obtain from running these operations on a single machine in that order. Obeys real time: for non-overlapping operations, if operation A commits before operation B begins, in our global total order A must come before B. | | :—- |
To inspect if a system’s consistency model, we visualize a system execution in a plot where:
Rx1 (meaning read x=1) or Wy2 (meaning write y = 2).
A single Rx1 operation.
Let us consider the following execution. Is it linearizable?
To understand the execution, we can ask ourselves, is there an order of all the operations that could be replayed on a single machine and get the same results (global total order). Furthermore, does that order obey real time?
Consider the following global total order of operations:
Wx1Rx2Rx2Wx3Rx3We can also see this obeys real time. Even though the second Rx2 starts after the beginning of Wx3, the two operations overlap, thus it is completely possible that Rx2 observes the system state before Wx3 was finally committed.
Another approach for visualizing if an execution is linearizable is to attempt to draw an atomic moment in time, where the operation occurs (within the operation’s duration). If you can– the execution is linearizable.
Consider the below diagram showing a single atomic moment for each operation to occur. Notice, these lines (read from left-to-right) correspond to the above global total order.
Let’s consider another example, is the following linearizable?
Let’s attempt to create a global total order that obeys real time.
Wx1 must precede Rx1 in the order (for our value to be read correctly). However, Rx1 must precede Rx0 in real time.
Thus, we are stuck! We can not have Rx1 Wx1 Rx0 Rx0 Rx1. Because the reads would not result in a 0, after Wx1. We will find no such global total order that satisfies our constraints, thus the execution is not linearizable.
Now consider the following execution, is this linearizable?
Yes! We can find a global total order of operations that satisfies real time constraints.
Wx1, Rx1, Wx2, Rx1.
Why are we allowed to order Rx1 before Wx2 in the order? Again, because they overlap we have no time constraints. We can imagine the instant in time each operation occurs as follows:
Let us consider if the consistency model offered by raft is linearizable.
Linearizability for committed log operations is guaranteed via the protocol’s use of a single leader to append and commit entries into the log.
Thus, we can see from the Raft protocol we get a single global total order of operations which is exactly the order in this log.
Furthermore, let’s consider Raft’s real time guarantees.
Thus, we can see that for two non-overlapping operations, we obey real time guarantees. An earlier committed entry will come first in the log.
Can we achieve linearizability for reads without committing them in the log?
Does Chain Replication guarantee linearizability?
Linearizability for reads is enforced by the fact that all reads go through a single tail server.
Linearizability for writes is enforced by the fact that all commits, and the eventual response to the client, go through the single tail server.
By the time the tail serves a read, it has already applied every acknowledged write: the tail’s state reflects the complete prefix of the global write sequence.
Furthermore, let’s consider CR’s real time guarantees.
Thus, for non-overlapping operations, we obey real-time guarantees, as they are all processed by this single tail server.
We may notice that there is an inherent trade-off:
The CAP Theorem recognizes this tradeoff:
CAP Theorem (2000): Consistency, Availability, Partition tolerance: pick 2 out of 3
However, network partitions are inevitable, so a more useful way to put it is “A distributed system can be either Consistent or Available when Partitioned”.
Thus, to increase a system’s availability, we may consider weaker consistency models than linearizability.
What if our clients don’t need the strength of the real-time guarantees provided by linearizability, and would prefer our system to be more available? We can weaken the consistency model used by our system, instead offering sequential consistency.
Sequential consistency has the same guarantee of appearing to have a single, agreed-upon, order of operations that could have been produced on a single machine. However, sequential consistency abandons the real-time guarantees that linearizability requires.
| An execution is sequentially consistent if there exists a global total order of operations in the system and that order obeys client ordering. Global Total Order: An order of all the operations/results within an execution which one could obtain from running these operations on a single machine in that order. Obeys client ordering: for two operations submitted by a single client, A and B. If operation A happens-before operation B, in our global total order A must come before B. | | :—- |
Consider the following execution, is it linearizable? Is it sequentially consistent?
The above execution is not linearizable as we must have Wxa precede Rxb in our global order due to real time constraints (Wxa completes before Rxb).
However, the execution is sequentially consistent with the following global total order:
B:WxbC:RxbD:RxbA:WxaC:RxaD:RxaThis order of events obeys each individual client’s orderings, and could occur on a single machine. Note, we are free to have A:Wxa placed fourth in this ordering, as there are no real-time constraints.
Aside: When might sequentially consistency occur in practice? If we have a system wherein clients read from geographically local replicas, which are periodically synchronized across regions, this could result in sequentially consistent executions. These executions emerge when we respond immediately to clients in our region, and reconcile requests at a slower rate between these regions.
Now consider the following execution, is it sequentially consistent?
As we try to create a global total order of operations, we find the following issue. Wxa must precede Wxb in the order (as we must respect the order of client operations). However, Rxb must precede Rxa in the order (to respect client C and D’s order of operations). Finally, remember it is impossible to Rxb before Wxb. Thus, it is impossible to create a total order which obeys all of the above constraints. The execution is not sequentially consistent.
In summary, with sequential consistency:
###
Some systems do not need a global total order of all operations across all servers. A reasonable consistency model could instead be to provide ordering guarantees only on operations with causal relationships. A system could then be causally consistent to meet this requirement.
An execution is causally consistent if operations with causal relationships are executed in a global order that obeys those causal orderings. Obeys happens-before relationships: Causal relationships include any operations that occur in a single client (we may not re-order a client’s submitted operations). Furthermore, writing a variable to a specified value has a happened-before relationship with reading that value. (i.e. Wx1 → Rx1)
Consider the following execution. Is it sequentially consistent? Is it causally consistent?
The above execution is not sequentially consistent. Consider that there is no single global total order that can have both: client C’s Rxc occurs before Rxb and client D’s Rxb before Rxc.
However, let’s consider the causal relationships shown in the execution:
A:Wxa → A:Wxc (client ordering)
C:Rxa → C:Rxc→C:Rxb(client ordering)
D:Rxa → D:Rxb→D:Rxc(client ordering)
A:Wxc → Rxc(value)
A:Wxa → Rxa(value)
B:Wxb → Rxb(value)
None of the above orderings contradict, thus we can say the execution is causally consistent.
We can see that because Wxc and Wxb are causally independent, we do not need to enforce a consistent global order on these two operations. Thus, no contradiction arises from having C:Rxc→C:Rxb and D:Rxb→D:Rxc.
Now, let’s consider the following execution. Is it causally consistent?
Like before, let us enumerate the causal relationships:
B:Rxa →B:Wxb(client ordering)A:Wxa → Rxa(value)B:Wxb → Rxb(value)C:Rxb → C:Rxa(client ordering)D:Rxa → D:Rxb(client ordering)By transitivity and (1) and (2), we can see that A:Wxa → B:Wxb.
However, a contradiction then arises with (4). There is no way for A:Wxa → B:Wxb, and C:Rxb → C:Rxa to be satisfied simultaneously.
In other words, there is a causal relationship between the Wxa and Wxb. Thus, we can not observe these two events in a different ordering (like on client C).
One key observation is that consistency problems occur because of conflicting concurrent writes.
However, in many real-world systems, reads are far more common than writes, and writes do not need to be instantly propagated.
Thus, a need arises for consistency models which encapsulate this real world need.
Eventual consistency can lend itself to highly available systems by compromising with comparatively quite weak consistency guarantee. Modern systems, with a focus on performance and availability, may use eventual consistency.
Eventual consistency says, in the presence of no future writes, eventually, all reads in the system will converge to the same value. The time to converge is undefined, and system specific.
In practice this can mean, with no updates, all replicas converge to the same internal state, eventually.
Consider the below execution.
The execution is not causally consistent (Client C observes Ryb, then Rxa, even though Wxb → Wyb). But, such temporary inconsistencies are allowed: eventually we see the system converge such that all reads of x, return b. Thus, the execution is eventually consistent.
Examples from http://nil.csail.mit.edu/6.824/2022/notes/l-raft2.txt
|-Wx1-| |-Wx2-|
|---Rx2---|
|-Rx1-|
This is linearizable with order: Wx1, Rx1, Wx2, Rx2. Since all operations overlap, we are free to order them in any fashion in our global total order. You can also mark, for each operation, an instant within the interval where the operation occurred. This also indicates the history is linearizable.
|-Wx1-| |-Wx2-|
|--Rx2--|
|-Rx1-|
This is NOT linearizable. Constraints: Wx1 → Rx1 (we must write before we read a value) Wx2 → Rx2 (again, value) Wx1 → Wx2 (in real time) Rx2 → Rx1 (in real time) There is no global total order that obeys the above constraints.
|--Wx0--| |--Wx1--|
|--Wx2--|
|-Rx2-| |-Rx1-|
This is linearizable with order: Wx0, Wx2, Rx2, Wx1, Rx1. We are free to perform Wx2 before Wx1 because the two operations overlap.
|--Wx0--| |--Wx1--|
|--Wx2--|
C1: |-Rx2-| |-Rx1-|
C2: |-Rx1-| |-Rx2-|
This is not linearizable. Constraints: C1 Rx2 → C1 Rx1 (real time) C2 Rx1 → C2 Rx2 (real time) Wx1 → C1 Rx1, C2 Rx1 Wx2 → C1 Rx2, C2 Rx2 There is no way, in a single global total order, to allow C1 to Rx2 before Rx1 AND have C2 Rx1 before Rx2.
|-Wx1-|
|-Wx2-|
|-Rx1-|
This is not linearizable. Constraints:
Wx1 → Wx2 (real time)Wx2 → Rx1 (real time)Wx1 → Rx1 (value)With real time constraints we must have the order of operations be Wx1, Wx2, Rx1. But the read would be Rx2, due to the most recent write.
C1: |-Wx3-| |-Wx4-|
C2: |-Rx3-------------|
This is linearizable. The order would be Wx3, Rx3, Wx4. We are free to perform the Rx3 and Wx4 in any order since they overlap in real time.
|-Wx1-|
|-Wx2-|
|--Rx2--||-Rx1-|
|--Rx1--||-Rx2-|
This is NOT linearizable or sequentially consistent because there is no single global total order of events that can satisfy the two clients reading x’s update in different orders. The execution is causally consistent. Consider all the causal relationships within the system. Wx1 → C3 Rx1, Wx1 → C4 Rx1 Wx2 → C3 Rx2, Wx2 → C4 Rx2 C3 Rx2 → C3 Rx1 C4 Rx1 → C4 Rx2
No contradiction emerges. There is no causal relationship between the Wx1 and Wx2, we can have client 3 and client 4 observe these writes in different orders.
|--Wx1--|
|-Rx1-||-Wy2-|
|-Ry2-||-Rx0-|
This is NOT causally consistent. C1 Wx1 → C2 Rx1 C2 Rx1 → C2 Wy2 C3 Ry2 → C3 Rx0 C2 Wy2 → C3 Ry2
Not only is there no global total order, but simplifying the above relations shows us there is a contradiction. C1 Wx1 → C2 Wy2 (simplify 1 and 2) C3 Ry2 → C3 Rx0 C2 Wy2→ C3 Ry2 C1 Wx1 → C2 Ry2 (simplify 1 and 3) C3 Ry2 → C3 Rx0
Combining these last two rules leaves us with the impossible: C1 Wx1→ C3 Rx0
Informally, we can note that C2’s read of x =1 may have CAUSED the write of y=2, so we should only see y=2, when we see the effect of Wx=1.