CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 13: Consistency Models

    Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau

    Up until this point, we have informally discussed one form of consistency– strong consistency.

    We described strong consistency informally, and as several ideas:

    1. we can not “lose committed values”
    2. our system should appear to be not-replicated
    3. we can not get stale reads.

    Weak Consistency Example

    Now, let’s expand our understanding of consistency and consider the following model.

    Consider the distributed data store with two replicas. This datastore is responsible for selling Taylor Swift tickets. Consider that Alice queries the datastore for the remaining number of tickets. She receives a response of 1. She then reserves the ticket. She messages Bob, her friend, that she just reserved the last ticket.

    A little while later, Bob comes along, and then requests the datastore himself for the number of tickets left. He, however, talks to a different underlying replica, gets a stale result, and sees that the number of tickets remaining is still one, despite Alice’s reservation.

    This example illustrates a possible execution of a system with a weak consistency model. On a single machine, we could not have the operations read tickets_left = 1, write tickets_left = 0, read tickets_left = 1 occur!

    Linearizability

    The strongest consistency model is linearizability.

    Informally, we can consider linearizability to mean:

    1. The execution (result of all operations) could have been produced on a single machine
    2. Every client has the same view of the data
    3. It is as if all operations are atomic and propagated to every replica instantly

    | An execution is linearizable if there exists a global total order of operations in the system and that order obeys real time. Global Total Order: An order of all the operations/results within an execution which one could obtain from running these operations on a single machine in that order. Obeys real time: for non-overlapping operations, if operation A commits before operation B begins, in our global total order A must come before B. | | :—- |

    To inspect if a system’s consistency model, we visualize a system execution in a plot where:

    1. We represent each client on the y-axis of the plot.
    2. We represent real-time as the x-axis.
      1. In these plots we do not consider issues in clock synchronization. This is a “birds-eye” view of the system.
    3. All data values are assumed to be initialized to zero.
    4. Operations are written in the notation Rx1 (meaning read x=1) or Wy2 (meaning write y = 2).
      1. Each operation occurs over some duration of time represented by a horizontal bar or box.

    A single Rx1 operation.

    Let us consider the following execution. Is it linearizable?

    To understand the execution, we can ask ourselves, is there an order of all the operations that could be replayed on a single machine and get the same results (global total order). Furthermore, does that order obey real time?

    Consider the following global total order of operations:

    1. Wx1
    2. Rx2
    3. Rx2
    4. Wx3
    5. Rx3

    We can also see this obeys real time. Even though the second Rx2 starts after the beginning of Wx3, the two operations overlap, thus it is completely possible that Rx2 observes the system state before Wx3 was finally committed.

    Another approach for visualizing if an execution is linearizable is to attempt to draw an atomic moment in time, where the operation occurs (within the operation’s duration). If you can– the execution is linearizable.

    Consider the below diagram showing a single atomic moment for each operation to occur. Notice, these lines (read from left-to-right) correspond to the above global total order.

    Let’s consider another example, is the following linearizable?

    Let’s attempt to create a global total order that obeys real time.

    Wx1 must precede Rx1 in the order (for our value to be read correctly). However, Rx1 must precede Rx0 in real time.

    Thus, we are stuck! We can not have Rx1 Wx1 Rx0 Rx0 Rx1. Because the reads would not result in a 0, after Wx1. We will find no such global total order that satisfies our constraints, thus the execution is not linearizable.

    Now consider the following execution, is this linearizable?

    Yes! We can find a global total order of operations that satisfies real time constraints.

    Wx1, Rx1, Wx2, Rx1.

    Why are we allowed to order Rx1 before Wx2 in the order? Again, because they overlap we have no time constraints. We can imagine the instant in time each operation occurs as follows:

    Linearizability in Raft

    Let us consider if the consistency model offered by raft is linearizable.

    Linearizability for committed log operations is guaranteed via the protocol’s use of a single leader to append and commit entries into the log.

    1. Only once an operation is sure to be spread to a majority of replicas in the current term do we consider it committed.
    2. Once a log entry is committed, it is never lost, nor are any preceding operations reordered or lost.

    Thus, we can see from the Raft protocol we get a single global total order of operations which is exactly the order in this log.

    Furthermore, let’s consider Raft’s real time guarantees.

    1. When is an operation considered to be started?
      1. When the client submits it to the leader.
    2. When is an operation completed?
      1. When it is committed and the client receives a response.

    Thus, we can see that for two non-overlapping operations, we obey real time guarantees. An earlier committed entry will come first in the log.

    Can we achieve linearizability for reads without committing them in the log?

    Linearizability in Chain Replication

    Does Chain Replication guarantee linearizability?

    Linearizability for reads is enforced by the fact that all reads go through a single tail server.
    Linearizability for writes is enforced by the fact that all commits, and the eventual response to the client, go through the single tail server.

    By the time the tail serves a read, it has already applied every acknowledged write: the tail’s state reflects the complete prefix of the global write sequence.

    Furthermore, let’s consider CR’s real time guarantees.

    1. When is an operation considered to be started?
      1. When the client submits it to the head (for write) or tail (for read)
    2. When is an operation completed?
      1. When the client receives a response from the tail.

    Thus, for non-overlapping operations, we obey real-time guarantees, as they are all processed by this single tail server.

    Trade-offs & CAP Theorem

    We may notice that there is an inherent trade-off:

    1. If an application requires linearizability, it will become unavailable under network partitions.
    2. If an application does not require linearizability, it can remain available even in the face of network issues.


    The CAP Theorem recognizes this tradeoff:
    CAP Theorem (2000): Consistency, Availability, Partition tolerance: pick 2 out of 3

    However, network partitions are inevitable, so a more useful way to put it is “A distributed system can be either Consistent or Available when Partitioned”.

    Thus, to increase a system’s availability, we may consider weaker consistency models than linearizability.

    Sequential consistency

    What if our clients don’t need the strength of the real-time guarantees provided by linearizability, and would prefer our system to be more available? We can weaken the consistency model used by our system, instead offering sequential consistency.

    Sequential consistency has the same guarantee of appearing to have a single, agreed-upon, order of operations that could have been produced on a single machine. However, sequential consistency abandons the real-time guarantees that linearizability requires.

    | An execution is sequentially consistent if there exists a global total order of operations in the system and that order obeys client ordering. Global Total Order: An order of all the operations/results within an execution which one could obtain from running these operations on a single machine in that order. Obeys client ordering: for two operations submitted by a single client, A and B. If operation A happens-before operation B, in our global total order A must come before B. | | :—- |

    Consider the following execution, is it linearizable? Is it sequentially consistent?

    The above execution is not linearizable as we must have Wxa precede Rxb in our global order due to real time constraints (Wxa completes before Rxb).

    However, the execution is sequentially consistent with the following global total order:

    1. B:Wxb
    2. C:Rxb
    3. D:Rxb
    4. A:Wxa
    5. C:Rxa
    6. D:Rxa

    This order of events obeys each individual client’s orderings, and could occur on a single machine. Note, we are free to have A:Wxa placed fourth in this ordering, as there are no real-time constraints.

    Aside: When might sequentially consistency occur in practice? If we have a system wherein clients read from geographically local replicas, which are periodically synchronized across regions, this could result in sequentially consistent executions. These executions emerge when we respond immediately to clients in our region, and reconcile requests at a slower rate between these regions.

    Now consider the following execution, is it sequentially consistent?

    As we try to create a global total order of operations, we find the following issue. Wxa must precede Wxb in the order (as we must respect the order of client operations). However, Rxb must precede Rxa in the order (to respect client C and D’s order of operations). Finally, remember it is impossible to Rxb before Wxb. Thus, it is impossible to create a total order which obeys all of the above constraints. The execution is not sequentially consistent.

    In summary, with sequential consistency:

    ###

    Causal Consistency

    Some systems do not need a global total order of all operations across all servers. A reasonable consistency model could instead be to provide ordering guarantees only on operations with causal relationships. A system could then be causally consistent to meet this requirement.

    An execution is causally consistent if operations with causal relationships are executed in a global order that obeys those causal orderings. Obeys happens-before relationships: Causal relationships include any operations that occur in a single client (we may not re-order a client’s submitted operations). Furthermore, writing a variable to a specified value has a happened-before relationship with reading that value. (i.e. Wx1 → Rx1)

    Consider the following execution. Is it sequentially consistent? Is it causally consistent?

    The above execution is not sequentially consistent. Consider that there is no single global total order that can have both: client C’s Rxc occurs before Rxb and client D’s Rxb before Rxc.

    However, let’s consider the causal relationships shown in the execution:

    A:Wxa → A:Wxc (client ordering)
    C:Rxa → C:Rxc→C:Rxb(client ordering)
    D:Rxa → D:Rxb→D:Rxc(client ordering)
    A:Wxc → Rxc(value)
    A:Wxa → Rxa(value)
    B:Wxb → Rxb(value)

    None of the above orderings contradict, thus we can say the execution is causally consistent.

    We can see that because Wxc and Wxb are causally independent, we do not need to enforce a consistent global order on these two operations. Thus, no contradiction arises from having C:Rxc→C:Rxb and D:Rxb→D:Rxc.

    Now, let’s consider the following execution. Is it causally consistent?


    Like before, let us enumerate the causal relationships:

    1. B:Rxa →B:Wxb(client ordering)
    2. A:Wxa → Rxa(value)
    3. B:Wxb → Rxb(value)
    4. C:Rxb → C:Rxa(client ordering)
    5. D:Rxa → D:Rxb(client ordering)

    By transitivity and (1) and (2), we can see that A:Wxa → B:Wxb.
    However, a contradiction then arises with (4). There is no way for A:Wxa → B:Wxb, and C:Rxb → C:Rxa to be satisfied simultaneously.

    In other words, there is a causal relationship between the Wxa and Wxb. Thus, we can not observe these two events in a different ordering (like on client C).

    Relaxing Consistency for Performance

    One key observation is that consistency problems occur because of conflicting concurrent writes.

    However, in many real-world systems, reads are far more common than writes, and writes do not need to be instantly propagated.

    Thus, a need arises for consistency models which encapsulate this real world need.

    Eventual Consistency

    Eventual consistency can lend itself to highly available systems by compromising with comparatively quite weak consistency guarantee. Modern systems, with a focus on performance and availability, may use eventual consistency.

    Eventual consistency says, in the presence of no future writes, eventually, all reads in the system will converge to the same value. The time to converge is undefined, and system specific.

    In practice this can mean, with no updates, all replicas converge to the same internal state, eventually.

    Consider the below execution.

    The execution is not causally consistent (Client C observes Ryb, then Rxa, even though Wxb → Wyb). But, such temporary inconsistencies are allowed: eventually we see the system converge such that all reads of x, return b. Thus, the execution is eventually consistent.

    Appendix: Additional Linearizability Examples

    Examples from http://nil.csail.mit.edu/6.824/2022/notes/l-raft2.txt

    Example History 1

      |-Wx1-| |-Wx2-|
        |---Rx2---|
          |-Rx1-| 
    

    This is linearizable with order: Wx1, Rx1, Wx2, Rx2. Since all operations overlap, we are free to order them in any fashion in our global total order. You can also mark, for each operation, an instant within the interval where the operation occurred. This also indicates the history is linearizable.

    Example history 2:

      |-Wx1-| |-Wx2-|
        |--Rx2--|
                  |-Rx1-|
    

    This is NOT linearizable. Constraints: Wx1 → Rx1 (we must write before we read a value) Wx2 → Rx2 (again, value) Wx1 → Wx2 (in real time) Rx2 → Rx1 (in real time) There is no global total order that obeys the above constraints.

    Example history 3:

    |--Wx0--|  |--Wx1--|
                |--Wx2--|
            |-Rx2-| |-Rx1-|
    

    This is linearizable with order: Wx0, Wx2, Rx2, Wx1, Rx1. We are free to perform Wx2 before Wx1 because the two operations overlap.

    example history 4:

    |--Wx0--|  |--Wx1--|
                |--Wx2--|
    C1:     |-Rx2-| |-Rx1-|
    C2:     |-Rx1-| |-Rx2-|
    

    This is not linearizable. Constraints: C1 Rx2 → C1 Rx1 (real time) C2 Rx1 → C2 Rx2 (real time) Wx1 → C1 Rx1, C2 Rx1 Wx2 → C1 Rx2, C2 Rx2 There is no way, in a single global total order, to allow C1 to Rx2 before Rx1 AND have C2 Rx1 before Rx2.

    Example History 5:

    |-Wx1-|
            |-Wx2-|
                    |-Rx1-|
    

    This is not linearizable. Constraints:

    With real time constraints we must have the order of operations be Wx1, Wx2, Rx1. But the read would be Rx2, due to the most recent write.

    Example history 6:

    C1: |-Wx3-|          |-Wx4-|
    C2:          |-Rx3-------------|
    

    This is linearizable. The order would be Wx3, Rx3, Wx4. We are free to perform the Rx3 and Wx4 in any order since they overlap in real time.

    Appendix: Additional Causal Consistency Examples

    Example history 1:

      |-Wx1-| 
             |-Wx2-|
                     |--Rx2--||-Rx1-|
                     |--Rx1--||-Rx2-|
    

    This is NOT linearizable or sequentially consistent because there is no single global total order of events that can satisfy the two clients reading x’s update in different orders. The execution is causally consistent. Consider all the causal relationships within the system. Wx1 → C3 Rx1, Wx1 → C4 Rx1 Wx2 → C3 Rx2, Wx2 → C4 Rx2 C3 Rx2 → C3 Rx1 C4 Rx1 → C4 Rx2

    No contradiction emerges. There is no causal relationship between the Wx1 and Wx2, we can have client 3 and client 4 observe these writes in different orders.

    Example history 2:

    |--Wx1--|
           |-Rx1-||-Wy2-|
                         |-Ry2-||-Rx0-|
    

    This is NOT causally consistent. C1 Wx1 → C2 Rx1 C2 Rx1 → C2 Wy2 C3 Ry2 → C3 Rx0 C2 Wy2 → C3 Ry2

    Not only is there no global total order, but simplifying the above relations shows us there is a contradiction. C1 Wx1 → C2 Wy2 (simplify 1 and 2) C3 Ry2 → C3 Rx0 C2 Wy2→ C3 Ry2 C1 Wx1 → C2 Ry2 (simplify 1 and 3) C3 Ry2 → C3 Rx0

    Combining these last two rules leaves us with the impossible: C1 Wx1→ C3 Rx0

    Informally, we can note that C2’s read of x =1 may have CAUSED the write of y=2, so we should only see y=2, when we see the effect of Wx=1.