CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 15: Transactions and Serializability

    Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau

    Why Transactions

    Consider the following example:

    A need in certain applications arises: we want to execute a group of operations as one logical unit, so that either all of them succeed or all of them fail.

    A transaction is a group of several reads and writes that will appear to run as one unit– either all succeeding or all failing.

    Distributed systems that support transactions make building applications easier. Clients of a system with distributed transactions can then be assured that the transactions they define will appear as logical units, uninterruptible and unbreakable.

    A client can use an API to mark the start and end of their transaction with TXN_START and TXN_END operations.

    Now let’s consider how our ticket service may want to utilize transactions:

    Alice submits a transaction which corresponds to reserving a ticket, consisting of a get tickets operations, a matching update to a specific seat to mark it as not available, and a final decrement of get tickets.

    Bob submits a get tickets transaction.

    Consider– how do we want these transactions to interact with each other?

    Ideally, it should be impossible for Bob’s transaction to see the status that 1 ticket is left, but B2 is not available. We also may not want it to be possible for Alice’s transaction to fail part way through– reserving B2 but not updating the ticket value.

    Both of the above behaviors are formally defined within a transaction’s ACID properties.

    Transaction Properties

    A transaction is specifically defined as a group of operations that have the following properties:

    These properties are often referred to by the acronym ACID.

    Isolation and Serializability

    Today we will focus on how a system could support isolated distributed transactions. We define isolated to mean that results of incomplete transactions will not be visible to other transactions.

    Consider that we have two concurrent transactions.
    Our starting conditions: x=100,y=100

    TXNs  
    T1:
    BEGIN TXN
    add(x,10\)
    add(y,\-10)
    print(y)
    END TXN
    T2:
    BEGIN TXN
    x1 = get(x)
    y1 = get(y)
    print(x1, y1)
    END TXN

    Consider that we have a T2 print that x=110 and y=100. How could this occur? It seems that what happened was between line 2 and 3 of T1, T2 ran on our distributed store and observed a partial result of T1. This would mean that our transactions are not isolated.

    Thus, in implementing distributed transactions we must build a system where such behavior is impossible to observe.

    Isolation is synonymous with a property known as serializability. Serializability is a consistency model that pertains to transactions. Linearizability and all other consistency models we discussed in the previous section only pertained to single operations and made no guarantees about transactions (groups of operations).

    Serializability guarantees that even if transactions execute concurrently, the results are the same as if each transaction had executed in some serial order. Strict Serializability is a stronger model, which also guarantees that the serial order for the transactions respects the real time the transactions were committed by the client.


    Implementing Serializability

    How do transactions enforce isolation and implement serializability? We will see two categories of methodologies.

    Pessimistic concurrency control (PCC) assumes most concurrent transactions acting on the same data objects will conflict (have unserializable results). Thus, PCC uses locks, or a similar mechanism, to enforce that only one transaction at a time (serially) can access some particular data object.

    Optimistic concurrency control (OCC) assumes most concurrent transactions will not conflict (have unserializable results). Thus, OCC lets two transactions run optimistically, verifies the results are serializable, then, if they are, finalizes/commits those transaction results.

    Pessimistic Concurrency Control and 2PL

    One strategy to ensure transactions are isolated is via pessimistic concurrency control (PCC).
    This approach assumes that transactions will conflict (they read/write the same data in a way that would not be isolated), and thus enforces a serial order in which the transactions must run.

    Locking is PCC as it enforces that only one thread of execution in our system can access some specific data.

    Two Phase Locking is one technique to achieve isolation via PCC.

    In 2PL:

    1. Each object has a lock that transactions need to acquire before they can execute their operations
    2. A lock can either be in shared mode (read lock) or in exclusive mode (write lock)
    3. To read an object x, a Txn must acquire a read lock for x (shared)
      1. Multiple Txns are allowed to share a read lock on x simultaneously
    4. To write an object x, a Txn must acquire a write lock for x (exclusive lock)
      1. No other txn can have the lock at the same time
      2. If another Txn has a write lock on x, all other transactions must wait

    The two phases in 2PL refer to the expanding phase (when locks are acquired) and the shrinking phase (when locks are released).

    After a txn has acquired the lock, it must continue to hold the lock until the end of the transaction (commit or abort)

    Consider the following two transactions:

    In 2PL several things could occur:

    1. T1 acquires get(y) read lock, T2 acquires get(x), get(y), get(z) readlock, T1 blocks waiting for exclusive write z lock until T2 finishes.
      1. Results the same as serial order: T2, T1
    2. T1 acquires get(y) read lock, T2 acquires get(x), get(y) readlock. T1 acquires a write lock for z. T2 blocks for read lock on z until T1 finishes.
      1. Results the same as serial order: T1, T2

    Consider that 2PL is vulnerable to deadlock.


    How could deadlock occur in the above example? What happens if T1 gets the exclusive lock for x and T2 gets the exclusive lock for y?

    Optimistic Concurrency Control

    https://15445.courses.cs.cmu.edu/fall2025/notes/19-timestampordering.pdf

    The main downside to PCC is performance. We have to spend time waiting for, acquiring and releasing locks. OCC aims to reduce this overhead by allowing transactions proceed concurrently and check for conflicts in the end

    The steps of OCC are as follows:

    1. Begin: Record the start time of the Txn.
    2. Modify: Perform operations on temporary state (tentative write).
    3. Validate: Check if any other Txns interfered, i.e. modified data that this transaction had read or written (may include those still active).
    4. Commit/Rollback: If conflict detected, abort the transaction, undo all changes, and retry. If there is no conflict, apply the temporary changes and commit.

    How do we approach validation in OCC?

    Approach 1: Centralized Validation Server

    In centralized validation, all clients send the results of their read/write operations (that comprise the transaction) to a centralized validation server.

    The centralized validation server then must assess the results of these received transactions, determine if they are serializable or not, and then respond back to the clients if they should abort or commit.

    Consider:

    TXNs  
    x=0 y=0 z=0
    T1: Rx0 Wx1
    T2: Rz0 Wz9
    T3: Ry1 Rx1
    T4: Rx0 Wy1
    Our centralized validation server can find the following serializable order: T4;T1;T3;T2 (note that T2 could go anywhere) Thus, our centralized validation server can respond to the txn coordinator(s) that T1,T2,T3,T4 can commit.
    x=0 y=0
    T1: Rx0 Wx1
    T2: Rx0 Wy1
    T3: Ry0 Rx1
    Our centralized validation server can find no serializable order: T1->T3: required (via x) T3->T2: required (via y) BUT T2,T1 required (via x) Thus, our centralized validation server must issue an “abort” response to at least one txn in order to remedy the situation.

    Approach 2: Distributed Validation?

    Ideally, we would not need a centralized authority in order to perform this validation step in OCC. Let’s instead propose that each individual server (that comprises some distributed data store) validates the transaction based only on receiving operations that pertain to data in their local store.

    Let’s consider what occurs in the following scenario:

    TXNs  
    x=0 y=0
    T1: Rx0 Wx1
    T2: Rx0 Wy1
    T3: Ry0 Rx1
    S1 stores x. S2 stores y. S1 finds serializable order T2, T1, T3 based on the operations pertaining only to x [Rx0], [Rx0 Wx1], [Rx1]. S2 finds serializable order T3, T2, T1 based on the operations pertaining to x. Notice, that the results of these transactions are not serializable, and thus should be aborted. Clients/txn coordinator will receive ok response from all servers and will commit the transaction incorrectly!

    We see that separating the information onto two servers means that no server has enough information to correctly abort the transactions. Can we include some additional information such that servers will always abort unserializable results?

    Approach 3: Distributed Validation with Timestamps

    Our final plan will be as follows:

    1. Coordinator/Client marks a Txn with a timestamp
    2. Validation now checks to see if reads and writes are in timestamp order
      1. Solves distributed problem: validating the same order — “Yes” will now refer to same order being valid

    The key observation with the above approach is that the metadata of a timestamp with each txn defines a particular serializable order, and every server in our system will check if the results of the operations are consistent given this particular order.

    If our timestamps correspond to real time– we get strictly serializable results. But notice for serializability alone we do not need this guarantee and using any unique, comparable, value as a timestamp will work.

    TXNs  
    x=0 y=0
    T1@100: Rx0 Wx1
    T2@110: Rx0 Wy1
    T3@105: Ry0 Rx1
    S1 stores x. S2 stores y. S1 enforces timestamp order T1, T3, T2 and notes that T2’s Rx0 is impossible with this order. S2 enforces timestamp order T1, T3, T2 and sees no issue. However, clients/txn coordinator receiving one S1 “abort” and S2 “ok” will abort the txn.
    x=0
    T1@100: Rx0 Wx1
    T2@50: Rx1 Wx2
    S1 stores x. S1 enforces timestamp order T2, T1 and notes that T1’s Rx0 is impossible with this order. The client/txn coordinator receiving S1 “abort” will abort the txn. However, note that the txns are serializable with the order T1, T2 and our approach will abort anyway.

    Notice, the above approach will always abort unserializable results. However, it may also abort serializable results at times as well. If the latter situation is rare enough, however, the benefit of distributed transaction validation is worth the few unnecessary “abort” results.