CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 10: Distributed Consensus I

    https://raft.github.io/
    https://thesecretlivesofdata.com/raft/

    Distributed consensus is the process of achieving agreement on a single value between multiple servers. Up until this point, our replication schemes lacked a true solution to distributed consensus, because, at their core, they all revolved around a single point of failure.

    All these aforementioned replication protocols required a separate centralized authority to determine which servers were live in order to upgrade primaries. What if we wanted to remove this centralized nature of the protocols and instead achieve distributed consensus? The core issue is it is impossible to distinguish between two servers inability to communicate over a broken network link or a server failing!

    A true real-world solution to distributed consensus should tolerate crash-recover failures, message omission, network partitions, and should remain available as long as any majority of servers is live and connected.

    Instead of a single, outside, authority to determine view, what if we rely on a majority opinion instead? If a majority of servers can communicate with another server, we can say that server is up and running.

    “Historical” Context: Paxos

    The first paper to propose a real-world solution to distributed consensus protocol was Paxos, published by Leslie Lamport.

    However, Paxos was originally written in a somewhat whimsical fashion, framing the protocol within the context of a Greek island (Paxos) and legislators voting on parliamentary decrees. Unfortunately, many readers found this presentation of Paxos to be troublesome to follow, sparking revision in Paxos Made Simple.

    Paxos, however, was still a general protocol for distributed consensus. Paxos in its simplest form only describes distributed consensus on a single value. RAFT, the following work, sought to make concrete what Paxos had not fully specified. Raft built the concept of a log and leader election into the protocol and fleshed out the complete design to achieve distributed consensus over a RSM.

    RAFT

    We will examine the techniques used by RAFT, a distributed consensus protocol. RAFT is a quorum protocol, meaning it is based around an election process to determine an up-and-running leader. Unlike our solution to primary-backup replication, there is no need for a configuration service or any other single point of failure.

    Client Interaction

    Clients submit all operations through the leader of the raft cluster. A client can arbitrarily contact any server in the raft cluster, and this server will point the client to who is the leader at this time.

    Clients then submit operations to the leader and await a response. Once the client gets a response, it can be sure that its operation has been communicated to replicas in such a way that this operation will not be lost. As before with primary-backup, this guarantee to the client is known as strong consistency.

    Logs

    Raft itself uses a replicated state machine (RSM) approach. The Raft protocol coordinates sending application operations to all replicas so that the application state is identical on all replicas.

    The basis of the structure of Raft is built upon the Raft log. The log contains every operation to replicate in the system. The log also contains information about the term in which that operation was added, as well as the index in the log.

    | Aside: As we move forward to discuss how leaders are elected, we will see why it is important to save the term. It is important to mark a term because for any term there can only be one leader. Thus we know all log entries with the same term passed through a single leader.

    Notice that if we replicate this log of operations, as with our discussion on replicated state machines, each server will perform the exact same operations in the exact same order. This guarantees that the application state matches on each replica.


    An example of Raft Logs and corresponding application state on a single variable, x.

    The application state refers to whatever the state of application running on raft is– a key value store, a flight booking database, a mapreduce master, etc.

    By performing these committed operations in the log, the result is that all replicas have the exact same state.

    When a client submits an operation, the current Raft leader will save whatever operation it its log. Then, the leader attempts to propagate the new log entries to every other server in the system with an AppendEntries request to all other servers.

    Committing Entries

    How do we know when we can respond back to the client that their operation was successful? Remember, the moment we communicate back with our client is known as the commit point of that operation.

    When the Raft leader gets enough successful responses to it’s AppendEntry requests, it knows a particular log entry exists on the majority of servers. At this point, this log entry is committed. Then, the leader can respond back to the client with their operation result or an “ok” message.

    The commitIndex state variable in each raft server is the highest known committed index in the log. CommitIndex is updated via the leader (who does bookkeeping to track if the majority of servers have a particular entry). The known commitIndex is propagated from leader to client in AppendEntry at a regular interval.

    Because this log entry exists on a majority of servers, after this point, it is impossible to lose this operation even if servers fail. No future client requests will be able to see stale information (from before the operation occured).

    If one follower, out of three raft servers, fails during an append entry, as long as a majority remains up and running, Raft continues to function as normal. (If a majority of servers are failed however, raft stalls and can not make progress).

    But what happens when the leader is the single failed server? This question leads into the discussion of Raft’s leader elections.

    ##