CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 11: Distributed Consensus II

    Now that we understand the basic structure of Raft– a log, a leader, and the AppendEntries RPC, we will endeavor to understand what happens when servers fail. In particular, we will ask, what happens when the leader server fails.

    We will explore how we carefully structure our election protocols so that even arbitrary leader failure does not threaten our consistency guarantees with respect to committed entries in the log.

    Leader Election

    Continually, the leader is sending these AppendEntries requests to all other servers in the system. The leader sends an AppendEntries request when a client performs an operation. The leader also continually sends AppendEntries requests as a heartbeat to tell the other servers they are still connected to the up-and-running leader.

    What happens when the leader crashes? Or if the network between the leader and a particular server goes down? At this point, the AppendEntries requests will stop arriving. After a specified election timeout, the server who has not received any requests will timeout and start an election.


    Note: This election timeout must be somewhat randomized so that most of the time, exactly one server starts an election at a time (for efficient leader election).

    In starting an election the server will:

    1. Become a CANDIDATE
    2. Increase the term number.
    3. Solicit votes.
    4. If we receive a majority of votes, become LEADER. Else, the election fails. Wait and repeat the process later.

    So how do other followers decide to cast a yes or no vote?

    There are two properties we wish to guarantee from our leader elections:

    1. There is at most one leader for a given term
    2. An elected leader will have all committed log entries.

    VotedFor

    It is the voter’s responsibility to ensure they do not vote twice. A follower maintains a votedFor variable which holds which (if any) candidate we voted for during currentTerm.

    As long as no voter votes twice in an election, there is no way for two candidates to both get a majority of the votes. Thus, we will have at most one leader per term.

    Up-To-Date Logs

    As a follower, it is our job to ensure that the candidate we vote for has all the log entries we do (or better). If all voters do this, we can ensure that elected leaders have all log entries present on a majority of servers, thus all newly elected leaders have all committed entries!

    The first time a server sees a RequestVote message for a new term they will cast their vote if the RequestVote comes from an up-to-date server. We will explain what up to date means in a moment. It is also critical that the server saves that it has cast its vote for that term. A server therefore will not vote for a different candidate in that same term. (This is important in determining that a majority will elect one and only one leader).


    An example where a candidate’s RequestVote is rejected because it is not up-to-date.

    Now what does up-to-date mean in the context of casting a vote? Well, it is extremely important that we do not lose any committed entries! If a server votes for another server who potentially does not have an entry that was propagated to the majority, we may lose an important entry.

    Thus, the rule is, a server will only cast a vote based on the requester’s log length and last term:

    1. If the logs have last entries with different terms, then the requester’s log has the later term.
    2. If the logs end with the same term, then the requester’s log is longer or equal in length.

    Therefore, comparing the last log term and index gives us a strong guarantee that we are voting for a leader with an up-to-date entire log of preceding entries as well.

    However, why is it enough to guarantee we will never elect a leader without a committed value? For example, some voters may also be of date, and may not have some entries, and they would grant a vote!

    Overlapping Majorities


    Notice, one important property of two majority groups, is that these two groups always overlap. Furthermore, because log entries are committed by being received by the majority and also, a leader is only elected by a majority.

    Thus, it must be that the majority who elects a new leader, contains a member who both 1) saw all committed entries, and 2) believes this new leader is up to date.

    This means that it is impossible to elect a leader that is missing a committed entry.

    Catching-Up Followers

    Consider the following scenario, a server goes down, misses several AppendEntry rpcs and then rejoins the raft cluster. At this point, the leader has just received a new operation in it’s log. Is it okay for the recovered server1 to add this new entry right away?

    We can see if we imagine which operation each log entry represents, it is important for the correctness of our RSM, that we append the same entries in the same order. So we must have a mechanism wherein server1 will reject this appendEntry RPC from the leader.

    Our implementation will be as follows:

    Why do we care only about prevLogTerm and prevLogIndex? Why do we not need to compare the entire preceding log?

    Raft ensures the log matching property always holds.

    Log Matching Property

    Why can we be certain of this property? The log matching property is a side effect of how we build up the raft log one entry at a time, in each term, through a single leader. Therefore, if we only agree to add a log entry, it means we agree on the preceding entry and therefore we build up identical logs step by step.

    An example of two failed appendEntry RPCs followed by a success:





    Finally, server1 is caught up.