Detailed Lecture Notes
Now that we understand the basic structure of Raft– a log, a leader, and the AppendEntries RPC, we will endeavor to understand what happens when servers fail. In particular, we will ask, what happens when the leader server fails.
We will explore how we carefully structure our election protocols so that even arbitrary leader failure does not threaten our consistency guarantees with respect to committed entries in the log.
Continually, the leader is sending these AppendEntries requests to all other servers in the system. The leader sends an AppendEntries request when a client performs an operation. The leader also continually sends AppendEntries requests as a heartbeat to tell the other servers they are still connected to the up-and-running leader.
What happens when the leader crashes? Or if the network between the leader and a particular server goes down? At this point, the AppendEntries requests will stop arriving. After a specified election timeout, the server who has not received any requests will timeout and start an election.
Note: This election timeout must be somewhat randomized so that most of the time, exactly one server starts an election at a time (for efficient leader election).
In starting an election the server will:
CANDIDATELEADER. Else, the election fails. Wait and repeat the process later.So how do other followers decide to cast a yes or no vote?
There are two properties we wish to guarantee from our leader elections:
It is the voter’s responsibility to ensure they do not vote twice. A follower maintains a votedFor variable which holds which (if any) candidate we voted for during currentTerm.
As long as no voter votes twice in an election, there is no way for two candidates to both get a majority of the votes. Thus, we will have at most one leader per term.
As a follower, it is our job to ensure that the candidate we vote for has all the log entries we do (or better). If all voters do this, we can ensure that elected leaders have all log entries present on a majority of servers, thus all newly elected leaders have all committed entries!
The first time a server sees a RequestVote message for a new term they will cast their vote if the RequestVote comes from an up-to-date server. We will explain what up to date means in a moment. It is also critical that the server saves that it has cast its vote for that term. A server therefore will not vote for a different candidate in that same term. (This is important in determining that a majority will elect one and only one leader).
An example where a candidate’s RequestVote is rejected because it is not up-to-date.
Now what does up-to-date mean in the context of casting a vote? Well, it is extremely important that we do not lose any committed entries! If a server votes for another server who potentially does not have an entry that was propagated to the majority, we may lose an important entry.
Thus, the rule is, a server will only cast a vote based on the requester’s log length and last term:
Therefore, comparing the last log term and index gives us a strong guarantee that we are voting for a leader with an up-to-date entire log of preceding entries as well.
However, why is it enough to guarantee we will never elect a leader without a committed value? For example, some voters may also be of date, and may not have some entries, and they would grant a vote!
Notice, one important property of two majority groups, is that these two groups always overlap. Furthermore, because log entries are committed by being received by the majority and also, a leader is only elected by a majority.
Thus, it must be that the majority who elects a new leader, contains a member who both 1) saw all committed entries, and 2) believes this new leader is up to date.
This means that it is impossible to elect a leader that is missing a committed entry.
Consider the following scenario, a server goes down, misses several AppendEntry rpcs and then rejoins the raft cluster. At this point, the leader has just received a new operation in it’s log. Is it okay for the recovered server1 to add this new entry right away?
We can see if we imagine which operation each log entry represents, it is important for the correctness of our RSM, that we append the same entries in the same order. So we must have a mechanism wherein server1 will reject this appendEntry RPC from the leader.
Our implementation will be as follows:
prevLogIndex the index of the log entry just before the entries to appendprevLogTerm the term of the log entry just before the entries to appendprevLogIndex, prevLogTerm.Why do we care only about prevLogTerm and prevLogIndex? Why do we not need to compare the entire preceding log?
Raft ensures the log matching property always holds.
Log Matching Property
Why can we be certain of this property? The log matching property is a side effect of how we build up the raft log one entry at a time, in each term, through a single leader. Therefore, if we only agree to add a log entry, it means we agree on the preceding entry and therefore we build up identical logs step by step.
An example of two failed appendEntry RPCs followed by a success:
Finally, server1 is caught up.