Detailed Lecture Notes
Replication is when we create a copy of an application or dataset on separate physical machines. Having replicas improves fault tolerance because if a single server fails, clients can interact with a different replica server instead, effectively masking failure.
However, by introducing replication into our system we must answer several questions:
Why is replication difficult? In some systems, replication seems trivial to add– consider an application to get_random_cat_fact(), replicating this service may be as simple as copying cat_fact.py onto another running server.
However, difficulties arise when we have system state which we must coordinate across replicas. The property of achieving agreement on replicated state is known as consensus.
Let’s imagine we have a simple key-value store which we want to replicate. We want to be sure that when clients perform put operations, the results of these operations are not lost and are propagated to all replicas to be viewed. However, as we learned in the previous section, at any time, any one, or multiple, of our servers may fail. Additionally, it is possible that messages are lost. How should we design such a system?
There are two main approaches to answer the question of how clients should contact/communicate with a replicated system.
This is the distinction between Active vs Passive Replication.
In Active Replication the client broadcasts its write requests to all servers. As we saw with our example of totally-ordered multicast, we must be concerned with the consistent ordering of incoming requests from clients on all servers.
In Passive Replication the client interacts with one server, who communicates with other replicas to keep them in sync.
For the purposes of this section, we will explore passive replication schemes, where we will designate one server for the client to communicate with, and our system will coordinate any messages that need to be sent to replicas.
What should we communicate in passive replication from the one contacted (primary) server to the replicas in order for the replicas to receive any state change updates?
In a Replicated State Machine (RSM) the server will send over each operation to all replicas. Replicas replay each operation in the same order, and thus have the same state. However, operations must be deterministic in order for replicas to remain identical.
An alternative to an RSM approach is State Transfer. State transfer means instead of sending over each operation, we may have a server send over a copy of its modified, resulting state. We might opt to use a state transfer approach if operations are costly and the state is small. Sending this small state over the network becomes feasible and could save compute resources on the replicas. However, in many systems it is common that the state is large (i.e. a database) and operations are simple (get/put requests). Thus, RSMs are an extremely popular approach.
Now let’s explore– when should we have servers communicate these operations or state transfers? How should we deal with a server failing? We will explore two simple schemes in this chapter: primary/backup replication and chain replication.
In Primary-Backup Replication:
When a client sends an update (a put request) the primary communicates the operation to all backups in parallel. Then, the primary waits and monitors responses from all backups.
Only once each replica has acknowledged the update, will the primary proceed to responding to the client and processing more requests. Once the primary has communicated with all backups and received acknowledgements, it will communicate back to the client that the operation is finished.
This final communication with the client is known as the commit point of that operation.
When can the primary respond to the client? Does it need to wait for all the backups to respond before this commit point? The answer to this question depends on what behavior we view as “correct”. Consider the following: if the primary does not have to wait for all backups, it is possible it will tell a client that an operation is “committed” but then the primary may fail right after, before backups receive the operation!
So what does it mean for our system to be correct?
The rules which formally describe correct behaviour of a system is known as a consistency model. We will discuss consistency in depth in later sections but for now, we will say, we wish to achieve strong consistency with primary/backup replication.
For now, strong consistency means, even with failures, once we pass the commit point of an operation, we do not want to lose the effect of that operation. As an effect of this rule, clients never see stale data. A read operation will always see the effect of the latest committed write operation.
For example, with strong consistency, it would be disastrous to have a client communicate a put(x,10) operation, get OK, then communicate a put(x,12) operation, get OK, then have another client communicate with our system and have get(x) = 10. After the commit point of put(x,12) we want our system to guarantee that all subsequent get(x) requests will see the effect of this operation. No stale reads!
Notice that it is possible that multiple clients can request the primary at once– but there will be one strict order of operations on all replicas defined by the order the primary chooses to serve the requests and forward them to backups.
Aside: The Importance of Order of Operations
If we have two requests arriving at roughly the same time: put(x,10) put(x,20) We want to ensure that on all replicas we agree on an order.
If one replica ran put(x,20), put(x,10) and the other ran put(x,10), put(x,20) they would not agree on the final value of x (one saying it is 10, the other 20).
So we can see it is important to agree on an order of operations.
Luckily, in primary backup, when our primary receives two requests, it decides on an order to process them. After sending out the first request, the primary waits until all replicas have processed that request before propagating the next request to replicas.
Therefore, all replicas will have an identical order of requests.
Now, what happens when the primary fails? Our goal is that if the primary fails, we want to promote a backup server to the primary server. But now the question becomes, how do we detect that a primary has failed?
For the purposes of this exploration, let’s say we have a single primary server and a single backup replica.
Okay, for our first attempt, let’s say, if the backup can’t contact the primary, it will promote itself to primary. However, a nasty problem arises with this strategy. What if the primary is up and running, but it is instead the network that is failed? When the backup can’t contact the primary, we can’t distinguish between network failure compared to a true primary server failure. With our current strategy the backup would upgrade itself to primary, allowing clients to send updates through itself, and the original primary would continue to operate, meaning that our system would diverge through these two primaries.
Our second attempt will then be to choose a third party, a separate server, to be a configuration service. This configuration service will not interact with clients (therefore it should not be a bottleneck), however, it will serve as a central authority on the group membership view.
Group Membership refers to which servers are up or down in our system at any given time. Based on the group membership, we must determine a configuration of servers– who is the primary or backup.
We will say if CFG can contact a server, this means that server is live. If CFG can not– the server is down. When a server is up but can’t communicate with CFG, the system treats that server as failed. This strategy may not maximize availability of our system (especially in the face of omission failures/partitions), but it will guarantee correct behavior (never allow for diverging replicas through multiple primaries).
The downside of this approach is that there is now a single point of failure in our system– if CFG fails, our system fails completely!
There are two main downsides to primary-backup replication which we will discuss in this section.
The first is the question of performance. You may ask yourself, what is the core limiting factor on how many requests our system can process in p/b replication?
The bottleneck (or limiting component) is the primary server. The primary server must respond to all client requests, which introduces a problem of scalability. As more clients interact with our system, the primary may be inundated with requests. Furthermore, as we discussed with our rule for the commit point, the primary must wait for communication with all backups. In fact, in order to get our guarantees on replication, our primary/backup system will be slower than just a single machine serving requests.
The second limitation of primary backups is that the story for how to change configurations when the primary fails in the middle of propagating an update is a bit complicated.
Consider the fact that a primary may have partially propagated an update to ⅔ of the backup servers. Now, the primary crashes. How do we deal with the fact that our backups are now in different states?
There are several reasonable plans to mitigate a partially propagated update, however, they all have some additional overhead to consider.
For example we may:
To address these two limitations, a protocol known as Chain Replication was proposed by Renesse and Schneider (OSDI 2004).
A downside of primary/backup replication is that though we have multiple servers running, we get no parallel request processing in our system. Our performance will be worse than just a single machine serving client requests. Chain replication seeks to be a high-throughput variant of primary/backup replication.
Chain replication is a variation of primary-backup replication, however, backups are organized in a chain topology and updates are communicated down the chain.
The basic flow works for a write operation is as follows:
Client read requests are sent to the tail of the chain and can be responded to directly.
The question that follows is, does chain replication offer strong consistency in the face of server failures? Remember, strong consistency informally means: after our commit point, our operation will never be lost, and all subsequent client requests will reflect that change.
Because results of an operation only become visible when the update reaches the tail server, we can guarantee that every server in the chain indeed has the update by the time the operation is committed (and we respond back to the client). Thus, even if any number of servers fail, all remaining live servers will have all committed operations.
By dividing up responsibility of read/write requests in this fashion chain replication offers better load balancing, using the processing power of the head and tail server, which in turn improves throughput.
Throughput is defined as the rate at which requests are processed in our system.
However, because updates are propagated serially instead of in parallel, chain replication has worse write latency than traditional primary-backup replication. The time it takes for a user to get a request is limited by the time it takes to propagate said request down the chain.
Latency is defined as the time it takes for a client to get a response to their request.
Notice, we only get this benefit in chain replication if our workload is mixed reads/writes. If all of the operations are read, we are 100% bottlenecked by the tail server. If all of the operations are writes, we are 100% bottlenecked by the head server.
Like primary-backup replication, chain replication requires the view to be determined by a separate configuration service/authority.
Note that inopportune failures may require some updates to be resent from the predecessor to a newly upgraded server. This resending needs to occur when a partially-propagated update had been only sent to the failed server, and not to the next in the chain. Chain replication requires some additional bookkeeping in order to ensure that this re-send happens.
Fail-Stop Servers: Most common failure is a complete server failure, easily detectable, without other erroneous behaviours.
Consider what if one server is not failed, but much slower than the others? Our entire system would slow to serve any write requests, but our CFG would not adjust to exclude the slow server from the chain.
Consider if the CFG thinks a particular server in the chain is up and running, but it can’t communicate with its neighbors in its chain (this would be an example of omission failure). Though chain replication would remain correct (it would never have a diverging state), it would not be available during this type of failure to clients.
Single Centralized Configuration Service does not fail. Thus, chain replication suffers from a single point of failure.
Server Failure is Easy to Detect.
Consider, what if it is hard to distinguish between temporary and permanent failures?
The core tradeoff becomes: