Detailed Lecture Notes
In previous courses you have taken, it may have been sufficient to write programs that did not consider failure. If you are writing a simple C program perhaps you did not consider what will happen to your program if your computer loses power.
Perhaps you reasoned that this possibility, of your machine seeing catastrophic failure, is rare enough not to consider. Or perhaps, you reasoned, there is no better plan for failure than simply having your user re-run your program once the machine recovers. These are fair plans for straightforward programs on single machines.
However, in a distributed system, both of these lines of reasoning are flawed. In distributed systems:
Partial Failures, where part of a system is failing (one machine or component) but the rest is operational, are common in distributed systems.
In some ways, partial failure is more painful to accommodate than total failure: we must account for the fact that other machines are up and trying to make progress while critical components may be in a failed state. In total failure, we can know that no progress is being made at all, which may be better than progress resulting in strange or inconsistent results.
Fault Tolerance is the property of a distributed system to remain operational in an “acceptable” way, while partial failures occur. Fault tolerance can be achieved via automatic recovery from these failures.
To discuss fault tolerance, we define two desirable properties:
Availability – is the property that a system is ready to be used immediately. Availability captures the fraction of time it is functioning compared to the time that it is failed. A highly available system, for example, could be said to be ready to be used 99% or 99.99% of the time a client makes a request.
Reliability – is the property that a system can run continuously without failure. A highly reliable system, for example, may fail for some period of time only once a year or once every 10 years.
Notice we can have a highly available system that is not reliable (it fails every minute, but recovers almost instantaneously). We can also have a highly reliable system that is not highly available (it doesn’t fail for years, but when it does, it takes a month to recover).
To measure failure we also define several metrics:
Mean time to failure (MTTF) is the average time until a component fails
Mean time to repair (MTTR) is the average time needed to repair a component.
Mean time between failures (MTBF) is MTTF + MTTR.
However, computing these metrics is non-trivial. To do so, we must be able to reliably detect when components are failing!
Failure detection is a difficult task. However, it is crucial to be able to detect failures in order to repair, or operate while they occur.
For example, consider we have three servers, P1, P2, P3, connected via a network. How, when P3 fails, can we be sure that P1 and P2 will discover this?
This is the problem of consensus on group membership.
The first mechanism we will discuss is heartbeats. Heartbeats are simple “are you alive” messages sent out from all machines to all other machines, at a regular interval. When a process does not receive a response, it suspects that the machine has failed.
Then, for example if P1 does not receive a response from P3 to its heartbeat, P1 knows that P3 has failed. Furthermore, if P2 does not receive a response from P3, P2 also knows that P3 has failed. Thus, in this case, we have achieved consensus on group membership: both P1 and P2 know that P3 has failed.
In a synchronous system, where message delivery times are bounded, we can be sure that a suspected failure is a real failure.
However, in the real world systems are partially-synchronous– it may be that a message fails when the machine itself is still up and running. Even worse, network partitions may arrive, where subsets of machines can communicate with each other, but not others.
These behaviors mean that a naive heartbeat protocol may result in false positives, where a machine believes another to be failed, when really it is operational. It would be disastrous for the correctness of our system, if nodes in our system have a different view on group membership. For example, in the above image, P1 may say the membership is [P1,P2] and P2 says it is [P1,P2,P3].
One remedy for this failure is: when a node notices a timeout, that node requests neighbors if others believe that server has failed.
This leads to our second mechanism: a gossip protocol. Gossip protocols involve communicating with other nodes in the system to disseminate information. In this case, we will use a gossip protocol to communicate what nodes are known to be up/down. To do this, in our heartbeats to other servers, we can also include information about what nodes we have communicated with recently.
If we see that a node does not respond to our heartbeats, but other nodes are communicating with that node, we can say that it has not failed. In the above example, P2 would communicate with P1 that P3 is up.
So now that we can detect failure, what is our plan to remain available for the duration of the failure?
The primary mechanism for masking component failure is redundancy. We can achieve redundancy in two ways. The first, is to repeat an action, which is known as time redundancy. When we see that an operation has failed, retransmit that operation to the server. The second, is physical redundancy, which is to set up extra equipment or hardware as back-ups for primary hardware.
For example, the internet is ripe with physical redundancy. For any one request from endpoint to endpoint, there are many, many redundant paths through the internet’s network of routers and switches to serve that request.
We will see how to achieve physical redundancy of a dataset or server in our discussion on replication.
Another aspect of fault tolerance is recovery. Not only do we want to mask failures and remain available when components fail, but we also may want to have a plan for returning to a consistent state once all components are functioning again.
One plan to achieve recovery is to “checkpoint” a state in our system and then, in the face of failure, install that checkpointed state onto every machine in our system.
The main problem then becomes: how do we establish a “safe” point on every machine, without interrupting the operation of our system?
We will explore one protocol to capture distributed snapshots in the next section.
For the past few lectures we’ve been discussing failure in distributed systems. Today, we will define several failure models which codify what types of failures a system is capable of handling.
We also will endeavor to define a protocol which could be used to recover from failures in distributed snapshotting. Distributed snapshots involve capturing and saving the state of a distributed system at some point in time when the system is operating correctly.
Crash-Stop Failure
Omission Failure
Crash-Recover Failure
Persistent state refers to stable storage, like a hard disk or SSD which can hold data when the machine is powered off. Volatile state refers to state that is not maintained without power, like RAM.
Byzantine Failure (out of scope)
Failure models can be understood hierarchically. For example, a system that can handle crash-recovery failures, is also tolerant to omission failures. A system that can handle omission failures, is also tolerant to crash-stop failures.
Crash recovery encapsulates omission failure, as it may omit any arbitrary messages while recovering/crashed while adding that system state (such as in volatile storage, RAM) may be lost.
Omission failure encapsulates all crash-stop, as crash-stop is equivalent to a simple case of omission failure: omitting all messages after the crash.