Detailed Lecture Notes
Now, we will move on to discuss failure recovery (such as in a crash-recover) model through distributed snapshotting. Again, we wish to capture some moment in our distributed system, saving the state of each component/machine, then when we install the snapshot, each component restores their state to the saved version.
Consider the following example:
Now, consider that all three servers fail and reboot, however, one had saved its state three hours ago, one had saved its state two hours ago, and another had saved its state just five minutes prior. Now, in totality, the map is a mismash of different maps from different times! Our system is inconsistent.
Inconsistent restored system state from our three image servers.
Instead– what we wish we could do is ensure that our three servers save or snapshot one true instance of the system in time. Particularly, you may note if you remember our lecture on time, we do not even want to capture the image servers at one instance in real time but instead in the moment that was the same point in causal time (the result of the same drone update).
Thus, a distributed snapshot may be useful!
Snapshotting is useful in system recovery, as we have some known, saved, working state to rollback to once all servers are online. Other applications of snapshotting exist, however.
So how can we capture a correct system state? One approach could resemble the following:
However, we note that this process requires us to halt our system entirely, harming our availability to clients which is not desirable.
Let’s aim to improve our snapshotting protocol to be able to capture snapshots while the system continues to operate. In order to understand such a procedure, we must deeply understand the goal of a snapshotting protocol.
Distributed snapshotting protocols aim to capture a consistent cut (recovery line) through our systems execution.
A cut is a subset of a system execution’s global history that is a union of prefixes of process histories. Thus, a cut is defined by a series of points, one in each process’s history.
The image above depicts a consistent cut.
We define an event to be pre-snapshot if it occurs before this cut point and an event to be post-snapshot if it occurs after the point.
A consistent cut is some cut where, for any event A, if A happens before B, and B is pre-snapshot, then A must also be pre-snapshot. Thus, a consistent cut preserves causality across the cut boundary.
An inconsistent cut would include an effect without the cause (i.e. if A happens before B, and only B is pre-snapshot, while A is post-snapshot).
An example of an inconsistent cut.
Let’s consider our previous example with a drone and three image servers.
We can visually represent our system events in the fashion below, and the following line defines a consistent cut. Event a represents the drone taking an image. b,d,e represent sending the partial-images to the image server. And c,g,e represent receiving the partial image on each server.
Notice: the below line would represent an inconsistent cut, because we include m without including j. It is not consistent to include a message receive event without including the corresponding message send.
Also note, sometimes a consistent cut will define certain messages to be still “in the air” so to speak. All this means is that our snapshot should also note what messages have been sent but not received. When we recover/restart the system from this snapshot, we will just have to resend those “in-transmission” messages.
Note here, we have defined such a consistent cut, where event b has occurred, the message is being sent to s1, but we snapshot the system before event g, when the message is received. Thus, this message is “in air” and would need to be replayed.
The Chandy-Lamport snapshot protocol is a snapshot algorithm that is used in distributed systems for recording a consistent global state of an asynchronous system.
It was invented by Mani Chandy and Leslie Lamport (of Lamport timestamp fame).
Assumptions:
Goals of Chandy-Lamport Protocol
A process initiating the snapshotting protocol:
A process receiving a marker for the first time:
Otherwise (not first time):
The snapshotting process is complete when all processes:
After this point, we can then collect the locally recorded states and construct a global snapshot.
Some intuition on why this snapshotting protocol works is as follows:
In the above example, we see a resulting snapshot. The pink highlighted area corresponds to all events saved in the snapshot. One message will have to be replayed (H->D), and is saved because p1 had not received a marker message from p2 before D.
Now, what happens when we want to restart our system from a snapshot?
A subtle question arises: when replaying from a snapshot are we guaranteed to get the exact same order of events? No! Restarting from a snapshot is not deterministic.
For example, when replaying the above snapshot, we may get the following:
It is certainly possible that in replaying the snapshot, post-snapshot events are re-ordered, such as D is delivered before event C occurs on P1. This could be an entirely valid, but different, execution which occurs from our snapshotted state.