CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 8: Failure Recovery and Snapshotting

    Now, we will move on to discuss failure recovery (such as in a crash-recover) model through distributed snapshotting. Again, we wish to capture some moment in our distributed system, saving the state of each component/machine, then when we install the snapshot, each component restores their state to the saved version.

    Consider the following example:

    Now, consider that all three servers fail and reboot, however, one had saved its state three hours ago, one had saved its state two hours ago, and another had saved its state just five minutes prior. Now, in totality, the map is a mismash of different maps from different times! Our system is inconsistent.


    Inconsistent restored system state from our three image servers.

    Instead– what we wish we could do is ensure that our three servers save or snapshot one true instance of the system in time. Particularly, you may note if you remember our lecture on time, we do not even want to capture the image servers at one instance in real time but instead in the moment that was the same point in causal time (the result of the same drone update).

    Thus, a distributed snapshot may be useful!

    Why Snapshots?

    Snapshotting is useful in system recovery, as we have some known, saved, working state to rollback to once all servers are online. Other applications of snapshotting exist, however.

    Naive Snapshotting Approach

    So how can we capture a correct system state? One approach could resemble the following:

    1. Pause the ingestion of all input messages
    2. Wait for all in-flight data to be completely processed
    3. Copy the state of each process to a remote, persistent storage
    4. Wait until all processes have finished their copies
    5. Resume processing and data ingestion

    However, we note that this process requires us to halt our system entirely, harming our availability to clients which is not desirable.

    Distributed Snapshotting Protocols

    Let’s aim to improve our snapshotting protocol to be able to capture snapshots while the system continues to operate. In order to understand such a procedure, we must deeply understand the goal of a snapshotting protocol.

    Distributed snapshotting protocols aim to capture a consistent cut (recovery line) through our systems execution.

    A cut is a subset of a system execution’s global history that is a union of prefixes of process histories. Thus, a cut is defined by a series of points, one in each process’s history.

    The image above depicts a consistent cut.

    We define an event to be pre-snapshot if it occurs before this cut point and an event to be post-snapshot if it occurs after the point.

    A consistent cut is some cut where, for any event A, if A happens before B, and B is pre-snapshot, then A must also be pre-snapshot. Thus, a consistent cut preserves causality across the cut boundary.

    An inconsistent cut would include an effect without the cause (i.e. if A happens before B, and only B is pre-snapshot, while A is post-snapshot).

    An example of an inconsistent cut.

    Drone and Map Example: Consistent Cuts

    Let’s consider our previous example with a drone and three image servers.

    We can visually represent our system events in the fashion below, and the following line defines a consistent cut. Event a represents the drone taking an image. b,d,e represent sending the partial-images to the image server. And c,g,e represent receiving the partial image on each server.

    Notice: the below line would represent an inconsistent cut, because we include m without including j. It is not consistent to include a message receive event without including the corresponding message send.

    Also note, sometimes a consistent cut will define certain messages to be still “in the air” so to speak. All this means is that our snapshot should also note what messages have been sent but not received. When we recover/restart the system from this snapshot, we will just have to resend those “in-transmission” messages.

    Note here, we have defined such a consistent cut, where event b has occurred, the message is being sent to s1, but we snapshot the system before event g, when the message is received. Thus, this message is “in air” and would need to be replayed.

    Chandy-Lamport Snapshot Protocol

    The Chandy-Lamport snapshot protocol is a snapshot algorithm that is used in distributed systems for recording a consistent global state of an asynchronous system.

    It was invented by Mani Chandy and Leslie Lamport (of Lamport timestamp fame).

    Assumptions:

    Goals of Chandy-Lamport Protocol

    Initiator Process

    A process initiating the snapshotting protocol:

    1. Records its own state
    2. Sends a marker out on each of its outgoing channels
    3. Starts recording all data (application) messages it receives on all of its incoming channels
    4. The marker is a special (control) message that is not recorded in the snapshot but enforces the exact point in the FIFO communication between processes where the cut occurs.

    Receiving a Marker

    A process receiving a marker for the first time:

    1. Records its own state
    2. Marks the channel that the marker came in on as “empty” (Future messages arriving on this channel will not be part of the snapshot)
    3. Sends markers to all its outgoing channels
    4. Starts recording incoming messages on all its incoming channels except the one marked as “empty”

    Otherwise (not first time):

    1. Stops recording on the channel it received the marker from

    Completing a snapshot

    The snapshotting process is complete when all processes:

    1. Have received a marker and recorded their local state
    2. Have received markers on all incoming channels and have recorded all channel states

    After this point, we can then collect the locally recorded states and construct a global snapshot.

    Intuition

    Some intuition on why this snapshotting protocol works is as follows:

    1. The marker being received defines the boundary between pre/post snapshot.
      1. The first marker received corresponds to the moment we capture our system state.
      2. However, other processes may not have received the marker, yet, thus it is possible messages we receive from other processes will need to be replayed. Thus, we record messages on our incoming channels (as these two may be pre-snapshot).
    2. Because the channels are FIFO (not reordered), the marker is a clear boundary also on each communication channel in the stream of messages from one process to another.
      1. When we receive a marker from an incoming channel, we know that we can stop monitoring messages on that channel, as they are now post-snapshot.

    In the above example, we see a resulting snapshot. The pink highlighted area corresponds to all events saved in the snapshot. One message will have to be replayed (H->D), and is saved because p1 had not received a marker message from p2 before D.

    Restarting from a Snapshot

    Now, what happens when we want to restart our system from a snapshot?

    1. In each process, we restore the saved state to our system state.
    2. Replay any saved messages from the snapshot.

    A subtle question arises: when replaying from a snapshot are we guaranteed to get the exact same order of events? No! Restarting from a snapshot is not deterministic.

    For example, when replaying the above snapshot, we may get the following:

    It is certainly possible that in replaying the snapshot, post-snapshot events are re-ordered, such as D is delivered before event C occurs on P1. This could be an entirely valid, but different, execution which occurs from our snapshotted state.