CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 6: Time, Causality, Lamport Clocks

    Why Time?

    Up until this point, we have yet to consider what is correct behavior for a distributed system and why achieving this correctness may be more difficult than on a traditional single-machine system.

    Most simply, we can define correct behavior, as exhibiting a result that could occur on a single machine, single cpu. When processes on different machines are interleaved in execution in a manner we do not anticipate, depending on our system design, this could cause us problems.

    One way we achieve correctness is through coordination. Coordination is the process of orchestrating distributed executions to achieve a desired behavior. Coordination can be achieved by performing operations in a specified order and/or by waiting for other endpoints.

    In distributed systems, determining or enforcing the order that events occurred in our system is a critical, and sometimes difficult problem.

    For example, when debugging a system, we may want to uncover the order of events to understand where our system went wrong. When recovering from failure, we may need to replay an agreed upon order of operations. Or, when we have multiple replicas of some data, we may need to enforce the same order of state changes on each, in order to keep the replicas in sync.

    Consider the following example, say we have two servers. These are two bank servers that implement data parallelism, each holding ½ of the users in our system. We also introduce task parallelism, where one server is responsible for issuing low balance warnings. Furthermore, on both machines, once per day, we perform a 1% interest operation on each machine.

    We want to check and perform the low-balance warning once per day, after interest has been applied. However, note that we must perform generate_interest on both servers.

    How do we reason about the order in which these operations occur in order to enforce that our flaw_low_balance operation occurs after apply_interest on both machines? As programmers accustomed to working on a single machine, we may think time is a trivial way to reason about order.

    After this process runs, we see the logs from SHARD A and SHARD B:
    A 12:01:03pm: GENERATED INTEREST
    A 12:02:14pm: QUERYING MACHINE B
    A 12:03:02pm: ISSUEING FLAGS

    We also see on B:
    B 11:59:01 am: GENERATE INTEREST

    Given these logs, can we be certain that Machine B’s step 1 occurs before Machine A’s step 2 (which was our correct, desired, behavior)?

    To understand this answer, we must understand the properties of the hardware clock in a typical machine, which is responsible for the local time on Machine A and Machine B.

    Physical Clocks

    Inside your machine, you have a hardware clock. Typically, this will be a quartz clock. The useful property of this small piece of quartz is that it generates an electronic signal periodically.
    The hardware in your computer uses this periodic signal to send an interrupt signal to the CPU. An interrupt stops execution of whatever is going on in your computer otherwise, to jump to a special piece of code called an interrupt handler.

    On top of this regularly triggering interrupt handler, we can implement a simple software clock by incrementing some counter each time the interrupt fires.

    int clock_handler() {
        time_of_day += 1; 
    }
    

    An example of a simple C interrupt handler for the clock.

    Problems with Distributed Clocks

    The problem of keeping hardware clocks in sync is difficult. Why? Physics! The quartz hardware clock is sensitive to environmental factors and has some natural variation. For example, with temperature variations clocks may run a small amount faster or slower.

    Clock Skew describes how much two clocks vary from each other. In fact, it is common that two machine’s internal clocks differ from one another!

    Bank with Physical Clock

    Consider our previous example where we are applying interest and flagging low balance accounts. Now consider, if Machine B’s clock runs slower than Machine A’s clock due to clock skew. What could happen?

    A’s log:
    A 12:01:03pm: GENERATED INTEREST
    A 12:02:14pm: QUERYING MACHINE B
    A 12:03:02pm: ISSUEING FLAGS

    B’s log (with an annotation from an all-knowing power):
    B 11:59:01 am (@12:04:10 from A’s POV): GENERATE INTEREST

    We see that B’s clock being slow could mean that Machine B’s interest operation was delayed until after Machine A’s query for user balances, falsely flagging accounts who would be >$50 after interest is applied.

    Thus, we see that using physical clocks for synchronization or ordering events can be hazardous for the correct behavior of our programs.

    Practical Solutions to get Better Clocks

    Atomic Clocks are a much larger, more expensive type of hardware clock than the one in a typical laptop computer. Atomic clocks are based around the atomic period of a particular element: caesium-133, and measuring the transition frequency of that atom. This is an extremely accurate measure of the length of a second.

    Unfortunately, not all computers can have atomic clocks, thus it would be useful to have a protocol wherein we can synchronize a group of distributed clocks with a remote atomic clock via networked communication.

    Achieving Synchronization: Metrics

    The first question in determining such a synchronization protocol, will be, how often must we synchronize our clocks in order to bound our clock skew? To understand the answer to this question, we must define several metrics.

    Precision (π) is the measure of how much two clocks differ from one another.
    Internal Synchronization represents a system where two clocks are precise: they are both synchronized to each other.

    ∀t, ∀p, q : |Cp(t) − Cq(t)| ≤ π

    Accuracy (α) is the measure of how much a clock differs from some external reference point.
    External Synchronization represents a system where two clocks are accurate: they both attempt to synchronize to some external reference point.

    ∀t, ∀p : |Cp(t) − t| ≤ α

    However, even if we have two clocks that are synchronized at one instant, they may not stay synchronized. This is because of clock drift.

    Clock Drift describes the rate difference between two clocks. If one clock is ticking faster than another, after some elapsed time the clock skew grows. How often we need to synchronize our clocks to bound our precision is a function of the clock drift.

    Drift Rate and Synchronization for Precision

    Say the maximum drift rate of a clock is ρ per unit of time. However, we want to bound the precision of the two drifting clocks to each other.

    At a maximum, the two clocks can be drifting exactly apart, thus their difference from each other grows by 2*ρ every unit of time. Then, to bound a maximum difference in clocks of π, we need to synchronize every π/(2*ρ) units of time.

    Network Time Protocol (NTP)

    Now that we have an understanding of how often we should perform synchronization, how do we determine by how much two remote clocks differ?

    Consider the following interaction of two remote computers A and B, who both send eachother a message at the specified times.

    Say we run the following protocol:

    1. Computer A sends Computer B a message containing T1.
    2. Computer B receives the message at T2, spends processing time, then sends a response at T3 containing T2, and T3.
    3. Computer A receives the message at T4.

    Round Trip Delay is the time in the protocol spent by sending/awaiting messages on the network. How can we calculate this time given we have T1,T2,T3,T4?

    D = (T4-T1) - (T3-T2)

    This is the total time minus the processing time on B, thus this represents the time where messages were being transmitted.

    Why can we not calculate the round trip delay as (T2-T1) + (T4-T3)? Remember, clocks across machines could be different due to clock drift. To make this intuitive, imagine that Computer B has a much slower clock than A. This could mean that T2 < T1! Thus, T2-T1 would be a negative value!

    However, we can use the above observation to deduce the clock difference between A and B.

    Notice, when we take T2-T1, we are calculating the RTD/2 + clock_difference. When we take T4 - T3 we are calculating RTD/2 - clock_difference.

    So, if we take:

    θ = [(T2 - T1) + (T3 - T4)] / 2

    We can isolate the clock difference between the two machines.

    Now, using this above observation we build up to the NTP (network time protocol). NTP operates by doing several rounds of the above operation (to average out noise) and get an accurate reading for θ.

    NTP also organizes a network of servers and atomic clocks (in satellites) into stratum. A server with a reference clock such as a UTC receiver or an atomic clock, is a stratum-1 server. When A contacts B, it will adjust its time only if its own stratum level is higher than that of B. After synchronization A’s stratum level will be one higher than that of B.

    NTP achieves accuracy in the range of 1-50ms. Is this good enough to reason about event ordering in distributed systems? Maybe for some applications, but it is also reasonable that a distributed system may handle thousands of requests per second. At this rate, 50 ms accuracy will not distinguish the order of these requests accurately.

    Away from Real-Time Clocks

    As we have seen, even with our best synchronized clocks, two clocks on different machines may differ by milliseconds (not to mention the computational annoyance of constantly synchronizing over the network).

    What we realize is that, for many distributed systems, we do not care about the precise instant in real time which an event occurs, but instead we care about the relative ordering of events. Also, we note that if two processes do not interact, never sending messages or sharing data, the ordering of their events to each other does not matter at all.

    Thus, for many distributed applications we opt to use logical timestamps instead of real-time timestamps. Logical timestamps encode happens-before or causal orders of events.

    Happened-Before Relation

    A → B is the happened-before relation. Read aloud as “A happens-before B”.

    Transitivity: If A→B, and B→C, we can say A→C.

    We call these two events concurrent (denoted as   ) if: A ⇏B and B⇏A. (⇏ means “not Happened-Before”).


    In the above example we can say: a→b, b→c, x→y,y→z from the ordering of events on individual processes.

    We can also say: b→y and z→c from the ordering of send/recieve events.

    This means, from transitivity, we can say a→y and b→z for example (and many more).

    However, when considering a and x, a ⇏ x and x⇏a. Thus, these two events are concurrent (x   a).

    Lamport Clocks

    A lamport clock is made up of timestamp pairs (Ci, id), where Ci is the highest known logical time, and id is the id of the process/server.

    Lamport clocks are updated on events, as follows:

    On internal event a:

    On event “send m”:

    On event “receive m” with timestamp ts(m) from another process:

    Lamport timestamps are comparable:

    (C1, id1) > (C2, id2) iff C1 > C2 or (C1 = C2 and id1 > id2).
    (C1, id1) = (C2, id2) iff C1 = C2 and id1= id2.

    Lamport timestamps correspond to the happens-before relation as follows:

    1. If A → B, then C(A) < C(B). C(x) is the lamport timestamp for event x.
    2. If C(A) >= C(B), then A ⇏ B.

    Note the converse of these two observations does not hold.

    The above example assigns lamport timestamps to 6 events.

    Consider in the above example that C(a) < C(e), however, we can not say that a → e. In fact, visually we can discern that these two events are concurrent (a ⇏ e and e ⇏ a). Furthermore, in this example, we can say that e ⇏ c, however, again we can not say that C(e) >= C(c) in fact C(c) = (3,2) > C(e).

    Lamport timestamps enforce a global total order. This means that for every unique set of events E, there is some ordering of events where C(e1) < C(e2) < … C(en), for all events in E, where C(x) is the lamport timestamp of event x.

    Totally Ordered Multicast

    For example, say we have a distributed banking system, with two bank servers. Each bank server contains a copy of the account dataset. (We can call this replication, however, we will cover replication in detail in a later section).

    Now, we allow multiple clients to interact with our system. For example we may have Alice deposit 1000 dollars into her account from a local server in BOS and then NY, and we may have a banker in NY increase Alice’s balance by 1% interest and then communicate with the BOS server.

    Now, arriving to these two servers, simultaneously, we have two operations:

    What is correct behavior in this situation?

    Either:

    1. we should have both BOS and NYC perform alice_money += 1000, then alice_money *= 1.01, leaving Alice with $1010.
    2. Or, we should have both BOS and NYC perform alice_money *= 1.01, then alice_money +=1000, leaving Alice with $1000.

    However, it would be catastrophic for our system to have our servers disagree on how much money Alice has. This can happen if BOS and NYC decide on different orders of operations.

    Notice: The key problem is how to order operations consistently across endpoints.

    Let’s try to use lamport timestamps in order to solve the problem.

    Assumptions:

    Protocol:

    Below is an illustration of the protocol, omitting self messages for simplicity.

    Notice, lamport timestamps allow us to enforce the same order of messages across every server’s queue.

    An example execution:

    1. Roughly simultaneously, +1000 arrives at S1, and %1 arrives at S2. Both are added to respective queues with timestamps (1,1) for S1 and (1,2) for S2.
    2. S1 sends a request for ACK +1000 to S2.
    3. S2 sends a request for ACK 1% to S1.
    4. S2 receives ACK, +1000 (1,1) is at the front of the queue.
      1. marks locally that 2 have acknowledged +1000, applies +1000
      2. Increments Lamport ts to (2,2)
      3. (Note that any new messages received at this point would be timestamped as (3,2) for example)
    5. S1 receives ACK %1(1,2), enqueues behind +1000 (1,1). Thus, does not apply anything.
    6. S2 receives ACK %1, applies %1.
    7. S1 receives ACK +1000, thus has ACKS=2, and can apply +1000, and then apply %1.

    ###

    Vector Clocks

    Lamport timestamps enforce a global total order. This means that for every unique set of events E, there is some ordering of events where C(e1) < C(e2) < … C(en), for all events in E, where C(x) is the lamport timestamp of event x.

    However, Lamport clocks have some downsides. Though they enforce a global total order, a bit, they do so arbitrarily, and we lose some information with lamport timestamps. Specifically, when two events a,b are concurrent one of them will still be assigned a lamport timestamp: such that either T(a) < T(b) or T(b) < T(a), depending on the execution.

    Vector clocks recognize this downside of lamport timestamps and devise a scheme that captures this concurrent relationship, by enforcing not a global total order but instead a partial casual order.

    To do so, vector timestamps capture total events known about that happened before some event, per process.

    A vector timestamp takes the form v(a) = [x1, x2, x3, … xn] where n is the number of processes in our system, and xj is the number of events that occurred before a on process j.

    Initially, before any events occur in the system, the vector timestamp starts as all zeros : [0,0,0 … 0]

    Vector timestamps are updated as follows:

    On internal event, “send m” event, or “receive m” event at process i, increment local counter in v:

    On “receive m” event at process i, update v as follows:


    Vector timestamps are also comparable as follows:

    vp < vq if and only if:

    Vector clocks imply:
    If v(a) < v(b), then a → b.
    If a → b, then v(a) < v(b).

    Vector clocks also imply:
    If and only if v(a) !< v(b) AND v(b) !< v(a), the time stamps are incomparable v(a) < > v(b), and we can say that a is concurrent to b.

    Thus, vector timestamps capture causality between events.

    An illustration of the categories of events as viewed by happened-before relationships, lamport timestamps, and vector timestamps.

    Notice that vector timestamps and happened-before relations have a 1-to-1 mapping of event sets. However, for lamport timestamps t(A) < t(B) is a superset of A->B, and t(A) >= t(B) is a subset of A !-> B.

    Causally-Ordered Multicast

    The totally ordered multicast example from before may be too strong in certain scenarios. For example, if two bank operations act on different accounts entirely, do we need to ensure that we run them in the same order on all bank machines? Because we are enforcing this global total order of events, we may hold items in the queue for longer than we need.

    Thus, to improve the performance of our system, we may say that causally-ordering events is good enough. Only if two events have a causal relationship, A → B, must we ensure that A is run before B on every machine in our system.

    Thus, we can modify our scheme of totally-ordered multicast with lamport timestamps to instead use vector timestamps to order the queue. This scheme would give us a causally-ordered multicast primitive.