CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 16: Dynamo

    Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau

    Dynamo is Amazon’s distributed key-value store.
    We elect to study Dynamo because it is an excellent case study on a bevy of ideas:

    1. Dynamo uses vector clocks to identify and reconcile merging inconsistent versions of data
    2. Dynamo uses quorum-like protocol for highly-available writes (with consistency guarantees only in the best case)
    3. Dynamo uses consistent hashing to shard data flexibly across servers
    4. Dynamo utilizes a Gossip-based protocol to communicate view/group-membership information
    5. Dynamo illustrates a design that favors availability while illustrating how this is in conflict with strong consistency models.
    6. Dynamo implements eventual consistency, showing how in practice even this extremely weak consistency guarantee can be usable.

    Context at Amazon

    Dynamo was built at Amazon. Critical to understanding Dynamo’s design goals is understanding the workload of Amazon. At Amazon, each write is a potential purchase– a user submitting an order, placing an item in a cart, etc.

    Thus, having a high availability for writes means making more money. If our system is temporarily unavailable, we may lose a customer. This reasoning is the backbone for why Amazon favors availability– it is better to try to accept the write and potentially create small inconsistencies (a user’s mobile device not showing the same check-out cart as their laptop), rather than rejecting the write altogether.

    Finally, the workload at Amazon is spikey– people buy products during specific times of day far more than other times, on black friday they receive much more traffic altogether. Thus, there is a need for our infrastructure to be incrementally scalable or elastic.

    Tail Latency and Service Level Agreement

    Dynamo is built to provide a Service Level Agreements (SLA), which is a guarantee on latency and availability of requests to the system. Each Dynamo instance can configure its own SLA, and in turn will tune various system parameters. One interesting feature of this SLA is it provides a claim about Dynamo’s tail latency.

    A tail latency is the time of the slowest requests that Dynamo serves. If we visualize all of the requests in Dynamo as a cumulative distribution function (CDF), where for each latency we report the percentage of requests that are completed at this latency or faster.

    Making a statement about the tail latency of the request distribution, makes a claim about this 99.9% or even 99.999% percentile of requests. For example, Dynamo’s SLA may state that 99.999% of requests will complete in 350 ms or less.

    Consider, why might it be favorable to make a claim about tail latency as compared to average latency?

    The core reason: Amazon notices it can be more powerful to make claims about the worst case than the average case. It may not be important to have small latencies on average if several outlier requests take much longer to complete.

    Big Picture at Amazon

    Amazon’s infrastructure is built upon multiple systems, each with their own SLAs. If Dynamo has a particular guarantee for tail latency, and another service is built upon Dynamo, these guarantees stack and inform the latency guarantees of the applications built upon Dynamo.

    Dynamo API

    Notably, dynamo is not a full relational database, instead it is just a key-value store. Dynamo only supports get and put requests, instead of SQL-like queries.

    get(k) = {object, context}
    put(k,object,context)

    Each data object in Dynamo also has associated context. This context contains versioning information, and it returns alongside a get request. Furthermore, clients must submit the context with their put requests.

    Dynamo Ring

    In Dynamo each server is symmetric– there is no designated “special” server such as a coordinator or a master. These servers are organized in a flat structure, specifically in a consistent hashing ring.

    Dynamo uses consistent hashing, as defined and discussed in previous sections.

    Remember: in consistent hashing we hash the key of a data object to map it onto a ring of the hash function’s output space. We also map servers onto the ring. Then, a data object’s “primary” replica can be considered to the next, clockwise, server in the ring.


    Data object w hashes to the blue location.
    Requests pertaining to w will be primarily handled by S, who will store w.

    The benefits of this organization is that it minimizes data shuffling as servers are added/removed from the ring, supporting Dynamo’s goals of elasticity and scalability.

    Routing in the Ring

    How then, does a client interact with Dynamo given they have a request to a specific piece of data?

    A naive attempt could be: Each server has a pointer to the next in the ring, and each server will re-route the client in the ring until they find the owner for a particular key. However, this would require a lot of traversal, and many network requests.

    A better approach, which Dynamo employs, is “zero-hop” routing. Through an underlying gossip protocol Dynamo servers establish an eventually consistent view on group membership.

    Thus, any one contacted server knows enough about the other live servers in the ring to correctly route clients to the host for the data pertaining to their get/put request. After only a single re-routing, a client can then be connected to a primary server for their data.

    Virtual Tokens

    Like we discussed in the previous sections, a common modification to consistent hashing is to use 2 or more “virtual tokens” per physical server and hash these to the ring. This means each physical server is responsible for several regions of the ring.


    Dynamo notes the benefits of using virtual tokens:

    | If a node becomes unavailable (due to failures or routine maintenance), the load handled by this node is evenly dispersed across the remaining available nodes. When a node becomes available again, or a new node is added to the system, the newly available node accepts a roughly equivalent amount of load from each of the other available nodes. The number of virtual nodes that a node is responsible can decided based on its capacity, accounting for heterogeneity in the physical infrastructure. Dynamo, Section 4.2 | | :—- |

    Replication

    Replication exists on top of this ring structure. When a client makes contact with a server, that server replicates that request across the next N-1 servers where N is a tunable system parameter for replication.

    This means that every server hosts not only the keys that hash to its virtual tokens, but also the N-1 predecessors in the ring.

    Failures and Availability

    Dynamo makes a big promise in regards to availability in the presence of failures. Dynamo wants to be always writable in the sense that Dynamo will accept a client’s write request even if the primary server and replicas are failed. In fact, Dynamo claims to be available even if almost all the servers are failed! Dynamo’s plan is to accept the write on any remaining live server.

    Dynamo’s plan to be available for writes in the face of temporary failures is as follows:

    1. If a server that would normally be one of the N replicas responsible for a write is unreachable, instead perform the request on the next server in the ring.
      1. This may mean different sets of N servers replicate a request, some of which are not normally responsible for this data object.
    2. A server who is not normally responsible for the data, but processes the request, will keep this data object separate and mark it accordingly.
    3. Then, this extra server will attempt to perform a hinted hand-off to the failed server once that server is reachable again. The extra server will hear about the failed servers recovery via the gossip protocol.

    Sloppy Quorum

    When no failures occur, however, Dynamo wants to attempt to appear consistent. Thus, Dynamo employs a quorum-like technique known as a sloppy quorum. The property of this sloppy quorum is that in the common case, reads will see the latest writes. However, we do not want to compromise availability, so we do this “sloppily”.

    There are three parameters in this sloppy quorum that are tuned per dynamo instance:

    R: The number of successful servers required during a read.
    W: The number of successful servers required during a write.
    N: The replication factor of each operation in the system.

    We notice an interesting effect when we set R,W,N such that: R + W > N. The effect is similar to that of overlapping majorities.

    If a read and a write occur across the same set of N servers (which is the case with no failures). If a write succeeds, then necessarily, a subsequent reading group will overlap with this writing group, and at least one replica will return the most up-to-date version of the data object.

    However, remember that our quorum is “sloppy” because in the case of failures, this R sized group and this W sized group may come from different sets of N servers in total. This property can cause us to return stale data some of the time.

    1) R=2,W=2,N=3. Client A submits a put operation, switching x to green. Two servers are unreachable, so Dynamo replicates the operation on G,F,E instead (W=3,N=3).
    2) The data object is updated on GFE. G,F mark the object for hinted handoff
    3) Before handoff occurs, client B performs a read, contacting the original 3 servers. However, E’s message is lost. The read succeeds with R=2, and the client retrieves old data.

    Consequences of Always Writeable Design

    As an effect of our usage of “sloppy quorums” we get the benefit that writes and reads will be highly available, succeeding most of the time. However, inconsistencies can emerge across data versions and replicas.

    For example, one client may get a stale version of the data, perform some change and now we have branching data versioning.

    Consider the above scenario: client 1 contacts two servers, reads that cart1 is empty and then adds a book to the cart. Then client 2 reads the same cart but from a different two servers, and gets a stale read (cart being empty). Client 2 then adds a lamp to the stale cart.

    Finally, we must ask: what should happen when client3 reads cart1? In this case our data version has forked– and really neither version is more ‘up-to-date’. In fact, we now have two causally independent branches of the data!

    Resolve Inconsistencies

    How should Dynamo resolve these diverging “forks” of data?

    In the above scenario, the desired behavior for this shopping application may then be to merge the two sets, and set cart1 to now be {book,cart1}. As such, Dynamo’s plan for diverging forks is to allow the application to resolve the inconsistency and submit a later put request with the fixed version of the data.

    To facilitate this behavior, Dynamo keeps track of versioning information in the data object’s associated context. Multiple versions of a single data object may exist within dynamo simultaneously.

    Note that in the case of no failures, which is the common case, subsequent writes subsume preceding writes and there is no such branching. Dynamo can automatically reconcile these two causally dependent versions by overwriting the older data version with the newer.

    However, in some cases like above, we have diverging data and Dynamo can not simply say one is “stale”. Dynamo decides to offload this reconciliation to applications in the case of casually independent data versions. Dynamo will maintain multiple versions of a data object if such diverging writes occur. Then, Dynamo will reconcile this divergence lazily.

    How can Dynamo tell that data versions are or casually dependent (can be automatically reconciled) or independent (need application assistance)?

    Dynamo’s versioning information is a variant of a vector timestamp where each index is the server that made a change and the counter associated with the server. Thus, if two vector timestamps are incomparable, we can know those data versions are causally independent.

    On a client read, Dynamo fetches all versions from the R nodes in the read quorum, collapses versions that have a causal ordering and returns only the incomparable versions to the client. So it may be that a client receives multiple data objects and they are left to reconcile the divergence themselves with a later put. After reconciliation, subsequent writes from the client will include a vector timestamp that indicates the reconciled version subsumes both branching versions.

    Figure 3 of the Dynamo paper illustrates an example of diverging data objects and the associated vector timestamp.

    What occurred:

    1. Sx handled an initial write, maybe put(cart1,{laptop}) D1={laptop}
    2. Sx handled a second write, maybe put(cart1,{laptop, book}) D2={laptop,book}.
      1. Notice that the vector timestamp for D2 > timestamp for D1, indicating that D2 subsumes D1.
    3. Sx fails temporarily, or is unreachable by some clients
    4. Sy handled a later write, maybe put(cart1,{laptop,book,kindle}) D3={laptop,book, kindle}
    5. Sz handled a second write, maybe put(cart1,{laptop,book,dvd}) D4={laptop,book, dvd}
    6. A client performs a read, obtaining D3 and D4. They reconcile the two sets with application logic, merging the carts.
      1. Note that the vector timestamps for D3 and D4 are incomparable.
    7. Sx handles the last write, maybe put(cart1,{laptop,book,dvd,kindle}) D5={laptop,book,dvd,kindle}.
      1. Notice that in terms of timestamps: D5> D3 and D5 > D4.

    Merkle Tree

    Consider that hinted handoff may not work in certain situations– what if a replica hosting a hinted handoff fails before the handoff completes? Replicas may diverge for an unbounded period of time, however, Dynamo includes anti-entropy techniques to eventually reconcile divergent data.

    In the background, on top of the gossip protocol between servers, we also wish to check if our replica data is divergent or matching. Most of the time, replicas will match, however, in the case of unfortunate failures, they may diverge. The question then becomes how can we quickly identify this divergence (for the purpose of reconciliation) without wasting time?

    A naive approach would be to cross-compare every data item, one by one, across two replicas. This would be an O(n) operation.

    Instead, Dynamo opts to use a merkle tree. A merkle tree is a tree of hashes. The parent node of two leaves is calculated as: H(left | right). If


    Finally, we produce one root hash. We can then send this root hash as metadata in the gossip protocol. If two replicas have diverging underlying data, this root hash will not match, and this check is quick– O(1). Reconciling the divergence then becomes a problem of traversing down the tree: is the right child hash mismatched? Then we can deduce the underlying inconsistency lies in this right branch. We continue traversing until we get to the final leaf that has the diverging data block. This reconciliation process is then O(log(N)).