CAS CS 351 - Distributed Systems

Detailed Lecture Notes

  • Lecture 1: Threads and Processes
  • Lecture 2: Communication Primitives
  • Lecture 3: Concurrency and Parallelism
  • Lecture 4: Sharding and Consistent Hashing
  • Lecture 5: MapReduce
  • Lecture 6: Time, Causality, Lamport Clocks
  • Lecture 7: Failure, Detection, Fault Tolerance, Recovery
  • Lecture 8: Failure Recovery and Snapshotting
  • Lecture 9: Replication
  • Lecture 10: Distributed Consensus I
  • Lecture 11: Distributed Consensus II
  • Lecture 12: Distributed Consensus III
  • Lecture 13: Consistency Models
  • Lecture 14: GFS and Weak Consistency
  • Lecture 15: Transactions and Serializability
  • Lecture 16: Dynamo
  • Lecture 17: Transactions and 2PC
  • Lecture 18 + 19: Spanner
  • Lecture 14: GFS and Weak Consistency

    Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau

    A consistency model that lacks a global total order of operations can be considered to be a weak consistency model. This means that clients of that system must design applications that can handle potentially stale data.

    We saw in the prior section that enforcing strong consistency will require sacrifices in availability or performance. Thus, some systems may elect to employ very weak consistency guarantees in exchange for a more available system or improved request throughput.

    To illustrate how and why a system may employ a weak consistency model, we will examine the Google File System (GFS).

    GFS Setting

    GFS is Google’s file system, meaning that it maintains a file structure of paths and directories, as well as the content of many files. GFS is built to handle massive datasets. Similar to the setting of MapReduce, these datasets exist to support Google’s main product of 2003, the search engine! Clients of GFS are other applications within Google. Thus, google developers will write applications that store their data in GFS.

    For example, MapReduce reads and stores data in GFS.

    The workload of GFS is specific:

    GFS Trade-offs and Weak Consistency

    Key Design Ideas

    GFS exhibits several mechanisms we have studied previously: replication, sharding, primary-backups, heartbeats and weak consistency. To understand where these mechanisms occur, we must examine three critical parts of the GFS architecture: chunks, chunkservers, and the GFS master.

    Chunks

    In GFS, each file is split into fixed sized chunks. Each of these chunks is fixed at some size, the paper specified 64MB. Each chunk is assigned a unique identifier handle.


    A visual representation of three files split into chunks.

    You may consider: what is the tradeoff between having large chunks compared to small chunks?

    Chunks are the base unit of data in GFS. You may notice that chunks are an application of sharding, a technique we discussed in a previous section.

    When a client wishes to perform an operation on a file, they must contact the master, who can calculate which chunk this request pertains to and return the mapping to the client. Chunkservers will host several of these file chunks.

    Chunkservers

    Chunkservers are the “worker” servers of GFS. Their job is to host chunks and respond to client requests pertaining to data reads and mutations.

    GFS also incorporates replication via assigning multiple chunkservers (say 3) to each host a replica copy of a particular chunk.


    Chunkservers hosting various chunks.
    Notice each chunk is replicated across chunkservers.
    Chunkservers host several chunks from several files.

    The GFS master maintains a mapping of which chunkservers host which chunks and propagates this information to clients.

    GFS Master

    The GFS master is responsible for storing information about the file structure. For example, information regarding filepaths, filenames, directories, etc., is all metadata stored within the GFS master. Mutations to this filestructure metadata are stored as a log of operations.

    The GFS master also stores which chunks make up a particular file.

    Finally, the master is also responsible for mapping chunks to which chunkservers host that data. This mapping is volatile state maintained as GFS master communicates regularly with chunkservers via heartbeats. In response to a heartbeat, a chunkserver communicates which chunks it has and the master updates the mapping.

    GFS Architecture

    Path of a Read

    When a client wishes to perform a read in GFS:

    1. The client contacts the master to obtain a list of chunkservers that host the data they are interested in
    2. The client then caches this list of chunkservers
    3. The client then contacts the closest chunkserver to perform the read (this optimizes the latency of the read, and splits several clients’ workload across chunkservers).

    Aside: Network Topology What does the “closest” chunkserver mean in GFS? Consider that all servers are connected to one another in a tree-like network topology consisting of routers, switches and network links. Every “hop” on the network takes some time. Thus, if we contact a close server in terms of network topology, this request will generally have a lower latency, as we need to take fewer “hops” on the network.

    Path of a Write

    When a client wishes to perform a write in GFS:

    1. The client contacts the master to obtain a list of chunkservers that host the data they are interested in.
    2. Since we are performing a write, the master will designate one chunkserver as the primary via a lease.
      1. We designate this primary for a specified chunk to enforce an order on mutation (writes) across replicas.
      2. Upon granting a lease, we increase the chunk version number.
    3. Clients “stages” the write data on all replicas– sending the data to write over the network.
      1. They may do this in any order, optimized for the “closest” replica.
    4. The client contacts the primary to confirm the write and apply it.
      1. The primary’s job is to enforce the order of writes from potentially several clients.
      2. At the point a backup replica receives word from the primary, it will apply the staged write, modifying the chunk data, and then respond back to the primary.
      3. The primary tracks the response of the other replicas, noting if one fails to respond.
    5. If all replicas respond to the primary, the primary responds with success to the client. Otherwise, the primary notes the operation failed and responds to the client with an error.
      1. Note that errors may leave replicas in inconsistent states, with some applying the mutation and others not!

    Failures

    What is the plan for failure in GFS? What if a write is partially propagated, but some replicas fail to receive it?

    GFS places a large responsibility on the client application to handle possible inconsistencies. GFS, upon noting that a replica does not respond to a write, responds with an ERROR to the client.

    It is the client’s responsibility to retry the write operation. Consider that a client wishes to append B to some chunk.

    The first time the client attempts this operation, it fails:

    Notice that now, our three replicas have diverging state. J and H have applied the append, but A has not!

    Now, the client retries the append. If no more failures occur, a successful append will be applied to the same offset on all three replicas. Notice that the three replicas have different chunk contents– J and H have the append twice and A has some padding –and GFS has no plan to reconcile this inconsistency.

    Instead, it is a requirement of applications built on top of GFS to be able to handle this type of behavior, manually filtering out duplicate sections or padding.

    So now we may ask ourselves: what consistency does GFS guarantee? We will see GFS does not even enforce eventual consistency, our weakest model, as replicas may diverge in content and this inconsistency may never be resolved.