Detailed Lecture Notes
Visuals by Vasiliki Kalavri and Anna Arpaci-Dusseau
A consistency model that lacks a global total order of operations can be considered to be a weak consistency model. This means that clients of that system must design applications that can handle potentially stale data.
We saw in the prior section that enforcing strong consistency will require sacrifices in availability or performance. Thus, some systems may elect to employ very weak consistency guarantees in exchange for a more available system or improved request throughput.
To illustrate how and why a system may employ a weak consistency model, we will examine the Google File System (GFS).
GFS is Google’s file system, meaning that it maintains a file structure of paths and directories, as well as the content of many files. GFS is built to handle massive datasets. Similar to the setting of MapReduce, these datasets exist to support Google’s main product of 2003, the search engine! Clients of GFS are other applications within Google. Thus, google developers will write applications that store their data in GFS.
For example, MapReduce reads and stores data in GFS.
The workload of GFS is specific:
GFS exhibits several mechanisms we have studied previously: replication, sharding, primary-backups, heartbeats and weak consistency. To understand where these mechanisms occur, we must examine three critical parts of the GFS architecture: chunks, chunkservers, and the GFS master.
In GFS, each file is split into fixed sized chunks. Each of these chunks is fixed at some size, the paper specified 64MB. Each chunk is assigned a unique identifier handle.
A visual representation of three files split into chunks.
You may consider: what is the tradeoff between having large chunks compared to small chunks?
Chunks are the base unit of data in GFS. You may notice that chunks are an application of sharding, a technique we discussed in a previous section.
When a client wishes to perform an operation on a file, they must contact the master, who can calculate which chunk this request pertains to and return the mapping to the client. Chunkservers will host several of these file chunks.
Chunkservers are the “worker” servers of GFS. Their job is to host chunks and respond to client requests pertaining to data reads and mutations.
GFS also incorporates replication via assigning multiple chunkservers (say 3) to each host a replica copy of a particular chunk.
Chunkservers hosting various chunks.
Notice each chunk is replicated across chunkservers.
Chunkservers host several chunks from several files.
The GFS master maintains a mapping of which chunkservers host which chunks and propagates this information to clients.
The GFS master is responsible for storing information about the file structure. For example, information regarding filepaths, filenames, directories, etc., is all metadata stored within the GFS master. Mutations to this filestructure metadata are stored as a log of operations.
The GFS master also stores which chunks make up a particular file.
Finally, the master is also responsible for mapping chunks to which chunkservers host that data. This mapping is volatile state maintained as GFS master communicates regularly with chunkservers via heartbeats. In response to a heartbeat, a chunkserver communicates which chunks it has and the master updates the mapping.
When a client wishes to perform a read in GFS:
Aside: Network Topology What does the “closest” chunkserver mean in GFS? Consider that all servers are connected to one another in a tree-like network topology consisting of routers, switches and network links. Every “hop” on the network takes some time. Thus, if we contact a close server in terms of network topology, this request will generally have a lower latency, as we need to take fewer “hops” on the network.
When a client wishes to perform a write in GFS:
What is the plan for failure in GFS? What if a write is partially propagated, but some replicas fail to receive it?
GFS places a large responsibility on the client application to handle possible inconsistencies. GFS, upon noting that a replica does not respond to a write, responds with an ERROR to the client.
It is the client’s responsibility to retry the write operation. Consider that a client wishes to append B to some chunk.
The first time the client attempts this operation, it fails:
Notice that now, our three replicas have diverging state. J and H have applied the append, but A has not!
Now, the client retries the append. If no more failures occur, a successful append will be applied to the same offset on all three replicas. Notice that the three replicas have different chunk contents– J and H have the append twice and A has some padding –and GFS has no plan to reconcile this inconsistency.
Instead, it is a requirement of applications built on top of GFS to be able to handle this type of behavior, manually filtering out duplicate sections or padding.
So now we may ask ourselves: what consistency does GFS guarantee? We will see GFS does not even enforce eventual consistency, our weakest model, as replicas may diverge in content and this inconsistency may never be resolved.