Detailed Lecture Notes
MapReduce is a system that exhibits massive data parallelism.
MapReduce’s goal is to make it easy for programmers with no distributed system’s experience to develop applications that have massive parallelism for simple data processing tasks. MapReduce wants to ensure that application developers don’t have to worry about failures, consensus, workload distribution, or any other distributed system problem. We will focus mainly on how MapReduce orchestrates parallelism at scale.
MapReduce was designed by google for a specific type of job– simple data analytics on very large input datasets.
For example:
What do these tasks have in common? A lot of these tasks are conceptually simple but would take hours or days running on a single machine.
First, we will examine how users interact with MapReduce. Users of MapReduce are application developers within Google who need to parse their datasets (for example, searching for occurrences of words, etc.)
To use MapReduce application developers need to write only two snippets of code: a MAP function and a REDUCE function.
A map function takes in an input document (in the form of a key,value pair) and produces a list of intermediate key value pairs. Oftentimes the name of the document is the key and the value is a blob of file text data. The output, for example, could be (word, freq) for each word in the document.
A reduce function takes in a single intermediate key and a list of associated values (from all runs of a map function that produced that intermediate key) and produces a final (key, final_value) output pair.
Then with the input data, map function, and reduce function, the user says “okay run!” and from there MapReduce system orchestrates the massive number of workers running. The workers run map tasks on sections of the input and then the workers run many reduce tasks on partitions of the intermediate output. This massive parallelism on different sections of the input data and different sections of the intermediate data means producing the final output happens in hours or minutes instead of days or weeks!
Below is an example of how the input data is transformed into intermediate key,value pairs, which are partitioned into reduce tasks– who produce the final output data. Note here we do not represent worker machines, only the TASKS that run and the input/output of each tasks. Below we have 2 map tasks and 3 reduce tasks.

So how is the MapReduce system designed?
MapReduce Coordinator orchestrates which workers run which tasks and when. In order to perform this orchestration, the coordinator must maintain a data structure of all tasks, if they are completed or not, and which worker is assigned to the task. The coordinator is also responsible for communicating with clients once the job is done.
MapReduce Worker performs either map tasks and reduce tasks. The worker regularly polls the coordinator for a task and performs the corresponding task on corresponding data. The worker is also responsible for storing intermediate key,value pairs that result from any and all map tasks that worker performs.
In a run of MapReduce:
This is done on-the-fly. Workers continually poll the coordinator for tasks. The coordinator maintains a data structure of each task, tracking which worker it is assigned to, as well as the state of each task complete, inprogress, or idle. When a worker polls the coordinator, if a task is available (idle), it will be assigned to the worker.
However, note, if there are no tasks available to assign (they are all already assigned to other workers), the coordinator responds that the worker should remain idle for now.
For efficiency purposes, ideally we want all workers to have a task assigned at all times. Having all workers busy is the best way to leverage the maximal amount of parallelism in our computation. However, if tasks are uneven in size or compute intensity, it is possible that some workers are waiting around after completing their shorter tasks without any assigned job.
The intermediate data is stored in local storage on the worker. This design choice is fast, however, leaves the data vulnerable to worker failure.
Notice, we can not assign a reduce task while there is an ongoing or incomplete map task. Map tasks are responsible for computing the necessary input intermediate data. Until all intermediate data is computed, we can not reduce anything (this is because for any map task, it is possible it will produce intermediate data relevant to any reduce task).
Again, now when a worker polls the coordinator, it will be assigned a reduce task.
The worker running a reduce task takes in as input, an entire partition of the intermediate keys. However, this partition is spread across all the workers’ local storage (remember, it is the output of the map tasks). So the reducing worker must acquire the intermediate data.
Reducing workers need to communicate across network to get intermediate data from other workers’ map tasks.
A two-worker example where a worker running reduce on partition 2 communicates with the other worker.
This is an “all-to-all” shuffle. All workers talk to each other in sending intermediate data for reduce tasks. In the worst case, all reduce tasks require some piece of information from every other worker (who had performed a relevant map tasks and stored the intermediate data). So we must perform this communication. The reducers also sort the intermediate data, so that the worker can call the client’s supplied reduce function on each unique key.
Finally, Reducing workers save their final output in some final storage medium (i.e. send across network to location in company’s permanent file system)
Once all reduce tasks are complete, the coordinator signals to the client that the computation is finished.
It is crucial that all occurrences of a single intermediate key go to a single worker. Thus, it is important partitioning happens in the same fashion across all mapping workers. Imagine if the key (red,1) and a different key (red,1) went to different reduce task partitions!
As we can see from the example above, in that case, we would not have correct frequencies at the end!
So how do we correctly partition?
By default, trivially, we can take the hash of a key.
Every time we call H(“red”) for example, we may get 2260571. Then we can take H(“red”) % 3 servers, and always map instances of intermediate key “red” to partition 1 (in this example). Then, when we assign a reducing worker partition 1, they can simply query for partition 1 on each other worker server and know for certain that they get all intermediate data associated with the key “red”.
A key feature of this hashing approach is that it is deterministic, thus all mappers will always assign identical keys to the same partition. The hashing approach also creates balanced partitions (assuming our hash function uniformly distributes our keys over the hash output space). It is important to have balanced partitions for reducers so that our reduce tasks are roughly the same size (for load balancing purposes).
A visualization of partitioning intermediate data within a mapping worker
via a hash modulo partitioning function.
During the duration of a run of MapReduce the coordinator constantly pings workers to check if they are up and responding.
If the coordinator notices a worker has failed, it must take action.
What happens if the failed worker had previously completed map tasks?
A future reduce task will need that intermediate map task’s output. However, the only copy of the failed worker’s intermediate data was in local storage on the worker. Thus, it is no longer accessible when the worker fails!
We need to re-run all failed worker’s map tasks on a new worker. Luckily, from the coordinator’s perspective, this is as easy as switching the tasks status to idle (which is the unassigned state of a task). The next time a running worker polls the coordinator, it can be assigned the map task to rerun.
What happens if it had completed reduce tasks?
In the suggested implementation of MapReduce, completed reduce tasks write the final output to some external, cloud storage location, not on the failed worker.
This means we do not need to re-run completed reduce tasks, as they are written to external storage.
What happens if the coordinator fails?
What happens, however, if a worker has not failed, but is performing computation much slower than other workers? We call this slow worker a straggler.
MapReduce is sensitive to stragglers. In one bad case, we have to wait for the slow worker to finish a map task before starting any reduce tasks. Or, at the end of our computation, again we must wait for the slow worker to finish it’s last reduce tasks.
To minimize the impact of stragglers, when the coordinator notes that a particular task is taking a long time on a worker, the coordinator re-issues the task to another worker. There is some potential then, however, that duplicates of a single task run and complete. See MapReduce Section 3.3 Semantics in the Presence of Failures to understand why the MapReduce system can handle tasks being run multiple times without affecting correctness.