Detailed Lecture Notes
Up until this point we have built up some basic primitives needed for implementing a distributed system– threads for computation and rpcs for communication. However, we have not tackled the question of why one might want to build a distributed system.
One primary reason is parallelism. Parallelism refers to doing work simultaneously. Intuitively, if we have an application that runs at a slow speed on a single computer, we may think, running that same application with the power of 100 computers should surely be faster? Right? However, if we can achieve such performance benefits is dependent on the design of our system.
Designing a distributed system such that we can leverage parallelism and achieve major speedups will be the focus of the next few sections.
One important terminology distinction is that concurrency and parallelism refer to two different concepts.
Concurrency is the ability to manage multiple running processes. However, it may be that these processes never actually run simultaneously, but instead these processes are interleaved on a single resource.
Parallelism is when multiple processes run simultaneously on multiple resources.
A system is said to be concurrent if it can support two or more actions in progress at the same time. A system is said to be parallel if it can support two or more actions executing simultaneously.
Let’s consider, for example, a simple machine with a single cpu core. Though it is not possible to achieve parallel execution (at any one time exactly one process may be running on the single core), it is possible to achieve concurrency. We may interleave execution of one process with another.
In the section on threads, we were actually discussing the idea of concurrent programming. Concurrent programming is structuring your program in such a way that smaller units of the program (in our case threads or goroutines) can be run in an interleaved or simultaneous fashion.
With parallelism, it is clear that running or more processes simultaneously on two cpus will complete faster than running them one after the other on a single cpu. But why is concurrency useful?
There will be no performance benefit if the two tasks require only CPU time (and no other resource, like i/o or memory). However, commonly, processes require some resources. For example, they may be blocked waiting for a response over the network.
Now, consider if we run these two tasks (with io time requirements) sequentially.
As you may notice, there is a lot of time where the CPU is idle as the running task waits for io.
However, what happens when we run these two tasks concurrently? Our operating system notices for example, that task A is blocked on IO and runs one time unit of Task B. Then Task A is ready to resume and we continue from there. There is only one instance in time where, unfortunately, both tasks are using io resources, and the CPU is idle.
However, with concurrent execution– interleaving running tasks –we complete both tasks in 8 time units as compared to the 11.5 time units of the sequential execution.
In fact, we’ve explored this idea already– remember our example of a command center orchestrating satellite averages? Our command center exhibited concurrency. Though the command center had only one CPU, it was able to achieve speedup by processing one satellite’s data when we were blocked waiting for other satellites to respond.
Data Parallelism refers to the same task executing on different data at the same time.
Oftentimes data parallelism is achieved via partitioning a dataset into chunks and running the task on each chunk simultaneously.
Task Parallelism refers to different tasks executing at the same time, operating on the same or different data.
Pipeline Parallelism occurs when different tasks operate in parallel forming a pipeline. The output of each task is passed as input to the next task in the pipeline. Each task can continue processing the next input by the time it pushes the result to the output.
Consider the below 3-stage pipelined task. Naively, we only complete a task once each of the 3 stages finished. Now, we can complete a task every time a single stage completes!
If each stage takes 3 seconds, without pipelining we complete one task every 9 seconds. With pipelining we complete one task every 3 seconds. From the perspective of a single task, it still takes 9 seconds for our system to output a response from the time it arrives in stage 1 to the time it exits stage 3. However, we can service more requests per unit of time. Pipelining improves our system’s throughput.
Notice, pipeline parallelism is most effective when each stage can run in parallel. Thus, it is important that stages use separate resources. If two stages, for example, are both competing for CPU resources, we will not see the full impact of running these stages in parallel.
Consider the unparalleled section of go code. Can we add task parallelism?
import "stats"
func main() { ...
mean, _ := stats.Mean(*s1)
median, _ := stats.Median(*s1)
p99th, _ := stats.Percentile(s2, 99)
}
One solution would resemble the implementation below. This solution uses channels to grab the results from each of the goroutines.
Task parallelism implementation with channels:
func main() {
meanCh := make(chan float64)
medianCh := make(chan float64)
p99Ch := make(chan float64)
go func(input *[]float64) {
mean, _ := stats.Mean(input)
meanCh <- mean
}(&data)
go func(input *[]float64) {
median, _:= stats.Median(input)
medianCh <- median
}(&data)
go func(input *[]float64) {
p99th, _ := stats.Percentile(input, 99)
p99Ch <- p99th
}(&data)
mean := <-meanCh
median := <-medianCh
p99 := <-p99Ch
}
Can we add data parallelism to the go code which normalizes our satellite data readings?
Serial implementation of normalizing 100,000 readings:
for i := 0; i < 100000; i++ {
data[i] = normalizeReading(data[i])
}
The example below breaks up our data into 12 batches and normalizes the data in parallel. Notice, because the goroutines work on non-overlapping sections of the input data, there is no data race!
Implementation with data parallelism and waitgroups:
batch := len(data) / 12 // Divide up data into 12 chunks, where 12 is # cores
var wg sync.WaitGroup
for start := 0; start < len(data); start += batch {
end := start + batch
if end > len(data) {
end = len(data)
}
wg.Add(1)
go func(s, e int) {
defer wg.Done()
for i := s; i < e; i++ {
data[i] = scaleReading(data[i])
}
}(start, end)
}
wg.Wait()
Finally, can we add pipeline parallelism to our entire command center code?
Implementation with pipelining via channels:
readingQueue := []string{"temp", "dist",...}
const N = 100000
for _, q := range readingQueue {
var data []float64
// Collect readings
for i := 0; i < N; i++ {
data = append(data, performReading(q))
time.Sleep(1 * time.Second)
}
// Normalize readings
for i := 0; i < len(data); i++ {
data[i] = normalizeReading(data[i])
}
// Compute stats
mean, _ := stats.Mean(data)
p99, _ := stats.Percentile(data, 99)
median, _ := stats.Median(data)
// Send to Earth
sendToEarth(q, data, mean, p99, median)
}
The below implementation uses channels between a go routine responsible for each of a 4 stage pipeline.
readingQueue := []string{"temp", "dist",...}
// Channels between stages
readCh := make(chan ReadingBatch)
statsCh := make(chan StatsResult)
// Stage 1: Reading goroutine(s)
go func() {
for _, q := range readingQueue {
var data []float64
for i := 0; i < 1000000; i++ {
data = append(data, performReading(q))
time.Sleep(1 * time.Second)
}
readCh <- ReadingBatch{q, data}
}
close(readCh)
}()
// Stage 2: Normalization goroutine
go func() {
for batch := range readCh {
for i := range batch.data {
batch.data[i] = normalizeReading(batch.data[i])
}
mean, _ := stats.Mean(batch.data)
p99, _ := stats.Percentile(batch.data, 99)
median, _ := stats.Median(batch.data)
statsCh <- StatsResult{...}
}
close(statsCh)
}()
// Stage 3: Send results to Earth
for result := range statsCh {
sendToEarth(result.q, result.data, result.mean, result.p99, result.median)
}
The implementation with pipeline parallelism will improve our system performance best cases where stages primarily use different system resources.
Another technique for coordinating goroutines is the go synchronization package’s WaitGroup primitive. A WaitGroup allows a main or coordinating goroutine to wait for all worker threads to complete or mark themselves as “done”.
var wg sync.WaitGroup; is how you make a waitgroup.
wg.Add(n) is how you add n to the “waiting” counter of the waitgroup.
wg.Done() is how you mark one worker as “done” on the waitgroup. Effectively, this reduces the “waiting” counter by 1. Oftentimes, you will see defer wg.Done() at the top of a function. The defer keyword specifies a line to run right before the function returns. This is useful so we do not have to remember to put wg.Done() before every return statement of a long function.
wg.Wait() blocks until the “waiting” counter is zero and all workers have completed.
What happens when we run the following two snippets of worker code simultaneously?
func workerA() {
ALock.Lock()
defer ALock.Unlock()
for i := 0; i < 1000; i++ {
if(A[i] == 0) {
BLock.Lock()
A[i] = B[i]
BLock.Unlock()
}
}
...
}
func workerB() {
BLock.Lock()
defer BLock.Unlock()
for i := 0; i < 1000; i++ {
if(B[i] == 0) {
ALock.Lock()
B[i] = A[i]
ALock.Unlock()
}
}
...
}
It is possible that we have the following pattern: workerA acquires Alock. workerB acquires Block. workerA tries to acquire BLock but can’t as workerB already holds the lock, and blocks. workerB tries to acquire ALock but can’t as workerA already holds the lock, and blocks. This problem is known as deadlock.
Deadlock is when two or more threads cannot progress because each is waiting for a resource the other holds.
A key feature of deadlock is that it will never be resolved– neither thread will ever give up their resource. Deadlock is defined by a circular dependency in resource requirements between who holds a resource and who requests a resource.
When running the following code we get:
func do_processing(batch int) {
log.Println(“Working on Batch: ”, batch)
}
func main() {
batch := 0
for i := 0; i< 5; i+= 1 {
batch += 1
go func() {
do_processing(batch)
}()
}
}
Upon running the code we get the following buggy output. Instead of having one call of do_processing per batch, we see something to the effect of:
Working on Batch: 5
Working on Batch: 5
Working on Batch: 5
Working on Batch: 5
Working on Batch: 5
Why is this?
Well, this is because batch is a shared variable across all threads, even as the threads start to execute. batch is in scope for all of our anonymous functions. Also, don’t forget, that threads may be scheduled any way the goruntime wishes. So what happens is:
Then, all the body code of each of the five threads executes! And we see, 5, the shared value of batch get printed out five times.
How can we fix this? We should instead pass in batch as a parameter to the anonymous function:
func do_processing(batch int) {
log.Println(“Working on Batch: ”, batch)
}
func main() {
batch := 0
for i := 0; i< 5; i+= 1 {
batch += 1
go func(batch int) {
do_processing(batch)
}(batch)
}
}
Since go is pass-by-value for integers, each thread will be started with whatever the value of batch was at the time of calling.
We notice a bug when running the below code:
func add(slice []int, val int) {
slice = append(slice, val)
log.Println(slice)
}
func main() {
slice := []int{1,2,3,4}
log.Println(slice)
add(slice, 5)
log.Println(slice)
}
For some reason, we get the output:
[1,2,3,4]
[1,2,3,4,5]
[1,2,3,4]
This is because go is pass-by-value for the slice header. Within the body of add, we reassign that copy of the slice header’s data point to a newly allocated array: [1,2,3,4,5]. However, when add returns, the original slice header in main remains unchanged, still pointing to an array of [1,2,3,4].
What the slice header in add resembles at the start of function call.
What the slice header in add resembles at the end of function call.
To address this, we must instead pass the slice header by reference, using a pointer.
func add(slice *[]int, val int) {
*slice = append(*slice, val)
log.Println(slice)
}
func main() {
slice := []int{1,2,3,4}
log.Println(slice)
add(&slice, 5)
log.Println(slice)
}
The syntax for go pointers is similar to C:
* as an operator to dereference (follow that pointer to the data object it refers to).& to get a pointer to a data object*type (.i.e. *int), to indicate the type of this expression is pointer-to-whatever-type or pointer-to-intSelect statements are useful programming structure for waiting on the result of several channels.
func main() {
…
for {
select {
case <-quit:
…
case <-ticker.C: //every 500ms
…
case <- msg_ready:
…
default:
log.Println(“Nothing to do!”)
}
}
}
However, if nothing is ready on any channel, the default case will run. If there is no default case, then the select statement will block. ((The above code causes a lot of messages to be printed!))
If more than one channel is ready, go will select at random a branch of the select statement to execute.
time.Ticker creates a ticker object with ticker.C being a channel which fires (sending a time object) every specified period.
Beware of using time.After in a select statement. Each time time.After is called, a new channel is created which will send a message after the specified time.
func main() {
…
for {
select {
case <-quit:
…
case <-time.After(500*time.Millisecond): //every 500ms
… //will never run!
default:
log.Println(“Nothing to do!”)
}
}
}
The above code will never enter the second select case! This is because we continually loop into the default case, which re-runs the time.After line of the code, which recreates the channel, effectively restarting the timer.
Instead, it is preferable to use time.Ticker in these cases.