PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run --bin t Finished dev [unoptimized + debuginfo] target(s) in 0.01s Running `target\debug\t.exe` R:100000000 r:24134332 PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run --bin t Finished dev [unoptimized + debuginfo] target(s) in 0.01s Running `target\debug\t.exe` R:100000000 r:24366177 PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run -r --bin t Finished release [optimized] target(s) in 0.01s Running `target\release\t.exe` R:100000000 r:10000004 PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run -r --bin t Finished release [optimized] target(s) in 0.01s Running `target\release\t.exe` R:100000000 r:10000004
abc@310a0ca1fb3d:~/workspace/t/src$ cargo run --bin t Finished dev [unoptimized + debuginfo] target(s) in 0.01s Running `/config/workspace/t/target/debug/t` R:100000000 r:74990565 abc@310a0ca1fb3d:~/workspace/t/src$ cargo run --bin t Finished dev [unoptimized + debuginfo] target(s) in 0.01s Running `/config/workspace/t/target/debug/t` R:100000000 r:74348213 abc@310a0ca1fb3d:~/workspace/t/src$ cargo run -r --bin t Compiling t v0.1.0 (/config/workspace/t) Finished release [optimized] target(s) in 2.51s Running `/config/workspace/t/target/release/t` R:100000000 r:10000004 abc@310a0ca1fb3d:~/workspace/t/src$ cargo run -r --bin t Finished release [optimized] target(s) in 0.01s Running `/config/workspace/t/target/release/t` R:100000000 r:10000004
PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run --bin t1 Compiling t v0.1.0 (C:\Users\rivers\Desktop\AtomicVariableDemo-master) Finished dev [unoptimized + debuginfo] target(s) in 0.59s Running `target\debug\t1.exe` PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run --bin t2 Compiling t v0.1.0 (C:\Users\rivers\Desktop\AtomicVariableDemo-master) Finished dev [unoptimized + debuginfo] target(s) in 0.52s Running `target\debug\t2.exe` PS C:\Users\rivers\Desktop\AtomicVariableDemo-master> cargo run --bin t3 Compiling t v0.1.0 (C:\Users\rivers\Desktop\AtomicVariableDemo-master) Finished dev [unoptimized + debuginfo] target(s) in 0.51s Running `target\debug\t3.exe`
而Aarch64下, 稍稍等一会(我这里跑了3分钟), 就可以看到t1, t2出现了异常值:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
abc@310a0ca1fb3d:~/workspace/t/src$ cargo run --bin t1 Compiling t v0.1.0 (/config/workspace/t) Finished dev [unoptimized + debuginfo] target(s) in 1.60s Running `/config/workspace/t/target/debug/t1` Get! V1.0=0 ^C abc@310a0ca1fb3d:~/workspace/t/src$ cargo run --bin t2 Compiling t v0.1.0 (/config/workspace/t) Finished dev [unoptimized + debuginfo] target(s) in 1.59s Running `/config/workspace/t/target/debug/t2` Get! V1.0=0 abc@310a0ca1fb3d:~/workspace/t/src$ cargo run --bin t3 Compiling t v0.1.0 (/config/workspace/t) Finished dev [unoptimized + debuginfo] target(s) in 1.61s Running `/config/workspace/t/target/debug/t3`
PS C:\Users\rivers\Desktop\example> cargo build Compiling example v0.1.0 (C:\Users\rivers\Desktop\example) error[E0277]: the trait bound `T: Clone` is not satisfied --> src\main.rs:9:10 | 9 | impl <T> MustBeClone for CloneByPtr<T> where T:MayNotClone{} | ^^^^^^^^^^^ the trait `Clone` is not implemented for `T` | note: required because of the requirements on the impl of `Clone` for `CloneByPtr<T>` --> src\main.rs:4:10 | 4 | #[derive(Clone)] | ^^^^^ note: required by a bound in `MustBeClone` --> src\main.rs:8:20 | 8 | trait MustBeClone: Clone {} | ^^^^^ required by this bound in `MustBeClone` = note: this error originates in the derive macro `Clone` (in Nightly builds, run with -Z macro-backtrace for more info) help: consider further restricting this bound | 9 | impl <T> MustBeClone for CloneByPtr<T> where T:MayNotClone + std::clone::Clone{} | +++++++++++++++++++
For more information about this error, try `rustc --explain E0277`. error: could not compile `example` due to previous error
In this lab you’ll build a MapReduce system. You’ll implement a worker process that calls application Map and Reduce functions and handles reading and writing files, and a coordinator process that hands out tasks to workers and copes with failed workers. You’ll be building something similar to the MapReduce paper. (Note: the lab uses “coordinator” instead of the paper’s “master”.)
You’ll fetch the initial lab software with git (a version control system). To learn more about git, look at the Pro Git book or the git user’s manual. To fetch the 6.824 lab software:
通过git拉取代码,如下:
1 2 3 4 5
$ git clone git://g.csail.mit.edu/6.824-golabs-2021 6.824 $ cd 6.824 $ ls Makefile src $
We supply you with a simple sequential mapreduce implementation in src/main/mrsequential.go. It runs the maps and reduces one at a time, in a single process. We also provide you with a couple of MapReduce applications: word-count in mrapps/wc.go, and a text indexer in mrapps/indexer.go. You can run word count sequentially as follows:
$ cd ~/6.824 $ cd src/main $ go build -race -buildmode=plugin ../mrapps/wc.go $ rm mr-out* $ go run -race mrsequential.go wc.so pg*.txt $ more mr-out-0 A 509 ABOUT 2 ACT 8 ...
Your job is to implement a distributed MapReduce, consisting of two programs, the coordinator and the worker. There will be just one coordinator process, and one or more worker processes executing in parallel. In a real system the workers would run on a bunch of different machines, but for this lab you’ll run them all on a single machine. The workers will talk to the coordinator via RPC. Each worker process will ask the coordinator for a task, read the task’s input from one or more files, execute the task, and write the task’s output to one or more files. The coordinator should notice if a worker hasn’t completed its task in a reasonable amount of time (for this lab, use ten seconds), and give the same task to a different worker.
We have given you a little code to start you off. The “main” routines for the coordinator and worker are in main/mrcoordinator.go and main/mrworker.go; don’t change these files. You should put your implementation in mr/coordinator.go, mr/worker.go, and mr/rpc.go.
Here’s how to run your code on the word-count MapReduce application. First, make sure the word-count plugin is freshly built:
以下,介绍如何使用你的代码运行word count.首先,确保word-count是最新构建出的:
1
$ go build -race -buildmode=plugin ../mrapps/wc.go
In the main directory, run the coordinator. 在main目录中,运行Coordinator程序
1 2
$ rm mr-out* $ go run -race mrcoordinator.go pg-*.txt
The pg-*.txt arguments to mrcoordinator.go are the input files; each file corresponds to one “split”, and is the input to one Map task. The -race flags runs go with its race detector.
When the workers and coordinator have finished, look at the output in mr-out-*. When you’ve completed the lab, the sorted union of the output files should match the sequential output, like this:
$ cat mr-out-* | sort | more A 509 ABOUT 2 ACT 8 ...
We supply you with a test script in main/test-mr.sh. The tests check that the wc and indexer MapReduce applications produce the correct output when given the pg-xxx.txt files as input. The tests also check that your implementation runs the Map and Reduce tasks in parallel, and that your implementation recovers from workers that crash while running tasks.
$ bash test-mr.sh *** Starting wctest. sort: No such file or directory cmp: EOF on mr-wc-all --- wc output is not the same as mr-correct-wc.txt --- wctest: FAIL $
The test script expects to see output in files named mr-out-X, one for each reduce task. The empty implementations of mr/coordinator.go and mr/worker.go don’t produce those files (or do much of anything else), so the test fails.
You’ll also see some errors from the Go RPC package that look like
你会在Go PRC包中看到类似于下面的报错:
1
2019/12/16 13:27:09 rpc.Register: method "Done" has 1 input parameters; needs exactly three
Ignore these messages; registering the coordinator as an RPC server checks if all its methods are suitable for RPCs (have 3 inputs); we know that Done is not called via RPC.
The map phase should divide the intermediate keys into buckets for nReduce reduce tasks, where nReduce is the argument that main/mrcoordinator.go passes to MakeCoordinator().
The worker implementation should put the output of the X’th reduce task in the file mr-out-X.
worker实现应该将第X个reduce任务的输出放在mr-out-X文件中
A mr-out-X file should contain one line per Reduce function output. The line should be generated with the Go “%v %v” format, called with the key and value. Have a look in main/mrsequential.go for the line commented “this is the correct format”. The test script will fail if your implementation deviates too much from this format.
You can modify mr/worker.go, mr/coordinator.go, and mr/rpc.go. You can temporarily modify other files for testing, but make sure your code works with the original versions; we’ll test with the original versions.
main/mrcoordinator.go expects mr/coordinator.go to implement a Done() method that returns true when the MapReduce job is completely finished; at that point,mrcoordinator.go will exit.
When the job is completely finished, the worker processes should exit. A simple way to implement this is to use the return value from call(): if the worker fails to contact the coordinator, it can assume that the coordinator has exited because the job is done, and so the worker can terminate too. Depending on your design, you might also find it helpful to have a “please exit” pseudo-task that the coordinator can give to workers.
One way to get started is to modify mr/worker.go’s Worker() to send an RPC to the coordinator asking for a task. Then modify the coordinator to respond with the file name of an as-yet-unstarted map task. Then modify the worker to read that file and call the application Map function, as in mrsequential.go.
If you change anything in the mr/ directory, you will probably have to re-build any MapReduce plugins you use, with something like go build -race -buildmode=plugin ../mrapps/wc.go
This lab relies on the workers sharing a file system. That’s straightforward when all workers run on the same machine, but would require a global filesystem like GFS if the workers ran on different machines.
The worker’s map task code will need a way to store intermediate key/value pairs in files in a way that can be correctly read back during reduce tasks. One possibility is to use Go’s encoding/json package. To write key/value pairs to a JSON file:
You can steal some code from mrsequential.go for reading Map input files, for sorting intermedate key/value pairs between the Map and Reduce, and for storing Reduce output in files.
Workers will sometimes need to wait, e.g. reduces can’t start until the last map has finished. One possibility is for workers to periodically ask the coordinator for work, sleeping with time.Sleep() between each request. Another possibility is for the relevant RPC handler in the coordinator to have a loop that waits, either with time.Sleep() or sync.Cond. Go runs the handler for each RPC in its own thread, so the fact that one handler is waiting won’t prevent the coordinator from processing other RPCs.
The coordinator can’t reliably distinguish between crashed workers, workers that are alive but have stalled for some reason, and workers that are executing but too slowly to be useful. The best you can do is have the coordinator wait for some amount of time, and then give up and re-issue the task to a different worker. For this lab, have the coordinator wait for ten seconds; after that the coordinator should assume the worker has died (of course, it might not have).
If you choose to implement Backup Tasks (Section 3.6), note that we test that your code doesn’t schedule extraneous tasks when workers execute tasks without crashing. Backup tasks should only be scheduled after some relatively long period of time (e.g., 10s).
To ensure that nobody observes partially written files in the presence of crashes, the MapReduce paper mentions the trick of using a temporary file and atomically renaming it once it is completely written. You can use ioutil.TempFile to create a temporary file and os.Rename to atomically rename it.
test-mr.sh runs all the processes in the sub-directory mr-tmp, so if something goes wrong and you want to look at intermediate or output files, look there. You can modify test-mr.sh to exit after the failing test, so the script does not continue testing (and overwrite the output files).
test-mr-many.sh provides a bare-bones script for running test-mr.sh with a timeout (which is how we’ll test your code). It takes as an argument the number of times to run the tests. You should not run several test-mr.sh instances in parallel because the coordinator will reuse the same socket, causing conflicts.