CS-310 Lecture 19 - Distributed Computing

Outline:
0:12 Last time – Computing Platforms
2:00 Easy vs. Hard scalability problems
4:10 “Big Data” problems
5:53 Distributed computing
7:22 Example: Find the largest sales order
9:29 Maximum aggregator is trivially parallelizable
13:35 Basic features of distributed computing
14:16 Maximum aggregator is trivially parallelizable
14:17 Basic features of distributed computing
16:07 Hadoop is a distributed computing platform
19:45 MapReduce
21:29 MapReduce programming model
23:17 MapReduce example: counting words
26:54 Counting words example
29:31 Communication between many nodes
30:32 The Hadoop framework provides:
31:49 Spark improves on MapReduce/Hadoop ideas
33:15 Spark innovations
34:25 Resilient Distributed Datasets (RDDs)
37:09 Spark example: Text Search
39:35 Example: Logistic Regression (ML classification)
42:15 Traditional Parallel Computing
44:23 Recap
45:57 Architectural concepts in this class:
46:14 Class Recap

[thinkthoughtthunk]

Thank you for posting these videos. I saw your post on HN and checked them out since I'm a professional dev but don't have much experience in distributed systems, and I just wanted to say you're an excellent teacher. It took me a bit to get through them, but you did a great job of providing the background rationale behind what motivated the development of a lot of these technologies in a way that I haven't been able to find in other places. Really appreciate it.