Week 5, Day 3 (Friday, February 7)

In class—sharding and replication

Suppose our image processing application (Assignment 2) included its own image storage function rather than using Amazon S3. Some of the images on our site might become very popular (say, a striking photograph of a celebrity). When an image is popular, we want to retrieve it with low latency. We also want to ensure our images are retained even if there are disk crashes. (Recall that many data centers intentionally use disks with relatively low reliabilities.)

How would you use the approaches of sharding and replication (from Week 2 Wednesday) to decrease the latency of your image retrievals? For your sharding, how might you group images?

Implementation (from You Are Building a Distributed System)

Build versus buy is not a simple choice

Testing is difficult

• Transient bugs are common
• Total network outage is rare—degraded performance is common
• Drive the system at expected load
• Induce failures (more on this in Week 8)

Operating the system requires careful monitoring (Week 9)

• If it moves, measure it
• Need both real-time and analytic data
• You will produce lots of data—which might require its own highly scalable, reliable system to manage

Anonymous poll

On an unsigned sheet of paper, briefly complete:

1. I most want to see the class start doing …
2. I most want to see the class continue doing …
3. I most want to see the class stop doing …

No readings over Reading Break

Work on Assignment 3 and review the previous readings.