#111 Impossible problems
Two decades ago, a new generation of software engineers discovered that the Internet created the potential for explosive growth. The sheer scale of what might be possible inspired these engineers to rethink and reinvent new ways to approach the really tough problems presented by massive scale. When it became impossible to get any bigger, they figured out how to get small things to work at scale. When data grew so large that it no longer could be fit onto one computer, they figured out how to distribute data effectively. When release cycles grew from months to years, they worked out a way to safely deliver software continuously. When complexity threatened to bring down every large system, they figured out how to make internet-scale systems more reliable than ever.
Necessity is the mother of invention. When things grow incrementally, you can expand current techniques incrementally. But when size explodes or market forces take a sudden unexpected turn, the incremental approach doesn’t work. When problems are so difficult that none of the existing paradigms will work, the stage is set for innovation.