Survivability of observability: Taking your DIY system to success

The goal of this talk is to provide a long view of making your DIY observability program a success.

Far more programs get started than end up flourishing in the long term. A not-so-uncommon story: initially, everyone loves your solution, your team receives extra headcount and budget, and you're busy evangelizing and training your organization on how to use it. But before you know it, the dreaded outages begin, all these new users start demanding more and more features, and overall satisfaction with the toolset starts to decline. You either limp along or start exploring commercial options. How did this happen?

Based on firsthand experience growing DIY and commercial observability systems from scratch to supporting thousands of users plus hundreds of thousands of servers, we'll share some reasons and winning strategies. We'll also discuss the observability maturity curve, how and why requirements evolve, usability, and integrations needed for a coherent toolset, monitoring-ops ("observability"), and challenges that come with rapid growth.