In the world of software development, observability has become a crucial concept, especially when dealing with complex systems. At QCon London 2026, Colin Douch, a site reliability engineer at DuckDuckGo, delved into the intricacies of self-hosted observability, offering a comprehensive guide to navigating this complex terrain. Douch's presentation was a thought-provoking exploration of the challenges and opportunities that come with building and maintaining your own observability stack.
The Complexity Demon
Douch began by addressing a familiar challenge faced by many developers: the ever-looming complexity demon. He argued that while we often develop hacks to escape this demon, it's essential to confront it head-on. Observability, he emphasized, is the tool to help us debug and understand complex systems. However, he also highlighted a paradox: the very observability stack we rely on to simplify debugging can become complex in itself.
Outsourcing vs. Self-Hosting
One of the key decisions organizations face is whether to outsource observability to SaaS vendors or run their own stack internally. Douch advised caution, suggesting that self-hosting should be considered only after exhausting all other options. He outlined the significant challenges, including the need for additional engineers and substantial financial investment. However, he also acknowledged the benefits of self-hosting, such as greater control and customization.
Building the Self-Hosted Stack
Douch provided a roadmap for constructing a self-hosted observability stack. He recommended Prometheus or VictoriaMetrics for metrics collection, praising their capabilities despite scaling challenges. For logs, he suggested VictoriaLogs or Loki, emphasizing the importance of structured storage in a columnar database. Douch also warned against the pitfalls of log ingestion, advocating for a structured approach to avoid data soup.
The Ecosystem and Modular Approach
The observability landscape is characterized by a modular ecosystem of open-source tools. Douch discussed the benefits of this modularity but also highlighted the operational overhead it introduces. He reviewed the current ecosystem, endorsing OpenTelemetry for traces, while recommending Prometheus Text Exposition and JSON for metrics and logs. Douch's perspective was that connecting logs, metrics, and traces is the true value of observability, rather than treating them as independent silos.
Designing the Telemetry Pipeline
A key takeaway from Douch's presentation was the importance of designing a coherent telemetry pipeline rather than simply selecting tools. He emphasized that observability is less about individual tools and more about the seamless flow of data between them. This approach ensures that logs, metrics, and traces are not isolated but rather interconnected, providing a holistic view of system health.
Personal Reflection
In my opinion, Douch's presentation was a refreshing take on observability, offering a balanced perspective on self-hosting. While he acknowledged the challenges, he also highlighted the benefits and opportunities. His emphasis on designing a coherent telemetry pipeline was particularly insightful, as it challenges the notion of observability as a collection of independent tools. Personally, I believe this approach can lead to more efficient and effective observability practices, especially in large-scale systems.
Broader Implications
Douch's talk raises important questions about the future of observability. As systems become more complex, the need for robust observability solutions will only grow. The choice between self-hosting and outsourcing will continue to be a critical decision for organizations, with each option presenting its own set of advantages and trade-offs. The modular ecosystem of open-source tools will likely continue to evolve, offering new possibilities for customization and integration.
Takeaway
In conclusion, Colin Douch's presentation at QCon London 2026 provided a comprehensive and thought-provoking guide to self-hosted observability. His emphasis on designing a coherent telemetry pipeline and his balanced perspective on self-hosting offer valuable insights for developers and organizations navigating the complex world of observability. As the landscape continues to evolve, Douch's insights will undoubtedly remain relevant, shaping the future of observability practices.
