OpenTelemetry has brought a sea change in the world of observability. The idea of the project was to standardize the instrumentation needed for generating telemetry. Teams shouldn’t need to change how they collect data if they want to try a new visualization/backend for the telemetry data. That was the vision.
When you think about observability? Do you just think of it as an insurance? Or do you think it can also drive growth for the business?
Working with many customers and users trying to get better at Observability, there are two mindsets I see:
Datadog has a huge product footprint with a sophisticated user experience, but any discussion of its usefulness must include a consideration of its significant costs. Datadog pricing is complex and has a lot of SKUs that a customer needs to understand. If you're not careful, you might end up blowing your Datadog bill.
In a recent reddit thread, I got into a conversation about justifying the cost of observability. It got to a really basic question about running a tech company: how do you know that any cost is justified? While a small number of expenses have clear and direct business values, a bunch of other costs, I would even say most costs, just aren’t that clear cut.
I’d like to write a bit about how Observability costs are significant, how these costs tend to be justified, and how precise amount a company spends on anything tends to be more subjective than you’d think. This article is not about how to reduce or control these costs, but rather how the costs are justified.
Everyone talks about the need for observability, it even comes up in board meetings. But let’s talk about the reasons you shouldn’t add observability to production.
Kubernetes and OpenTelemetry are both CNCF projects, and both are closely associated with modern microservice architecture. Despite their connection, there isn’t a single cohesive solution to monitoring your Kubernetes cluster with OpenTelemetry.
Large teams that use complex clusters in production have generally ended up building their own tools for monitoring both their infrastructure and application code. See Intuit’s talk from the recent Open Source Summit on how they built tools to easily summarize golden signals.
While reading a GitHub issue on the OpenTelemetry Collector about trying to send two versions of a metric, one with higher and one with lower cardinality, it occurred to me that we’ve never written on this blog about what is high-cardinality data exactly and how it matters to your OpenTelemetry observability.
Cloud has become the de-facto standard for new application development. Kubernetes solves many problems of modern-day cloud infrastructure. It has made microservices-based distributed software systems possible, enabling organizations to provide on-demand scaling. But at the same time, Kubernetes has also increased operational complexity.
Observability vs monitoring has been a common topic in DevOps recently. There has been a lot of debate, and I have learned a lot from them when I started my observability journey. Most literature on observability is associated with a particular product or shares a textbook definition.
Cloud computing and containerization have brought many benefits to IT systems, like speed to market and on-demand scaling. But it has also increased operational complexity. Applications built on dynamic and distributed infrastructure are challenging to operate and maintain. A robust observability framework can help application owners stay on top of their software systems.