The Hidden Economics of Observability Cost Growth

A platform team at a mid-sized SaaS company runs a straightforward exercise every quarter: plot revenue, plot infrastructure spend, plot the observability vendor invoice, all indexed to the same starting point. Infrastructure spend tracks revenue reasonably well — new customers need more compute, more storage, more database capacity, and the line rises roughly in proportion. The observability line does not behave the same way. It rises in steps, often triggered by nothing anyone would call a business event. A new microservice ships with default instrumentation. A support incident prompts someone to add a customer-ID tag to a metric "just for this investigation." A team turns on debug-level logging to chase a bug and forgets to turn it back down. None of these are usage growth. All of them show up on the same invoice as usage growth, indistinguishable from it unless someone goes looking.

This is not a story about vendors overcharging, and it is not a story about observability being a wasteful category of spend that should be cut. Visibility into a running system is one of the more defensible categories of engineering investment there is — the alternative to paying for observability is not paying nothing, it is finding out about problems later, more expensively, from customers instead of dashboards. The actual problem is narrower and more tractable: the unit economics of telemetry are structurally different from the unit economics of the systems telemetry describes, and almost no engineering organization treats instrumentation as a decision with a cost attached, in the moment the decision is made. A database schema change gets reviewed. An API contract change gets reviewed. A new metric with an unbounded label, added in a five-line pull request, typically does not — even though, as this article will show with real vendor pricing, it can cost more per month than the database change it was added to observe.

The purpose of this analysis is to make that cost visible and specific: to explain exactly which instrumentation decisions drive observability spend, why they interact multiplicatively rather than additively, how the major pricing models (per-host, per-GB-ingested, per-metric-series, per-indexed-event) translate instrumentation choices into invoice line items, and what a cost-aware instrumentation discipline actually looks like in practice — not as an argument for instrumenting less, but as an argument for instrumenting deliberately.

Why Observability Cost Doesn't Scale With the System It Watches

Start with a distinction that is easy to state and consistently ignored in practice: telemetry volume is a function of code paths and labeling decisions, not a function of user-facing load. A service that handles ten times more traffic without any instrumentation changes will generate roughly ten times more log lines, roughly the same number of distinct metric series (assuming label cardinality doesn't change), and roughly the same trace structure, just sampled from a larger population. That is proportional growth, and it is exactly the kind of cost growth a finance team expects and budgets for.

What actually drives disproportionate growth is a different category of change entirely: decisions that alter the shape of telemetry rather than its volume at constant shape. Adding a label to a metric doesn't add data linearly — it multiplies the number of distinct time series that label's metric produces, because most observability backends store one time series per unique combination of metric name and label values. Turning on debug logging doesn't add a constant amount of log volume — it adds volume proportional to every request the debug statements sit on, which in a hot code path can dwarf everything else the service logs combined. Raising a trace sampling rate from 1% to 10% doesn't cost 10% more — for backends that charge per ingested span or per GB, it costs roughly ten times more, applied to every trace in the system, whether or not that trace turned out to be interesting.

The mechanism common to all three examples is that observability pricing is built around dimensions that scale combinatorially with instrumentation choices, not dimensions that scale linearly with business activity. This is the core economic fact this article is built around, and it explains a pattern most engineering leaders have felt but rarely named precisely: the bill that grows in discrete jumps tied to code changes, not smooth curves tied to customer growth.

The Four Cost Levers, and Why They Compound

Four instrumentation decisions drive the overwhelming majority of avoidable observability cost, and they are worth naming individually before looking at how vendors price around them.

Metric cardinality — the number of unique label-value combinations a metric produces. This is the single highest-leverage lever in the entire cost structure, because pricing models built around "distinct time series" or "custom metrics" charge per combination, not per metric name, and combinations multiply.

Log volume at a given verbosity level — how much text a service emits per request, and at what severity threshold. This is the most familiar lever and the easiest to reason about, but also the easiest to leave misconfigured indefinitely, because nothing forces a revisit once an incident that justified verbose logging is closed.

Trace sampling rate and strategy — what fraction of distributed traces get kept, and whether that decision is made before or after a trace's outcome is known. This lever interacts with both ingestion-based and indexed-event-based pricing, and it is the lever with the most nuanced trade-off, because sampling too aggressively can hide exactly the rare, expensive failures an organization most needs visibility into.

Retention period — how long ingested data stays queryable at full fidelity before being deleted, downsampled, or moved to cheaper storage. Retention interacts multiplicatively with every other lever: a cardinality problem retained for 13 months instead of 15 days costs roughly 26 times more to store, even though the underlying instrumentation decision never changed.

These four levers do not act independently. A new metric with a customer_id label (a cardinality decision) that is also retained at full resolution for a year (a retention decision) compounds two multipliers against each other. A debug log level (a volume decision) that also gets fully indexed rather than routed to cold storage (effectively a retention/tier decision) compounds two more. This compounding is why observability bills are notoriously difficult to predict from a simple "traffic times average cost" model, and why the mitigation strategies later in this article target each lever individually rather than offering one universal fix.

How Metric Cardinality Actually Multiplies Cost

Cardinality is worth explaining mechanically before discussing cost, because the cost consequence only makes sense once the underlying data model is clear.

A metric like http_requests_total is not, to a time-series backend, a single thing. It is a family of time series, one for every unique combination of label values attached to it. Prometheus's own documentation states this directly: "every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored," and it explicitly warns against using labels for "dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values" — guidance that predates the current wave of cost-driven attention to the topic and reflects a storage-engine constraint, not a pricing artifact. Prometheus — Instrumentation: Metric and Label Naming

Concretely: a metric with a route label (say, 40 distinct API routes) and a status_code label (say, 6 distinct values grouped into classes) produces at most 240 time series — bounded, predictable, and cheap. Add a method label with 4 values and the ceiling rises to 960 — still bounded. Add a customer_id label, and the ceiling disappears. If the platform serves 50,000 customers, that one metric can produce up to 50,000 × 240 = 12,000,000 time series, and every one of those series is stored, indexed, and — depending on the vendor's pricing model — billed independently, whether or not most of those customers generate meaningful traffic.

This is not a hypothetical failure mode; it is the specific mechanism OpenTelemetry's own project documentation was updated to address directly. OpenTelemetry's metrics SDK now enforces a default limit of 2,000 unique attribute combinations per metric stream specifically because, as the project's engineering blog explains, "including raw URLs, user IDs, session tokens, or request identifiers as attributes can rapidly multiply combinations from hundreds into millions," and unbounded cardinality growth "protects your process from unbounded memory growth" at the SDK level, before the data even reaches a backend. OpenTelemetry Blog — Metric cardinality limits in OpenTelemetry: a practical guide

That same OpenTelemetry guidance is worth reading carefully for what it says about the failure mode of hitting a cardinality limit, because it is not simply "the extra data gets dropped." When a metric stream exceeds its cardinality limit, the SDK does not drop the excess measurements — it folds them into a single overflow data point marked otel.metric.overflow=true. The aggregate total remains correct, but any query that filters or groups by the attribute that caused the overflow will silently undercount, because the original attribute values on the overflowed measurements are gone. This matters economically as well as operationally: a team that raises cardinality limits reflexively to "fix" overflow warnings, without first asking whether the attribute belongs on the metric at all, is choosing to pay for the cardinality rather than eliminating the reason it exists.

How Vendors Actually Bill for Cardinality

The mechanism above explains why cardinality matters technically. What makes it an economics problem rather than a purely technical one is that every major commercial observability vendor bills, directly or indirectly, on the number of distinct time series a customer's metrics produce.

Datadog's own billing documentation is explicit about the unit of account: "Your monthly billable custom metrics usage is calculated by taking the total of all distinct custom metrics for each hour in a given month, and dividing it by the number of hours in the month," where a "distinct custom metric" is defined as a unique combination of metric name and tag values, including the host tag. A request.latency metric submitted with tags for endpoint and status can generate as few as four billable time series or as many as tens of thousands, depending entirely on how many unique combinations those tags actually take on in practice — the metric name is the same in both cases, but the bill is not. Datadog — Custom Metrics Billing

Grafana Cloud bills even more directly on the same unit: its Pro plan pricing is stated as a dollar rate "per 1,000 active series," with a base allocation and pay-as-you-go pricing beyond it, and Enterprise agreements priced per-series at volume. Grafana Labs — Pricing There is no ambiguity in this model about what a cardinality decision costs — the price is quoted directly in the unit that cardinality inflates.

New Relic and Honeycomb price primarily on ingested data volume (per GB) rather than per distinct series, which changes the mechanism but not the outcome: a high-cardinality metric with many unique combinations produces more distinct data points to transmit and store per collection interval, so it still inflates a GB-based bill, just through volume rather than through a distinct-series line item. New Relic's public pricing confirms a straightforward per-GB model above its included allowance — $0.40 per GB ingested beyond the free 100 GB/month tier on its standard data option, or $0.60/GB on its extended-governance "Data Plus" option. New Relic — Pricing Honeycomb, whose product is built specifically around high-cardinality, wide-event data as a design philosophy rather than a discouraged edge case, still ultimately prices on ingested event volume, meaning a team that adopts wide, high-cardinality events without any cost discipline around which attributes get attached is trading a metrics-cardinality cost problem for an events-volume cost problem, not eliminating the trade-off. Honeycomb's own documentation on the subject treats high cardinality as a deliberate architectural choice with real value for debugging, not a mistake to be avoided outright — but a choice with a cost, made consciously. Honeycomb Docs — High Cardinality

The practical implication of comparing these models side by side is that the specific pricing mechanism changes which mitigation technique pays off first, a point developed further in the mitigation section below: a per-series model (Grafana Cloud) rewards eliminating unnecessary label combinations directly; a per-GB model (New Relic, Honeycomb) rewards reducing overall payload size and event frequency in addition to cardinality; a hybrid model with separate ingestion and indexing costs (Datadog) rewards decoupling "collect everything" from "index and query everything," which is precisely what Datadog's own "Metrics without Limits" feature is built to do — ingest broadly, index a curated allowlist of queryable tag combinations, and charge the higher indexing rate only on what's actually indexed.

An Illustrative Cost-Scaling Scenario: One Label, Compounding

The following scenario uses hypothetical, illustrative numbers to demonstrate the arithmetic of cardinality growth. The figures are constructed for this article, not drawn from any real customer, vendor benchmark, or measured incident. The per-series and per-GB rates used are approximations based on the publicly documented Datadog and Grafana Cloud pricing structures cited above, simplified for clarity of illustration — actual contracted rates vary by volume commitment and plan tier.

Consider a payments platform instrumenting a single metric, payment_processing_duration_seconds, as a histogram, with three labels already in place: payment_method (6 values: card, ACH, wire, wallet, buy-now-pay-later, crypto), status (3 values: succeeded, failed, pending), and region (4 values). That's a ceiling of 6 × 3 × 4 = 72 unique label combinations — small, predictable, and cheap by any pricing model.

An engineer investigating a support ticket adds a merchant_id label to help isolate which merchant's transactions are slow, intending it as temporary. The platform serves 8,000 active merchants. The metric's ceiling jumps from 72 combinations to 8,000 × 72 = 576,000.

The table below walks through what happens to estimated monthly cost under each of three illustrative billing approximations, holding the platform's actual transaction volume constant — the only thing that changed is the label, not the business.

What this scenario demonstrates: a single label decision, made by one engineer for a legitimate short-term debugging need, can move a metric's cost from effectively free to a four-figure-a-month line item without any change in the platform's actual transaction volume, and a second, independently reasonable-sounding decision (retention extension for compliance) compounds against the first rather than adding to it. Neither decision was unreasonable in isolation. Neither decision, in most organizations, went through any review process that would have surfaced the interaction before it reached a bill. This is the mechanism, not an edge case — it is the same mechanism the OpenTelemetry cardinality-limits documentation describes as the standard cause of unexpected memory and cost growth, just expressed in dollar terms.

A simplified text view of the compounding pattern:

Baseline metric:           72 series   ████
+ merchant_id label:    576,000 series ████████████████████████████████████████ (8,000x)
+ 13-month retention:   576,000 series, ~3x storage multiplier on top
+ request_id label:     effectively unbounded — SDK overflow / backend rejection

Log Volume: The Cost Lever Everyone Recognizes but Rarely Governs

If cardinality is the least visible cost lever, log volume is the most visible and still, in most organizations, the least governed. Every engineer knows debug logging is verbose. Very few organizations have a mechanism that reliably reverts a debug-level change once the incident that justified it is closed.

The mechanism is straightforward to state: log ingestion pricing, across every major vendor, is charged by volume — bytes or events ingested, scanned, or indexed — not by how useful the resulting logs turn out to be. Datadog's log pricing is a clean illustration of a multi-stage model: ingestion is billed at a flat rate per GB regardless of what happens to the data afterward ($0.10 per ingested or scanned GB per month, per its public pricing page), while real-time indexing for search and alerting is billed separately and far more expensively, per million log events rather than per GB, and historical "Flex Storage" for logs that don't need real-time search sits at a fraction of the indexing cost. Datadog — Pricing The economic lesson embedded in that structure is explicit, even if Datadog doesn't phrase it this way: not all logs deserve the same treatment, and a flat "log everything at INFO, index everything" policy pays the most expensive rate for data that mostly doesn't need it.

Datadog's own engineering guidance on high-volume log optimization recommends a specific, layered set of techniques rather than a single fix: filtering and normalizing at the edge (dropping redundant metadata and null fields before data leaves the environment), disciplined log-level configuration in production (capturing warnings, errors, and critical events while suppressing verbose debug output), sampling successful/routine transactions while retaining full detail on failures, generating metrics directly from high-volume logs (such as CDN or WAF logs) instead of storing every raw line, and routing low-priority logs to cheap archive storage while reserving expensive hot-tier indexing for logs that are actually queried in practice. Datadog Blog — How to optimize high-volume log data without compromising visibility None of these techniques is exotic. What makes them uncommon in practice is that none of them happens automatically — each requires someone to have decided, deliberately, that a given log stream doesn't need the most expensive treatment by default.

The Honeycomb team's analysis of observability cost structure makes an adjacent and important point about why uncontrolled log growth compounds badly for reasons beyond the direct bill: in a fragmented observability stack that separately stores metrics, logs, traces, APM data, and real-user-monitoring data, a single logical request can generate telemetry that is stored, in effect, five separate times across five separate tools, each with its own ingestion cost — what the article characterizes, attributing the framing to Honeycomb co-founder Charity Majors, as a "5x multiplier" effect on the cost of observing a single unit of application behavior. The article's sharper point is about diminishing returns rather than the multiplier alone: "as your logging bill goes up, the value goes down," because higher-volume, lower-signal logs make searches slower and correlation across tools harder, so a team can be paying more for less usable visibility, not more. Honeycomb Blog — The Cost Crisis in Observability Tooling

The Specific Failure Pattern: Incident Logging That Never Gets Reverted

The most common version of uncontrolled log growth is not a deliberate over-logging policy — it is a temporary change made under incident pressure that outlives the incident. An engineer investigating an intermittent bug raises a service's log level from info to debug, and separately adds full request-and-response body logging to the specific endpoint under suspicion, in order to capture enough detail to reproduce the problem. The incident resolves within a day or two. The logging changes are not reverted, because reverting a debug-logging change is rarely part of a team's incident-closure checklist, and no automated check flags elevated log volume as something requiring sign-off the way a schema migration or a dependency upgrade would.

For a service handling meaningful request volume, the effect compounds quietly rather than dramatically: full request/response body logging at debug level can increase the average logged payload per request by an order of magnitude, and because the increase applies against a baseline that is already growing with legitimate customer volume, it does not present as the kind of sharp step-change spike that a cost anomaly alert is tuned to catch. It shows up, months later, as a gradually elevated baseline that nobody can point to a specific cause for — which is precisely the failure mode a deliberate log-level governance process, discussed later in this article, is built to prevent.

Trace Sampling: Head, Tail, and the Cost Trade-off Neither One Fully Solves

Distributed tracing has the sharpest version of the cost-versus-visibility trade-off of any telemetry signal, because a trace's value is concentrated disproportionately in a small number of unusual traces — the ones with errors, the ones with unusually high latency, the ones exercising a rare code path — while a trace's cost is roughly uniform per trace regardless of how interesting it turns out to be. This mismatch is exactly why sampling strategy matters more for traces than for metrics or logs, and why the two dominant sampling approaches make fundamentally different trade-offs.

OpenTelemetry's own documentation frames the underlying rationale plainly: sampling is "one of the most effective ways to reduce the costs of observability without losing visibility," built on the principle that "the more data you generate, the less data you actually need to have a representative sample" — a smaller, well-chosen dataset can represent a much larger population without materially degrading the insight available from it. OpenTelemetry — Sampling

Head sampling makes the keep-or-discard decision at the beginning of a trace — typically at the root span, often using a simple probabilistic rule (keep 1 in every N traces) applied before anything about the trace's outcome is known. Its defining property is that the decision is cheap and can be made independently at any service in a distributed system without coordination, because the outcome of the sampling decision doesn't depend on anything that hasn't happened yet.

Tail sampling defers the decision until after all spans in a trace have completed, which means the sampling policy can key on the trace's actual outcome — keep every trace containing an error, keep every trace exceeding a latency threshold, keep a small probabilistic sample of everything else. OpenTelemetry's engineering blog on the subject is direct about the appeal: tail sampling lets a team "see only the traces that are of interest to you" and "lower data ingest and storage costs because you're only exporting a predetermined subset of your traces," selected by outcome rather than by chance. OpenTelemetry Blog — Tail Sampling with OpenTelemetry

The same source is equally direct about tail sampling's costs, which are frequently underestimated by teams adopting it purely for its cost-reduction promise:

• Memory and buffering overhead. A tail-sampling collector must hold every span of a trace in memory until a decision is made, governed by a decision_wait window (commonly defaulting to around 30 seconds) and a maximum number of buffered traces — meaning the collector infrastructure itself has real, ongoing resource cost that a head-sampling setup doesn't need.
• A scalability constraint that shapes deployment topology. All spans belonging to one trace must arrive at the same collector instance to be evaluated together, which prevents naive horizontal scaling of the collector tier and typically requires a load-balancing layer specifically designed to route by trace ID — an architectural cost, not just a runtime one.
• Loss of statistical extrapolation. Because OpenTelemetry doesn't propagate sampling-probability metadata end-to-end by default, backend systems that try to extrapolate from a sampled trace count back to a true total (for example, to estimate real error rates or real request volume from sampled data) lose accuracy, since the sampling wasn't uniform across trace types.
• Trace fragmentation risk. Spans that arrive after the decision window has already closed can be dropped or orphaned, producing incomplete traces for the exact long-running or delayed operations that the extra buffering was arguably trying to capture.

The practical conclusion is not that tail sampling is superior to head sampling, or the reverse — it is that the two solve different problems, and the right choice depends on what visibility actually matters for a given service. A payments API where every failed transaction needs a complete trace, but where the overwhelming majority of successful transactions look identical to each other, is a strong candidate for tail sampling despite its infrastructure cost, because the value of the data being kept is highly concentrated in a small, identifiable subset. A high-throughput, relatively uniform internal service where any given trace is about as informative as any other is a better candidate for head sampling, because tail sampling's buffering overhead buys little additional insight when outcomes are not the differentiator. Many mature observability practices run both: head sampling as a coarse, cheap default across most services, layered with tail-sampling policies specifically for the small number of services where outcome-based selection meaningfully changes what gets kept.

Retention: The Multiplier That Compounds Every Other Decision

Retention period is the least glamorous of the four cost levers and arguably the most consequential, precisely because it applies as a multiplier against whatever cardinality, volume, and sampling decisions already exist, rather than being a decision made in isolation.

Grafana Cloud's published pricing structure makes this multiplicative relationship explicit in its rate card rather than leaving it implicit: logs and traces are billed separately for processing, writing, and retaining, at roughly $0.05/GB to process, $0.40/GB to write, and $0.10/GB per month to retain, on its Pro plan. Grafana Labs — Pricing The retention component is a recurring monthly charge for every GB kept in queryable storage, which means a decision to retain a high-volume log stream for 90 days instead of 14 does not cost roughly 6.4x more once — it costs that multiple every month, indefinitely, until someone revisits the retention policy specifically.

The mistake this most commonly produces is treating retention as a single, organization-wide setting rather than a decision that should vary by data type and by how the data is actually used. Debug-level application logs rarely have investigative value more than a few weeks after they were written, because incident investigation happens close to the incident. Audit logs and compliance-relevant records may need to be retained for months or years, but for evidentiary purposes rather than for interactive querying, which makes them strong candidates for cheap cold storage rather than expensive hot-tier indexing — the same logical distinction Datadog draws between its real-time indexing tier and its Flex Storage tier for historical data, priced at a fraction of the indexing rate specifically because it trades query latency for storage cost. High-cardinality metrics used for real-time alerting need only as much full-resolution retention as the alerting window requires; long-term capacity-planning trends can be served by downsampled, aggregated data retained far longer at a fraction of the storage footprint of the raw series.

An Illustrative Chart: What Retention Period Alone Does to Storage Cost

Assume a service ingests a steady 40 GB of logs per day — roughly 1,200 GB per month — and that volume does not change. The only variable in the chart below is how long that data stays in queryable, full-fidelity storage before being deleted or moved to a cheaper cold tier. Because retention cost accrues per GB per month it is stored, not per GB ingested once, the total footprint under management at any given time grows in proportion to the retention window, even though the daily ingestion rate never changes.

A simple text representation of the same growth:

14 days:   $56    █
30 days:   $120   ██
90 days:   $360   ██████
365 days:  $1,460 █████████████████████████

What this chart demonstrates: unlike a cardinality decision, which multiplies cost through the number of distinct series, a retention decision multiplies cost through time, and it does so against whatever ingestion volume already exists — meaning a retention extension made for an unrelated reason (a compliance request, an auditor's preference, an engineer's instinct to "keep it just in case") compounds against every cardinality and volume decision already baked into that data stream, not against some smaller, isolated baseline. This is precisely why the earlier cardinality scenario treated a retention change as a second, independent multiplier rather than a separate, unrelated cost: the two levers stack multiplicatively whenever they apply to the same underlying data.

The practical implication is that retention deserves the same "does this specific need justify this specific cost" scrutiny as a cardinality decision, and it deserves that scrutiny applied per data type rather than as a single global setting. Extending retention for an entire log stream because one field in it has occasional compliance relevance is the retention equivalent of adding an unbounded label to an entire metric because one investigation needed per-customer detail — the fix in both cases is to scope the expensive treatment to the specific data that actually needs it, not to apply it uniformly to everything nearby.

A Comparison of How Four Major Platforms Price Telemetry

The pricing mechanics described individually above are easiest to reason about side by side. The table below summarizes the primary billing dimension each platform uses for each signal type, based on each vendor's own published pricing documentation as of this writing. Contracted enterprise rates vary and are not reflected here; this is a comparison of billing mechanism, not a claim about which vendor is cheapest for any specific workload.

Sources: Datadog Pricing, Datadog Custom Metrics Billing, Grafana Labs Pricing, New Relic Pricing, Honeycomb — High Cardinality. Rates reflect each vendor's publicly listed self-service pricing at the time of research and are vendor-reported figures, not independently audited; enterprise and committed-use pricing typically differs by negotiated contract.

The pattern worth taking away from this comparison is not "which vendor is cheapest" — that depends entirely on a specific workload's signal mix — but that the billing unit shapes which engineering behavior gets rewarded. A per-series pricing model (Grafana Cloud) creates a direct, visible incentive to control cardinality specifically, because cardinality is the exact unit being billed. A consolidated per-GB model (New Relic) blurs metrics, logs, and traces into one undifferentiated cost signal, which can make it harder to tell which telemetry type is actually driving a spend increase without additional internal tagging or cost-allocation work. An events-based model (Honeycomb) removes the metrics-versus-logs-versus-traces distinction entirely but does not remove the underlying need to think about what belongs in an event and how often events fire. No pricing model eliminates the need for engineering discipline; each one simply routes the consequence of missing discipline to a slightly different place on the invoice.

Three Scenarios Showing How This Plays Out

The following three scenarios are illustrative and hypothetical, constructed for this article to demonstrate realistic mechanisms across different industries. None describes an actual QAtronic client, engagement, or measured outcome.

Scenario one: the fintech platform and the per-transaction label

Initial situation. A fintech company processing card and ACH payments instruments its transaction-processing service with detailed metrics to support its fraud and risk team's real-time dashboards. Early instrumentation is disciplined: metrics are labeled by payment_method, status, and merchant_category_code — all bounded, low-cardinality dimensions.

The hidden assumption. When the risk team requests the ability to drill into fraud patterns by individual merchant rather than by category, an engineer adds a merchant_id label directly to the existing high-frequency transaction-latency metric, reasoning that it is "just one more label" and that the team already has metrics infrastructure in place to handle it. Nobody evaluates what "one more label" means multiplied against the platform's actual merchant count, because the request came from a legitimate business need and the change looked small in the pull request diff.

The consequence. The platform serves several thousand active merchants. The metric's cardinality ceiling jumps by that same multiple, and because the metric is a histogram (which OpenTelemetry and Prometheus both implement by generating multiple underlying series per histogram — one for the count, one for the sum, and one for each configured bucket boundary), the actual series count multiplies again on top of the merchant-count multiplier. The observability vendor's monthly invoice for custom metrics rises by an amount large enough that finance flags it in a routine spend review, several weeks after the change shipped, with no immediate way to identify which of dozens of recent deploys caused it.

The decision point. Engineering leadership has to choose between three options: revert the label and lose per-merchant fraud drill-down entirely, keep the label and accept the ongoing cost increase indefinitely, or find a middle path that preserves the underlying business capability at lower cost.

The better approach. The middle path exists and is a direct application of the cardinality-versus-cost trade-off: rather than attaching merchant_id to a high-frequency histogram queried by every dashboard, the team routes per-merchant fraud detail through a separate, lower-frequency, purpose-built metric (or, better, through log-derived or trace-derived analysis scoped to flagged transactions only, since fraud investigation is inherently a low-volume, high-detail activity rather than a high-frequency dashboard metric). The high-frequency operational metric reverts to its original bounded label set. The business capability the risk team actually needed — investigate a specific merchant's fraud pattern — is preserved through a data path shaped for that specific, lower-volume use case rather than bolted onto a metric shaped for a different, higher-frequency use case.

Scenario two: the e-commerce platform and the sampling rate nobody revisited

Initial situation. An e-commerce platform adopts distributed tracing across its checkout flow ahead of a peak shopping season, configuring head-based probabilistic sampling at 100% specifically to get complete visibility during the highest-risk period of the year, with an explicit plan to reduce the rate afterward.

The hidden assumption. The plan to reduce the sampling rate after the peak period depends on someone remembering to do it and having a defined target rate to reduce it to. Neither exists as a tracked action item — the 100% sampling configuration is treated as a deploy like any other, not as a temporary measure with an expiration.

The consequence. Baseline, non-peak traffic continues to be traced at 100% for months afterward. Because trace ingestion at this platform is billed per span ingested, and checkout flows generate a large number of spans per transaction across payment, inventory, tax calculation, and fulfillment services, the sustained 100% sampling rate produces trace ingestion costs many times higher than the platform's steady-state visibility needs actually require. The cost increase is gradual and sustained rather than a single spike, which is exactly the profile that evades both human attention (nothing "looks wrong" on any single day) and automated cost-anomaly detection tuned to catch sudden deviations from a historical baseline rather than a permanently elevated one.

The decision point. Once identified — in this hypothetical, during an unrelated infrastructure cost review — the team has to decide what sampling strategy to adopt going forward, not simply revert to whatever arbitrary rate existed before the peak-season change.

The better approach. Rather than picking a new flat percentage, the team adopts a tail-sampling policy for the checkout flow specifically: keep 100% of traces that contain an error or exceed a defined latency threshold (the traces most likely to represent an actual customer-impacting problem), and apply a much lower probabilistic rate (in the low single digits) to traces that complete successfully within normal latency bounds. This preserves complete visibility into the failure modes that matter most for a checkout flow specifically, while reducing the cost of tracing the large volume of unremarkable, successful transactions that a flat 100% rate was capturing without added investigative value. The team also adds an explicit expiration and review step to any future "temporary" sampling-rate change, addressing the process failure as well as the immediate configuration.

Scenario three: the healthcare SaaS platform and the debug log that outlived the incident

Initial situation. A healthcare scheduling SaaS platform experiences an intermittent bug in its appointment-reminder notification pipeline. During incident response, an on-call engineer raises the notification service's log level from info to debug across all instances, and adds full payload logging (including patient appointment metadata, handled under the platform's existing data-handling controls) to the specific code path under investigation, in order to gather enough detail to reproduce the intermittent failure.

The hidden assumption. The team's incident-closure process includes confirming the underlying bug is fixed and validating the notification pipeline behaves correctly again. It does not include a step to confirm that diagnostic changes made during the incident — the log-level change and the added payload logging — have been reverted, because those changes were treated as part of the investigation rather than as a production configuration change requiring its own review.

The consequence. The notification service handles a high, steady volume of scheduling events continuously, and debug-level payload logging increases its average logged data volume substantially. Because the observability platform in use bills primarily on ingested log volume, the service's monthly log-ingestion cost rises and stays elevated, but the rise is gradual against a baseline that is also naturally growing with the platform's genuine customer growth, so it does not present as an anomaly large enough to trigger investigation on its own for some time.

The decision point. When the elevated cost is eventually noticed — during a routine vendor invoice review — the team faces a choice about how to prevent recurrence, not just how to fix this one instance.

The better approach. The specific fix (reverting the log level and removing the payload logging) is trivial once identified. The durable fix is procedural: treating a debug-logging change made during incident response as a production change with its own expiration, the same way a feature flag enabled for an investigation would be expected to carry a default expiry rather than persisting indefinitely. A lightweight mechanism — an automated reminder tied to the original log-level change, or a policy requiring elevated logging to be re-justified after a fixed window (a small number of days, not open-ended) — closes the gap that let a one-day incident produce a months-long, unreviewed cost increase.

The Governance Gap: Why Nobody Reviews Instrumentation Like They Review Code

Every scenario above shares a structural feature worth naming directly: the instrumentation change that caused the cost increase was individually reasonable, made by a competent engineer solving a real, immediate problem, and shipped through the same code review process as everything else the team writes. Standard code review caught nothing, because standard code review is looking for correctness, security, and maintainability — not for what a new label does to a metric's cardinality ceiling, or what a debug-log change does to ingestion volume six weeks after the incident that justified it closes.

This is the actual gap this article is arguing needs to close: not "engineers need to care more about cost" — most already do, in the abstract — but that instrumentation decisions need a review lens that current code review does not apply, because the cost consequence of an instrumentation change is often invisible in the diff itself. A merchant_id label is one line. Its cost consequence depends on a number (merchant count) that does not appear anywhere in the pull request.

A Lightweight Instrumentation Review Framework

The following is not a proposal to add a new committee or a new approval gate that slows down every deploy. It is a small, specific checklist that targets exactly the decisions this article has shown to be high-leverage, designed to run in minutes as part of existing code review rather than as a separate process.

Step 1 — Classify the change by cardinality risk before merge. A new or modified metric label falls into one of three categories: bounded and small (fewer than roughly a few dozen possible values, known in advance — HTTP methods, status code classes, environment names), bounded but large (hundreds to low thousands of possible values, known but sizable — region codes, plan tiers, feature-flag names), or unbounded (the value space grows with the business — customer IDs, user IDs, session tokens, request IDs, raw error messages, raw URLs with path parameters). Only the third category requires the steps below; the first category needs no review at all, and the second needs a brief sanity check on expected total combinations, not a full review.

Step 2 — For any unbounded-risk label, require an explicit alternative before approval. The reviewer's question is not "should we forbid this," it is "does this specific need require attaching this specific value to a high-frequency metric, or would a lower-frequency metric, a log field, a trace attribute, or an exemplar serve the same investigative purpose at a fraction of the cardinality cost." Trace attributes and log fields are far cheaper homes for high-cardinality, per-request detail than metric labels are, precisely because they are not pre-aggregated into stored time series the way metrics are — this single substitution resolves the majority of accidental high-cardinality-metric cases without sacrificing the underlying investigative capability.

Step 3 — Attach an estimated series or volume impact to the pull request, not just a description of the change. This does not require precise measurement — an order-of-magnitude estimate ("this label has roughly N possible values, multiplying this metric's series count by roughly that factor") is enough to make the cost visible to a reviewer at the moment of decision, which is the entire point. OpenTelemetry's own cardinality-limits guidance recommends exactly this kind of estimation exercise — calculating expected combinations from known dimensions before shipping — as the practical alternative to discovering the real number after the fact via an overflow warning or a bill. OpenTelemetry Blog — Metric cardinality limits in OpenTelemetry: a practical guide

Step 4 — Require an explicit, tracked expiration for any diagnostic-only instrumentation change. Debug-level logging enabled for an active investigation, a temporarily elevated trace sampling rate, or a temporary high-cardinality label added to debug a specific incident should carry the same default-off discipline as a feature flag — an expiration date or an automated follow-up, not a manual "remember to revert this" that depends on human memory during a period when attention is already consumed by the incident itself.

Step 5 — Route new high-cardinality or high-volume telemetry changes through collector-side controls rather than relying on source discipline alone. This is the step that turns the review process from a purely human gate into one backed by infrastructure. The OpenTelemetry Collector is explicitly designed to sit between instrumented services and observability backends for exactly this purpose — the project's own documentation describes transforming, filtering, and aggregating telemetry at the collector layer "for data quality, governance, cost, and security reasons," using processors that can drop unwanted attributes, filter out entire classes of telemetry before they reach a billed backend, rename or restructure metrics, and enrich or normalize data consistently across every service that reports through the collector, rather than depending on every individual service's instrumentation code to get it right. OpenTelemetry — Transforming telemetry A CNCF-published case study on building cost-effective observability with OpenTelemetry makes the same point from a different angle: it describes deliberately sequencing collector processors (a memory limiter first, then enrichment, then filtering and transformation, with batching last for efficient delivery) as the mechanism that let one organization address metric multiplication at its root cause — in that case, a scrape-topology misconfiguration causing a documented 20–40x metric multiplication — rather than papering over it with aggressive sampling after the fact. CNCF Blog — How to build a cost-effective observability platform with OpenTelemetry A collector-enforced allowlist of approved high-cardinality label combinations, or a collector-side rule that strips any attribute matching a known-unbounded pattern (raw email addresses, UUIDs in unexpected fields) before export, closes the gap for the cases that slip past code review entirely.

A minimal illustration of what this looks like in practice, using the Collector's transform and filter processors to strip a known-unbounded attribute before metrics ever reach a billed backend:

processors:
  transform/strip_unbounded_labels:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user_id")
          - delete_key(attributes, "session_token")
          - delete_key(attributes, "request_id")
  filter/drop_debug_verbosity:
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_INFO and resource.attributes["deployment.environment"] == "production"'

service:
  pipelines:
    metrics:
      processors: [transform/strip_unbounded_labels, batch]
    logs:
      processors: [filter/drop_debug_verbosity, batch]

This configuration enforces two of the review framework's principles as infrastructure rather than as policy documentation: the transform processor removes a fixed list of known-unbounded attributes from every metric passing through the collector, regardless of which service or engineer emitted them, and the filter processor drops below-info log records specifically in production, closing the gap left by a debug-level change that a code reviewer might miss or a developer might forget to revert. Neither rule depends on every engineer remembering the policy at the moment they write instrumentation code — the enforcement point moves from "hope everyone read the guidelines" to "the collector won't forward it regardless." This is also, worth noting, exactly the sequencing the CNCF case study referenced above recommends: enrichment and filtering ahead of batching, so that attribute-stripping and severity-filtering happen before the more expensive batching and export stages process data that was never going to be kept anyway.

This five-step framework deliberately does not attempt to make every instrumentation change go through heavyweight approval. The overwhelming majority of metrics, logs, and traces any team adds are low-risk by the classification in step one and need no additional process at all. The framework exists specifically to catch the minority of changes — new unbounded labels, elevated verbosity, expanded sampling — that carry disproportionate cost risk, and to catch them at the point of code review rather than the point of the monthly invoice.

A Cardinality and Volume Budget, by Maturity Level

Different organizations need different starting points, and a framework calibrated for a fifty-person scale-up will either be ignored as overkill by a five-person startup or be dangerously under-specified for a regulated enterprise. The table below sketches reasonable defaults across three maturity levels, built specifically for this article rather than adapted from a generic capacity-planning template.

A Before-and-After Walkthrough: Migrating a Noisy Metrics Setup to a Cardinality Budget

The following walkthrough is a labeled hypothetical, constructed for this article to show what an actual migration looks like end to end. The service, its metric names, and its numbers are illustrative, not drawn from a real QAtronic client or engagement.

A mid-sized logistics-SaaS platform runs a shipment_status_service that has accumulated instrumentation the way most services do: additively, over roughly two years, by different engineers solving different immediate problems, with no single person ever looking at the metric set as a whole. A platform engineer doing a routine spend review decides to audit it before deciding whether to raise the team's Grafana Cloud series allocation or fix the underlying problem.

Before. The audit turns up one histogram, shipment_status_update_duration_seconds, carrying five labels: carrier (12 values), status_code (8 values), region (6 values), shipment_id (effectively unbounded — one value per shipment, and the platform processes several million shipments a month), and warehouse_id (340 values, added eighteen months earlier by a different team investigating a warehouse-specific slowdown and never removed). The shipment_id label alone means this single histogram is generating, in effect, a new time series for every shipment the platform has ever handled that is still within the metric's retention window — not a bounded number in any meaningful sense, and the exact failure mode Prometheus's own naming guidance warns against by name. The service's Grafana Cloud bill for this one metric, estimated from the platform's per-1,000-series rate, accounts for a disproportionate share of the team's entire observability spend, a fact nobody had isolated before the audit because the invoice arrives as one undifferentiated total, not broken out per metric.

The audit. Applying the review framework's step one classification to each label produces a clear split: carrier, status_code, and region are bounded and small — no action needed. warehouse_id is bounded but large at 340 values, large enough to be worth a second look but not inherently a problem on its own. shipment_id is squarely unbounded, and it is also the label nobody actually queries the histogram by in practice — the dashboards built against this metric group by carrier and region, never by individual shipment, meaning the highest-cost label in the entire setup is not providing observable value proportional to what it costs.

The migration. The team removes shipment_id from the metric entirely, replacing per-shipment investigative detail with what it was actually being used for in the rare cases anyone needed it: a trace attribute on the corresponding span, queryable on demand for a specific shipment without pre-aggregating a stored time series for every shipment that ever existed. The warehouse_id label stays, but the team adds a Collector-side transform rule capping it to the current set of active warehouses and folding any warehouse retired from the network into an other bucket, preventing slow, unbounded growth as the warehouse network expands. The histogram's remaining label set — carrier × status_code × region × warehouse_id (bounded to active warehouses) — produces a ceiling in the low tens of thousands of series instead of an effectively unbounded count, a reduction of several orders of magnitude.

After. The dashboards the team actually uses continue to work without modification, because none of them ever grouped by shipment_id in the first place — the removed label was pure cost with no corresponding analytical use. Per-shipment investigation, on the rare occasion it's needed, now happens through trace lookup rather than through a metric dimension, which is both cheaper and, for that specific use case, a better fit for the access pattern (looking up one shipment on demand, not aggregating across all of them continuously). The team documents the warehouse_id cap and the shipment_id removal as the new baseline in its service's instrumentation notes, so the next engineer who wants to add per-entity detail to this metric sees a clear record of why that specific pattern was removed rather than rediscovering the same problem eighteen months from now.

The general lesson this walkthrough is meant to generalize, not just illustrate once: the highest-cardinality label in an existing metric is frequently not the one providing the most query value, precisely because unbounded labels tend to get added for a narrow, one-off investigative reason and then never get used the way the metric's core, bounded labels are used every day on every dashboard. An audit that checks actual query patterns against actual label cardinality, rather than assuming every label pulls its weight, routinely finds this mismatch — and finds it faster than restructuring an entire team's instrumentation practice from scratch would.

When This Discipline Is the Wrong Investment

None of the above is free to build, and it is worth being direct about when it does not pay for itself. A five-person startup pre-product-market-fit, running a handful of services with a monitoring bill measured in low hundreds of dollars a month, does not need a formal cardinality review process — the entire framework above costs more in process overhead than it would plausibly save, and a founder or lead engineer glancing at the bill monthly is a perfectly adequate substitute for formal governance at that scale.

The same is true for genuinely low-stakes internal tooling regardless of company size: an internal admin dashboard used by a handful of employees does not need the same instrumentation rigor as a customer-facing payments flow, and applying the five-step review uniformly regardless of what a service actually is turns a targeted cost discipline into indiscriminate friction. The judgment call in step one of the review framework — classifying cardinality risk — is meant to filter out exactly this kind of low-stakes case automatically, by recognizing that most metrics never carry unbounded-risk labels in the first place.

There is also a more subtle failure mode worth naming: an organization that becomes so cost-conscious about telemetry that it under-instruments genuinely important systems, sampling away the rare failure traces or suppressing the exact log detail that would have made a real incident faster to diagnose. Cardinality and volume discipline is not the same goal as minimizing observability spend to zero; it is the goal of making sure spend tracks value rather than accident. A tail-sampling policy that discards error traces to save money is optimizing the wrong variable — the entire argument in this article's sampling section is that error and outlier traces are disproportionately the ones worth keeping, precisely because they are disproportionately informative, not despite it.

Frequently Asked Questions

Does adding tags or labels to a metric always increase cost? Not necessarily. Datadog's own billing documentation notes that a new tag only increases billable custom metrics when it introduces genuinely new information — if a tag is redundant with information already captured by an existing tag, adding it does not create new distinct combinations. The cost risk is specifically in tags whose values are numerous and not already implied by existing labels, particularly anything approaching one distinct value per customer, user, or request.

Is tail sampling always better than head sampling for controlling trace costs? No. Tail sampling reduces the volume of exported traces by selecting based on outcome, but it requires buffering every span of a trace in memory until a decision is made, which adds real infrastructure cost and a routing constraint (all spans of one trace must reach the same collector instance). For services where outcomes don't vary meaningfully between traces, or where the operational cost of tail-sampling infrastructure exceeds the value of outcome-based selection, a simple, cheap head-sampling rate is often the better trade-off.

What's the fastest way to find out if we already have a cardinality problem? Query your metrics backend for the highest-cardinality metrics currently reporting, most vendors and open-source tools (including Prometheus-compatible tooling such as Mimirtool, referenced in Grafana Labs' own guidance on this exact problem) expose this directly. Look specifically for any metric whose label set includes an identifier that scales with your customer, user, or request count — that is the pattern to investigate first, before doing any broader audit.

Should observability cost review sit with engineering, platform/SRE, or finance? The review step itself belongs with whichever team already owns telemetry infrastructure and code review — typically platform engineering or SRE — because the judgment involved (is this label bounded, does this service need this level of trace detail) is a technical judgment, not a financial one. Finance and engineering leadership should own the budget and the retention/tiering policy that the technical review operates within, similar to how a cloud infrastructure budget is set by leadership but enforced through engineering practice day to day.

Can OpenTelemetry's cardinality limit alone solve this problem for us? It solves a narrower, important problem: it prevents an individual metric stream from causing unbounded memory growth in the process that emits it, which is a real and valuable protection. It does not solve the backend billing problem, because the overflow behavior it triggers (folding excess combinations into a single overflow data point) preserves correct totals but silently degrades the usefulness of any query that filters or groups by the attribute that overflowed — meaning a team that treats the SDK limit as sufficient governance is likely to discover the cost and correctness trade-off only when a specific query stops returning the breakdown they expect.

Is switching to an events-based platform like Honeycomb, or a wide-event architecture in general, a way to avoid this problem entirely? It changes where the trade-off lives rather than removing it. A wide-event or "observability 2.0" architecture, as advocated in Honeycomb's own writing on the topic, consolidates what would otherwise be separate metrics, logs, and trace data into fewer, richer events, which genuinely reduces the "5x multiplier" cost of storing the same underlying request information five separate times across five separate tools. It does not remove the need to think about how many distinct events fire and how much detail each one carries, because event-ingestion volume is still the billed unit in that model. The architectural choice can reduce redundant storage; it does not substitute for the review discipline this article describes.

The Distinction That Matters

The recurring failure this article has traced through cardinality, log verbosity, sampling, and retention is not that engineers instrument too much. It is that instrumentation decisions get made without anyone in the loop who can see their cost consequence at the moment the decision is made — the same structural gap that lets a fourteen-line code change silently triple a cloud bill, applied here specifically to the telemetry layer, where the mechanism (combinatorial multiplication of stored time series, volume amplification of verbose logs, ingestion-proportional trace costs) is unusually direct and unusually easy to estimate in advance, if anyone stops to do it.

The distinction worth carrying back to an engineering organization is this: observability spend that tracks real system growth is a cost of doing business; observability spend that tracks instrumentation decisions unrelated to system growth is a cost of not reviewing those decisions. The first category deserves a budget, a forecast, and patience. The second category deserves a checklist, a named owner, and the same review discipline already applied to schema changes and API contracts — because, as the numbers in this article show, an unreviewed label can cost more per month than the system it was added to observe.

The next time a metric, a log line, or a trace attribute gets added to a pull request, the question worth asking is not whether the data would be nice to have — almost all telemetry would be nice to have, considered in isolation. The question is whether anyone has estimated what it costs to have it, multiplied against the actual scale of the system it will run on, before it ships rather than after the invoice arrives.

A Note on Where QAtronic Fits

Instrumentation review is, at its core, a quality practice applied to a system's diagnostic layer rather than its functional layer, and it draws on the same discipline QAtronic applies to test strategy and release engineering more broadly: identifying where a change carries disproportionate downstream risk, and building a lightweight, specific check for exactly that risk rather than a heavyweight process applied uniformly. Teams that already have strong code review and release-gating practices are typically a short distance from extending that same rigor to telemetry cost; teams building both at once benefit from treating instrumentation review as part of the same discipline from the start, rather than bolting it on after the first surprising invoice. If your organization is trying to work out where an observability bill's growth is actually coming from, or wants a second set of eyes on how new services are instrumented before that instrumentation reaches production, that diagnostic work sits squarely inside the kind of engineering quality review QAtronic does.