Software Engineering
opentelemetry observability distributed-systems software-engineering java

I Don't Start an OpenTelemetry Migration at the SDK

What Atlassian's metrics rewrite made me write down about UDP contracts, streamID hashing, and delta aggregation.

2026-09-17 ยท Vineet Kumar Loyer

Atlassian published OpenTelemetry everywhere: Migrating a metrics platform at scale on the CNCF blog. The headline is 100k hosts, 14 regions, a 99.95% SLO, and a metrics pipeline that used to run on gostatsd.

I stared at the diagrams and wrote a rude note in the margin: so where is the SDK?

That is the question this post is for.

Why this post?

Every OpenTelemetry plan I have watched on a whiteboard starts at instrumentation. Swap the StatsD client for the OpenTelemetry SDK. Teach every team OTLP. Flip a flag. The calendar then fills with years, and the pipe that pages people is the one you are ripping out.

I have been around data pipelines long enough to know the packet that fires an alert is the packet you cannot drop while you modernize. Atlassian said the same thing with fleet numbers attached. I wanted to write that down in language I would actually use at a whiteboard.

What did a service owner actually sign?

A metrics pipeline has two ends.

One end is a sentence: send StatsD over UDP to this address, and the points show up in the backend. The other end is collection, ingest, aggregation, and forward. Teams live on the first sentence. They page on it.

They kept that sentence. They replaced the machinery after the datagram. An org-wide client migration became a platform-team migration.

I would do the same thing, and I would do it for a boring reason. The SLO lives on the old packets until the new pipe is proven.

Why gostatsd was finished even though it was fine

gostatsd is Atlassian’s StatsD implementation in Go. For a decade it did two jobs: sidecar on the host, aggregator at the far end. Nobody thought about it. That is usually the compliment.

The rest of the industry standardized on OpenTelemetry. More of what fed their pipeline was already emitting OTel they could not ingest. gostatsd spoke UDP. It had no traces, no logs. Every useful Collector component the community shipped was another thing they would have to rebuild by hand.

I have tried to keep a private StatsD-shaped fork current with Collector features. I ran out of hours before I ran out of features.

They also already ran the OpenTelemetry Collector for tracing, as the pipeline core and as a host-metrics sidecar. “Is it production-ready at our scale?” was a solved question inside the company. That matters more than the blog spends time on. You do not introduce a new runtime and a new protocol in the same quarter if you can avoid it.

Four binaries, one Collector

They put purpose-built Collector distributions at four stages: collection, ingest, aggregation, forward. A distribution is a compiled Collector with a chosen set of receivers, processors, and exporters. You build it with ocb. You can ship a change to ingest without touching aggregation.

I like that split. A pipeline this large is four different reliability problems pretending to be one service.

Collection

They replaced the gostatsd sidecar with the same Collector the tracing team already shipped. Applications still fire StatsD over UDP. Day one, no team noticed.

The payoff is mechanical. You stop running two sidecars on every host. Folding metrics into the tracing sidecar and killing the StatsD one saved about 3.9% CPU on average per service across their priciest Micros hosts, roughly a 30% cut in sidecar cost at fleet scale. They also turned on an OTLP receiver, so the same process can take native OTel metrics when a team is ready.

The statsd receiver keeps the UDP contract alive. The docs say it is meant to run in agent mode. Horizontally scaled Collector deployments of that receiver are unsupported, so it lives next to the app. The OTLP receiver is how native OTel metrics enter the same process.

Ingest

This is the part I circled.

Aggregation is stateful. Every datapoint for a time series has to hit the same aggregator, or you split a counter across shards and lie. Ordinary load balancing will do that lie for you, politely, at line rate.

For years an in-house proxy called nomad hashed (service, environment) onto a shard. Their service-to-metric load is a long tail. The shards that owned the biggest services became hot. Idle replicas sat next to them. Autoscaling could not shrink the pool because the fat keys were pinned.

The contrib loadbalancingexporter can hash by streamID: the identity of an individual time series (resource, scope, metric, datapoint attributes). One fat service smears across the pool. Any given series still always lands on the same shard. The exporter uses consistent hashing over the backend list, so two ingest replicas with the same config agree on the destination.

After the change, per-shard CPU went flat. Even load means a tighter autoscaling band, real off-peak scale-down, and fewer hot-shard pages.

I want that on a whiteboard as a type. I will get there in a minute.

Aggregation

This is the stage that makes the numbers affordable. They take in about 4.8 billion datapoints a minute and land about 220 million. A 96% reduction.

Most of their metrics are delta temporality. StatsD has always been a delta protocol: you send the increment since last flush. Cumulative backends expect a running total from a start timestamp. You cannot convert one into the other without remembering the previous point, which is why the cumulativetodelta and deltatocumulative processors both warn about statefulness.

Upstream did not aggregate deltas the way their users expected, so they wrote atlassian-aggregation-processor and published it under atlassian-labs. It sits on the contrib interval processor. Over a configurable window it sums non-monotonic delta sums, merges delta histograms, keeps the last gauge, and passes the rest through. Same traffic. The aggregation tier now runs on about half the CPU. They stopped parsing gostatsd, the load is even, and they inherit the community’s tuning.

If two aggregators both see http.server.duration{status=500} as a delta, you get two partial windows and a dashboard that under-counts incidents. That is why ingest hashing and this processor are the same design. The processor assumes a stream already has a home.

Forward

A bespoke internal forwarder became a stateless Collector they call metrics-gateway, built on upstream exporters. Fan-out to SignalFx, S3, and whatever is next. Retries, queuing, and backpressure come with the exporter helper. Adding a destination is a config change.

They called this the easy one. I believe them. Stateless fan-out is the part of a metrics pipe I would want to be boring.

Lambda

Serverless cannot run a sidecar. They built an OTel Lambda extension with the same StatsD address and the same env vars. gostatsd already had an experimental Lambda mode that flushes at the end of an invocation, because the extension freezes when the function returns. The point is the contract survives even when the process model changes.

The hash key is the architecture

I think in Java. Records are a decent way to freeze an invariant so I cannot accidentally widen it.

The types are small because the invariant is small: a series has one home, or the aggregate is a lie.

public record SeriesIdentity(
        String resourceFingerprint,
        String scope,
        String metricName,
        String datapointAttributes
) {}

public record StreamRoute(
        SeriesIdentity series,
        int shard
) {
    public static StreamRoute of(SeriesIdentity series, int poolSize) {
        // sketch. production uses consistent hashing over the endpoint list
        int shard = Math.floorMod(series.hashCode(), poolSize);
        return new StreamRoute(series, shard);
    }
}

public record ServiceRoute(
        String service,
        String environment,
        int shard
) {
    public static ServiceRoute of(String service, String environment, int poolSize) {
        int shard = Math.floorMod((service + "/" + environment).hashCode(), poolSize);
        return new ServiceRoute(service, environment, shard);
    }
}

public enum Stage {
    COLLECTION,   // sidecar / Lambda extension, StatsD + OTLP
    INGEST,       // stateful routing
    AGGREGATION,  // delta window
    FORWARD       // stateless fan-out
}

ServiceRoute pins every series of a fat service to one replica. StreamRoute pins one series, and a fat service smears. The affinity I actually need is the series. The load I actually have is the long tail of services. Those two facts only agree if the routing key is the series.

streamID in the loadbalancingexporter is identity.OfStream(...): resource plus scope plus metric plus datapoint attributes. That is SeriesIdentity with a better name. The PR that added it is worth reading if you have ever hashed the wrong thing and then paged on CPU.

Routing key What shares a shard Failure mode on a long-tail fleet
(service, environment) every series of a large owner hot shard, idle replicas, stuck autoscaling
metric name every service emitting that name a popular counter becomes its own outage
streamID one time series even smear, a larger map of keys per batch

What the cost numbers are actually saying

gostatsd aggregators and nomad together were about 38% of CPU requests in the metrics clusters. Nomad alone was about 13% of total resources. Sidecar merge saved 3.9% CPU per service on the expensive Micros fleet. Aggregation dropped 96% of datapoints before storage.

I have seen teams celebrate an OTel SDK rollout and then discover the bill moved from the app to the backend. Dropping wasteful datapoints at ingest is the cheapest place to do it. They said that. I believe them because 4.8 billion to 220 million is the whole business case.

One codebase for every stage also changes how you add a hop. You write a component. Standing up a new service for “the S3 exporter” is how you get another nomad.

What I would steal before I started

Their own list is short. Mine is the same list, said the way I would tape it to a monitor.

  1. Pick the people who hurt. Dev and staging first. The teams who already hate the current pipe will iterate with you. They will also forgive a bad Tuesday.
  2. Profile in production. Benchmarks lie about Collector cost. Continuous profiling under real cardinality is the only signal I trust. A processor that looks cheap on a synthetic series explodes when attribute keys multiply.
  3. Keep the operational primitives. A migration this size runs for months. You operate two systems. If paging, dashboards, and rollout tooling diverge, the dual-run is the outage.
  4. Ramp where failure is cheap. 1% to 10% to 50% to 100%. Lower environments first. Find the bug on a tier-2 service.

None of that is OpenTelemetry-specific. It is how you replace a pipe that alerts fire on.

What I am still unsure about

The next move they named is the one I would worry about. The pipe can take OTLP today. The clients still speak UDP, through DogStatsD and the StatsD libraries they have carried for years. Getting those clients onto the OpenTelemetry SDK is the remaining contract.

I do not know how their cardinality behaves when resource attributes and datapoint attributes replace StatsD tags. streamID hashing gets more expensive as the key gets wider. A 4.8-billion-point ingest layer that splits every batch by series identity is a map, and maps have a tail.

I also do not know how they treat UDP loss versus OTLP retry. StatsD over UDP is best-effort. OTLP with the exporter helper will queue. The volume and the latency SLO will fight each other the first week a large service flips.

Until the routing key is the series, and the aggregator speaks delta the way the users already think, I would leave the UDP address alone.

References

More reading on the pieces I only sketched:

CATALOG
    FEATURED TAGS
    aws data-engineering distributed-systems java software-engineering