Most OpenTelemetry migrations don't fail technically. They stall — six services instrumented, two backends running in parallel, a dashboard nobody trusts, and a legacy agent that was supposed to be gone two quarters ago. The difference between the migrations that land and the ones that drift is almost entirely sequencing: what you change first, what you deliberately leave alone, and how you prove nothing broke before you delete anything.
This is the checklist we work through with teams. It assumes you already have some monitoring in place and people relying on it, which is the harder and more common case.
Phase 0 — Decide what you're actually migrating
- Inventory every telemetry producer: application SDKs, agents, sidecars, log shippers, cron jobs, serverless functions, and anything writing StatsD.
- List the dashboards and alerts that are load-bearing during an incident. These are your regression tests, and there are usually fewer than twenty.
- Name the backend end state (OTLP-native vendor, Grafana stack, or both during transition). Migrating instrumentation and backend at the same time doubles the blast radius.
- Agree a semantic-convention baseline (service.name, service.version, deployment.environment) and write it down. Retrofitting resource attributes later is the most tedious part of any migration.
Phase 1 — Instrumentation strategy
- Start with auto-instrumentation for HTTP, gRPC and database clients. It gets you 80% of trace value with near-zero code churn.
- Add manual spans only where auto-instrumentation is blind: queue consumers, batch jobs, business-critical transactions, and anything with retry logic.
- Run OpenTelemetry alongside your existing agent rather than replacing it. Dual-emit for one full on-call rotation before you switch anything off.
- Ban high-cardinality attributes at review time (raw user IDs, full URLs with IDs, request bodies). Cardinality is a cost and performance decision, not a style preference.
- Pin SDK and instrumentation-library versions per language, and own the upgrade cadence — the ecosystem moves quickly and stability guarantees differ by signal.
Phase 2 — Collector topology
- Deploy the Collector rather than exporting straight from apps to a vendor. It is what lets you change backend, sampling or redaction without redeploying services.
- Use an agent Collector per host/pod for resource detection and batching, plus a gateway Collector tier for tail sampling, enrichment and fan-out.
- Configure batch and memory_limiter processors from the start; an unbounded Collector under load becomes the incident.
- Put redaction and PII filtering in the gateway, not in application code, so the policy is enforced in one place.
- Give the Collector its own dashboard and alerts: queue length, refused spans, export failures. An observability pipeline you can't observe will fail silently.
Phase 3 — Trace, metric and log correlation
- Emit trace_id and span_id into structured logs via log correlation in each SDK — this single change is the largest day-to-day debugging win of the whole migration.
- Standardise on exemplars so a latency spike on a histogram links to an actual trace.
- Keep resource attributes identical across all three signals. Mismatched service.name between logs and traces is the most common reason correlation quietly doesn't work.
- Decide sampling deliberately: head sampling for volume control, tail sampling for keeping errors and slow requests. Never sample errors out.
- Verify correlation end-to-end on one service before rolling further — click from an alert to a trace to the logs for that request, and time how long it takes.
Phase 4 — Rollout without breaking dashboards
- Migrate one non-critical but non-trivial service first — something with real traffic that won't page anyone if telemetry gaps appear.
- Dual-write metrics under the new names while keeping old ones alive, then move dashboards panel by panel and diff the graphs over the same window.
- Rebuild alerts against the new metrics in a silenced state, run both for a rotation, and compare firing behaviour before cutting over.
- Freeze old-agent instrumentation changes during transition so you aren't chasing two moving targets.
- Set an explicit decommission date for the legacy agent. Migrations that never finish are more expensive than either end state.
Common pitfalls
- Renaming metrics without a translation layer, then discovering three months of dashboard history is gone.
- Treating OpenTelemetry as a cost-reduction project. It reduces lock-in and improves correlation; volume usually goes up before it goes down.
- Letting each team pick its own attribute names. Without a shared convention you get five spellings of environment and no cross-service queries.
- Skipping the Collector to save a hop, then needing a full redeploy of every service to change sampling.
- Ignoring logs. Teams migrate traces and metrics, leave logs on the legacy shipper, and lose the correlation that justified the work.
- No owner. OpenTelemetry migrations stall when they're everyone's side quest — name one accountable engineer with time allocated.
What "done" looks like
You're finished when three things are true. Every production service emits traces, metrics and logs through the Collector with consistent resource attributes. Every load-bearing alert and dashboard runs on the new data, and the old ones are deleted rather than muted. And the legacy agent is uninstalled, so there is no second pipeline quietly costing money and diverging from reality.
A realistic timeline for a scale-up with twenty to forty services is one to two quarters with a named owner spending roughly half their time on it — faster if auto-instrumentation covers most of your stack, slower if you have legacy runtimes or a large body of hand-rolled StatsD metrics. Plan the decommission date up front and treat it as the actual deliverable.