A microservice extraction can look successful while the people carrying the pager see slower requests, harder incidents, and more ways to lose a transaction. I would judge a legacy-to-microservices change against a production user journey, not against the number of services deployed. If the journey does not improve without exceeding its failure and operating-cost limits, the extraction is not working.
The user journey is the unit of success, not the new service
Microservices first breaks prod on legacy apps you must run makes a useful warning, but I would not treat a new service as a failure merely because it adds a network hop: the production question is whether that hop makes a complete user action less reliable. Conversely, Monolith Scaling Pains Modernize to Microservices identifies a plausible motive, but lower CPU use in the old process does not prove users got more capacity; contention may simply have moved to a database or queue.
Pick one journey before changing its ownership. For checkout, start the clock when an accepted request reaches the edge and stop it only when the order reaches a durable, queryable state. Count an HTTP 200 followed by a lost Kafka event as a failure, because the customer did not receive a completed order. Count an eventual completion separately if its delay matters to the customer. This definition prevents a service team from reporting a fast API while an operator investigates stranded messages.
Write down the baseline as a small set of decision metrics: completed journeys per minute, the fraction completed correctly, end-to-end p95 latency, and the rate of duplicate or missing outcomes. Add cost per completed journey if the extraction is meant to relieve a scaling problem. These metrics belong to the journey rather than to a pod, because Kubernetes can report healthy replicas while a downstream dependency prevents completion.
Make the outcome observable across the old and new paths. Propagate W3C Trace Context through HTTP and Kafka headers; use OpenTelemetry spans to connect the edge request, database writes, and asynchronous consumer. Give each attempt a stable journey ID and record its final state in an auditable store, because sampled traces alone cannot count every missing transaction. OpenTelemetry documentation publishes OTLP ports 4317 for gRPC and 4318 for HTTP; whichever transport you configure, verify that the edge and worker preserve the same trace context before trusting a distributed trace.
Comparable traffic matters more than a dramatic before-and-after graph
A release-day graph mixes architecture effects with traffic effects, because request mix, promotions, deploys, and database maintenance can change at the same time. Keep a control path where possible: route eligible requests to either the monolith-owned flow or the extracted flow using a stable hash of customer or account ID. Hashing by request is a poor choice for stateful journeys, because one customer could alternate between implementations during a retry.
Define eligibility before looking at results. Both groups should contain the same journey types and roughly comparable request sizes, regions, and dependency load. Record the assigned variant at the edge and retain it through completion; filtering by which backend ultimately answered would hide failures that never reached that backend. Use Envoy routing or Argo Rollouts for the split, but check that retries do not silently switch variants.
For a gate you can tune, start the new path at 5% of eligible traffic and hold each step long enough to see a representative peak and an asynchronous completion cycle. Five percent is not a universal safety threshold: it may produce too few requests to estimate a tail percentile, or too many if a failure can corrupt durable state. If the journey is low-volume, prioritize completion counts and inspect individual failures rather than declaring victory from a noisy p95.
Compare each variant over the same clock intervals and report both the absolute difference and request volume. In an illustrative, invented dataset, suppose the control completes 99.95% of 200,000 eligible journeys while the extracted path completes 99.80% of 10,000. The smaller path is worse by 0.15 percentage points in that example, even if its median latency looks better. Investigate the failed journeys before increasing traffic, because a completion deficit can represent durable customer harm rather than transient slowness.
When parallel routing is impossible, compare matched weeks with the same hour-of-day windows and annotate deploys and traffic changes. That comparison is weaker than a concurrent control because unrelated events can explain the difference; say so in the release decision rather than presenting a before-and-after percentage as proof.
Tail latency and error budgets expose failures that service dashboards hide
Publish a Prometheus histogram for the complete journey, labeled by route and assigned variant, and aggregate buckets before computing a percentile. The following Bash command queries the last 30 minutes for a classic histogram named journey_duration_seconds; it requires curl, jq, and a reachable Prometheus API.
#!/usr/bin/env bash
set -euo pipefail
: "${PROM_URL:?set PROM_URL}"
for variant in monolith extracted; do
query="histogram_quantile(0.95, sum by (le) (rate(journey_duration_seconds_bucket{variant=\"${variant}\",path=\"checkout\"}[30m])))"
curl -fsSG "$PROM_URL/api/v1/query" \
--data-urlencode "query=$query" |
jq -r --arg v "$variant" '"\($v) p95 seconds: \(.data.result[0].value[1] // "no data")"'
done
The query is useful for comparing populated cohorts, but it cannot establish correctness: Prometheus interpolates within histogram buckets, and a request that never completes may never enter the duration histogram. Pair it with a counter of accepted journeys and a counter of journeys reaching a terminal correct state. Reconcile those counts against durable IDs, allowing for the documented completion delay, so an expanding gap becomes an alert rather than an invisible success.
Set an SLO on correct completion and its deadline, then translate misses into an error budget. A proposed 99.9% completion objective allows 0.1% of eligible journeys to miss; it is a policy choice to review with the service owner, not a score the architecture earns by default. Alert on sustained budget burn for the extracted cohort as well as on missing outcomes, because a modest error rate can consume the allowance quickly under heavy traffic. Grafana can display the cohort and budget, while Tempo traces can help explain individual misses; neither replaces the durable outcome count.
Inspect the shape of latency rather than only its p95. Compare time spent in Envoy, the extracted handler, PostgreSQL, and the Kafka consumer, because a fast handler can still leave customers waiting on locks or a backlog. PostgreSQL’s pg_stat_statements can reveal a newly frequent query, while consumer lag can reveal work accumulating after an apparently successful HTTP response. Keep labels such as journey and variant bounded in cardinality; putting a unique journey ID on a Prometheus metric creates an expensive time series per request, so keep IDs in logs or traces instead.
Independent scaling wins only when its full operating cost is lower
There are two defensible responses to a monolith’s hot path. Monolith resource tuning wins when profiling finds a shared query, lock, or allocation bottleneck that can be fixed without adding a distributed failure path; it costs engineering time in the existing codebase and may leave unrelated workloads coupled. Service extraction wins when the hot path can scale independently and its boundary has stable ownership; it costs network latency, deployment coordination, observability, and on-call work. Compare those costs per correctly completed journey at the same traffic level, because raw pod counts say little about capacity delivered.
Measure the claimed scaling gain under load. Hold the journey mix fixed, raise offered traffic with k6, and plot correct completions per second against CPU, memory, database connections, queue lag, and total infrastructure spend. A new service has not raised useful capacity if its Kubernetes Horizontal Pod Autoscaler adds replicas while PostgreSQL connections saturate and correct completions flatten. Test overload behavior too: timeouts, bounded retries, and idempotent writes matter because unbounded retries can amplify an upstream slowdown into a wider incident.
I would not approve the extraction on deployment frequency or mean time to recovery alone, because teams can deploy a broken journey frequently and recover its pods without repairing lost outcomes. Track those operational measures alongside the journey result: pages per completed journey, time to identify the failing owner, rollback duration, and manual reconciliation work. An architecture that improves throughput but doubles reconciliation work may still be a bad trade for the team supporting it.
Put the release decision in writing before widening the canary. State the minimum acceptable completion rate, the maximum tolerable latency regression, the traffic level at which scaling should improve, and the rollback trigger. Precommitting matters because a persuasive CPU graph is easy to favor after the fact, even when the metric customers experience has deteriorated.
The first useful step is to define a completion record
Before extracting or expanding anything, choose one production journey and identify the durable record that proves it completed correctly. Join that record to the edge request ID for the existing path, then count accepted, completed, missing, and duplicate journeys over a representative peak. If you cannot make those counts agree today, fix that measurement first: otherwise the next release can change the architecture without telling you whether it helped.
