The measure of a delivery pipeline is not what it can do. It is whether an engineer merging a change thinks about it at all.
By that measure most pipelines fail in the same three ways: they are slow enough to break concentration, they differ per service for no deliberate reason, and rollback is improvised. None of these are exotic problems, and the fixes are ordinary work.
Duration changes behaviour, not just throughput
There is a threshold — different per team, usually somewhere between five and fifteen minutes — past which engineers stop waiting for a pipeline and switch tasks. Once that happens, the cost is not the wasted minutes. It is that changes get batched, so each deployment contains more, and a failure is harder to attribute.
Most slow pipelines are slow for unremarkable reasons:
- Dependencies reinstalled from scratch on every run, because caching was never configured or the cache key is wrong and always misses.
- Sequential stages that have no dependency on each other. Lint, unit tests and container builds usually do not need to be ordered.
- Container images rebuilt from the base layer because the Dockerfile copies source before installing dependencies, invalidating every cached layer on every commit.
- Integration tests that wait on fixed sleeps rather than readiness checks.
Before optimizing anything, look at where the time actually goes. Teams routinely spend a week parallelizing test execution when 60% of the run was dependency installation.
Consistency beats capability
In an organization with a dozen services, it is common to find a dozen slightly different pipelines. Each was reasonable when written. Collectively they mean nobody can reason about deployment in general — only about the specific service they last touched.
The fix is a small number of shared, versioned pipeline definitions that services reference, rather than copy. Reusable workflows and shared pipeline templates exist in every major CI system for this purpose.
Two rules keep it honest. First, the shared definition should cover the common path well rather than every possible case; a service with genuinely unusual needs can opt out explicitly. Second, when the shared definition changes, existing services should keep working — which means versioning it, not editing it in place.
The payoff is transferable knowledge. An engineer who understands one service's deployment understands all of them.
Rollback is a feature, and it needs rehearsing
Ask a team how they roll back. The honest answer is often: revert the commit, wait for the pipeline, hope. During an incident, at speed, under pressure.
Rollback should be a deliberate, tested operation:
- Deploy immutable artifacts. A build produces an image with a digest; the deployment references that digest. If the pipeline rebuilds from a branch, you cannot deploy a known-good version — you can only rebuild something and hope it matches what worked.
- Keep the previous version available. Do not garbage-collect the image you might need in four minutes.
- Make rollback a single action that does not require the full build path. Redeploying a previous digest should take as long as a deployment, not as long as a build plus a deployment.
- Rehearse it. Roll back a real service in a real environment, deliberately, on a quiet afternoon. If nobody has done it this quarter, it does not work — you just do not know that yet.
Database changes are the genuine exception, and they deserve their own discipline: forward-compatible migrations, deployed separately from the code that depends on them, so that rolling back the application does not require rolling back schema.
Separate build from deploy
A surprising number of pipelines conflate the two: the same job that builds also deploys, so deploying anywhere means building again. That makes promotion between environments impossible to reason about, because staging and production are running artifacts that were built at different times from — possibly — different dependency versions.
Build once, produce a versioned artifact, then deploy that same artifact to each environment. Staging becomes a genuine test of the thing that will run in production, rather than a test of something similar.
Make failures readable
A red pipeline should tell you what went wrong without opening five collapsed log sections. Small things carry a lot of the load here: named steps rather than raw commands, test results surfaced as structured output rather than buried in stdout, and failure notifications that go to the person who made the change rather than to a channel everybody has muted.
If a pipeline fails intermittently and the team's response is to re-run it, the flake has become a feature. Quarantine or fix the test; do not train people to ignore red.
Where to start
If you inherit a pipeline situation and need an order: measure where duration goes and fix the largest single cost. Then make rollback a real, rehearsed operation. Then converge the per-service differences onto a shared definition.
That order is deliberate. The first change buys attention back for the team, the second reduces the cost of being wrong, and the third makes both improvements apply everywhere instead of one service at a time.