Every engineer has been there. You push a small fix, open a PR, and then wait. And wait. Twenty minutes later, the build finishes. Thirty minutes after that, the test suite passes. You merge, and then the deployment pipeline adds another fifteen minutes. That tiny change cost you nearly an hour of wall-clock time, and you spent most of it context-switching to other work, losing your train of thought in the process.
Slow CI pipelines are not just an inconvenience. They are a real drag on developer productivity and team morale. The good news is that most of the slowness I see in production pipelines is self-inflicted and fixable. Let me walk through the common culprits and what you can actually do about them.
You Are Running Too Much
The number one reason pipelines are slow: they do things that provide no value on a given run. I have lost count of how many pipelines I have reviewed that run the entire integration test suite on every single commit, including documentation changes. Or pipelines that rebuild every container image from scratch even when the base layers have not changed.
This is usually a failure of pipeline design, not tooling. People tend to start with a single pipeline that does everything, and as the project grows, nobody goes back to split it up. The result is a monolithic pipeline that takes forty minutes regardless of whether you changed a CSS file or rewrote the authentication module.
Solution: Conditional Steps and Path-Based Triggers
Most CI systems let you conditionally run steps. In GitHub Actions, you can use paths filters on workflow triggers. In GitLab CI, you use only:changes. In CircleCI, you can use path-filtering orbs. Stop running your full E2E suite when someone updates a README. It is that simple.
If you have a monorepo, this becomes even more important. You should be able to build and test only the packages that actually changed. Tools like Nx or Turborepo exist specifically for this reason, and they are worth the setup cost if you are working in a monorepo of any size.
Dependency Management Is Wasteful
Here is a pattern I see constantly: a pipeline starts, the first step installs dependencies from scratch, and that step takes three to five minutes. Multiply that by every job in your pipeline, and you have just burned ten to fifteen minutes on downloading and extracting packages that have not changed since the last run.
This happens because people do not cache their dependency installations. Or, almost as bad, they cache them incorrectly. A common mistake is caching node_modules directly instead of caching the package manager’s global cache. The former means you are storing thousands of small files, which is slow to restore. The latter means you store a few compressed archives and unpack them locally, which is much faster.
Solution: Cache the Right Layer
For npm projects, cache ~/.npm or ~/.cache/yarn, not node_modules. For pip projects, cache ~/.cache/pip. For Maven, cache ~/.m2/repository. The general principle is: cache the download cache, not the installed artifacts. And make sure your cache key includes the lock file hash so you get cache invalidation when dependencies actually change.
If you are using Docker, use BuildKit and its --mount=type=cache feature. It lets you persist package manager caches across builds without having to bake them into the image layers. This alone can cut your Docker build times in half.
Your Tests Are Slow Because You Let Them Be
I have strong opinions on this one. Slow test suites are almost always slow because of lack of discipline, not because of the nature of the application. Here are the patterns that kill pipeline speed:
- Network calls in unit tests. If your unit tests are making HTTP requests to external services, they are not unit tests. They are integration tests that happen to be slow and flaky. Mock the external calls. Use tools like nock, responses, or wiremock.
- Database setup and teardown per test. If every test case spins up a fresh database, you are paying seconds per test. Use transaction rollbacks instead. Start a transaction, run the test, roll it back. This is a well-solved problem in most frameworks.
- Running tests serially that could run in parallel. Most test runners support parallel execution. Use it. If your tests have shared state that prevents parallelism, fix the tests. Shared state in tests is a bug, not a feature.
- No test prioritization. Not all tests are equally valuable. Your fast unit tests should run first. If they pass, then run integration tests. If those pass, then run E2E. Do not run the slowest, flakiest tests upfront on every commit.
Solution: The Test Pyramid Still Works
Mike Cohn’s test pyramid is not a new concept, but it is still ignored in practice. You should have a large base of fast, isolated unit tests, a smaller set of integration tests, and a very small set of end-to-end tests. If your pyramid is inverted, your pipeline will be slow, and there is no tooling fix for that. You have to write better tests.
A practical starting point: measure how long each test file takes. Sort by duration. Now look at the top twenty slowest files. Are they actually testing something that matters on every commit? If not, move them to a nightly or pre-merge-only pipeline. You will be surprised how much time this saves.
You Are Rebuilding What Has Not Changed
Incremental builds are not optional in a mature CI setup. Yet I regularly see pipelines that destroy their workspace between jobs, rebuild every artifact from source, and repackage everything from scratch. This is especially common in Docker-based pipelines where each job starts with a clean slate.
The problem is partly cultural. Engineers often treat CI as a black box that should rebuild everything to ensure reproducibility. There is a kernel of truth there: your pipeline should be able to build from a clean state. But running that full build on every commit is wasteful. You need a pipeline that can do a full rebuild on demand and does an incremental build by default.
Solution: Layered Caching and Build Artifacts
First, cache your build outputs between pipeline runs. Most CI systems support artifact storage. Build your binary once, store it as an artifact, and download it in subsequent jobs rather than rebuilding it. This is not cutting corners; it is basic efficiency.
Second, structure your Docker builds to take advantage of layer caching. Put rarely-changing instructions early in the Dockerfile. Install system packages before copying application code. Use multi-stage builds so your final image is small and your build stages can be cached independently.
Third, if you are using a compiled language, use proper incremental compilation. Gradle, Bazel, and even make are designed for this. If your build tool cannot do incremental builds, seriously consider switching. Build times scale with project size, and you do not want to be in a position where full rebuilds are your only option.
Resource Allocation Is an Afterthought
Most CI pipelines run on whatever default runner the CI system provides. For GitHub Actions, that is a 2-core, 7GB RAM machine. For many projects, that is fine. But if you are compiling a large codebase or running a compute-heavy test suite, you are leaving massive speed gains on the table by sticking with the defaults.
I worked with a team that had a Java build taking twenty-two minutes on the default GitHub Actions runner. We switched to a larger self-hosted runner with 8 cores and 32GB RAM, and the build dropped to six minutes. The cost difference was negligible compared to the time saved across the entire engineering team.
Solution: Right-Size Your Runners
Look at your pipeline metrics. Find the jobs that consume the most CPU and memory. Either upgrade your runner size for those jobs or use self-hosted runners with appropriate specs. The major CI providers all offer larger runners, and the cost is usually justified when you factor in developer waiting time.
If you are running parallel test suites, make sure your runner has enough cores to actually benefit from the parallelism. Running eight parallel processes on a 2-core machine is not going to be eight times faster. It might even be slower than running four processes due to context-switching overhead.
The Pipeline Is Not the Bottleneck, the Process Is
Sometimes the CI pipeline itself is not the problem. The problem is that the team requires every PR to go through a full pipeline run, including stages that could be deferred. Does every PR need a full security scan? Does every PR need to deploy to a staging environment? Does every PR need to run the performance benchmark suite?
Probably not. You can run security scans on a schedule or on the main branch after merge. You can deploy to staging only when the PR is approved and ready for final review. You can run performance benchmarks nightly or on release branches.
The key insight is: the PR pipeline should verify that the change works, not that the entire system works. The latter is what your main branch pipeline and scheduled pipelines are for. Conflating the two is the source of much wasted time.
Quick Wins Checklist
If you want to start fixing your slow pipeline today, here is what I would do, in order:
- Measure. Add timing to every step. You cannot fix what you cannot see. Most CI systems have built-in timing reports. Use them.
- Cache dependencies. This is usually the easiest win. Cache the package manager download cache, not the installed directory.
- Split your pipeline. Separate fast, required checks from slow, optional ones. Run the fast checks on every PR and the slow checks on merge or on a schedule.
- Make tests faster. Remove network calls from unit tests. Use transaction rollbacks instead of database rebuilds. Run tests in parallel.
- Use incremental builds. Store build artifacts between runs. Structure your Docker builds for layer caching.
- Upgrade your runners. If your build is CPU or memory bound, a larger runner will pay for itself immediately.
None of these are exotic or complicated. They are engineering discipline applied to the CI pipeline, which is one of the most overlooked pieces of infrastructure in any organization. Fix it, and your team will ship faster with less frustration. It really is that straightforward.
FAQ
How do I measure which CI steps are the slowest?
Most CI platforms provide step-level timing in their UI. GitHub Actions shows duration per step in the workflow run summary. GitLab CI shows_duration for each job. If your platform does not give you this, add timestamp logging at the start and end of each step in your pipeline config. You can also use tools like CircleCI’s test splitting and timing features to get granular data on test performance.
Should I use self-hosted runners or cloud-provided runners?
It depends on your workload. If your build steps are CPU or memory intensive, self-hosted runners with better specs will save you significant time. If your pipeline is mostly I/O bound, like downloading dependencies and running lightweight tests, the default cloud runners are probably fine. The real win for self-hosted runners comes when you can keep warm caches on the machine between runs, which eliminates cold-start costs entirely.
Is it worth caching Docker image layers in CI?
Yes, absolutely. Docker layer caching is one of the most impactful optimizations for pipelines that build containers. Use BuildKit with --mount=type=cache for package manager caches, and structure your Dockerfiles so that stable layers like system dependencies are built before volatile layers like application code. If your CI provider supports remote layer caching, like GitHub Actions’ container registry caching, enable it. The first build will be slow, but subsequent builds will reuse unchanged layers and skip minutes of work.