A build runner sitting at 15% CPU while the wall clock climbs is not a mystery. It is a measurement problem. The instrument you reached for — a CPU profiler — is designed to sample threads that are running. When the build is slow because a linker is waiting on a page fault, a package manager is blocked on a lock, or a compiler process is queued behind forty siblings on a four-core runner, there is nothing on-CPU to sample. The profiler reports an idle machine and you conclude the machine is fine.
It is not fine. It is waiting. Off-CPU analysis is the discipline of measuring that waiting, and bpftrace is a reasonable way to do it on Linux because it can aggregate in the kernel instead of dumping every scheduler event to userspace. But the tool is the easy part. The hard part is asking the right question, because “off-CPU” is two different problems wearing one name, and conflating them sends you to the wrong fix.
Two questions, two tracepoints, two different tickets
Brendan Gregg’s off-CPU analysis write-up draws the basic line: on-CPU time is where threads execute, off-CPU time is where they wait while blocked on I/O, locks, timers, paging, and similar events. CPU sampling, he notes, “doesn’t gather data when the application has blocked and is waiting off-CPU.” That is the whole reason a busy-looking runner can hide a slow build.
Within off-CPU time there are two sub-questions that get merged in casual conversation and should not be:
- Run-queue latency. The task is ready to run — it has been woken — but no core is free. It is waiting for the scheduler to give it a turn. The remediation is capacity or scheduling policy: more cores, fewer concurrent jobs, cgroup weights, CPU pinning.
- Blocked time. The task voluntarily yielded the CPU to wait for something: a read to return, a lock to release, a sleep to finish. The remediation is in the I/O path, the lock, or the dependency the build is waiting on.
Both are read from the same two scheduler tracepoints, sched_wakeup and sched_switch, but the arithmetic is different and so is the conclusion. If you measure only one and call it “off-CPU,” you will spend a sprint adding cores to a runner that is actually blocked on a network filesystem, or you will go hunting for a lock that does not exist while the real problem is that you scheduled twelve linkers onto four cores.
The minimal shape of each measurement
bpftrace programs follow a simple structure: a probe, an optional predicate, and an action block. The bpftrace documentation describes the language as a high-level tracing language and runtime for Linux based on eBPF, supporting static and dynamic tracing for both kernel and user-space. For scheduler work you want the static tracepoints, not kprobes on scheduler internals — tracepoints are the stable API, and the kernel’s event tracing documentation describes sched_wakeup as a trace event carrying fields including comm, pid, prio, success, and cpu.
Run-queue latency is the interval from sched_wakeup (the task becomes ready) to sched_switch picking that same pid as the next task. Record a timestamp keyed by pid on wakeup, and on switch, if the incoming pid has a stored timestamp, take the difference and delete the entry. The result is a histogram of how long ready tasks waited for a core. A long tail here is CPU contention, full stop.
Blocked time is the interval from when a task is switched out to when it is switched back in. Record a timestamp keyed by prev_pid on sched_switch, and when next_pid matches a stored entry, take the difference. The result is a histogram of how long tasks sat blocked. A long tail here is I/O, locks, or sleeps — and to find out which, you key the histogram on the kernel stack at the moment of switch-out instead of on the pid.
That last point is the one that turns a number into a diagnosis. A histogram of blocked time by pid tells you which build process is stuck. A histogram of blocked time by kernel stack tells you where it is stuck. The first is a triage signal; the second is the thing you paste into the design doc.
If you would rather not write the script, the BCC toolkit ships offcputime, which summarizes off-CPU time by kernel stack trace, and runqlat, which summarizes run-queue latency as a histogram. Those are reasonable starting points. They are not the only way, and they are not a substitute for understanding what they measure — a tool you cannot explain is a tool you cannot trust when its output contradicts your expectation.
Why in-kernel aggregation is not a micro-optimization
The scheduler is one of the hottest paths in the kernel. Gregg’s overhead warning is blunt: scheduler events can be very frequent, “in extreme cases, millions of events per second,” and although a tracer may add only a tiny amount per event, at that rate the overhead accumulates. Tracers that dump every event to userspace for post-processing, he writes, “can easily become prohibitive, creating Gbytes of trace data per minute.”
This matters more on a shared build fleet than on a single database server, because the thing you are measuring is the thing you are perturbing. If your trace adds enough load to change the scheduling behavior you are trying to observe, the histogram is a picture of the tracer, not the build. In-kernel aggregation — histograms in BPF maps, summarized before anything crosses into userspace — is what makes the measurement admissible. It is not an optimization for its own sake; it is the difference between a result you can act on and a result you have to apologize for.
Two practical consequences follow. First, bound the trace window. A ten-second capture on a representative build is usually enough to see the shape of the distribution; a ten-minute capture on a production fleet is usually enough to cause an incident. Second, measure the tracer’s own effect. Run the same build with and without the trace attached and compare wall clock. If the difference is inside your measurement noise, say so and move on. If it is not, shorten the window or narrow the predicate until it is.
From one-liner to runbook
Here is the contrarian part. The one-liner is not the deliverable. The one-liner is the thing that gets pasted into a Slack thread, half-remembered three months later, and re-derived from scratch by the next person who hits the same wall. That is not a toolchain; that is folklore with a shell history.
If your internal developer platform is run as a product — named owners, a published SLA, a deprecation calendar — then off-CPU measurement belongs in it, with the same discipline as any other capability. Concretely, that means:
- A named owner. Someone is accountable for the runbook being correct against the kernel and bpftrace versions actually deployed on the fleet. bpftrace’s own repository warns that when using a distribution package you should verify
bpftrace --versionbefore referencing documentation, because behavior and syntax move between releases. An unversioned script in a wiki is a trap. - A published runbook. Which question does this script answer, which tracepoints does it attach to, what does the histogram tail mean, and what is the next step for each shape of tail. The runbook should be readable by someone who has never run bpftrace.
- A bounded trace window. Documented in the runbook, with the reasoning: scheduler events are frequent, in-kernel aggregation keeps the cost down, but the window is still bounded on purpose.
- A deprecation calendar. When a canned tool supersedes a hand-rolled script, the script gets a retirement date. When the kernel or bpftrace version on the fleet changes, the script gets re-validated or retired. Scripts nobody owns become the reason the next investigation starts from zero.
This is the same lifecycle you already apply to a build tool or a linter. The fact that the artifact is a twenty-line tracing script instead of a compiled binary does not exempt it.
The decision matrix
Before you attach a probe, decide which question you are answering. The matrix below is the one worth putting in the runbook.
| Question | Probe shape | Output | What a long tail means | Next action |
|---|---|---|---|---|
| Are ready tasks waiting for a core? | Timestamp on sched_wakeup keyed by pid; diff on sched_switch when next_pid matches |
Histogram of run-queue latency | CPU contention: more runnable tasks than cores | Fix capacity or concurrency: more cores, fewer parallel jobs, cgroup weights, pinning |
| Are tasks blocked, and on what? | Timestamp on sched_switch keyed by prev_pid; diff when next_pid matches |
Histogram of blocked time | Waiting on I/O, locks, or sleeps | Re-run keyed on kernel stack to identify the blocking path |
| Which build process is stuck? | Blocked-time script with a predicate on comm or pid |
Histogram per process | One compiler, linker, or package manager dominates | Narrow the investigation to that process’s I/O and lock behavior |
| Where in the kernel is it stuck? | Blocked-time script keyed on kstack at switch-out |
Histogram by kernel stack | A specific blocking path (filesystem, network, futex, page fault) | Fix the I/O path, the lock, or the dependency; or change the build graph to stop waiting on it |
The matrix also tells you when to stop tracing. If the run-queue tail is long and the blocked-time tail is short, you have a capacity answer and no amount of stack-trace archaeology will improve it. If the blocked-time tail is long and the run-queue tail is short, you have a waiting answer and adding cores will not help. If both tails are short and the build is still slow, the problem is on-CPU — go back to the profiler you started with, because this time it is the right instrument.
What this does not tell you
Off-CPU tracing tells you where time went. It does not tell you whether that time was necessary. A build that blocks for two seconds on a network filesystem read is not automatically a build with a problem; it may be a build that is correctly waiting for a dependency it cannot avoid. The histogram is evidence, not a verdict. The verdict comes from comparing the observed blocking against what the build graph says should be happening — and that comparison is a design-doc exercise, not a tracing exercise.
It also does not replace per-target build timing. If your build orchestrator already reports which targets are slow, start there. Off-CPU tracing is for the case where the orchestrator says a target took forty seconds and the CPU profiler says the process used two seconds of CPU — the thirty-eight-second gap is the thing you are chasing, and the scheduler tracepoints are how you see into it.
FAQ
Do I need root to run these scripts?
Attaching to kernel tracepoints with bpftrace typically requires elevated privileges. The exact capability requirements depend on your kernel version and how bpftrace was built; check bpftrace --info on the actual runner image rather than assuming. If your build agents run unprivileged, that is a constraint to design around, not a detail to discover during an incident.
Can I run this on every build?
No. Scheduler events are frequent, and even with in-kernel aggregation the cost is not zero. Treat off-CPU tracing as a diagnostic you attach to a representative build when you have a question, not as a permanent telemetry stream. If you want continuous visibility, the right artifact is a periodic sampled capture with a documented window, owned by the platform team, not a trace attached to every job.
Why not just use perf?
You can, and for some questions you should. The trade-off is in how the data is handled: event-dumping tracers push volume to userspace and post-process it, while in-kernel aggregation summarizes before anything crosses the boundary. On a shared build fleet where perturbation matters, the aggregation approach is usually the better fit. On a single machine where you want full event detail, the calculus may differ. Measure the overhead of whichever you pick; do not assume.
What if the histogram is all in the first bucket?
Then most context switches are fast and the tail is where the story is. A distribution dominated by sub-millisecond entries with a thin tail out at seconds is normal for a build that is mostly fine and occasionally blocked. Read the tail, not the mode. If the tail is empty too, the build is not off-CPU-bound and you should look elsewhere.
How do I know the trace did not change the result?
Run the same build with and without the trace and compare wall clock. If the difference is within your normal run-to-run variance, the trace is admissible. If it is not, shorten the window or narrow the predicate. Document the comparison in the runbook so the next person does not have to re-derive it.
The point
An idle runner and a slow build are not a contradiction. They are a signal that you are measuring the wrong thing. The fix is not a better CPU profiler; it is a second instrument that sees the time the first one cannot. bpftrace is a capable instrument for that job, but the capability that matters is the discipline around it: separate run-queue latency from blocked time, aggregate in the kernel so the measurement stays admissible, and run the whole thing as an owned capability with a runbook and a retirement date rather than a one-liner that lives in someone’s shell history. The tool is a liability until the benchmark says otherwise. The benchmark, in this case, is a histogram with a tail you can explain.