Averages hide the tail
One healthy average can coexist with a handful of slow events that users actually feel.
eBPF-powered Linux performance diagnostics
Find the cause behind the latency.
rand-fast measures scheduler latency, CPU, disk I/O, off-CPU waits, and memory for a single process, reports TCP as unavailable when it cannot prove ownership, then ranks what it found. A vague slowdown becomes a signal you can investigate.
$ sudo fast diagnose --pid 1234 --duration 10s
Example output
PID: fast-workload (363)Duration: 4s
Measured
scheduler500 us p95
off-cpu1.46 s total
cpu12.5 % of host
Ranked causes
on a futexLock contention 100.0%
share of off-CPU time waiting on a futex: 100.0% (threshold 33.0%)
Every ranked line carries the measurement and the threshold behind it.
The problem
Latency is the symptom.
The queue is the clue.
CPU graphs can tell you that a service is busy. They rarely tell you why one request waited longer than the next. Scheduler latency makes that hidden wait measurable.
Host-wide data loses the culprit
Start with the process that regressed, then follow its threads and the CPUs they wait for.
Kernel signals add context
Wakeups and context switches show the gap between work becoming runnable and actually running.
How it works
From kernel events to a ranked answer.
One eBPF load, every collector on its own stream, one ranking built from what they measured.
tracepointsThe kernel reports it
sched_wakeup, sched_switch, block_rq_issue, sched_process_exit, and page-fault hooks. TCP tracepoints are intentionally not attached. Startup validates the scheduler and block-request BTF layouts against fixtures captured from the pinned 6.1, 6.6, 6.12 and 7.2 kernels; unknown layouts are rejected rather than misreported. No application instrumentation.
one loadEvery stream runs at once
One reader thread per stream per CPU, keyed to the target's threads. Block-I/O completions pair by live request identity, while sched_process_exit clears pending wait state before a TID can be reused. TCP tracepoints are not attached: neither the softirq current task nor a remembered pointer proves socket ownership, so TCP is reported as unavailable rather than guessed at.
rankedA cause, with the evidence
Each cause scores against a documented threshold and the ranking shows the measurement and the threshold together, so it can be argued with.
Checked, not claimed
Every number below came out of
a verification run.
These are cross-checks against a second source or against a fixture built to break one thing. Where a measurement could not be taken, it is reported as unavailable rather than as zero.
TCP attribution
off
no guessed RTT or loss rate; unavailable is explicit
Page faults
100.00%
eBPF counters against /proc, at 1.0M faults per second
Diagnosis scenarios
9/9
declared QEMU scenarios, each with fixture-reported evidence
Kernel cells
5cells
pinned CI cells: 6.1, 6.6, 6.12, 7.2 and a PSI-off negative
What it measures
Six signals, one process, one answer.
Start with the process you care about. Keep the output small enough to act on.
01Process-first
Measure the workload that hurts
Focus on one PID and its threads instead of drowning in host-wide averages. Threads are re-synced during collection, exit cleanup prevents stale wait state from crossing a reused TID, and --progress reports the collection window on stderr.
02Every signal at once
One load, every collector
Scheduler, CPU, block I/O, off-CPU, and memory are observed together from a single eBPF load, with the streams in parallel. TCP is reported as unavailable rather than guessed at, and startup rejects any layout outside the pinned 6.1-7.2 kernel matrix.
03Ranked with evidence
A cause, not a dashboard
Every ranked line carries the measurement behind it and the threshold it crossed, so the ranking can be argued with.
04Built for later
Record, then keep the evidence
A flight recorder watches scheduler, I/O, and pressure thresholds, measures each interval on its own, and writes a bounded bundle you can read at 3am. By default it runs until interrupted or its target exits; --duration sets a bound.
Quick start
Point it at one process.
fast diagnose collects every signal from one eBPF load and ranks the causes. For a single-signal run, use --pid with a bounded --duration; pass --format json to feed another tool, and --progress for a stderr timeline. The flight recorder runs until interrupted or its target exits by default.
shell
$ sudo fast diagnose --pid 1234 --duration 10sThe first measurement is the fastest way in.