Averages hide the tail
One healthy average can coexist with a handful of slow events that users actually feel.
eBPF-powered Linux performance diagnostics
Find the cause behind the latency.
rand-fast measures scheduler latency, CPU, disk I/O, off-CPU waits, and memory for a single process, reports TCP as unavailable when it cannot prove ownership, then ranks what it found. A vague slowdown becomes a signal you can investigate.
$ sudo fast diagnose --pid 1234 --duration 10s
Example output
PID: fast-workload (363)Duration: 4s
Measured
scheduler500 us p95
off-cpu1.46 s total
cpu88.3 % of one
Ranked causes
on a futexLock contention 100.0%
share of off-CPU time waiting on a futex: 100.0% (threshold 33.0%)
Every ranked line carries the measurement and the threshold behind it.
The problem
Latency is the symptom.
The queue is the clue.
CPU graphs can tell you that a service is busy. They rarely tell you why one request waited longer than the next. Scheduler latency makes that hidden wait measurable.
Host-wide data loses the culprit
Start with the process that regressed, then follow its threads and the CPUs they wait for.
Kernel signals add context
Wakeups and context switches show the gap between work becoming runnable and actually running.
How it works
From kernel events to a ranked answer.
One eBPF load, every collector on its own stream, one ranking built from what they measured.
tracepointsThe kernel reports it
sched_wakeup, sched_switch, block_rq_issue, sched_process_exit, and page-fault hooks. TCP tracepoints are intentionally not attached. Startup validates the scheduler and block-request BTF layouts tested on Linux 6.12.105 and 7.2.0-rc6; unknown layouts are rejected rather than misreported. No application instrumentation.
one loadEvery stream runs at once
One reader thread per stream per CPU, keyed to the target's threads. Block-I/O completions pair by live request identity, while sched_process_exit clears pending wait state before a TID can be reused. Retransmissions are attributed by the socket's 5-tuple because they arrive in softirq context where the running thread is unrelated.
rankedA cause, with the evidence
Each cause scores against a documented threshold and the ranking shows the measurement and the threshold together, so it can be argued with.
Checked, not claimed
Every number below came out of
a verification run.
These are cross-checks against a second source or against a fixture built to break one thing. Where a measurement could not be taken, it is reported as unavailable rather than as zero.
TCP attribution
off
no guessed RTT or loss rate; unavailable is explicit
Page faults
100.00%
eBPF counters against /proc, at 1.0M faults per second
Diagnosis scenarios
5/5
one bottleneck each, the intended cause ranked first
Recorder cost
4%
of one core on a steady server, measured not asserted
What it measures
Six signals, one process, one answer.
Start with the process you care about. Keep the output small enough to act on.
01Process-first
Measure the workload that hurts
Focus on one PID and its threads instead of drowning in host-wide averages. Threads are re-synced during collection, and exit cleanup prevents stale wait state from crossing a reused TID.
02Every signal at once
One load, every collector
Scheduler, CPU, block I/O, off-CPU, and memory are observed together from a single eBPF load, with the streams in parallel. TCP is reported as unavailable rather than guessed at, and startup rejects unverified tracepoint and block-request layouts.
03Ranked with evidence
A cause, not a dashboard
Every ranked line carries the measurement behind it and the threshold it crossed, so the ranking can be argued with.
04Built for later
Record, then keep the evidence
A flight recorder watches scheduler, I/O, and pressure thresholds, measures each interval on its own, and writes a bounded bundle you can read at 3am. By default it runs until interrupted or its target exits; --duration sets a bound.
Quick start
Point it at one process.
fast diagnose collects every signal from one eBPF load and ranks the causes. For a single-signal run, use --pid with a bounded --duration; pass --format json to feed another tool. The flight recorder runs until interrupted or its target exits by default.
shell
$ sudo fast diagnose --pid 1234 --duration 10sThe first measurement is the fastest way in.