Averages hide the tail
One healthy average can coexist with a handful of slow events that users actually feel.
eBPF-powered Linux performance diagnostics
Find the cause behind the latency.
rand-fast measures scheduler latency, CPU, disk I/O, network, off-CPU waits, and memory for a single process, then ranks what it found. A vague slowdown becomes a signal you can investigate.
$ sudo fast diagnose --pid 1234 --duration 10s
Example output
PID: fast-workload (363)Duration: 4s
Measured
scheduler500 us p95
off-cpu1.46 s total
cpu88.3 % of one
Ranked causes
on a futexLock contention 100.0%
share of off-CPU time waiting on a futex: 100.0% (threshold 33.0%)
Every ranked line carries the measurement and the threshold behind it.
The problem
Latency is the symptom.
The queue is the clue.
CPU graphs can tell you that a service is busy. They rarely tell you why one request waited longer than the next. Scheduler latency makes that hidden wait measurable.
Host-wide data loses the culprit
Start with the process that regressed, then follow its threads and the CPUs they wait for.
Kernel signals add context
Wakeups and context switches show the gap between work becoming runnable and actually running.
How it works
From kernel events to a ranked answer.
One eBPF load, every collector on its own stream, one ranking built from what they measured.
tracepointsThe kernel reports it
sched_wakeup, sched_switch, block_rq_issue, tcp_probe, tcp_retransmit_skb, sched_process_exit, and page-fault hooks. Startup validates the scheduler, TCP, and block-request BTF layouts tested on Linux 6.12.105 and 7.2.0-rc6; unknown layouts are rejected rather than misreported. No application instrumentation.
one loadEvery stream runs at once
One reader thread per stream per CPU, keyed to the target's threads. Block-I/O completions pair by live request identity, while sched_process_exit clears pending wait state before a TID can be reused. Retransmissions are attributed by the socket's 5-tuple because they arrive in softirq context where the running thread is unrelated.
rankedA cause, with the evidence
Each cause scores against a documented threshold and the ranking shows the measurement and the threshold together, so it can be argued with.
Checked, not claimed
Every number below came out of
a verification run.
These are cross-checks against a second source or against a fixture built to break one thing. Where a measurement could not be taken, it is reported as unavailable rather than as zero.
TCP RTT
43us
against 45us from ss -ti on the same socket
Page faults
100.00%
eBPF counters against /proc, at 1.0M faults per second
Diagnosis scenarios
5/5
one bottleneck each, the intended cause ranked first
Recorder cost
4%
of one core on a steady server, measured not asserted
What it measures
Six signals, one process, one answer.
Start with the process you care about. Keep the output small enough to act on.
01Process-first
Measure the workload that hurts
Focus on one PID and its threads instead of drowning in host-wide averages. Threads are re-synced during collection, and exit cleanup prevents stale wait state from crossing a reused TID.
02Every signal at once
One load, every collector
Scheduler, CPU, block I/O, TCP, off-CPU, and memory are observed together from a single eBPF load, with the streams in parallel. Startup rejects unverified tracepoint and block-request layouts instead of reporting misleading measurements.
03Ranked with evidence
A cause, not a dashboard
Every ranked line carries the measurement behind it and the threshold it crossed, so the ranking can be argued with.
04Built for later
Record, then keep the evidence
A flight recorder watches for scheduler, I/O, retransmission, and pressure thresholds, and writes a bundle you can read at 3am. By default it runs until interrupted or its target exits; --duration sets a bound.
Quick start
Point it at one process.
fast diagnose collects every signal from one eBPF load and ranks the causes. For a single-signal run, use --pid with a bounded --duration; pass --format json to feed another tool. The flight recorder runs until interrupted or its target exits by default.
shell
$ sudo fast diagnose --pid 1234 --duration 10sThe roadmap
One signal at a time.
One diagnosis closer.
Scheduler latency was the first instrument. Each release added one more signal and one more check that would catch a regression in it, which is why every row below names what was measured rather than what was intended.
Scheduler
Runnable-to-running latency from sched_wakeup and sched_switch, with percentiles and a slow-event count.
- Scheduler latency
- PID-level CLI
- Percentile statistics
- Idle vs contention shows a clear increase
CPU
On-CPU usage from the accounting counters and hot stacks, symbolized. Sample count tracks frequency times CPU time: 3.96x for 4x the rate.
- CPU usage
- Symbolized hot stacks
- Per-CPU breakdown
- Rate scaling checked
I/O
Block I/O latency is attributed by live request identity per device and operation, so concurrent requests to the same sector cannot collide. The slow-I/O table retains the slowest events.
- Per-request attribution
- Per-device latency
- Slow-I/O table
- Pairing checked
Network
TCP round-trip time measured from tcp_probe, retransmissions counted and attributed by the socket 5-tuple rather than by the running thread. Cross-checked against the kernel.
- TCP RTT
- Socket-keyed retransmissions
- Endpoint ranking by p95
- RTT and cwnd agree with ss
Off-CPU
The blocking stack is captured at switch-out, while the frame that blocked the thread still exists. Wait reasons are classified from the stack, and thread-exit cleanup prevents stale pending state from crossing TID reuse.
- Stack captured at switch-out
- Wait reasons from the stack
- Ranked by total wait time
- A CPU-bound process reports zero waits
Memory
Page faults and direct reclaim counted in kernel maps, because per-fault events cannot keep up with a million faults a second. Agreement with /proc measured at 100.00%.
- Faults and reclaim in kernel maps
- Verdict with documented thresholds
- PSI where the kernel has it
- Absent PSI reported as unavailable
Diagnose
One eBPF load, every stream in parallel, then a ranking built from what was measured. Five fixtures, one bottleneck each, rank the intended cause first.
- One load, parallel streams
- Ranked causes with evidence
- JSON on every subcommand
- Five ranking scenarios checked
Flight Recorder
A rolling window with triggers on scheduler, I/O, retransmission, and pressure thresholds, writing an incident bundle. By default it runs until interrupted or its target exits; --duration sets a bound. Measured cost: 4% of one core on a steady server, and more when the events it watches arrive faster.
- Multi-signal triggers
- Incident bundles with a diagnosis
- Valid history restored; corrupt intervals skipped
- Unbounded default, stopped by Ctrl-C or target exit
- Disk use bounded by a cap
The first measurement is the fastest way in.