Averages hide the tail
One healthy average can coexist with a handful of slow events that users actually feel.
eBPF-powered Linux performance diagnostics
Find the wait behind the latency.
rand-fast measures scheduler latency for a specific process, turning a vague slowdown into a signal you can investigate.
$ sudo fast sched --pid 1234 --duration 10s
Example output
PID: api-server (1234)Duration: 10s
Scheduler latency
p5021 µs
p95840 µs
p997.2 ms
max84.1 ms
Slow events> 1ms 421> 10ms 38> 50ms 4
From runnable to running, measured at the kernel boundary.
The problem
Latency is the symptom.
The queue is the clue.
CPU graphs can tell you that a service is busy. They rarely tell you why one request waited longer than the next. Scheduler latency makes that hidden wait measurable.
Host-wide data loses the culprit
Start with the process that regressed, then follow its threads and the CPUs they wait for.
Kernel signals add context
Wakeups and context switches show the gap between work becoming runnable and actually running.
How it works
Follow the scheduling path.
Three kernel events. One answer to the question: how long did this process wait?
sched_wakeupWork becomes runnable
rand-fast records when a target thread is woken and ready to compete for CPU time.
runnableThe wait is measured
The elapsed time between wakeup and execution becomes a scheduler latency event.
sched_switchWork starts running
Events are aggregated by percentile, threshold, process, and CPU for a useful signal.
v0.1 / Scheduler
A focused signal for a noisy system.
Start with the process you care about. Keep the output small enough to act on.
01Process-first
Measure the workload that hurts
Focus on one PID and its threads instead of drowning in host-wide averages.
02Tail-aware
See the latency tail
p50, p95, p99, and max expose the waits that an average keeps hidden.
03CPU context
Keep the CPU in the picture
Break scheduler latency down by CPU to reveal where runnable work is waiting.
04Kernel signal
Observe the scheduling path
Trace wakeups and switches with eBPF, without adding application instrumentation.
Quick start
Start with one process.
Point fast at a PID, give it a measurement window, and inspect the scheduler latency tail.
shell
$ sudo fast sched --pid 1234 --duration 10sThe roadmap
One signal at a time.
One diagnosis closer.
Scheduler latency is the first instrument. The roadmap expands rand-fast from a focused probe into a complete Linux performance flight recorder.
Scheduler
Validated in QEMU (7.2.0-rc6): idle vs contention shows clear scheduler latency increase.
- Scheduler latency
- PID-level CLI
- Percentile statistics
- QEMU smoke test
CPU
Validated: CPU usage, on-CPU hot stacks, and scheduler correlation.
- CPU usage
- On-CPU profiling
- Hot stacks
- QEMU verified
I/O
Validated: disk latency per device and per-process I/O bottlenecks.
- Disk latency
- Block devices
- Process I/O
- Configurable threshold
Network
Validated: TCP RTT, retransmissions, and slow endpoints.
- TCP RTT
- Retransmissions
- Connection latency
- Local fixture
Off-CPU
Validated: futex/lock waits and off-CPU stacks.
- Futex waits
- Lock contention
- Off-CPU stacks
- QEMU verified
Memory
Validated: PSI, page faults, reclaim, and swap pressure.
- PSI
- Page faults
- Swap pressure
- Controlled fixture
Diagnose
Validated: one-command ranked diagnosis with evidence.
- Signal correlation
- Ranked causes
- One command
- Deterministic ranking
Flight Recorder
Validated: low-cost rolling history with incident preservation.
- Continuous capture
- Incident preservation
- Low overhead
- Configurable trigger
The first measurement is the fastest way in.