Real-time latency on Linux: measuring, understanding and fixing it with PREEMPT_RT
"We enabled PREEMPT_RT and we still see 800 microsecond spikes." That is the sentence you hear most often, and it reflects a misunderstanding: the patch does not reduce latency, it makes the kernel preemptible. What remains afterwards is your own non-preemptible sections, and the hardware.
What PREEMPT_RT actually changes
Three transformations, and nothing else.
Spinlocks become priority-inheriting mutexes. In the mainline kernel, spin_lock() disables preemption: while the critical section runs, no higher-priority task gets in. Under PREEMPT_RT, most spinlock_t are converted to sleeping rt_mutex. A high-priority task can preempt the lock holder, and priority inheritance prevents inversion. What stays unconverted is raw_spinlock_t, used where the kernel cannot sleep: the scheduler, low-level IRQ handling, entry code.
Interrupt handlers become threads. A handler turns into a schedulable kernel task, SCHED_FIFO priority 50 by default. Only handlers marked IRQF_NO_THREAD stay in interrupt context. One direct and often overlooked consequence: your real-time task at priority 80 preempts the network controller's interrupt thread. Leave your task at 40 and the opposite happens.
Softirqs are threaded too, which removes the classic case where sustained network traffic blocked the return to user space for hundreds of microseconds.
Measure before you fix
Without a number, every optimisation is a belief. The reference tool is cyclictest, from the rt-tests package.
cyclictest --mlockall --priority=80 --interval=200 \
--distance=0 --histogram=2000 --duration=6h \
--affinity=3 --quiet
Three choices matter in that line. --mlockall locks pages in memory, otherwise a major page fault pollutes the measurement. --distance=0 keeps a constant interval between threads instead of staggering it. And above all --duration=6h: a ten minute run tells you nothing. The worst case happens on a rare event, a log rotation, a flash garbage collection cycle, a DRAM controller refresh.
What matters is never the average. An 8 microsecond average with a 900 microsecond maximum makes a system unusable for a 10 kHz control loop. Read the histogram, not the summary.
Measure under load, and under a load that resembles yours: stress-ng --cpu 4 --io 2 --vm 2, plus sustained network traffic and disk writes. An idle system always produces good numbers.
Finding the culprit
When the maximum is bad, ftrace answers the question "who held on".
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer
echo 200 > /sys/kernel/debug/tracing/tracing_thresh
echo 1 > /sys/kernel/debug/tracing/tracing_on
The wakeup_rt tracer measures the delay between a real-time task being woken and actually running, and only records cases above the threshold, here 200 microseconds. The trace file then holds the stack that caused the delay.
To hunt down interrupt-disabled sections, irqsoff gives you the offending function and its duration. That tracer is what exposes badly written drivers, the ones polling a hardware register with interrupts masked.
The five real causes, by frequency
CPU power management. This is the first cause, by a wide margin. Exiting a deep C-state costs tens to hundreds of microseconds. Set the governor to performance and forbid deep states:
cpupower frequency-set -g performance
cpupower idle-set -D 0
On an embedded platform, also check the idle states declared in the devicetree: idle-states with a high exit-latency-us cancels every software effort you make.
Badly ordered priorities. List what runs above you: ps -eLo pid,tid,class,rtprio,comm | grep FF. It is common to find irq/24-eth0 at 50 and your own task at 40. Raise your task, or lower the interrupt threads that do not concern you.
Core sharing. Isolate. On the kernel command line, isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3 removes those cores from the general scheduler, drops the periodic tick when a single task runs, and offloads RCU callbacks. Then pin your task with taskset or sched_setaffinity, and push non-critical interrupts elsewhere through /proc/irq/*/smp_affinity.
Page faults. Call mlockall(MCL_CURRENT | MCL_FUTURE) at startup, pre-touch your stack and pre-allocate everything you will use. A malloc inside the real-time loop is a defect, not a missed optimisation.
The hardware itself. On x86, SMIs are invisible to the kernel and can steal hundreds of microseconds: hwlatdetect measures them. On ARM, the equivalent culprit is often a memory bus saturated by a video DMA, or task migration on a big.LITTLE topology.
Achievable orders of magnitude
| Configuration | Typical worst case |
|---|---|
| Mainline kernel, voluntary preemption | a few milliseconds |
Full PREEMPT, untuned | 300 microseconds to 1 ms |
| Tuned PREEMPT_RT, isolated core, ARM Cortex-A | 30 to 80 microseconds |
| Tuned PREEMPT_RT, x86 without SMIs, isolated core | 10 to 30 microseconds |
| Bare-metal Cortex-M | 1 to 3 microseconds |
The figures in this section are observed orders of magnitude, not specifications. They depend on silicon, compiler and configuration. Measure your own before sizing a product on them.
These figures assume a long measurement under load. They are orders of magnitude, not a guarantee: the same configuration on two different SoCs can vary by a factor of three depending on the memory controller.
The last row deserves a pause. If your constraint is below 10 microseconds, Linux is not the right answer, even with PREEMPT_RT. The pattern that works is to move the critical loop onto a coprocessor, a Cortex-M on a heterogeneous SoC or an isolated core under a dedicated executive, and keep Linux for everything else.
What not to do
Do not put your task at SCHED_FIFO priority 99. You then sit above the kernel watchdogs and the migration threads; a loop with no blocking point freezes the machine. Priority 80 leaves headroom.
Do not use SCHED_FIFO without RLIMIT_RTTIME or sched_rt_runtime_us as a safety net during development. The first infinite-loop bug will cost you a hardware reset.
And never draw a conclusion from a ten minute run.
References
- Linux Foundation, Real-Time Linux wiki, PREEMPT_RT and lock conversion
- rt-tests and cyclictest
- Kernel.org, ftrace,
wakeup_rtandirqsofftracers - Kernel.org, Kernel parameters,
isolcpus,nohz_full,rcu_nocbs - Kernel.org, NO_HZ
Going further
These settings make sense once you have written a driver that respects the constraints of a preemptible kernel, and read an irqsoff trace of your own code. That is what our courses on real-time and multicore programming, and on Linux drivers, cover.