Why the Scheduler, Not the App, Decides When Your System Stalls
Archi3 min read
Contents
- The Stall Nobody Can Reproduce
- Where does a freeze like that actually start?
- Is that a tuning problem or a design problem?
- What about memory? Everyone points at swap.
- Can you see this without a profiler?
- Why do defaults make this worse?
- What would you change first?
- Is there a case where the app really is at fault?
The Stall Nobody Can Reproduce#
A machine freezes for two seconds, the cursor jumps, the audio glitches, and then everything is fine. Users report it as an app problem. It usually is not.
We spoke with a kernel developer who has spent a decade on scheduler and memory management code in production Linux distributions. The conversation has been edited for length and clarity.
Where does a freeze like that actually start?#
The run queue. When more threads are runnable than there are cores, the scheduler has to pick a loser. If a thread holding a lock gets preempted and the thread that needs that lock keeps running, you get priority inversion. The high priority task sits idle while a low priority one finishes. On a desktop with 16 threads competing for 8 cores, that inversion can stretch into tens of milliseconds. Stretch a few of those together and the compositor misses a frame. The user sees a stutter and blames the browser.
Is that a tuning problem or a design problem?#
Both, and they trade off. A fair scheduler gives every thread a slice. A latency-focused scheduler gives interactive threads a boost. You cannot maximize both. The kernel ships with heuristics that guess whether a thread is interactive based on sleep patterns. Guess wrong and a background compiler gets a boost it did not need, or a video call gets deprioritized because it polls in an unusual rhythm.
What about memory? Everyone points at swap.#
The reclaim path is where stalls get long. When free pages run low, the kernel has to evict. Evicting clean file pages is cheap. Evicting dirty anonymous pages means writing to disk first. If the writeback queue is full, the allocation stalls until it drains. A process asking for one page can block for hundreds of milliseconds behind a flush it never asked for. That is the freeze. The app did nothing wrong.
Can you see this without a profiler?#
Pressure stall information is the honest signal. It reports the percentage of time tasks were stalled waiting on CPU, memory, or IO. If memory pressure sits at 20 percent while the machine feels slow, the problem is reclaim, not the app. Most users never look at it because it is not in the task manager by default.
Why do defaults make this worse?#
Defaults are chosen for the median workload, which means a server profile can land on a laptop. Transparent huge pages are a good example. Mapping memory in 2MB chunks cuts TLB misses and helps databases. It also increases the cost of a single fault and can inflate memory use on small machines. A distro enables it globally, a desktop workload suffers, and the bug report says the desktop environment is slow.
What would you change first?#
Make the cost visible. Show reclaim stalls and run queue latency in the same panel where people see CPU percentage. A user who sees 400 milliseconds of memory stall will stop reinstalling their browser. Beyond that, keep the interactive heuristics conservative and let people opt into aggressive settings. The kernel already knows the data. It just does not surface it where the person holding the mouse will find it.
Is there a case where the app really is at fault?#
Plenty. An app that spawns one thread per request will drown any scheduler. An app that touches every page of a large file at startup will trigger reclaim no matter how the kernel is tuned. But those are architectural mistakes, and they show up as consistent load, not as intermittent two-second freezes. The intermittent ones almost always trace back to a scheduling decision or a reclaim stall. The app is the victim, not the cause.
More articles