Cache coherency and DMA on ARM: why your buffers get corrupted

The symptom is always the same. The DMA transfer completes, the done flag is raised, and yet the buffer holds the data from the previous transfer, or a mixture of both. You suspect the peripheral, you reread the datasheet, you add a volatile. The culprit is almost always the cache, and the fix depends on the direction of the transfer.

The problem in one sentence

The processor sees memory through its data cache. The DMA controller, unless stated otherwise, writes straight to DRAM. Those are two views of the same address, and nothing synchronises them automatically.

Two symmetric mistakes follow, and they must be told apart because they do not have the same fix.

Device to memory. The DMA writes to DRAM. The processor may still hold cache lines for those addresses, loaded before the transfer. It reads from the cache and gets stale data. The fix is to invalidate the cache over the range, after the transfer completes and before reading.

Memory to device. The processor writes into the buffer. With a write-back cache, which is the default on Cortex-A, those writes stay in the cache. The DMA reads DRAM, which has not been updated, and sends stale bytes. The fix is to clean the cache, that is, force dirty lines out to memory, before starting the transfer.

Getting these two the wrong way round is the most frequent error. Invalidating before an outgoing transfer throws away your data; cleaning after an incoming transfer writes stale data over what the DMA just delivered.

The shared line trap

This is the defect you only find after several days, because it only shows up occasionally.

Cache operations apply to whole lines, typically 32 or 64 bytes depending on the core. If your DMA buffer does not start on a line boundary, or if its size is not a multiple of the line size, the first and last lines also hold neighbouring data that is not part of the transfer.

Invalidating those lines destroys pending writes to those neighbours. The adjacent structure loses a field, non-deterministically, depending on what the scheduler happened to be doing.

So the rule is absolute: align the buffer on the cache line size, and round its size up to a multiple of it.

#define CACHE_LINE 64
static uint8_t rx_buf[512] __attribute__((aligned(CACHE_LINE)));

And never put a DMA buffer inside a structure next to other fields, nor on the stack.

Barriers, and what they do not do

ARM memory barriers are regularly offered as a cure for the cache problem. They do not solve it. These are two distinct mechanisms.

DMB guarantees the observed order of memory accesses: accesses before the barrier are observed before those after it. DSB is stronger: it waits for the actual completion of all outstanding accesses, including cache maintenance operations. ISB flushes the instruction pipeline and forces following instructions to be refetched, which is needed after changing system configuration, page tables or cache enabling for instance.

So the correct order to start an outgoing transfer is:

1. write the data into the buffer
2. clean the cache over the buffer range
3. DSB          (wait for the clean to actually complete)
4. write the DMA start register

The DSB at step 3 is mandatory. Without it, the command register write can be observed by the peripheral before the clean has finished, and the DMA starts on still-incoherent memory.

Cortex-M: the same problem, one layer fewer

On Cortex-M0 to M3 there is no data cache, so the problem does not exist. That is why a codebase proven on an M4 breaks when it moves to an M7.

On Cortex-M7 there is a data cache, and the fix goes through the CMSIS functions:

SCB_CleanDCache_by_Addr((uint32_t *)tx_buf, sizeof(tx_buf));      /* before sending */
SCB_InvalidateDCache_by_Addr((uint32_t *)rx_buf, sizeof(rx_buf)); /* after receiving */

These functions require an address aligned on 32 bytes, the M7 line size, exposed by CMSIS as __SCB_DCACHE_LINE_SIZE. An unaligned address is silently rounded down, which brings back exactly the shared line trap.

The clean alternative on Cortex-M is the MPU: declare the DMA buffer region as Device, or as Normal, non-cacheable. You lose processor access throughput, but the problem disappears by construction, which beats an invalidation forgotten in a rare error path.

Under Linux, do none of this

If you are writing a Linux driver, the DMA API does the work, and bypassing it is a defect.

dma_alloc_coherent() returns an uncached buffer, or a hardware-coherent one depending on the platform: no maintenance to do. It is the right choice for control structures, descriptor rings, anything accessed often by both sides.

For data buffers, dma_map_single() and dma_unmap_single() with the correct direction, DMA_TO_DEVICE or DMA_FROM_DEVICE, trigger the right cache operations for the target architecture. The point to remember: between the map and the unmap, the buffer belongs to the device. Touching it is a bug, even for reading. If you must access it in between, use dma_sync_single_for_cpu() then dma_sync_single_for_device().

Writing your own cache operations in a Linux driver almost always signals a misuse of the API.

What makes this hard to diagnose

These defects are not reproducible on demand, and that is what costs time.

The transfer works when the buffer was not in the cache, on the first pass for instance. It fails when another part of the code has just read it. On a multicore system it depends on which core runs. Debugging it with a breakpoint makes it disappear, because the pause gives write-back time to drain.

One useful reflex: if a DMA bug goes away when you disable the data cache, the cause is established, and what is left is finding which range is not being maintained correctly.

References

• Arm, CMSIS-Core, D-Cache Functions, SCB_CleanDCache_by_Addr, __SCB_DCACHE_LINE_SIZE
• Arm, Armv8-A memory model, barriers, DMB, DSB and ISB semantics
• Kernel.org, Dynamic DMA mapping guide
• STMicroelectronics, Maintaining cache coherence for DMA buffers

Going further

These mechanisms become clear once you have read the description of a write-back cache, followed an access through the MMU, and used maintenance operations on real hardware. That is what our courses on ARM Cortex-A and R architecture, Cortex-M architecture, and Linux drivers cover.