Driver I/O, interrupts & DMA

Driver code sits at a dangerous boundary: userspace supplies untrusted requests, devices operate asynchronously, and callbacks can run while teardown is trying to free state. This chapter follows one fictional transfer through those boundaries. The mechanisms are real Linux concepts; the transfer is a teaching model, not a driver to load.

Before you start#

Read Drivers and device trees, The kernel boundary, C/C++ source reading, and Concurrency and lifetime. Documentation references use Linux 6.1 where possible. Required practice is source/data analysis or an unprivileged Python example; it does not access device nodes, DMA memory, or interrupts.

Learning goals#

Trace open → request validation → user copy → device submission → interrupt/deferred completion → result → teardown. Distinguish CPU virtual, physical, and DMA addresses; explain ownership/cache/order rules; and identify why a successful ioctl return, interrupt, or buffer allocation does not independently establish a correct transfer.

A device node is an interface, not the device itself#

Opening /dev/example resolves a filesystem object, checks applicable permissions/policy, and reaches a registered driver/subsystem path. A resulting descriptor is a reference to an open file object. Operations dispatch through supported file/subsystem methods. The pathname alone does not identify which callback handles a request or whether hardware is ready.

An ioctl is an interface-specific command operation. It is useful when an operation does not fit ordinary read/write semantics, but it creates a long-lived userspace ABI. A command number, structure layout, field width, reserved fields, and compatibility behavior all matter. Kernel-internal pointers and long layout are poor accidental ABI choices across 32-/64-bit userspace. See Linux 6.1 ioctl guidance.

Why not expose arbitrary registers? A subsystem API lets the kernel preserve ownership, validate requests, coordinate power/lifetime, and mediate competing clients. A raw register interface can let userspace corrupt state, bypass sequencing, or access other memory. Existing input, networking, graphics, storage, and other subsystem contracts are preferable to inventing a private command protocol without need.

First boundary: untrusted request memory#

Suppose a fictional command requests count records of fixed size. The driver must validate the command and copied fields before using them:

• Does the command select a supported operation and structure version?
• Are lengths, flags, indexes, and reserved fields valid?
• Can count * record_size overflow the kernel's arithmetic or exceed a resource limit?
• Does the caller have authority over the referenced buffer/device/context?
• Can removal, cancellation, or another request invalidate that state?

copy_from_user() and copy_to_user() are fault-aware helpers; they do not mean the copied values are semantically trusted. Their return is the number of bytes not copied, not a conventional negative errno. A handler must not accidentally treat a partial copy as a complete initialized request. Also do not repeatedly reread an untrusted structure's fields and assume userspace cannot change them between checks; copy/validate a bounded request according to the interface's ownership rules.

User-copy helpers can fault and sleep in ordinary contexts. They are not appropriate inside arbitrary hard-IRQ or spinlock-held sections. The kernel user-access documentation describes the API; exact context rules apply to the selected helper.

Bounded length example

HOST Python 3; complete original teaching code, not a kernel implementation. The arithmetic is written in the overflow-avoiding form a fixed-width implementation would need, although Python integers themselves do not overflow. No allocation or hardware I/O occurs.

def transfer_bytes(count, record_size, limit):
    if count < 0 or record_size <= 0 or limit < 0:
        raise ValueError("invalid size")
    if count > limit // record_size:
        raise ValueError("transfer exceeds limit")
    return count * record_size

assert transfer_bytes(0, 16, 4096) == 0
assert transfer_bytes(256, 16, 4096) == 4096
for args in [(257, 16, 4096), (-1, 16, 4096), (1, 0, 4096)]:
    try:
        transfer_bytes(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(args)
print("5 length-validation cases passed; no device accessed")

A real C ABI also needs declared integer widths, conversion checks, a chosen maximum, and tested compat handling. This model does not parse user memory or prove a real ioctl safe.

Second boundary: three address spaces#

An IOMMU can translate and restrict device memory accesses. It is not guaranteed to be present, enabled, or configured identically on every board. A driver uses DMA APIs rather than casting a CPU pointer into a device register. The API accounts for platform mapping, masks, and cache requirements; drivers must check allocation/mapping errors. See the DMA API HOWTO.

DMA exists so devices can transfer data without the CPU copying each byte itself. It does not mean no CPU work, no cache synchronization, no security checks, or instant completion. A zero-copy design can still have mapping, ownership, cache, and scheduling costs.

Third boundary: who owns the buffer now?#

For a streaming mapping, the driver specifies direction, hands the appropriate buffer ownership to the device, and synchronizes/unmaps at the required transitions. The CPU cannot casually rewrite a buffer while the device is reading it or consume device output before the transfer/synchronization rules permit it.

Coherent allocation provides a different cache-visibility contract, but does not remove ordering requirements. Publishing a “ready” flag before descriptor fields are visible can let a device consume half-initialized work. Correct memory barriers, DMA API calls, and device protocol are required; declaring a pointer volatile is not a portable substitute. Conversely, sprinkling cache flushes everywhere without an ownership model can be both wrong and expensive.

Fictional receive transfer:

1. Driver allocates/maps a supported buffer and validates resources.
2. Driver prepares/publishes a descriptor according to device ordering rules.
3. Device owns the transfer interval and writes output.
4. Device reports completion through its supported mechanism.
5. Driver establishes completion and the required CPU visibility.
6. CPU validates/consumes the result; the buffer can later be recycled or released.

The exact direction and synchronization calls depend on the mapping type and device protocol. This is an ownership sequence, not code to apply to an arbitrary controller.

Interrupts notify; deferred work finishes#

A hard interrupt handler runs under restrictions: it must not perform operations that can sleep, and it should bound its work. It commonly acknowledges/records the event and defers slower handling. Threaded IRQ handlers or workqueues provide other execution contexts with different rules. “Runs after an interrupt” does not identify which context you are in.

A completion interrupt is not necessarily proof that every requested byte is valid. The driver still needs to interpret status, handle errors/partial results, synchronize memory as required, and match the completion to a live request. Spurious/shared interrupts, stale completions, device reset, and timeout can change the interpretation.

The earlier GPIO-key example supplies a real smaller case: gpio_keys.c separates interrupt notification from debounce/work and cancels asynchronous state during cleanup. It is not a DMA driver, but its callback lifetime lesson transfers directly.

Teardown is part of the transfer protocol#

Consider removal while DMA is outstanding. Freeing the buffer first can let the device write into memory reassigned to another object. Cancelling one work item is insufficient if hardware or an IRQ path can queue more work afterward.

A correct device-specific teardown generally needs to stop new submissions, mark state so racing callers cannot restart work, stop/quiesce hardware, synchronize relevant interrupts/work/completions, release mappings, and free state only after no user remains. The exact ordering is device/subsystem-specific. Waiting while holding the lock a completion needs can deadlock, so “wait for everything” also requires a lock/context design.

Managed devm_* allocation helps register resource cleanup; it does not invent this asynchronous protocol. Hardware may continue independently of a C++/C pointer's lexical lifetime. The generic IRQ API and DMA documentation define relevant primitives, not a universal board-reset recipe.

Practice#

Analyze this synthetic transfer log:

t0 request 18 accepted, buffer B mapped
t1 descriptor published, device starts
t2 removal begins; B freed
t3 interrupt for request 18
t4 deferred completion tries to consume B

Identify two hazards: device access after B is freed and deferred CPU access after B is freed. Propose the evidence needed to establish quiescence before release: submission gate, device stop/completion state, interrupt synchronization, deferred-work drain, and mapping lifetime. Do not say only “add a mutex”; the device is not a CPU thread that acquires it.

Then change t3 to a timeout rather than an interrupt. Can you free B immediately? No: a timeout describes the software wait, not proof the device stopped. A recovery/reset protocol must establish that the device can no longer access the buffer. This is why correct error paths are as important as the successful transfer.

Run the length model and compare the boundary and first-rejected case. On paper, record one driver ABI, one user-copy, one address, one ownership, and one teardown invariant. No actual kernel transfer is claimed.

Knowledge check#

1. What does a nonzero copy_from_user result mean?
2. Can a CPU pointer be assumed to equal the DMA address?
3. Does coherent memory eliminate all ordering barriers?
4. Does a software timeout prove hardware has stopped writing?

Answers#

1. Some bytes were not copied; it is not a successful complete request or a negative errno.
2. No. Use the appropriate DMA mapping API and handle its errors/constraints.
3. No. Visibility/coherency and ordering are different obligations.
4. No. Device-specific quiescence or reset evidence is required before releasing its buffers.

Checkpoint#

Ready when the five model cases pass and your transfer/teardown timeline states ownership, execution context, and release conditions at each step. Revise if an IRQ, timeout, volatile, or devm is treated as universal synchronization. Return to the first unsafe ownership transition and write the missing invariant. Real DMA, interrupt timing, and board correctness remain not yet evidenced without supported hardware, a real implementation, and tests.

Sources#

• Linux 6.1 ioctl guidance — persistent userspace ABI design and layout concerns.
• Linux 6.1 user-memory access API — user-copy semantics and restrictions.
• Linux 6.1 DMA API HOWTO and DMA API — mapping, masks, directions, synchronization, and errors.
• Linux 6.1 generic IRQ API — interrupt/context/synchronization primitives.
• Pinned ACK gpio_keys.c — real notification/deferred-work/cleanup example, not a DMA implementation.