Debug the kernel

Before you start#

Required earlier lessons: The kernel boundary, GKI and kernel builds, Drivers and device trees, Concurrency and object lifetime, and ADB and a safe lab. You should know the difference between a process and kernel task, and understand that device debug access is permission/configuration dependent. No crash is induced in this lesson.

Learning goals#

You will choose an observation method based on a question; distinguish text logs, structured trace events, and statistical sampling; do a first-pass reading of an explicitly synthetic oops; explain why exact build symbols matter; and report evidence without mistaking correlation or a stack frame for proven root cause.

Practice#

Ask a question before choosing a tool

Start by stating what failed, when, what changed, and what observation could distinguish likely causes. “The phone is slow” is too broad. “A test process blocks during file reads after a reproducible action” suggests recording syscall duration, scheduling, and I/O. Record device/model, Android build fingerprint, kernel release (uname -a where permitted), reproduction steps, timing, and whether it survives reboot.

The kernel is shared by apps and system services. A kernel failure can freeze or reboot the device. Logs/traces may expose sensitive data, consume resources, or perturb timing. Android production builds may restrict kernel logs and tracing; the exact access policy is device- and build-specific, and permission denial is a policy outcome, not an invitation to bypass it. The AOSP GKI debugging page says that debug features such as KASAN, UBSAN, and debugfs are disabled in GKI and may be unsuitable for production; it does not document log or trace access policy. Use an emulator, authorized development image, or source-only exercise.

Choose an evidence type

• printk/dmesg: human-readable messages in a kernel message buffer. Messages can be filtered, rate-limited, overwritten, or inaccessible. A missing line does not prove code never ran. Adding high-volume logs can alter timing, flood buffers, and leak details.
• Dynamic debug: where configured and permitted, selectively enables existing pr_debug/dev_dbg call sites (dynamic debug, v6.1). Availability depends on configuration and policy; do not expose debug interfaces on production hardware for a tutorial.
• ftrace/tracepoints: structured event/function timing data. Narrow filters and short captures reduce overhead and volume. Buffers can lose events (for example, through overwrite or overrun; see ftrace, v6.1); tracing is not a perfectly passive or complete view.
• perf/sampling: statistical evidence of where CPU time is spent. It may not explain a rare crash or one blocked operation. Android's documented system-tracing workflow can differ from generic Linux tools.

Choose the lowest-impact method that can answer the question, and record settings and limitations. A timestamped log can establish that a callsite emitted a message; a scheduler trace can correlate task transitions; neither alone proves why a delay occurred. Cross-check a hypothesis with an independent observation when possible.

A safe observation exercise

On a Linux host or disposable emulator you are authorized to inspect, run only these read-only commands:

uname -r
cat /proc/uptime
dmesg --level=err,warn

Expected: kernel release; uptime values; and either permitted warning/error messages or a permission-denied result. dmesg --level selects message levels (dmesg(1)). A denied command is a valid recorded result. Do not change permissions, mount debugfs, enable tracing, trigger a crash, or write to kernel control files. Neither empty logs nor denial means “no bug.”

First-pass oops triage (SYNTHETIC teaching sample)

The following is invented for instruction. It is not copied from a real device, not an ACK crash, not a reproducible incident, and its symbols/values do not identify real kernel source. It exists so learners can practice the reading order without creating a crash.

SYNTHETIC ONLY — not a real kernel log
Kernel: 6.1.0-demo (build-id DEMO-0001; symbols DEMO-0001.debug)
CPU: 2  PID: 412  Comm: demo-worker  Tainted: G        W
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
pc : demo_copy+0x24/0x80 [demo_module]
lr : demo_worker+0x58/0xa0 [demo_module]
Call trace:
  demo_copy+0x24/0x80 [demo_module]
  demo_worker+0x58/0xa0 [demo_module]
  process_one_work+0x.../0x... [kernel]
Modules linked in: demo_module

Read the lines in this order:

1. Identity: kernel version/build ID, module list, configuration/toolchain/artifact metadata if available. Without identity, symbolization cannot be trusted.
2. Context: CPU, task PID/name, and taint flags are clues about where/what was running, not proof of the cause.
3. Fault report: the address and faulting program counter (pc) suggest a null-ish dereference at an offset, but the real report would need architecture/register/fault details and the exact source/binary to interpret.
4. Trace: call frames are a path of execution, not a verdict. demo_copy is where this synthetic sample reports the fault; a caller may have supplied invalid state, a race may have corrupted it earlier, or an unrelated defect may have occurred. Do not say “the first frame is root cause” without evidence.
5. Next evidence: preserve the entire report and exact matching unstripped vmlinux/module symbols, build ID, config, compiler/toolchain, modules, and reproduction. Form a falsifiable hypothesis; compare a safe reproduction or source path. Do not symbolize this synthetic text against real ACK symbols.

An oops is a serious kernel error report; depending on state/configuration it may continue in a compromised state or lead to a panic (the panic_on_oops setting controls the latter; see sysctl kernel, v6.1). A panic halts or restarts because safe continuation is not possible. For real line-level symbolization, use the exact matching image and debug data. KASLR (kernel address-space layout randomization) can relocate code, so runtime addresses may require correct relocation metadata/offset (with CONFIG_RANDOMIZE_BASE set, the boot option nokaslr disables it; see kernel parameters, v6.1). A “similar” build is not good enough: wrong-build symbolization can produce plausible but invalid source lines. Preserve exact artifact identity. Linux bug hunting · Reporting issues

pstore can retain selected console/crash records across reboot when configured with a supported backend (for example, ramoops, v6.1, which needs persistent RAM). It is not present on every device, and missing pstore data does not prove no crash occurred. Treat collected logs as sensitive and use the device's authorized collection process.

Preserve the scheduler tracepoint source example

At the pinned ACK commit, include/trace/events/sched.h declares a sched_switch event. This excerpt shows the event prototype and selected stored fields; surrounding trace macros are omitted and formatting compacted:

TRACE_EVENT(sched_switch,
    TP_PROTO(bool preempt, struct task_struct *prev,
             struct task_struct *next, unsigned int prev_state),
    TP_ARGS(preempt, prev, next, prev_state),
    TP_STRUCT__entry(
        __array(char, prev_comm, TASK_COMM_LEN)
        __field(pid_t, prev_pid)
        __field(int, prev_prio)
        __field(long, prev_state)
        __array(char, next_comm, TASK_COMM_LEN)
        __field(pid_t, next_pid)
        __field(int, next_prio)
    )

TP_PROTO describes supplied values; TP_ARGS names them; TP_STRUCT__entry describes recorded fields. It can help answer which task was switched out and which selected next. It does not, by itself, explain why a task became non-runnable or whether a delay was harmful. A declaration in source is not proof that a target kernel enabled the event or that policy allows a user to collect it. Runtime availability depends on config, tracefs, filters, buffers, and access policy.

Contrast with the file-level pr_fmt(fmt) in pinned kernel/printk/printk.c: #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt. This prefixes text messages; it does not attach typed task fields. Both examples are source facts, not instructions to activate tracing. Pinned sched_switch · Pinned pr_fmt

Investigation loop and evidence worksheet

1. Reproduce once in an authorized, non-production environment; record frequency and steps.
2. Capture exact Android/kernel/build identity and unmodified evidence through permitted tools.
3. Write one hypothesis and an observation that could disprove it.
4. Choose the lowest-impact evidence source; record filters, permissions, timestamps, and possible event loss.
5. Change one variable at a time in development; preserve before/after artifacts.
6. Validate against the original reproduction and check regressions.
7. Report what is observed, inferred, ruled out, and still unknown.

Source worksheet: open the pinned include/trace/events/sched.h, find TRACE_EVENT(sched_switch, and record the path, event name, two stored task IDs, one limitation, and one runtime fact this declaration cannot establish. Then open pinned kernel/printk/printk.c, find pr_fmt, and state what its prefix does not contain.

Artifact: add the four-line worksheet plus a first-pass triage of the clearly synthetic sample: (a) identity required, (b) reported fault site, (c) at least two plausible causes still open, (d) next safe evidence. Expected: no root cause is claimed from the first stack frame; no runtime trace is claimed from a source declaration. If source access fails, label those evidence rows blocked.

Knowledge check#

Answer before reading the key.

1. A process is slow and an empty dmesg is returned. What does that establish, and what does it not establish?
2. A synthetic crash names demo_copy as the faulting frame. Why is “demo_copy caused the bug” too strong?
3. A developer symbolizes a crash against a similar kernel build and gets a source line. Is that valid evidence? Why?
4. The source declares sched_switch, but the device trace has no event. Give at least three explanations besides “no context switches occurred.”
5. Which method best suits (a) a rare crash artifact, (b) timing/order of scheduler events, and (c) statistical CPU hot spots?

Answers#

1. It establishes only that the permitted command returned no messages. Logs may be unavailable, filtered, overwritten, or the event may not log; it does not establish absence of a failure.
2. A frame locates execution at fault time, not necessarily the earlier cause. Invalid input/state, prior corruption, or a race remain possible until checked against exact artifacts and evidence.
3. No. Exact matching image/build symbols and relocation data are required; similar-build symbolization can produce plausible but false lines.
4. Event may not be compiled/enabled, trace capture/filter may exclude it, access may be denied, buffer may lose events, or capture may be incomplete.
5. Preserve full crash and exact symbols; narrow trace events for ordering/timing; sampling for statistical CPU attribution. Tools answer different questions and have different limits.

Checkpoint#

• Ready: you can separate observation from hypothesis, identify exact build artifacts needed for symbolization, and state at least two limitations of the evidence.
• Revise: you infer no event from missing logs, or declare the first frame the root cause; revisit the synthetic triage and investigation loop.
• Not yet evidenced: your artifact omits build identity, uncertainty, or next safe evidence. Complete it without inducing a crash or enabling privileged tracing.

Sources#

• AOSP kernel debugging with GKI — GKI constraints on intrusive debug features (KASAN, UBSAN, debugfs); it does not cover log or trace access policy.
• Linux 6.1 docs: ftrace, trace events, dynamic debug, bug hunting, reporting issues, ramoops, sysctl kernel, and kernel parameters.
• dmesg(1) — --level option used in the host observation.
• Pinned ACK sched_switch schema and pr_fmt — source-only examples; no runtime claim.
• Next: Verified Boot and encryption — returns to Android's boot trust and data-protection model.