Investigate native crashes

This lab teaches a bounded native-userspace investigation: make a harmless host process fail in a controlled way, distinguish a fault from its cause, then analyze a clearly synthetic Android tombstone. You will not induce a crash on Android, change a device, or need root. A crash trace is evidence about one execution—not a root-cause verdict.

Before you start#

• C/C++ source reading — functions, pointers, compiling, and the difference between source and a running process.
• Java and native interfaces — ABI/process boundaries and why a native crash is not a Java exception.
• Debugging and contributing — reproduction, log privacy, and exact-build symbol matching.
• Linux files and processes — child processes, exit status, and temporary files.

If one of these is unfamiliar, revisit it before the exercise. The examples below are host-only. Do not use adb, su, a device crash trigger, or kernel controls for this lab.

Learning goals#

By the end, you can (1) compile and run a deliberately faulty but bounded C program; (2) state what a sanitizer did and did not prove; (3) extract competing hypotheses from a synthetic tombstone and ANR; (4) explain why library frames and the first application frame are not automatically the root cause; and (5) propose a discriminating reproduction, narrowly scoped patch, and regression matrix tied to the exact build.

Practice#

1. A deliberately wrong index, bounded to one child process

Save this complete program as index_bug.c in a scratch directory. The array is initialized, the loop is bounded, and the faulty index is deliberately one past the valid range on its final iteration. It may print an incorrect value or appear to work in an uninstrumented build: C does not promise a useful result after out-of-bounds access. It is not an exploit or a kernel/device test.

#include <stdio.h>

int main(void) {
    int values[4] = {10, 20, 30, 40};
    for (int i = 0; i <= 4; ++i) {  // BUG: valid indexes are 0 through 3
        printf("values[%d]=%d\n", i, values[i]);
    }
    return 0;
}

First check that a host C compiler exists (cc --version). If it does not, stop here: installing a compiler is outside this lab. Compile the plain version, then run it as a child and record its exit status:

cc -g -O0 -Wall -Wextra -o index_bug index_bug.c
./index_bug
printf 'exit=%s\n' "$?"

The plain run is a control, not proof that the code is safe. Its output can vary with compiler, optimization, stack layout, and platform; do not treat a particular garbage value or signal as expected. The program has one small, fixed stack array and a finite loop, so this example does not intentionally exhaust the stack.

Now build a separate instrumented executable. AddressSanitizer is a compiler/runtime instrumentation technique for finding some memory errors; this flag does not alter the host kernel or any Android device.

cc -g -O1 -fno-omit-frame-pointer -fsanitize=address \
  -Wall -Wextra -o index_bug_asan index_bug.c
./index_bug_asan
printf 'asan-child-exit=%s\n' "$?"

An ASan diagnostic is expected on a supported compiler/runtime, but exact wording and exit status vary. If the runtime is unavailable, record that as an environment limitation; do not install random binaries or escalate privileges. ASan may stop at the first detected error (it exits on the first detected error by design), changes memory layout and timing, has blind spots, and is not a proof of correctness. Keep each run separate; the expected failure is the child process, not the shell or host. You may delete only these two executables and the source file when finished.

Run recorded for this lesson revision (host only): In a fresh /tmp/opencode/android-native-crash-* directory, GCC cc 16.2.1 compiled both variants. The plain binary exited 0 after printing four valid elements and an extra meaningless values[4] integer; that observed extra value is not portable. The ASan binary exited 1 and reported stack-buffer-overflow, a 4-byte read at index_bug.c:5, identifying values as the stack object overflowed. Replacing i <= 4 with i < 4 compiled and exited 0 with exactly four expected lines. These are this host's observed results, not Android/AOSP test results.

Make the source fix yourself by changing only the loop bound:

-    for (int i = 0; i <= 4; ++i) {  // BUG: valid indexes are 0 through 3
+    for (int i = 0; i < 4; ++i) {   // indexes 0 through 3

Rebuild under a new name. Save the four correct lines in expected.txt, then check exact output and exit status with ./index_bug_fixed > actual.txt && diff -u expected.txt actual.txt; claim success only if diff exits zero. This repairs this loop's bound; it does not prove arbitrary input bounds safe.

2. Read a synthetic artifact as evidence, not truth

Everything in the following sample is invented teaching data. It is not copied from an Android device, does not describe a real product, and does not assert that these exact fields always appear. Imagine a bug report whose reporter matched the crashing process to the Android 14 build and supplied its unredacted-to-them, privacy-reviewed tombstone excerpt:

*** SYNTHETIC TRAINING TOMBSTONE — NOT A REAL DEVICE REPORT ***
Build fingerprint: 'example/device14/emu:14/UP1A.000000.000/eng.test:userdebug/test-keys'
ABI: 'arm64'
Process: example.decoder (pid 2417, tid 2450, thread name: DecodeWorker)
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0000000000000018
    #00 pc 0000000000012a40  /system/lib64/libcodec_stub.so (read_header+32)
    #01 pc 0000000000008b10  /system/lib64/libcodec_stub.so (decode_packet+112)
    #02 pc 0000000000005c20  /system/lib64/libcodec_jni.so (nativeDecode+80)
    #03 pc 0000000000123400  /system/framework/arm64/boot-framework.oat (...)
Build ID: libcodec_stub.so = 0123456789abcdef (SYNTHETIC)

Read each field with a bounded interpretation:

• The banner says the entire artifact is synthetic; no command or device produced it.
• The fingerprint labels an example Android 14 userdebug image, but its placeholder build ID is not a retrievable product identity.
• ABI: arm64 says which instruction/data ABI the hypothetical process uses; symbols for a different ABI would not be suitable.
• Process, PID, TID, and thread name identify the hypothetical crashing process and faulting thread for correlation. IDs can be reused and are not global identities.
• Signal 11 / SIGSEGV, SEGV_MAPERR (sigaction(2) lists the code name), and address 0x18 describe a memory access fault in the sample. They do not name a source-level pointer or prove null dereference.
• #00 is the reported program counter at the fault; #01 and #02 are unwound callers as symbolized in this example. The offsets are synthetic and must not be resolved against an actual library.
• Build ID is explicitly synthetic. A real investigation would compare the identifier against the exact library and matching symbols, not assume this value is valid.

The low fault address and signal make a null-or-near-null dereference a reasonable hypothesis, not a proof of which pointer or why it was invalid. The top frame (read_header) is where execution was reported to fault. Frame 1 suggests a caller in the same library; frame 2 suggests a JNI boundary; neither proves the bad value originated there. The managed frame may be truncated/compiled/omitted in this invented excerpt. Do not infer a particular Java call path from an abbreviated symbol.

The build fingerprint and ABI narrow the target identity. A symbol is trustworthy only if the unstripped library/debug symbols match the exact crashing build and architecture. A similar version, same filename, or symbol package from another build is insufficient. A Build ID is a useful identity check, not a substitute for obtaining the matching artifact. In Android 14 debuggerd/crash_dump.cpp calls dump_backtrace(...) when the request is a backtrace and otherwise runs engrave_tombstone(...) for a tombstone. The latter implementation is in debuggerd/libdebuggerd/tombstone.cpp, not debuggerd/tombstone.cpp (that path returns HTTP 404 at the tag). The call passes unwinder, thread, target-thread, process, and open-file evidence. These source links and symbols were checked at the tag; the source establishes the collector implementation, not that this synthetic report came from it. Compare with the Android native debugging guidance. Even a real diagnostic does not magically identify the source-level root cause.

Consider this separate, also synthetic symptom:

*** SYNTHETIC ANR TRAINING NOTE — NOT A REAL DEVICE REPORT ***
12:00:00.000 example.decoder: request 81 submitted on main thread
12:00:05.000 ActivityManager: input dispatch timeout for example.decoder
12:00:05.020 main thread sample: waiting in Future.get()
12:00:05.021 DecodeWorker sample: blocked in read(fd=17)

An ANR (application-not-responding event) is a responsiveness timeout, not a native crash. These samples suggest a possible wait chain: main thread waits on a future while worker waits in I/O. But the note lacks ownership of fd 17, I/O completion context, lock state, and time-series evidence. A blocked read may be normal, stuck, or downstream of storage/network conditions. The crash and ANR could be unrelated; matching names are not causal evidence.

3. Build hypotheses, then choose the smallest test

Write at least two competing explanations before looking for a patch:

Do not “fix” the first visible frame by adding a null check without following input validation, ownership, callers, and intended contract. A guard may prevent a crash but silently turn corrupt state into bad output. A useful report distinguishes: observed signal/timeout; source-confirmed call path; inferred mechanism; unknowns; and next falsifying experiment.

4. A targeted reproduction and patch/test matrix

For a real authorized AOSP investigation, keep the target a disposable emulator or an approved development build. Start with a minimal input and the exact image fingerprint, ABI, process, library Build ID, and symbols. Preserve a privacy-reviewed tombstone before any cleanup. Trace the native function into its callers and JNI/interface contract at the matching source revision. Reproduce with the narrowest existing native test or a local host unit test that exercises the same parser contract; do not claim a host test proves device integration.

A defensible candidate patch might reject a packet whose declared header is shorter than the fields read_header accesses, and return a documented error before dereference. That is only appropriate if source and contract establish the invariant. Test dimensions should include:

Record command, test/module, source revision, build identity, pass/fail, and untested boundaries. Sanitizer success on the host is not Android ABI coverage; a device run is not proof for other vendor implementations.

Knowledge check#

1. The plain executable prints four values and exits zero. Is the indexing correct? What result changes your conclusion?
2. In the synthetic tombstone, which frame is the first fault location, and what additional evidence would establish the faulty invariant?
3. Why is a library with the right filename but wrong Build ID unsafe for symbolization?
4. Give one measurement that would separate the ANR wait-chain hypothesis from “native crash caused the ANR.”
5. What does ASan add, and what does it not establish?

Answers#

1. No. Index 4 is outside the four-element array regardless of the printed value or exit status. A clean run cannot validate undefined behavior; the instrumented report or source-level bounds reasoning exposes the defect.
2. Frame 0 (read_header) is the reported fault location. Matching binary/symbols plus disassembly/source mapping, caller/input contract, and a minimal reproducible input are needed to establish why the address was invalid.
3. The same name can refer to different code/layouts; offsets and symbols can map to the wrong function or line. Match architecture and exact build identity/Build ID.
4. Reproduce the ANR while suppressing/avoiding the crash path, correlate one request ID across timestamps, identify fd 17’s owner and completion, or show the main-thread wait persists after the native process is absent. A correlation-only log line is not enough.
5. ASan instruments selected memory accesses and can detect some out-of-bounds/lifetime errors in an instrumented process. It changes execution and has coverage limits; it neither proves absence of bugs nor validates Android integration, production build behavior, or other ABIs.

Checkpoint#

Submit a one-page incident record: synthetic-vs-observed label; exact target identity or “not applicable”; fault location; at least two hypotheses with falsifiers; symbol/build match evidence; minimal reproduction; proposed contract-level fix; and test matrix. Ready: all claims are separated into observation, source fact, inference, and unknown, and tests address both the bug and valid behavior. Revise: a test is missing, symbols are not matched, or the patch merely hides the symptom. Not yet: you treat a frame/value as proof or propose crashing a device. Remediation: revisit debugging and contributing, C/C++ source reading, then redo the hypothesis table without changing the sample.

Sources#

• Linux sigaction(2) — lists the SEGV_MAPERR signal-code name used in the synthetic sample.
• AOSP crash_dump.cpp, Android 14.0.0_r1 — verified calls to dump_backtrace and engrave_tombstone.
• AOSP debuggerd/libdebuggerd/tombstone.cpp, Android 14.0.0_r1 — verified engrave_tombstone implementation (note libdebuggerd/ in the path).
• AOSP debuggerd/tombstone_handler.cpp, Android 14.0.0_r1 — separate tombstoned handler boundary, not the formatter above.
• Debug native Android platform code — Android native debugging and symbolization guidance; choose tooling for the exact release/build.
• AddressSanitizer documentation — instrumentation behavior and limitations; host compiler support varies.
• Earlier: debugging, C/C++ source reading, and Linux files and processes.