When a platform change builds but behaves unexpectedly, collect evidence before changing code. This chapter teaches a repeatable debugging loop, shows how to identify an actual Android 14 native test target, and explains what logs and Gerrit can and cannot establish. The following chapter performs a controlled, reversible fault injection in a real pinned utility and its existing test.
Before you start#
- Find your way around AOSP — search definitions, callers, modules, and tests.
- Build and run AOSP — pinned source,
userdebug, and device identity. - Soong and your first change — targeted module build and safe diff review.
The diagnosis exercises require the Android 14 checkout. Device-based tests require a disposable matching Cuttlefish device; a source-only learner can still inspect declarations and reason through the expected failure without claiming a test ran.
Learning goals#
- Record a reproducible symptom and verify source/image identity.
- Treat logs, stack traces, and tombstones as evidence with limits.
- Derive a real test module and test selector from pinned test source and
Android.bp. - Preflight Repo branch operations before changing a shared workspace.
- Distinguish local test/build evidence from upstream review and merge.
Practice#
Record a symptom and establish the layer
- Reproduce and identify
Record expected/actual behavior, target build, source state, and minimal steps.
- Trace and predict
Follow evidence to callers, implementation, module, and a relevant assertion.
- Test one proposed cause
Establish a baseline; change one variable; preserve complete results and failures.
- Check repair and regressions
Repeat the original case, a negative case, and relevant neighboring behavior.
Write down expected behavior, actual behavior, exact steps, frequency, clean-boot state, device, and ro.build.fingerprint. A reproducer is the smallest sequence that reliably shows the issue. A stale image can make a correct source change look ineffective. Before editing, run repo status at the workspace root and git diff -- in the affected project. If you cannot account for existing changes, stop.
Classify the first evidence instead of immediately opening a random file:
- UI/app crash: process and Java/Kotlin exception; inspect app logs and source.
- Framework/service failure: identify the service process (often but not always
system_server) and its caller/API path. - Native daemon crash: native process, signal, and tombstone/stack trace.
- Kernel failure: kernel logs/oops and kernel-side evidence; userspace log silence is not proof.
- Test/infrastructure failure: runner, device preparation, timeout, or transport diagnostics may fail before product behavior is exercised.
These are starting hypotheses, not a diagnostic verdict. A stack’s top frame is where failure surfaced, not necessarily the root cause. Logs may expose account/device/network identifiers; capture privately and redact before sharing.
Read bounded logs and native crash evidence
Confirm the device and capture a finite snapshot:
adb devices -l
adb -s SERIAL logcat -d -t 500
Replace SERIAL with the intended device serial. Find the time window around the reproduction, then filter only after seeing relevant tags/text; broad filters can hide context. adb logcat -c clears diagnostic evidence, so do not run it before preserving what you need.
A native tombstone may contain signal, thread, registers, and stack frames. On a development build it is commonly under /data/tombstones/ (the native crash guide notes the tombstone is written there), but access and path vary. Do not root a device or change permissions just to retrieve it. Symbolize only with the exact binaries and symbols from the crashing build; mismatched symbols can misidentify functions. Follow the release-specific Android native debugging guidance rather than copying a command for a different build layout.
Discover and run a real Android 14 test
Do not infer an atest identifier from a filename. At android-14.0.0_r1, system/core/init/Android.bp declares the native cc_test module CtsInitTestCases, with source rlimit_parser_test.cpp. That file defines GTest case rlimit and test method RlimitSuccess; the Atest GTest selector format combines these as follows:
source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
adb devices -l
SERIAL='replace-with-the-intended-disposable-cuttlefish-serial'
atest -s "$SERIAL" CtsInitTestCases:rlimit#RlimitSuccess
This is a real tag-source-derived target and test selector, not a placeholder. The test module declares require_root: true, so use the matching disposable userdebug test device and replace SERIAL with its exact value from adb devices -l. Atest's -s/--serial option selects one device (Atest docs: "One device can be tested at a time"). Atest may build the test and prepare/run it on that device; it can take time. Do not infer success from command launch. For a broader check, atest -s "$SERIAL" CtsInitTestCases runs the module, which is larger than this selected method.
Output varies by Trade Federation and build. Evidence to retain is: the resolved test/module name, build/install/run stages, a per-test PASSED or FAILED result (or a clearly identified setup/infra error), and the command exit status. A build-only message or device-preparation failure is not a passing test. The atest selector syntax is documented for GTests as module:test-case#test-method; the pinned source supplies CtsInitTestCases, rlimit, and RlimitSuccess. No success transcript is printed here because this lesson has not executed the test.
If the target is not recognized, re-open the tag-local Android.bp, confirm the checkout tag, and run atest --help; do not substitute a filename, directory, or guessed module name. For other features, repeat this evidence chain: test source → TEST(case, method) or class/method → enclosing build declaration’s name → documented runner selector. Generated source is not the test’s source of truth; inspect its generator/interface and corresponding implementation/test.
Branch and review safety
Repo branches are project-specific. Before starting one, inspect state and existing names; these commands do not alter the checkout:
repo status
git -C system/core status --short
git -C system/core branch --show-current
git -C system/core branch --list 'fix-small-issue-*'
Proceed only if the project path is correct, local changes are understood, and the branch name you choose is unused. Then use a unique name:
repo start fix-small-issue-rlimit system/core
repo status
git -C system/core branch --show-current
If the project has unfamiliar edits or the branch state is unexpected, stop before repo start; ask the owner rather than switching, resetting, or overwriting it. Review only your intended project paths with git diff --check and git diff. Stage specific files, not git add -A.
Before an upstream submission, read AOSP CLA/header requirements. A signed-off commit (git commit -s) and repo upload are part of the review workflow, not proof of acceptance. Check the CLA and license header requirements first. Gerrit upload creates/updates a review; reviewers and presubmit still decide whether it is ready. Never claim a test passed if it was not run.
Knowledge check#
- What can a stack trace tell you, and what can it not prove?
- How is
CtsInitTestCases:rlimit#RlimitSuccessderived? - Why must the matching symbols come from the same image as a native crash?
- What is the safe action if
repo statusshows unfamiliar changes before branching?
Answers#
- It shows a path of calls and where a failure surfaced, but not necessarily its root cause or whether the observed path is the only cause.
CtsInitTestCasesis thecc_testname in the pinnedAndroid.bp;rlimitandRlimitSuccessare the GTest case and method inrlimit_parser_test.cpp.- Code layout and addresses vary between builds; symbols from another build can map the same address to the wrong source.
- Stop and identify/coordinate ownership. Do not start a branch, stage, or modify state you cannot explain.
Checkpoint#
Pass when: you can show a reproducible symptom record; explain which evidence identifies the failing layer; derive the actual test selector from its source and Android.bp; and demonstrate a preflight before any branch creation. For the device run, retain the selected serial, test result, and exit status. A failed test is useful evidence; distinguish product failure from setup/infra failure rather than relabeling it.
Revise: if you cannot derive the module or selector, return to source-to-module navigation and inspect the tagged build declaration and test macro together. If the runner stops before executing the selected case, revisit build and target readiness; retain the setup error instead of treating it as a test failure. If you cannot run the large test, complete the source-tracing portion and mark execution not run. Retry only after identifying the blocker. Next: AOSP investigation lab.
Sources#
- Android 14
init/Android.bpandrlimit_parser_test.cpp— exact test module, root requirement, source and GTest names. - Run tests (Atest) — module/class/method and GTest selector syntax.
- Debug native Android platform code — native crash evidence and symbolization.
- AOSP test overview — locally built images and test-device workflow.
- Submit code changes —
git commit -sandrepo uploadsteps (Gerrit review; acceptance is not implied). - Contributor license agreements and headers — CLA and file-header requirements.
- Repo command reference —
repo status,repo start, andrepo uploadbehavior.