Platform tests, CTS & VTS

Android compatibility is established with suites of tests, but a test result has a scope: a particular build, configuration, device, test selection, and run. This lesson uses a synthetic result packet and one source-verified module example. It does not claim to run CTS or VTS.

Before you start#

Required earlier lessons: Debug, test & contribute, HAL, Treble, and the vendor boundary, and Find your way around AOSP. Know the difference between a source tree and a running target, and between a test module and a whole suite. No build/device is required for the supplied exercise.

Learning goals#

You will distinguish CTS’s app/platform compatibility focus from VTS’s vendor/HAL/kernel-facing coverage, select a narrow relevant test rather than promising to run a huge suite, interpret pass/fail/infrastructure/not-executed outcomes, and propose a regression plan with limits.

Practice#

Two complementary suites

The Android Compatibility Test Suite (CTS) checks compatibility of Android behavior exposed to apps and the platform. Its official overview describes the test harness running from a desktop machine and automated tests executing on an attached device or emulator; host-side setup and reporting do not mean a device test ran. CTS does not test every possible app, workload, vendor, or physical condition. The Vendor Test Suite (VTS) exercises lower-level vendor-facing areas, including HAL and kernel interfaces, through Trade Federation (Tradefed) test infrastructure. Exact plans and prerequisites depend on the release and target. A CTS pass is not a VTS pass; neither is a general proof that a device is defect-free.

Trade Federation (Tradefed) is a host-side test framework for running tests on Android devices. Atest is a command-line tool that builds, installs, and runs Android tests locally, without requiring knowledge of the Tradefed harness. Before running a test, check its Android.bp, test configuration, source, and documented target requirements. A test name alone does not establish that your current checkout builds it, your target supports it, or it covers your bug.

“Unit” and “integration” describe test scope, not a separate guarantee carried by a CTS/VTS label. A unit-style test isolates a component and can be fast, but may replace dependencies with fakes. An integration test exercises multiple real components and can catch wiring/contract defects, but has more setup and failure sources. CTS modules include different test shapes; VTS focuses on lower-level vendor interfaces and may use a real target or a specific test configuration. Inspect each case: a mock-based test does not validate vendor registration, while an integration test’s pass still covers only the exercised setup and assertions.

One pinned, conditional module example

In AOSP cts/tests/tests/os/Android.bp at android-14.0.0_r1, the android_test module is named CtsOsTestCases. This is a verified example of the module declaration, not a claim that it tests a particular vibrator regression. In a matching AOSP checkout, after the build environment and appropriate target are configured, the conditional invocation is:

atest CtsOsTestCases

The Atest docs use the same module-name form (for example, atest FrameworksServicesTests). Run it only if that exact module exists in your checkout and its test configuration is suitable for your target; inspect the module’s source/test configuration first. It is a broad module, not a promise that every test inside is useful for your question. For a real vibrator defect, search the pinned CTS/VTS tree and the relevant Android.bp for a focused test, verify its actual test case and prerequisites, then substitute that exact module or test filter. Do not infer the module name or claim the command succeeded merely because it appears here. A source-only learner stops at inspecting the declared module and marks execution “not run.”

Synthetic result packet — not an actual test run

The following fictional report is for interpretation practice only:

SYNTHETIC PLAN: focused-vibrator-regression (not a real plan name)
Target: emulator build fingerprint ending in test-build-01 (invented)
CTS/API permission case: PASS (1 test)
VTS interface-version case: FAIL (expected interface version 2, observed 1)
VTS callback-timing case: INFRA_ERROR (device disconnected before assertion)
Physical actuator latency case: NOT_EXECUTED (test requires supported hardware)

Do not average these four statuses into “mostly passed.” The CTS pass only supports the permission assertion for this synthetic configuration. The VTS failure is a compatibility clue that needs source/manifest and test-log investigation; it does not by itself prove the vendor implementation is the root cause. The infra error is not a behavioral failure because the assertion did not complete. The latency case is not a pass or fail: it was not run. Since this is a fictional packet, none of the statuses is evidence about a real Android build.

Build a regression plan

For an actual issue, write down: the behavior and affected boundary; exact release/build and target; test source/module and what its assertion covers; required setup; one focused test; relevant broader suite or compatibility checks; expected result; cleanup; and what remains untested. For an API permission bug, an app-facing CTS test may be appropriate. For a stable AIDL interface or HAL mismatch, a VTS/compatibility test may be more relevant. For physical latency, a supported hardware test and measurement protocol are required. A test that does not exercise the changed path is not meaningful regression evidence.

Select a test by the question it answers

Before invoking a suite, translate the suspected defect into an observable assertion. For example, “the app permission check rejects an unauthorized request” is an app/framework contract question; “the vendor service implements the declared method/version” is an interface question; “the motor reaches a target timing under load” is a device behavior and measurement question. These may need different modules, fixtures, and hardware. A single large test plan can be both expensive and less informative than a correctly chosen targeted test.

In the checkout, inspect the test module’s Android.bp and configuration to identify its declared name, device requirements, test runner, and dependencies. Find the test method or case that actually exercises the behavior. Check if it is a host-side test, instrumentation test, Tradefed test, or test requiring a specific hardware feature. Read any setup rules before running it. A module may build successfully while being impossible to execute on an unsupported emulator; an installed test app may also skip tests when the required feature is absent. Skips should be represented as skips, not as evidence that the feature passed.

The named CtsOsTestCases example is intentionally not labeled as a vibrator test. It demonstrates how to verify a module name in a pinned Android.bp and how a conditional invocation is written. For a real vibration regression, do not blindly run it as the “right test.” Search the pinned CTS and VTS trees for relevant vibrator modules and test source, verify the exact module/configuration, and scope the run to the specific case if its runner supports that. If no relevant test exists, say so and propose a focused test only after learning that repository’s conventions.

Interpret result states without collapsing them

Reports commonly distinguish behavioral failures from execution failures and tests that did not run. Exact labels vary by harness and report version, so read the report details rather than relying only on the status word:

An overall suite status can conceal that some modules were skipped or failed infrastructure. Read the result summary and individual case records. Conversely, a single failure does not establish a platform-wide incompatibility until the target, test preconditions, expected result, and failure are understood. Preserve the original report and distinguish a test bug from a product bug where evidence permits; if it does not, mark the cause unknown.

Regression evidence packet

For a reviewable regression claim, save a compact record containing:

• AOSP tag/manifest or product build fingerprint and target type.
• The module and individual test case (or filter) that ran, linked to the source/configuration inspected.
• Exact invocation and relevant environment/setup; redact device identifiers and personal data.
• Per-case result, not only the suite summary; include skipped/failed-to-run cases.
• The expected behavior and the assertion that encodes it.
• Whether the test exercises the modified path; what it omits (for example, physical hardware timing).
• The outcome of a repeat run if flakiness or infrastructure instability is suspected.

This is not a demand to run the entire Android test catalog. CTS contains many modules and test cases; VTS also varies by device and target. A targeted result is often the right first experiment, but it must be scoped honestly. Broader CTS/VTS plans can be appropriate for release qualification or a compatibility claim, which is beyond this single-module learning exercise.

Knowledge check#

1. What does a PASS for one CTS test establish? Name two limits.
2. A Tradefed report says INFRA_ERROR before the test assertion. Should the code be called failed?
3. Why is NOT_EXECUTED different from PASS when hardware is absent?
4. The VTS interface-version case fails in the synthetic packet. Give two follow-up checks before assigning cause.

Answers#

1. The named test’s assertions passed in the recorded configuration. It does not prove the whole suite passed, other devices behave the same, or unrelated behavior is correct.
2. No. The test did not produce a behavioral result; repair/retry the infrastructure and retain the failure as an execution problem.
3. No assertion ran, so there is no positive evidence. State that the scenario is untested rather than treating absence of a failure as success.
4. Confirm the exact interface declaration/version and device VINTF manifest/matrix; inspect full test logs, target build identity, and implementation registration. The single failed assertion is a clue, not root-cause proof.

Model interpretation

For the synthetic packet: one CTS result passes its narrow permission assertion; one VTS assertion reports a version mismatch; one VTS test has no behavioral outcome due to disconnection; the hardware latency case is untested. Next, verify the version expectation and actual manifest/source at the same build revision, then rerun the narrow relevant test only in a supported, stable environment. Do not report an overall pass and do not attribute the synthetic failure to a real vendor.

Checkpoint#

Pass when your artifact separates each outcome, ties a test to the boundary/assertion it covers, and names one limitation and next check. Revise if INFRA_ERROR or NOT_EXECUTED is counted as a pass, or if one module result is called a full compatibility certification. Not yet evidenced is mandatory for the synthetic report’s runtime claims: it is not an execution record.

If you need the local targeted-test workflow, revisit Debug, test & contribute. For interface declarations versus implementation health, revisit HAL, Treble, and the vendor boundary. For source-tree module navigation, revisit Find your way around AOSP.

Sources#

• CTS overview — verified: CTS runs from a desktop machine and executes tests on attached devices or an emulator; it also lists platform API tests.
• VTS overview and VTS systems testing — verified: VTS covers HAL and kernel; the Tradefed harness runs on the host and test cases run on the device.
• Tradefed overview — verified: a continuous test framework that runs on a host computer and drives tests on Android devices.
• Atest — verified: command-line tool that builds, installs, and runs tests; module-name usage such as atest FrameworksServicesTests.
• AOSP cts/tests/tests/os/Android.bp, Android 14.0.0_r1 — source verification for module CtsOsTestCases; the command above is conditional guidance, not an executed test.