GitLab

PTS is now a part of CTS

bug: 11561456
Change-Id: I577296fe7735790e285aa4747dc7832bc8ae5b6d

Change-Id: Iab2fdae3692e3d4a716093749b65a3b4a7c5c735

- pts still have its own package list : PtsBenchmarkingList
- except that all others are relying on CTS
- CTS plan now includes PTS
- SDK plan corresponds to the old CTS (no PTS)
- No more distinction of PTS / CTS mode in several places

Change-Id: Ifb2245880a92e0d37a57591f41cdb840010e36a4

buildCts.py was the central script that generated all the
test package XMLs each time CTS was built. This had a
couple problems:

1. All the XML files for ~40 packages needed to be made
   every time CTS was made. Even if those packages were
   not touched at all.

2. Couldn't shard the XML generation process across the
   available cores on a machine. A pool was added to the
   python script, but it was set to a fixed number.

This change moves the test XML generation into a
smaller Java program called "cts-java-scanner" and
the doclet it relies upon to scan the Java files
into "cts-java-scanner-doclet.jar". The output of
the scanner like "cts-native-scanner" for native EXEs
is piped to the existing cts-xml-generator to
produce the test XMLs.

New CTS specific rules replace the standard
BUILD_PACKAGE and BUILD_HOST_JAVA_LIBRARY. They just
add extra rules for the package XML. The BUILD_CTS_PACKAGE
rule also adds a rule for copying the "package.apk"
to something more like "CtsFooTestCases.apk" to the
test case out directory. All the apks, exes, and xmls
are now thrown into a "cts-testcases" directory, before
they are copied to the final CTS distribution.

This change should prevent rebuilding the XMLs
unnecessarily and make rebuilding CTS quicker while
writing tests.

There are still the libcore tests that are always rebuilt,
but they can be adapted to fit into this model someday...

Change-Id: I52a916aa37fd679057e2709bb0ccec694c9fca01

Bug 4970300

CTS uses XML files to determine what tests to run. The buildCts.py
script runs JavaDoc over Java files to create these listings. However,
this doesn't work for native tests which are executables written in
C++. This adds a Java tool called cts-native-xml-generator that
scans the C++ source files to find tests and then generates a
XML file that buildCts.py can pull in to form the CTS test plans.

The generate XMLs don't use a lot of the existing attributes
were intended for Java. I have also added a new "testType" attribute,
because each new test has so far added another boolean like
"vmHostTest=true" or "isHosttest=true"...

New build rules call cts-native-xml-generator in test_executable.mk.
Replace calls to include BUILD_EXECUTABLE with BUILD_CTS_EXECUTABLE
to not only create the executable like usual but to define a rule
that will call upon cts-native-xml-generator to create the test
package XML. An additional variable called LOCAL_CTS_TEST_PACKAGE
is required to give the native tests a parent test package which
is required by CTS.

CtsNativeTestCase.mk is a Makefile that defines a temporary location
to store the generated XMLs that are then copied to the final
location by build/core/cts.mk. This Makefile is included by the main
CTS Android.mk so that BUILD_CTS_EXECUTABLE can be used throughout
the project.

CtsTestCaseList also includes CtsNativeTestCase.mk to convert the
simple test executable names like "CtsNativeTestcases" to its
full path and the corresponding generated XML's path. These paths
are then passed to build/core/cts.mk. Both end up in the testcases
directory. We define two new variables, CTS_TEST_CASES and
CTS_TEST_XMLS. This is for future planning as discussed later.

Finally, GeeTest and GeeTestResultParser are added to handle
running the native tests. While TradeFed already has similar
classes, these two get around the limitations to work on user
builds and append the package name defined back as
LOCAL_CTS_TEST_PACKAGE. Integrating these changes back to the
TradeFed project will need to be done in a separate change.

A lot of this could have been done by adding more to buildCts.py,
but I didn't want to add to what is a single serial monolithic
step in the CTS build process. By having a separate rule to
handle each XML, make can take advantage of the cores on the
command line rather than sitting there on one core waiting
for the buildCts.py script to finish.

The CTS_TEST_CASES and CTS_TEST_XMLS are made more general to
handle copying any file type whether executable or APK to
the testcases directory. The XMLs could also be for native
packages or they could be for Java tests too. They are made
general on purpose so we do not need to modify
build/core/tasks/cts.mk everytime we add a new test type.
We just need to update CtsTestCaseList.mk without ever
having to do a multiproject change. Furthermore, this
generality allows us to easily convert over the existing
Java test cases.

Change-Id: I68f784bfedda5182b9795de2e6b2a5b7ed49626e

Adds a few make rules to generate coverage reports that use
dexdeps to see what methods from the API are directly
called from the tests rather than using the usual approach
of using EMMA.

    make cts-test-coverage

This rule will generate coverage of the API from the test
cases in CtsTestCaseList's CTS_COVERAGE_TEST_CASE_LIST list.

    make cts-verifier-coverage

This rule will generate coverage by the CtsVerifier.

Change-Id: I28c8d825cff5b1c6b293b786e28a472059a7259a