GitLab

Change-Id: Icbf3f1c332a2a06dfe0eb9037d32379329a63a4c

Required by the libziparchive static dependency. Also fix
a few tabs in the makefile.

Change-Id: Iaa7aad6e7f3dfc6014651385d7182c009e182914

Revert change to dex rotation algorithm made as part of commit:
845d9d0e

Bug: https://code.google.com/p/android/issues/detail?id=161887


Change-Id: I255aa52b74c921a78e77671ae6187157fef011e1
Signed-off-by: Peter Jensen <jensenp@google.com>

This version includes "Support --num-threads with --multi-dex"

Change-Id: I83b6d54ab96de1c0f8b94075f11696095d9192e7

We have to parse descriptors and treat the types they reference as direct
reference as we do for other direct type references. This is required because
Dalvik verifier is checking those descriptor the same way as it does for other
type references. See:
https://android.googlesource.com/platform/dalvik/+/kitkat-mr2.2-release/vm/analysis/CodeVerify.cpp#637

Bug: 19626444
Change-Id: Ibdc8c3c129148994b3b28882db1c2ec38e9bbea8

* commit '366e2045':
  Support --num-threads with --multi-dex (take 2)

* commit 'b3ef71c0':
  Revert "Support --num-threads with --multi-dex"

* commit '6e28e432':
  Support --num-threads with --multi-dex

  With fix for regression introduced in original commit.

The current dx implementation supports options --multi-dex, for applications
not fitting within the dex format limitations; and --num-threads=N, triggers
concurrent processing of multiple input files. However, the implementation
has the following limitations:

The --num-threads option is disabled when used together with --multi-dex.
The --num-threads option implements concurrency at the level of classpath
entries, and does nothing when the classes to be translated are specified
with a single classpath element (e.g. single jar output from Proguard).
The existing --num-threads implementation may produce indeterministic output.
The heuristic used by the --multi-dex option to determine when to rotate the
dex output file is overly conservative.

The primary objective of this change is:
Concurrent translation of classes, independently of input specification format.
Support --num-threads=N in both mono- and multi-dex mode.
Deterministic class output order.
Near optimal use of dex file format capacity.

This is accomplished by reorganizing the dx workflow in a pipeline of
concurrent phases.

read-class  | parse-class | translate-class | add-to-dex | convert-dex-to-byte[];
output-dex-files-or-jar

To manage dex file rotation (i.e. --multi-dex support), the parse-class and
add-to-dex phases are synchronized to prevent forwarding classes to the
translate-class phase if it could potentially result in breaking the dex
format limitations. The heuristic currently used to estimate the number of
indices needed for a class is improved, to minimize the amount of serialization
imposed by this feedback mechanism, and to improve the use of dex file capacity.

The translate-class and convert-dex-to-byte[] phases are further parallelized
with configurable (--num-threads=N option) thread pools. This allow translating
classes concurrently, while also performing output conversion in parallel.
Separate collector threads are used to collect results from the thread pools
in deterministic order.

Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads.
The taskset command was used to experimentally establish that running with more
than 8 hardware threads does not provide any additional benefit.

Experiments shows that the argument to --num-threads should not exceed the
lesser of the number of available hardware threads, and 5. Setting it to a
higher value results in no additional benefit.

The gain is generally larger for larger applications, and not significant for
small applications with less than a few thousands classes. Experiments with
generated classes shows that for large applications gains as high as 50% may
be possible.

For an existing real-life application with more than 11k classes, and requiring
2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware
threads, 4g Java heap). A speedup of 31% was observed for another application
with ~7 classes.

For small applications, use of --num-threads=N>1 doesn’t provide significant
benefit. Running with --num-threads=1, the modified dx is slightly faster,
but no significant gain is observed unless the application requires multiple
dex files.

The one case where a significant regression may be observed is when using
--num-threads=N>1, with a single hardware thread. This is an inappropriate
configuration, even with the current implementation. However, because of
the limitations of the current implementation, such configurations may exist.
For instance, a configuration using both --multi-dex and --num-threads=5 will
currently generate a warning about using the two options together. With the
new implementation, the options can legitimately be used together, and could
result in an ~20% regression running on a single hardware thread.
Note: the current dx implementation, without --num-threads option, is already
approximately 50% slower  with 1 hardware thread, compared to running with 2
or more. With 2 hardware threads the implementations are practically at par
(a little better, or a little worse, depending on the application).

Testing:
Tested with 6 existing applications ranging in size from 1K - 12K classes.
Updated and tested with relevant existing unit tests (one test changed to
account for better dex rotation heuristic).
Added unit test to test deterministic output.
Added unit performance test. By default run script merely validates that
--multi-dex and --num-threads can be used together (fast). However, the test
is configurable to perform performance test, over sets of generated classes.
Signed-off-by: Peter Jensen <jensenp@google.com>

(cherry picked from commit 845d9d0e)

Change-Id: I721effa31c3b1a8b427d3a18ec554a19c5e9765b

This reverts commit 845d9d0e.

Bug: 19313927
Change-Id: Ia6582a3914cc33762aef74da1f5a6a153c8c0ab2

* commit '438b9164':
  Removing dalvik/hit, replaced by tools/base/perflib

Bug: 18326212
Change-Id: I1286fe00f35ecb6d3be9965f949fff13bba8958b

* commit '38246b3b':
  Don't discard directory entries in jar files.

This is a structural change, and breaks code that looks up
directory resource names (icu4j for example).

This change also includes a minor cosmetic change to use a
while() loop instead of for(;;).

bug: 19108324
Change-Id: Ib12c3c1d55f14a089702e5e668d7a704f298e1f4

* commit '64d5b033':
  Fix printf format specifiers.

Fixes:

dalvik/tools/hprof-conv/HprofConv.c:233:61: warning: format specifies type 'int' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
            fprintf(stderr, "ERROR: read %d of %d bytes\n", actual, count);

dalvik/tools/hprof-conv/HprofConv.c:256:58: warning: format specifies type 'int' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat]
        fprintf(stderr, "ERROR: write %d of %d bytes\n", actual, pBuf->curLen);

dalvik/tools/hprof-conv/HprofConv.c:537:26: warning: format specifies type 'int' but the argument has type 'long' [-Wformat]
                subType, buf - origBuf);

Change-Id: If9926900417d57971fa25a4c7f465a9a0c50405e

* commit 'c0221e7e':
  Allow to disable forced keep of annotated

We force all classes annotated with runtime annotation to be kept
in the main dex list. This allow to workaround a Dalvik bug when
accessing the annotations. But this increase the pressure in the
main dex indexes. The new option is to allow to disable this
workaround for application facing the index limit in their main
dex and that don't access annotation at runtime or that have only
annotations not subject to the resolution bug.

Bug: 18928046
Change-Id: Ibcd7e579a7fef3451ec8aeb266ea67514d82cd50

The current dx implementation supports options --multi-dex, for applications
not fitting within the dex format limitations; and --num-threads=N, triggers
concurrent processing of multiple input files. However, the implementation
has the following limitations:

The --num-threads option is disabled when used together with --multi-dex.
The --num-threads option implements concurrency at the level of classpath
entries, and does nothing when the classes to be translated are specified
with a single classpath element (e.g. single jar output from Proguard).
The existing --num-threads implementation may produce indeterministic output.
The heuristic used by the --multi-dex option to determine when to rotate the
dex output file is overly conservative.

The primary objective of this change is:
Concurrent translation of classes, independently of input specification format.
Support --num-threads=N in both mono- and multi-dex mode.
Deterministic class output order.
Near optimal use of dex file format capacity.

This is accomplished by reorganizing the dx workflow in a pipeline of
concurrent phases.

read-class  | parse-class | translate-class | add-to-dex | convert-dex-to-byte[];
output-dex-files-or-jar

To manage dex file rotation (i.e. --multi-dex support), the parse-class and
add-to-dex phases are synchronized to prevent forwarding classes to the
translate-class phase if it could potentially result in breaking the dex
format limitations. The heuristic currently used to estimate the number of
indices needed for a class is improved, to minimize the amount of serialization
imposed by this feedback mechanism, and to improve the use of dex file capacity.

The translate-class and convert-dex-to-byte[] phases are further parallelized
with configurable (--num-threads=N option) thread pools. This allow translating
classes concurrently, while also performing output conversion in parallel.
Separate collector threads are used to collect results from the thread pools
in deterministic order.

Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads.
The taskset command was used to experimentally establish that running with more
than 8 hardware threads does not provide any additional benefit.

Experiments shows that the argument to --num-threads should not exceed the
lesser of the number of available hardware threads, and 5. Setting it to a
higher value results in no additional benefit.

The gain is generally larger for larger applications, and not significant for
small applications with less than a few thousands classes. Experiments with
generated classes shows that for large applications gains as high as 50% may
be possible.

For an existing real-life application with more than 11k classes, and requiring
2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware
threads, 4g Java heap). A speedup of 31% was observed for another application
with ~7 classes.

For small applications, use of --num-threads=N>1 doesn’t provide significant
benefit. Running with --num-threads=1, the modified dx is slightly faster,
but no significant gain is observed unless the application requires multiple
dex files.

The one case where a significant regression may be observed is when using
--num-threads=N>1, with a single hardware thread. This is an inappropriate
configuration, even with the current implementation. However, because of
the limitations of the current implementation, such configurations may exist.
For instance, a configuration using both --multi-dex and --num-threads=5 will
currently generate a warning about using the two options together. With the
new implementation, the options can legitimately be used together, and could
result in an ~20% regression running on a single hardware thread.
Note: the current dx implementation, without --num-threads option, is already
approximately 50% slower  with 1 hardware thread, compared to running with 2
or more. With 2 hardware threads the implementations are practically at par
(a little better, or a little worse, depending on the application).

Testing:
Tested with 6 existing applications ranging in size from 1K - 12K classes.
Updated and tested with relevant existing unit tests (one test changed to
account for better dex rotation heuristic).
Added unit test to test deterministic output.
Added unit performance test. By default run script merely validates that
--multi-dex and --num-threads can be used together (fast). However, the test
is configurable to perform performance test, over sets of generated classes.

Change-Id: Ic2d11c422396e97171c2e6ceae9477113e261b8e
Signed-off-by: Peter Jensen <jensenp@google.com>

* commit 'e2808cf4':
  Do not keep the DexFile field in Main between runs

Bug: 18486731
Change-Id: I76dbdd709182c4663f2bbf42179f55d0184e97c2

* commit 'acc22d6b':
  Keep annotated classes in main dex list

* commit '8e88bab0':
  Keep annotated classes in main dex list

Only classes annotated with a runtime visible annotation are to be kept.
See https://code.google.com/p/android/issues/detail?id=78144

Bug: 18385117

(cherry picked from commit 893795fc)

Change-Id: I46ceef4aab5119ba4515a07e8f17577e15931dbd

* commit 'c9548518':
  Keep annotated classes in main dex list

Only classes annotated with a runtime visible annotation are to be kept.
See https://code.google.com/p/android/issues/detail?id=78144

Bug: 18385117
Change-Id: Id377b3945b22201ebad05e7cd026af57acc60292