GitLab

Fix potential buffer overrun. am: 38025bc7 am: 805b415b am: 0e17e184 am: 8e37aa19 am: e5c4e849 am: 46464a20 am: 3bbe8817 am: 63e01f73 am: a0af385a
am: f984fa7d

* commit 'f984fa7d':
  Fix potential buffer overrun.

Change-Id: I6c172e5a2e5b1f236b61351d92dc98357a22b892

Fix potential buffer overrun. am: 38025bc7 am: 805b415b am: 0e17e184 am: 8e37aa19 am: e5c4e849 am: 46464a20 am: 3bbe8817 am: 63e01f73
am: a0af385a

* commit 'a0af385a':
  Fix potential buffer overrun.

Change-Id: Ic52a3476c569227eb2146b4ac653fb8a3f58e068

Fix potential buffer overrun. am: 38025bc7 am: 805b415b am: 0e17e184 am: 8e37aa19 am: e5c4e849 am: 46464a20 am: 3bbe8817
am: 63e01f73

* commit '63e01f73':
  Fix potential buffer overrun.

Change-Id: I9da77753de3597886bd2aa9443f1e0f49962920d

Fix potential buffer overrun. am: 38025bc7 am: 805b415b am: 0e17e184 am: 8e37aa19 am: e5c4e849 am: 46464a20
am: 3bbe8817

* commit '3bbe8817':
  Fix potential buffer overrun.

Change-Id: I4b76d9d081fe87c207fc1a6f4832e3af969d7816

Fix potential buffer overrun. am: 38025bc7 am: 805b415b am: 0e17e184 am: 8e37aa19 am: e5c4e849
am: 46464a20

* commit '46464a20':
  Fix potential buffer overrun.

Change-Id: I180bcd9b425711745406725a2906f4854b7f3706

am: e5c4e849

* commit 'e5c4e849':
  Fix potential buffer overrun.

Change-Id: Icdcdd512ce2cde9b7a317dcd948cb2539e03f8d6

am: 8e37aa19

* commit '8e37aa19':
  Fix potential buffer overrun.

Change-Id: I30bc81276880e8816387a6b3d242015988a2b305

am: 0e17e184

* commit '0e17e184':
  Fix potential buffer overrun.

Change-Id: I2d0f7e326f21fd0fd2e8f03b2c7eda302730f4f9

am: 805b415b

* commit '805b415b':
  Fix potential buffer overrun.

Change-Id: I5c6959756d219ddcdb200318869d781bc3d99e93

am: 38025bc7

* commit '38025bc7':
  Fix potential buffer overrun.

Change-Id: I60b170a299f5c1f35e233ef41db528981453f40b

BUG=27840771

Change-Id: I240f188b2e8f4b45d90138cfb3b14869cf506452

Changes merging from quadratic in the number of classes to linear.
This has a tremendous speed up while merging many dexes at the same time.
A sample test (iosched app) with 29 dexes goes from 6 seconds to 1.

Change-Id: Iff02a0dc44d098b0878e88d18f5f4083804a6495

* commit '1396cdeb':
  Ensure deterministic multidex partitioning

* commit '72353f83':
  Ensure deterministic multidex partitioning

Change-Id: Ie70b72005a6d3df47adb0e5b1b21b9d7b99b8ab0

This is done by sorting inputs before processing them.

Bug: 23305646

Change-Id: Ia55c481f4eb342bc925bc100299324ea470bf9a0

Required by the libziparchive static dependency. Also fix
a few tabs in the makefile.

Bug: http://b/21558406
Change-Id: Iaa7aad6e7f3dfc6014651385d7182c009e182914
(cherry picked from commit a06ce75c)

Was only used by the fingerprinter, which has now been
deleted.

Change-Id: I52c251a60ab6a85167fe734626573bc83f86cfcf

* commit 'b8a003c7':
  Bump dx version to 1.11

* commit '1d40d9ea':
  Fix potential dex index overflow in dx

* commit '00d77c91':
  Support --num-threads with --multi-dex (take 2)

* commit '010ac971':
  Bump dx version to 1.11

* commit '7b153a1d':
  Fix potential dex index overflow in dx

Change-Id: Id603b84943b1cb244b69272b4c8607a520c517b2

* commit 'a6bf3001':
  Do not keep the DexFile field in Main between runs

Change-Id: I906a5fa2ecbdc43656dd64098f38cdf60438e1e3

* commit '5ff9033e':
  Command line option for disabling warnings

* commit 'ae1d7c49':
  Don't discard directory entries in jar files.

* commit 'f052d52e':
  Command line option for disabling warnings

* commit '90fcf359':
  Don't discard directory entries in jar files.

* commit '65c602a4':
  Descriptor references are direct references

This version includes "Support --num-threads with --multi-dex"

(cherry picked from commit 7d97556f)

Change-Id: I712145094205d56358ef0b9ff51f137907d972e9

Revert change to dex rotation algorithm made as part of commit:
845d9d0e

Bug: https://code.google.com/p/android/issues/detail?id=161887

Signed-off-by: Peter Jensen <jensenp@google.com>

(cherry picked from commit d024c067)

Change-Id: Iede742620854cef3542bf98b77e66aa8a305195e

  With fix for regression introduced in original commit.

The current dx implementation supports options --multi-dex, for applications
not fitting within the dex format limitations; and --num-threads=N, triggers
concurrent processing of multiple input files. However, the implementation
has the following limitations:

The --num-threads option is disabled when used together with --multi-dex.
The --num-threads option implements concurrency at the level of classpath
entries, and does nothing when the classes to be translated are specified
with a single classpath element (e.g. single jar output from Proguard).
The existing --num-threads implementation may produce indeterministic output.
The heuristic used by the --multi-dex option to determine when to rotate the
dex output file is overly conservative.

The primary objective of this change is:
Concurrent translation of classes, independently of input specification format.
Support --num-threads=N in both mono- and multi-dex mode.
Deterministic class output order.
Near optimal use of dex file format capacity.

This is accomplished by reorganizing the dx workflow in a pipeline of
concurrent phases.

read-class  | parse-class | translate-class | add-to-dex | convert-dex-to-byte[];
output-dex-files-or-jar

To manage dex file rotation (i.e. --multi-dex support), the parse-class and
add-to-dex phases are synchronized to prevent forwarding classes to the
translate-class phase if it could potentially result in breaking the dex
format limitations. The heuristic currently used to estimate the number of
indices needed for a class is improved, to minimize the amount of serialization
imposed by this feedback mechanism, and to improve the use of dex file capacity.

The translate-class and convert-dex-to-byte[] phases are further parallelized
with configurable (--num-threads=N option) thread pools. This allow translating
classes concurrently, while also performing output conversion in parallel.
Separate collector threads are used to collect results from the thread pools
in deterministic order.

Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads.
The taskset command was used to experimentally establish that running with more
than 8 hardware threads does not provide any additional benefit.

Experiments shows that the argument to --num-threads should not exceed the
lesser of the number of available hardware threads, and 5. Setting it to a
higher value results in no additional benefit.

The gain is generally larger for larger applications, and not significant for
small applications with less than a few thousands classes. Experiments with
generated classes shows that for large applications gains as high as 50% may
be possible.

For an existing real-life application with more than 11k classes, and requiring
2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware
threads, 4g Java heap). A speedup of 31% was observed for another application
with ~7 classes.

For small applications, use of --num-threads=N>1 doesn’t provide significant
benefit. Running with --num-threads=1, the modified dx is slightly faster,
but no significant gain is observed unless the application requires multiple
dex files.

The one case where a significant regression may be observed is when using
--num-threads=N>1, with a single hardware thread. This is an inappropriate
configuration, even with the current implementation. However, because of
the limitations of the current implementation, such configurations may exist.
For instance, a configuration using both --multi-dex and --num-threads=5 will
currently generate a warning about using the two options together. With the
new implementation, the options can legitimately be used together, and could
result in an ~20% regression running on a single hardware thread.
Note: the current dx implementation, without --num-threads option, is already
approximately 50% slower  with 1 hardware thread, compared to running with 2
or more. With 2 hardware threads the implementations are practically at par
(a little better, or a little worse, depending on the application).

Testing:
Tested with 6 existing applications ranging in size from 1K - 12K classes.
Updated and tested with relevant existing unit tests (one test changed to
account for better dex rotation heuristic).
Added unit test to test deterministic output.
Added unit performance test. By default run script merely validates that
--multi-dex and --num-threads can be used together (fast). However, the test
is configurable to perform performance test, over sets of generated classes.
Signed-off-by: Peter Jensen <jensenp@google.com>

(cherry picked from commit 845d9d0e)

(cherry picked from commit dd140a22)

Change-Id: I33a8ea0451efc0af7eb1d72e80cb926d6583d569

This is a structural change, and breaks code that looks up
directory resource names (icu4j for example).

This change also includes a minor cosmetic change to use a
while() loop instead of for(;;).

bug: 19108324

(cherry picked from commit 7736e8ff)

Change-Id: Ifcbc60cc7808334cdfdd4f27bf137a8f6455117d