author Gregory Szorc <gps@mozilla.com>
Wed, 30 Sep 2015 11:33:04 -0700
changeset 265430 4a19c58da5928924ab276092de5e00fabcff28b8
parent 242929 3e46e2f117699b4897528d471b5d4a96299adbe8
child 303847 dd8ef51f4920c5f4370afd0aebf43d1113363fc3
permissions -rw-r--r--
Bug 1208320 - Decrease compression level of test zip archives; r=glandium Compressing C++ unit tests is a long pole when writing test archives. Experimenting with various levels of compression revealed that compression level 9 was providing minimal space savings for significantly longer archiving times and greater CPU usage. Results of our experimentation of `make -sj8 package-tests` on OS X with various levels of compression are below. Note: these numbers were accidentally obtained without JS tests being archived. This skews the results a little but doesn't impact the analysis below. ARCHIVE SIZE WALL CPU (L=9) cppunittest 76,806,629 30.6s mochitest 61,276,928 9.4s reftest 31,204,396 11.0s ALL 228,146,761 31.2s 75.9s (L=8) cppunittest 76,851,593 24.1s mochitest 61,279,322 8.9s reftest 31,207,867 10.4s ALL 228,228,096 24.9s 64.7s (L=7) cppunittest 77,102,292 14.3s mochitest 61,305,147 8.2s reftest 31,260,359 9.4s ALL 228,717,803 15.0s 49.1s (L=6) cppunittest 77,321,408 11.5s mochitest 61,336,539 8.2s reftest 31,303,604 9.2s ALL 229,123,307 12.2s 44.7s (L=5) cppunittest 78,226,404 8.2s mochitest 61,483,804 7.6s reftest 31,509,349 8.8s ALL 230,725,600 9.6s 39.7s (L=4) cppunittest 79,733,669 6.3s mochitest 61,825,519 7.6s reftest 31,924,171 8.4s ALL 233,669,991 9.0s 36.4s (L=3) cppunittest 82,380,731 5.8s mochitest 62,554,431 7.1s reftest 32,696,415 8.1s ALL 239,180,168 8.9s 34.6s Levels lower than 3 resulted in larger archives with no decreae in wall time and marginal decrease in CPU time. As we can see, lowering the compression level reduces archiving time by >3x while only increasing total archive size by ~2.5 MB or ~1% for compression level 5. Total time hits a plateau around levels 4 and 5. After that, file size increases faster for little decrease in wall time. I suspect that we're hitting Python limits from having to process thousands of files: there's only so fast Python can do I/O and make function calls. I think choosing 4 or 5 for the new compression level are acceptable. I went with 5 because the wall time savings from 5 to 4 are marginal and the archive size does start to increase a bit faster at 4. That being said, 4 does consume 10% less CPU. I could easily just 4 as well. 5 is more conservative. We can always change to 4 after seeing results in the wild. The end result of this change is `make package-tests` is much faster: Before: 228,146,761 bytes; 31.2s wall; 75.9s CPU After: 230,725,600 bytes; 11.4s wall; 45.0s CPU Delta: +2,578,839 bytes; -19.8s wall; -30.9s CPU When you take the whole series into consideration: Before: 44.2s wall; 84.6s CPU After: 11.4s wall; 45.0s CPU Lowering CPU is impressive considering we switched from the C `zip` implementation to Python! Keep in mind we were at ~78s wall before e87b74b3db43 introduced concurrent archive generation! And we still haven't eliminated the staging of JS tests, which are several thousand files and a few dozen MB!

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

import imp
import os
from StringIO import StringIO
import shlex
import sys

old_bytecode = sys.dont_write_bytecode
sys.dont_write_bytecode = True

path = os.path.join(os.path.dirname(__file__), 'mach')

if not os.path.exists(path):
    path = os.path.join(os.path.dirname(__file__), 'config.status')
    config = imp.load_module('_buildconfig', open(path), path, ('', 'r', imp.PY_SOURCE))
    path = os.path.join(config.topsrcdir, 'mach')
mach_module = imp.load_module('_mach', open(path), path, ('', 'r', imp.PY_SOURCE))

sys.dont_write_bytecode = old_bytecode

def FlagsForFile(filename):
    mach = mach_module.get_mach()
    out = StringIO()
    out.encoding = None
    mach.run(['compileflags', filename], stdout=out, stderr=out)

    flag_list = shlex.split(out.getvalue())

    # This flag is added by Fennec for android build and causes ycmd to fail to parse the file.
    # Removing this flag is a workaround until ycmd starts to handle this flag properly.
    # https://github.com/Valloric/YouCompleteMe/issues/1490
    final_flags = [x for x in flag_list if not x.startswith('-march=armv')]

    return {
        'flags': final_flags,
        'do_cache': True