.ycm_extra_conf.py
author Gregory Szorc <gps@mozilla.com>
Mon, 08 May 2017 17:19:05 -0700
changeset 361962 11ba6213be5a745d70e5c6fdcd036bafb1ac2718
parent 308778 578ebd592745ff2b7647b205576f20b5b36f3327
child 455155 2a22490b8a06c9a5eef192ef3192386f8e7ff74f
permissions -rw-r--r--
Bug 1359965 - Support and generate tar.gz WPT archive; r=glandium Several years ago there was a single zip file for all test files. Clients would only extract the files they needed. Thus, zip was a reasonable archive format because it allowed direct access to members without having to decompress the entirety of the stream. We have since split up that monolithic archive into separate, domain-specific archives. e.g. 1 archive for mochitests and one for xpcshell tests. This drastically cut down on network I/O required on testers because they only fetched archives/data that was relevant. It also enabled parallel generation of test archives, we shaved dozens of seconds off builds due to compression being a long pole. Despite the architectural changes to test archive management, we still used zip files. This is not ideal because we no longer access specific files in test archives and thus don't care about single/partial member access performance. This commit implements support for generating tar.gz test archives. And it switches the web-platform archive to a tar.gz file. The performance implications for archive generation are significant: before: 48,321,250 bytes; 6.05s after: 31,844,267 bytes; 4.57s The size is reduced because we have a single compression context so data from 1 file can benefit compression in a subsequent file. CPU usage is reduced because the compressor has to work less with 1 context than it does with N. While I didn't measure it, decompression performance should also be improved for the same reasons. And of course network I/O will be reduced. mozharness consumers use a generic method for handling unarchiving. This method automagically handles multiple file extensions. So as long as downstream consumers aren't hard coding ".zip" this change should "just work." MozReview-Commit-ID: LQa5MIHLsms

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

import imp
import os
import shlex
import sys
try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO

old_bytecode = sys.dont_write_bytecode
sys.dont_write_bytecode = True

path = os.path.join(os.path.dirname(__file__), 'mach')

if not os.path.exists(path):
    path = os.path.join(os.path.dirname(__file__), 'config.status')
    config = imp.load_module('_buildconfig', open(path), path, ('', 'r', imp.PY_SOURCE))
    path = os.path.join(config.topsrcdir, 'mach')
mach_module = imp.load_module('_mach', open(path), path, ('', 'r', imp.PY_SOURCE))

sys.dont_write_bytecode = old_bytecode

def FlagsForFile(filename):
    mach = mach_module.get_mach()
    out = StringIO()

    # Mach calls sys.stdout.fileno(), so we need to fake it when capturing it.
    # Returning an invalid file descriptor does the trick.
    out.fileno = lambda: -1
    out.encoding = None
    mach.run(['compileflags', filename], stdout=out, stderr=out)

    flag_list = shlex.split(out.getvalue())

    # This flag is added by Fennec for android build and causes ycmd to fail to parse the file.
    # Removing this flag is a workaround until ycmd starts to handle this flag properly.
    # https://github.com/Valloric/YouCompleteMe/issues/1490
    final_flags = [x for x in flag_list if not x.startswith('-march=armv')]

    return {
        'flags': final_flags,
        'do_cache': True
    }