Bug 1367852 - Defer file read to reduce peak memory use in mozjar when dumping symbols. r=ted
authorChris Manchester <cmanchester@mozilla.com>
Tue, 30 May 2017 12:11:21 -0700
changeset 409498 463ad37994b0fd9b3eec67755e9f6a450af3a1fc
parent 409497 445cb4d47d4f80d58f10eb263035e0045f6e7b93
child 409499 2c8b721cffdb112e6cf377a393f101b8b6fdc006
push id7391
push usermtabara@mozilla.com
push dateMon, 12 Jun 2017 13:08:53 +0000
treeherdermozilla-beta@2191d7f87e2e [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersted
bugs1367852
milestone55.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1367852 - Defer file read to reduce peak memory use in mozjar when dumping symbols. r=ted Passing a file object intead of a string of the file's contents defers reading the file to a place Python can free the resulting string earlier, reducing peak memory consumption when packaging symbols and avoiding a MemoryError on Windows 32 builders in automation. MozReview-Commit-ID: H0R6BbjwhOu
python/mozbuild/mozbuild/action/symbols_archive.py
--- a/python/mozbuild/mozbuild/action/symbols_archive.py
+++ b/python/mozbuild/mozbuild/action/symbols_archive.py
@@ -20,17 +20,17 @@ def make_archive(archive_name, base, exc
         compress = ['**/*.sym']
     archive_basename = os.path.basename(archive_name)
     with open(archive_name, 'wb') as fh:
         with JarWriter(fileobj=fh, optimize=False, compress_level=5) as writer:
             for pat in include:
                 for p, f in finder.find(pat):
                     print('  Adding to "%s":\n\t"%s"' % (archive_basename, p))
                     should_compress = any(mozpath.match(p, pat) for pat in compress)
-                    writer.add(p.encode('utf-8'), f.read(), mode=f.mode,
+                    writer.add(p.encode('utf-8'), f, mode=f.mode,
                                compress=should_compress, skip_duplicates=True)
 
 def main(argv):
     parser = argparse.ArgumentParser(description='Produce a symbols archive')
     parser.add_argument('archive', help='Which archive to generate')
     parser.add_argument('base', help='Base directory to package')
     parser.add_argument('--exclude', default=[], action='append', help='File patterns to exclude')
     parser.add_argument('--include', default=[], action='append', help='File patterns to include')