Bug 1518726 - Apply https://reviews.llvm.org/D56475 to clang. r=froydnj
authorMike Hommey <mh+mozilla@glandium.org>
Thu, 10 Jan 2019 00:18:28 +0000
changeset 510307 33784a7ae90baf030b7a41b11f828a8767fa51ae
parent 510306 3e99aed6c14d423119aebd1cb2214f3270008faf
child 510308 463d00cf41fdc9baf09ab663c55e680ec3dce6b6
push id10547
push userffxbld-merge
push dateMon, 21 Jan 2019 13:03:58 +0000
treeherdermozilla-beta@24ec1916bffe [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersfroydnj
bugs1518726
milestone66.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1518726 - Apply https://reviews.llvm.org/D56475 to clang. r=froydnj Firefox uses multiple processes. It has intentional leaks, and when running with ASAN, we have suppressions to eliminate those. When running ASAN builds through CI tests, when Firefox exits, each of the processes (parent and child) exits and goes through its leaks and when there are (which is a given), the ASAN runtime runs llvm-symbolizer to symbolicate and match against suppressions. So each process runs llvm-symbolizer. At the same time. Some of the addresses to symbolicate are in libxul. Which contains all DWARF info, making it a ~1GB monster. Oh, and because you're lucky, things align perfectly such that libxul size is a multiple of the page size. That makes llvm-symbolizer pread() the file instead of mmap()ing it. Did I say there are multiple processes? So suddenly you have n processes simultaneously allocating and filling 1GB of memory each, on CI machines that have enough memory for the job they usually run, but not enough for a sudden rush of n GB. And things go awry. When you're lucky and the OOM killer didn't take care of killing the CI entirely, symbolication couldn't happen and the suppressions are not matched, and leaks are reported. This all turns out it originates in how llvm-symbolicate chooses between pread() and mmap(), which turns out is just defaults not being made for binary files. Differential Revision: https://phabricator.services.mozilla.com/D16010
build/build-clang/clang-7-android.json
build/build-clang/clang-7-linux64.json
build/build-clang/clang-7-macosx64.json
build/build-clang/clang-trunk-mingw.json
build/build-clang/clang-win64.json
build/build-clang/r350774.patch
--- a/build/build-clang/clang-7-android.json
+++ b/build/build-clang/clang-7-android.json
@@ -23,11 +23,12 @@
           "/builds/worker/workspace/build/src/android-ndk/sysroot/usr/include/arm-linux-androideabi",
           "/builds/worker/workspace/build/src/android-ndk/sysroot/usr/include"
         ]
       }
     },
     "patches": [
       "static-llvm-symbolizer.patch",
       "find_symbolizer_linux.patch",
-      "rename_gcov_flush_.patch"
+      "rename_gcov_flush_.patch",
+      "r350774.patch"
     ]
 }
--- a/build/build-clang/clang-7-linux64.json
+++ b/build/build-clang/clang-7-linux64.json
@@ -13,11 +13,12 @@
     "python_path": "/usr/bin/python2.7",
     "gcc_dir": "/builds/worker/workspace/build/src/gcc",
     "cc": "/builds/worker/workspace/build/src/gcc/bin/gcc",
     "cxx": "/builds/worker/workspace/build/src/gcc/bin/g++",
     "as": "/builds/worker/workspace/build/src/gcc/bin/gcc",
     "patches": [
       "static-llvm-symbolizer.patch",
       "find_symbolizer_linux.patch",
-      "rename_gcov_flush_.patch"
+      "rename_gcov_flush_.patch",
+      "r350774.patch"
     ]
 }
--- a/build/build-clang/clang-7-macosx64.json
+++ b/build/build-clang/clang-7-macosx64.json
@@ -18,11 +18,12 @@
     "as": "/builds/worker/workspace/build/src/clang/bin/clang",
     "ar": "/builds/worker/workspace/build/src/cctools/bin/x86_64-darwin11-ar",
     "ranlib": "/builds/worker/workspace/build/src/cctools/bin/x86_64-darwin11-ranlib",
     "libtool": "/builds/worker/workspace/build/src/cctools/bin/x86_64-darwin11-libtool",
     "ld": "/builds/worker/workspace/build/src/clang/bin/clang",
     "patches": [
       "static-llvm-symbolizer.patch",
       "compiler-rt-cross-compile.patch",
-      "compiler-rt-no-codesign.patch"
+      "compiler-rt-no-codesign.patch",
+      "r350774.patch"
     ]
 }
--- a/build/build-clang/clang-trunk-mingw.json
+++ b/build/build-clang/clang-trunk-mingw.json
@@ -9,10 +9,13 @@
     "lld_repo": "https://llvm.org/svn/llvm-project/lld/trunk",
     "compiler_repo": "https://llvm.org/svn/llvm-project/compiler-rt/trunk",
     "libcxx_repo": "https://llvm.org/svn/llvm-project/libcxx/trunk",
     "libcxxabi_repo": "https://llvm.org/svn/llvm-project/libcxxabi/trunk",
     "python_path": "/usr/bin/python2.7",
     "gcc_dir": "/builds/worker/workspace/build/src/gcc",
     "cc": "/builds/worker/workspace/build/src/gcc/bin/gcc",
     "cxx": "/builds/worker/workspace/build/src/gcc/bin/g++",
-    "as": "/builds/worker/workspace/build/src/gcc/bin/gcc"
+    "as": "/builds/worker/workspace/build/src/gcc/bin/gcc",
+    "patches": [
+      "r350774.patch"
+    ]
 }
--- a/build/build-clang/clang-win64.json
+++ b/build/build-clang/clang-win64.json
@@ -16,11 +16,12 @@
     "patches": [
       "workaround-issue38586.patch",
       "r342649-hotpatch-8-byte-nops.patch",
       "r342652-unpoison-thread-stacks.patch",
       "r343123-pin-asan-dll.patch",
       "aarch64-vastart-checking.patch",
       "downgrade-mangling-error.patch",
       "r346300-compiler-rt-windows-mmap.patch",
-      "loosen-msvc-detection.patch"
+      "loosen-msvc-detection.patch",
+      "r350774.patch"
     ]
 }
new file mode 100644
--- /dev/null
+++ b/build/build-clang/r350774.patch
@@ -0,0 +1,14 @@
+diff --git a/llvm/lib/Object/Binary.cpp b/llvm/lib/Object/Binary.cpp
+index d7c25921ec3..fe41987f5c2 100644
+--- a/llvm/lib/Object/Binary.cpp
++++ b/llvm/lib/Object/Binary.cpp
+@@ -88,7 +88,8 @@ Expected<std::unique_ptr<Binary>> object::createBinary(MemoryBufferRef Buffer,
+ 
+ Expected<OwningBinary<Binary>> object::createBinary(StringRef Path) {
+   ErrorOr<std::unique_ptr<MemoryBuffer>> FileOrErr =
+-      MemoryBuffer::getFileOrSTDIN(Path);
++      MemoryBuffer::getFileOrSTDIN(Path, /*FileSize=*/-1,
++                                   /*RequiresNullTerminator=*/false);
+   if (std::error_code EC = FileOrErr.getError())
+     return errorCodeToError(EC);
+   std::unique_ptr<MemoryBuffer> &Buffer = FileOrErr.get();