Bug 1562686 - revert remaining unnecessary bit of bug 1187245; r=glandium a=tomprince Differential Revision: https://phabricator.services.mozilla.com/D42644

Bug 1516325. r=glandium, a=lizzard Differential Revision: https://phabricator.services.mozilla.com/D22638

Bug 1531176 - Split the Google key management between gls and safe browsing r=glandium a=RyanVM Differential Revision: https://phabricator.services.mozilla.com/D21459

Bug 1531176 - Split the Google key management between gls and safe browsing r=glandium a=RyanVM Differential Revision: https://phabricator.services.mozilla.com/D21459

Bug 1492664 - update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium

Bug 1492664 - update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium

Bug 1492664 - use {artifact-reference: ..} in diffoscope; r=glandium

Bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium Eventually, workers will provide these variables directly (https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this ensures that TASKCLUSTER_ROOT_URL is set everywhere in production, and TASKCLUSTER_PROXY_URL is set wherever the proxy is active. The taskgraph Taskcluster utils module gets a `get_root_url()` that gets the root URL for the current run, either from an environment variable in production or, on the command line, defaulting to https://taskcluster.net for user convenience. When the production instance's URL changes, we can simply change that default. Other changes to use this function are reserved for later commits. This changes the docker build process propagate TASKCLUSTER_ROOT_URL into the docker images where necessary (using %ARG), specifically to create URLs for debian repo paths.

Bug 1513320 - SQLite package backport for Debian 7; r=glandium The SQLite in Debian 7 (3.7.13) lacks support for common table expressions (the WITH keyword), which was introduced in SQLite 3.8.3. The Mercurial SQLite storage backend currently relies on CTEs. Even if a future Mercurial doesn't require CTE, it is likely that it will still use CTE if available for performance reasons. So, it is in our best interest to give Mercurial access to a modern SQLite. Plus, using a modern SQLite and avoiding potential bugs in old versions seems prudent. This commit introduces a SQLite package backport for Debian 7 so we can use the new SQLite feature. We had to minimally patch the build to work with an older version of TCL that isn't using multiarch. I observed libsqlite3 being installed as part of building various other packages (such as Python). I initially added the package as a dependency so packages would be built against a more modern SQLite. But glandium doesn't believe it matters, since nothing should be doing build-time feature detection. Python is the most important downstream package (since Mercurial uses its SQLite). I audited the CPython build system and did not see any build-time SQLite feature detection or version sniffing. So I think we'll be fine building against an older SQLite. Differential Revision: https://phabricator.services.mozilla.com/D14194

Bug 1531655 - Emit a configure error if rustc version >= 1.33 and --enable-rust-simd used. r=glandium a=RyanVM

Bug 1471132 - Handle 3-column nm/readelf output in check_binary. r=glandium, a=jcristau This patch backports only one small part of 1471132 to handle certain types of nm/readelf output.

Bug 1481633 Resolve kPStaticModules undefined symbols in MinGW Clang. r=glandium, a=jcristau clang can handle MSVC-like codepaths generally, so we want to use those when building with clang for Windows. So we switch _MSC_VER over to _WIN32 to pick up those codepaths when compiling for Windows with clang. Additionally, we relax the ordering of sections for the same scenario. Note that we do need to tell clang to use -fms-extensions with the MSVC code, we do that in the mingw clang build job patch. Differential Revision: https://phabricator.services.mozilla.com/D3526

Bug 1473291 - handle valueless dynamic section entries; r=glandium, a=jcristau Some dynamic entry types, like DT_BIND_NOW, are printed without a value by readelf. Handle such types gracefully.

Bug 1519209 - Disable NSS_ALLOW_SSLKEYLOGFILE in beta and release. r=glandium, a=RyanVM This disables NSS_ALLOW_SSLKEYLOGFILE in beta in release in order to avoid shutdown hangs until the NSS project has time to fix the root cause of the issue.

Bug 1498640 - deploy latest image_builder image r=glandium This uses the latest image_builder image (on docker hub) to build even the image_builder image. The change to `docker.py` handles a new API response (`aux`) from the Docker daemon. It's unclear what this key means, but displaying it is simple. Differential Revision: https://phabricator.services.mozilla.com/D8441

Bug 1434316 - Add 64 Bit Windows Build Targets. r=glandium, a=RyanVM

Bug 1434316 - Bump MinGW version. r=glandium, a=RyanVM

Bug 1434316 - Add 64 Bit Windows Build Targets. r=glandium, a=RyanVM

Bug 1467605 - Disable processing of fetch dependencies; r=glandium a=tomprince 90dca0906337 accidentally broke `mach artifact toolchain --from-build` because that code is attempting to load toolchain tasks in isolation. The new "use_fetches" transform added to toolchain tasks requires that "fetch" tasks are already processed and their references are available to toolchain tasks. This commit adds a mechanism to effectively disable the "use_fetches" transform when called by `mach artifact toolchain`. It is a hack. I suspect future planned work around artifacts/fetches will necessitate additional changes to the `mach artifact toolchain` code. But this can be deferred to a later day: this commit unbusts `mach artifact toolchain` and isn't super hacky, so it seems more reasonable than backing out fetch tasks completely. Differential Revision: https://phabricator.services.mozilla.com/D1588

Bug 1445943 - Add Enterprise Policy support for macOS to ESR. This includes patches in bug 1445943, bug 1497408, bug 1497948, bug 1498032. r=mstange,felipe,glandium,spohl, a=lizzard

Bug 1507614 - Identify ESR with MOZ_APP_VERSION_DISPLAY. r=glandium, a=lizzard Differential Revision: https://phabricator.services.mozilla.com/D12067

Bug 1445943 - Add Enterprise Policy support for macOS to ESR. This includes patches in bug 1445943, bug 1497408, bug 1497948, bug 1498032. r=mstange,felipe,glandium,spohl, a=lizzard

Bug 1475566 - Disable #pragma comments for MinGW Builds. r=glandium, a=lizzard In the MinGW browser build job, we're going to use -fms-extensions, which will tell clang to start processing these comments. Clang cannot process them correctly (it's an upstream bug) but it doesn't need to, because we include the libs we need in moz.build files. So we exclude them for MinGW builds. mingw-clang gets them wrong and mingw-gcc (which doesn't even work anymore on -central) ignored them. In the future, with a llvm fix, we could clean up the moz.build files and re-enable these comments.

Bug 1491788 - Use clang 7final to build mingw r=glandium Differential Revision: https://phabricator.services.mozilla.com/D6018

Bug 1490564 Add a x86 MinGW-clang toolchain job r=glandium Differential Revision: https://phabricator.services.mozilla.com/D5720

Bug 1490567 - Update ffi build to handle x86 MinGW-clang builds. r=glandium, a=lizzard Differential Revision: https://phabricator.services.mozilla.com/D5643

Bug 1489744 - Fix a bounds violation crash in the prefs parser. r=glandium, a=RyanVM Currently, if a get_char() call returns EOF, the index moves beyond the buffer's bounds and get_char() cannot be called again without triggering a panic. As a result, everywhere that encounters an EOF and then does subsequent parsing ungets the EOF... except there was one place that failed to do that: the match case for CharKind::Slash in get_token(). This meant that a single '/' at the end of the input could trigger a bounds violation (but only if it is the start of a new token). This EOF-unget requirement is subtle and easy to get wrong, so this patch eliminates it. get_char() now can be called repeatedly after an EOF, and will return EOF on each subsequent call. This means that some of the existing unget_char() calls can be removed. Some others are still necessary to get line numbers correct in error messages, but the outcome of mishandled cases is now much less drastic -- an incorrect line number in an error message instead of a panic. The patch also clarifies a couple of related comments.

Bug 1460777 - Taskgraph tasks for retrieving remote content; r=dustin, glandium Currently, many tasks fetch content from the Internets. A problem with that is fetching from the Internets is unreliable: servers may have outages or be slow; content may disappear or change out from under us. The unreliability of 3rd party services poses a risk to Firefox CI. If services aren't available, we could potentially not run some CI tasks. In the worst case, we might not be able to release Firefox. That would be bad. In fact, as I write this, gmplib.org has been unavailable for ~24 hours and Firefox CI is unable to retrieve the GMP source code. As a result, building GCC toolchains is failing. A solution to this is to make tasks more hermetic by depending on fewer network services (which by definition aren't reliable over time and therefore introduce instability). This commit attempts to mitigate some external service dependencies by introducing the *fetch* task kind. The primary goal of the *fetch* kind is to obtain remote content and re-expose it as a task artifact. By making external content available as a cached task artifact, we allow dependent tasks to consume this content without touching the service originally providing that content, thus eliminating a run-time dependency and making tasks more hermetic and reproducible over time. We introduce a single "fetch-url" "using" flavor to define tasks that fetch single URLs and then re-expose that URL as an artifact. Powering this is a new, minimal "fetch" Docker image that contains a "fetch-content" Python script that does the work for us. We have added tasks to fetch source archives used to build the GCC toolchains. Fetching remote content and re-exposing it as an artifact is not very useful by itself: the value is in having tasks use those artifacts. We introduce a taskgraph transform that allows tasks to define an array of "fetches." Each entry corresponds to the name of a "fetch" task kind. When present, the corresponding "fetch" task is added as a dependency. And the task ID and artifact path from that "fetch" task is added to the MOZ_FETCHES environment variable of the task depending on it. Our "fetch-content" script has a "task-artifacts" sub-command that tasks can execute to perform retrieval of all artifacts listed in MOZ_FETCHES. To prove all of this works, the code for fetching dependencies when building GCC toolchains has been updated to use `fetch-content`. The now-unused legacy code has been deleted. This commit improves the reliability and efficiency of GCC toolchain tasks. Dependencies now all come from task artifacts and should always be available in the common case. In addition, `fetch-content` downloads and extracts files concurrently. This makes it faster than the serial application which we were previously using. There are some things I don't like about this commit. First, a new Docker image and Python script for downloading URLs feels a bit heavyweight. The Docker image is definitely overkill as things stand. I can eventually justify it because I want to implement support for fetching and repackaging VCS repositories and for caching Debian packages. These will require more packages than what I'm comfortable installing on the base Debian image, therefore justifying a dedicated image. The `fetch-content static-url` sub-command could definitely be implemented as a shell script. But Python is readily available and is more pleasant to maintain than shell, so I wrote it in Python. `fetch-content task-artifacts` is more advanced and writing it in Python is more justified, IMO. FWIW, the script is Python 3 only, which conveniently gives us access to `concurrent.futures`, which facilitates concurrent download. `fetch-content` also duplicates functionality found elsewhere. generic-worker's task payload supports a "mounts" feature which facilitates downloading remote content, including from a task artifact. However, this feature doesn't exist on docker-worker. So we have to implement downloading inside the task rather than at the worker level. I concede that if all workers had generic-worker's "mounts" feature and supported concurrent download, `fetch-content` wouldn't need to exist. `fetch-content` also duplicates functionality of `mach artifact toolchain`. I probably could have used `mach artifact toolchain` instead of writing `fetch-content task-artifacts`. However, I didn't want to introduce the requirement of a VCS checkout. `mach artifact toolchain` has its origins in providing a feature to the build system. And "fetching artifacts from tasks" is a more generic feature than that. I think it should be implemented as a generic feature and not something that is "toolchain" specific. I think the best place for a generic "fetch content" feature is in the worker, where content can be defined in the task payload. But as explained above, that feature isn't universally available. The next best place is probably run-task. run-task already performs generic, very-early task preparation steps, such as performing a VCS checkout. I would like to fold `fetch-content` into run-task and make it all driven by environment variables. But run-task is currently Python 2 and achieving concurrency would involve a bit of programming (or adding package dependencies). I may very well port run-task to Python 3 and then fold fetch-content into it. Or maybe we leave `fetch-content` as a standalone script. MozReview-Commit-ID: AGuTcwNcNJR

Bug 1443471 - Support mingw clang in skia moz.build. r=glandium, a=RyanVM MozReview-Commit-ID: 4H8bkHyczGM

Bug 1443471 - Take clang mingw into account in moz.build files. r=glandium, a=RyanVM MozReview-Commit-ID: 2vKiHjmI9Hn

Bug 1443471 - Use correct rust target for mingw clang. r=glandium, a=RyanVM MozReview-Commit-ID: G40IanxGWXv

Bug 1471556 - Support mingw clang in configure scripts. r=glandium, a=RyanVM MozReview-Commit-ID: GKLbHvYgXnL

Bug 1471643 - Use -std=gnu89 for libpng compilation on mingw. r=glandium, a=RyanVM MozReview-Commit-ID: 3JnNunChnv

Bug 1471293 - Support using llvm-objdump in dependentlibs.py. r=glandium, a=RyanVM MozReview-Commit-ID: 8b0o06EQh3n

Bug 1469676 - Reduce max-run-time for various build tasks; r=glandium The max-run-time for tasks appears to have been mostly cargo culted. In particular, a value of 36000 (10 hours) is quite absurd: no single task takes that long to execute. This commit reduces the max-run-time of several tasks to more reasonable values. The goal here is to prevent excessively long-running tasks. There is definitely room to further tweak values to further mitigate long-running tasks. But let's bite off the biggest chunk first, since that doesn't require much mental effort. There is a possibility I overshot on some of these tasks. If we get timeouts, we can always increase the timeout again. Differential Revision: https://phabricator.services.mozilla.com/D1716

Bug 1460777 - Extract GPG keys to standalone files; r=glandium After this change, we consistently import GPG keys from files in the GCC build scripts. MozReview-Commit-ID: BcyvCQoGbMS

Bug 1466746 - Install python-zstandard in debian-base; r=glandium Let's install python-zstandard for both Python 2 and Python 3 in all our Debian-based images so it is readily available for use. MozReview-Commit-ID: 1L8zDc5MYXA

Bug 1466746 - Debian packages for python-zstandard; r=glandium python-zstandard's 0.9.1 source distribution contains a debian/ directory. On Squeeze, producing a Debian package is straightforward. On Wheezy, we need to hack up Build-Depends because Wheezy doesn't have a package for the Hypothesis fuzzing library. This package is only used for testing and our package building disables testing, so we don't even need to further hack up the packaging to disable tests. MozReview-Commit-ID: 6raXjdzggCH

Bug 1466746 - dh-python backport for wheezy; r=glandium dh-python isn't available in Wheezy. Let's backport it so we can build Python packages that use it. Fortunately for us, the package builds without any modifications. The only customization we need is to ensure our custom Python packages are present in order to satisfy Build-Depends. MozReview-Commit-ID: CqZtwvosA6K

Bug 1460451 - Add /usr/bin/python3 to Debian images; r=glandium The python3-minimal package provides /usr/bin/python3 on Debian. This commit installs this package so a `python3` executable is provided. This required backporting the package to wheezy. The final patch is trivial. But I wasted a bit of time figuring out why `mk-build-deps` wasn't working. It would no-op and exit 0 and then the build would complain about missing dependencies! glandium's theory is that the ":any" multiarch support on wheezy isn't complete. Removing ":any" seems to make things "just work." MozReview-Commit-ID: FBicpK4SmkQ

Bug 1449629 - Install Python 3.5 in debian-base; r=glandium We want Python 3.5+ to be available everywhere so various processes can start using it. The debian-base Dockerfile is shared by Debian 7 and 9 images. Debian 9 ships with Python 3.5 and after the previous commit, we have a Python 3.5 package for Debian 7. So we simply install the "python3.5" package to get Python on all the Debian images. MozReview-Commit-ID: 9ZmoSxtHWTZ

Bug 1449629 - Install Python 3.5 in Debian 7 base image; r=glandium Debian 7 ships Python 3.2 by default. That's too old for our upcoming build requirement of Python 3.5. This commit adds a Python 3.5 package for wheezy that backports the Python 3.5 from a much later Debian version. The patch was inspired by the existing patch for Python 2.7. However, it needed additional work. The changes and reasons should all be documented in the changelog file as part of the package diff we apply. I'm a bit disappointed we had to disable PGO. But it was reliably segfaulting during the build. I didn't feel like going down that rabbit hole. MozReview-Commit-ID: ABpHW1KYFQP

Bug 1460720 - Do not define _aligned_malloc - instead define _aligned_malloc_impl and export _aligned_malloc. r=glandium, a=jcristau MozReview-Commit-ID: 3EwAd81Iz7r

Bug 1420350 - Enable jemalloc on MinGW. r=glandium, a=jcristau MozReview-Commit-ID: 6YUeFAJocHj

Bug 1448749 - Resolve undefined references to '_imp__mozalloc_abort' in the MinGW build. r=glandium, a=jcristau MozReview-Commit-ID: 5enjU5Xp8G9

Bug 1443823 - Apply no-keep-inline-dllexport to MinGW x64 also. r=glandium, a=jcristau MozReview-Commit-ID: 2Nyw738ZHou

Bug 1457598 - Add MinGW and GCC scripts to the resources of fxc2 and nsis to ensure they get rebuilt. r=glandium, a=jcristau

Bug 1440013 - For MinGW build, pass -Wa,-mbig-obj to solve 'too many sections' errors. r=glandium, a=jcristau MozReview-Commit-ID: 9ObJnrcpeKe

Bug 1422930 - Fix SpiderMonkey includedir installs. r=glandium, a=jcristau Somehow the header files were being installed directly into $prefix/include, rather than $prefix/include/mozjs-60. Something else changed somewhere that affected this, since this code was the same in older mozjs versions, but this seems the most logical place to fix it.

Bug 1440461 - Disable titlebar rendering for Linux/Firefox 59. r=glandium, a=lizzard The titlebar rendering on Linux/Gtk+ is recently enabled at Beta59 but with many bugs fixed at Nightly. Let's disable this feature for Beta / Release 59 and ship it at Firefox 60 where majority of the issues are fixed. MozReview-Commit-ID: FQL7tNhcvUG