Bug 1617794 - Properly set the target flag for bindgen when cross compiling for Windows. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D64292

Bug 1606922 - Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=tomprince,chmanchester We don't need to package tests for builds that we don't actually run tests from, but it is tricky to align this correctly by setting MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can set the environment variable in the task definition, and use a full taskgraph verification check to ensure that the flag is only set on builds that have tests. The one tricky area is the win64-aarch64 builds, which have a workaround by specifying the new skip-verify-test-packaging attribute. In one case, win64-aarch64-shippable has tests that run against it, but it copies those tests from a win64-aarch64-shippable-no-eme task, which itself has no tests. Both of those tasks need to skip the verify check as a result. In another case, the win64-aarch64-eme task is an artifact build that grabs test packages from the win64-aarch64 build. Since the win64-aarch64 build doesn't have tests, it needs to skip the verify check. Differential Revision: https://phabricator.services.mozilla.com/D59426

Bug 1606922 - Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=tomprince,chmanchester We don't need to package tests for builds that we don't actually run tests from, but it is tricky to align this correctly by setting MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can set the environment variable in the task definition, and use a full taskgraph verification check to ensure that the flag is only set on builds that have tests. The one tricky area is the win64-aarch64 builds, which have a workaround by specifying the new skip-verify-test-packaging attribute. In one case, win64-aarch64-shippable has tests that run against it, but it copies those tests from a win64-aarch64-shippable-no-eme task, which itself has no tests. Both of those tasks need to skip the verify check as a result. In another case, the win64-aarch64-eme task is an artifact build that grabs test packages from the win64-aarch64 build. Since the win64-aarch64 build doesn't have tests, it needs to skip the verify check. Differential Revision: https://phabricator.services.mozilla.com/D59426

Bug 1617207 - Enable LTO on Android PGO builds. r=chmanchester Turns out we have one more copy of the PGO-initializing mozconfig code for Android, and that one was not enabling LTO. Differential Revision: https://phabricator.services.mozilla.com/D63721

Bug 1495394 - add `mach install-moz-phab` command; r=firefox-build-system-reviewers,chmanchester Install moz-phab using the correct command for the current operating system as per moz-phab's documentation. Differential Revision: https://phabricator.services.mozilla.com/D59139

Bug 1613958 - set correct treeherder platform names for macOS webrender and condprof tasks. r=chmanchester Some webrender and condprof tasks on macOS had raw/'ugly' platform names and were listed at the top of a treeherder push due to that and not together with the other OS X tasks. With this patch applied, they are shown together with the other tasks on that platform. The platform name should reference either cross compile or the platform version it is running on. Differential Revision: https://phabricator.services.mozilla.com/D62051

Bug 1613649 - Explicitly opt-out of Rust's local ThinLTO for DEVELOPER_OPTIONS builds. r=firefox-build-system-reviewers,chmanchester By default the Rust compiler will perform a limited kind of ThinLTO on each crate. For local builds this additional optimization is not worth the increase in compile time. Differential Revision: https://phabricator.services.mozilla.com/D61821

Bug 1613247: Simplify sccache config; r=firefox-build-system-reviewers,chmanchester The logic for finding the sccache configuration to use has gone through several iterations. At this point, all the build workers set `TASKCLUSTER_WORKER_LOCATION`, so remove the code for dealing with other configurations. Also don't set sccache configuration if it is disabled, even if the bucket somehow gets set. Differential Revision: https://phabricator.services.mozilla.com/D61635

Bug 1613247: Use calculated GCS bucket in mozconfig.cacche; r=firefox-build-system-reviewers,chmanchester The code goes to some length to calculate the bucket to use, but then hard-codes the bucket in the credential URL it uses. Differential Revision: https://phabricator.services.mozilla.com/D61634

Bug 1611990 - Firefox omits Rust log messages of less than WARN level unless compiled in debug mode r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D61254

Bug 1604621 - Remove HAS_MISC_RULE; r=firefox-build-system-reviewers,chmanchester Two of these no longer have any misc:: rules associated with them, and can be removed outright. The remaining misc:: rule in toolkit/components/telemetry is already traversed because it contains EXTRA_COMPONENTS, which are processed during misc in the RecursiveMake backend. So we can remove HAS_MISC_RULE entirely even though we still have a custom misc rule to process addons (until bug 988938 is fixed). Differential Revision: https://phabricator.services.mozilla.com/D59425

Bug 1603557: Sanitize the environment before running xpcshell tests. r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D60062

Bug 1609422 - Clean up an old rustc version check for pgo r=firefox-build-system-reviewers,chmanchester We require rustc 1.39 nowadays. Differential Revision: https://phabricator.services.mozilla.com/D60008

Bug 1606922 - Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=firefox-build-system-reviewers,tomprince,chmanchester We don't need to package tests for builds that we don't actually run tests from, but it is tricky to align this correctly by setting MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can set the environment variable in the task definition, and use a full taskgraph verification check to ensure that the flag is only set on builds that have tests. The one tricky task is win64-aarch64-shippable/opt, which copies tests from another build rather than building them itself. For this reason, it explicitly sets MOZ_AUTOMATION_PACKAGE_TESTS: '0' in the environment even though that is now the default. This is why the exception is only raised if MOZ_AUTOMATION_PACKAGE_TESTS is not set at all, rather than checking that it is set to 1. Differential Revision: https://phabricator.services.mozilla.com/D59426

Bug 1608326 - don't unset WASM_SANDBOXED_LIBRARIES for Linux artifact builds; r=firefox-build-system-reviewers,chmanchester Otherwise we won't package the necessary libraries, which leads to peculiar runtime failures. Depends on D59555 Differential Revision: https://phabricator.services.mozilla.com/D59556

Bug 1608326 - only check for wasm tools when compiling; r=firefox-build-system-reviewers,chmanchester ...so that we can correctly declare that we are using sandboxed library(ies) from our "parent" build, but not check for all the tools to compile code. Depends on D59554 Differential Revision: https://phabricator.services.mozilla.com/D59555

Bug 1608326 - move most wasm sandboxing configury under an only_when; r=firefox-build-system-reviewers,chmanchester This change eliminates a lot of repetition, and paves the way for the next change. Differential Revision: https://phabricator.services.mozilla.com/D59554

Bug 1601691 - Remove FIXMEs for strict-alias which is WONTFIX r=firefox-build-system-reviewers,chmanchester Bug 414641 was marked WONTFIX, so let's remove these comments from configure Differential Revision: https://phabricator.services.mozilla.com/D55993

Bug 1607278 - specify memory and guard page sizes for lucetc; r=firefox-build-system-reviewers,chmanchester ...and pull lucetc options out into a separate variable for readability. Differential Revision: https://phabricator.services.mozilla.com/D58830

Bug 1605189 - update in-tree robustcheckout with latest version r=firefox-build-system-reviewers,chmanchester This commit vendors `robustcheckout` from commit ff8a7c9c843a9c in version-control-tools. Updates include the removal of some dead code, compatibility with the latest Mercurial, and Python 3 compatibility. Differential Revision: https://phabricator.services.mozilla.com/D57836

Bug 1605189 - update in-tree robustcheckout with latest version r=firefox-build-system-reviewers,chmanchester This commit vendors `robustcheckout` from commit ff8a7c9c843a9c in version-control-tools. Updates include the removal of some dead code, compatibility with the latest Mercurial, and Python 3 compatibility. Differential Revision: https://phabricator.services.mozilla.com/D57836

Bug 1604881 - fail if user attempts to set CC or CXX with "mk_add_options" in .mozconfig r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D57656

Bug 1601578 - mach - doc environment with python 3 r=ahal,firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D55937

Bug 1601578 - mach - doc environment with python 3 r=ahal,firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D55937

Bug 1603561 - [mozbuild] Ignore all potential objdirs in BuildReader._relevant_mozbuild_finder, r=firefox-build-system-reviewers,chmanchester Currently BuildReader._relevant_mozbuild_finder attempted to exclude objdirs by ignoring the pattern 'obj*'. However this can cause problems for developers who don't prefix their objdirs with the string 'obj'. This attempts to ignore all objdirs by looking for 'config.status' files in the topsrcdir. This wasn't a huge issue before because BuildReader._relevant_mozbuild_finder was only used in a handful of obscure edge cases. But now it's being invoked by the taskgraph and consequently |mach try|. So we'll see this issue pop up for people more frequently. Differential Revision: https://phabricator.services.mozilla.com/D57113

Bug 1594344 - Part 2: Fetch dump_syms toolchain on Windows builds. r=firefox-build-system-reviewers,chmanchester This installs the rust-based `dump_syms` toolchain on Windows builds. Differential Revision: https://phabricator.services.mozilla.com/D53319

Bug 1601704 - Work around sccache/rust confusion about mozglue-static. libxul part. r=firefox-build-system-reviewers,chmanchester While it doesn't seem to be happening (yet), the same problem that happened with jsrust can just as well happen with other leaf staticlib crates that indirectly use mozglue-static. Differential Revision: https://phabricator.services.mozilla.com/D56159

Bug 1602568 - Enable Rust PGO on linux32 and Windows r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D56584

Bug 1594344 - Part 2: Fetch dump_syms toolchain on Windows builds. r=firefox-build-system-reviewers,chmanchester This installs the rust-based `dump_syms` toolchain on Windows builds. Differential Revision: https://phabricator.services.mozilla.com/D53319

Bug 1557788 - Remove unused mozharness build variant; r=firefox-build-system-reviewers,chmanchester This is no longer used as of bug 1553065. Differential Revision: https://phabricator.services.mozilla.com/D56118

Bug 1557788 - Rename MOZ_PGO_PROFILE_USE to TASKCLUSTER_PGO_PROFILE_USE; r=firefox-build-system-reviewers,chmanchester MOZ_PGO_PROFILE_USE is used to enable certain features in the automation mozconfigs for profile-use builds, including the MOZ_PROFILE_USE configure option. They aren't easily combined into one flag due to the extra settings that are enabled only in automation mozconfigs. Renaming the former variable to TASKCLUSTER_PGO_PROFILE_USE makes it slightly more obvious where it comes from and avoids confusion. Differential Revision: https://phabricator.services.mozilla.com/D56117

Bug 1557788 - Remove spurious MOZ_PROFILE_USE ifndefs; r=firefox-build-system-reviewers,chmanchester One of these appears to be mistakenly leftover from bug 861178 which was intended to avoid purging dist/, but the code inside the ifndef no longer does that. The other is from bug 1246881 to avoid re-writing the buildid.h file during the profile-use build, but now that there are two separate builds, they will each have their own buildid.h / source-repo.h files. Differential Revision: https://phabricator.services.mozilla.com/D56116

Bug 1557788 - Remove OBJS_VAR_SUFFIX & .i_o suffix for instrumented builds; r=firefox-build-system-reviewers,chmanchester In 1-tier PGO builds that shared the objdir between the instrumented and profile-use builds, the instrumented build objects used a different suffix (.i_o) to separate them from the profile-use build (which uses the default .o suffix). These builds are now always in separate objdirs, and don't need special suffix rules anymore. As a bonus, this helps fix an issue with buildid.cpp continually rebuilding because libxul_so.list always lists the inputs as *.o, which don't exist if we're using a .i_o suffix. Make would always re-create buildid.cpp and therefore libxul.so in the instrumented build even when nothing has changed. Differential Revision: https://phabricator.services.mozilla.com/D56115

Bug 1557788 - Remove pgo.relink touches; r=firefox-build-system-reviewers,chmanchester This was used for Windows MSVC PGO builds that re-used the objdir. We don't use MSVC for PGO anymore, and we don't re-use the objdir. Differential Revision: https://phabricator.services.mozilla.com/D56114

Bug 1557788 - Remove pgomerge.py; r=firefox-build-system-reviewers,chmanchester pgomerge.py was needed for Windows MSVC PGO builds. Now that we use clang for these builds, it's no longer used. Differential Revision: https://phabricator.services.mozilla.com/D56113

Bug 1557788 - Remove MOZ_1TIER_PGO; r=firefox-build-system-reviewers,chmanchester MOZ_1TIER_PGO was a temporary hack to support 1-tier PGO builds while they were being ported to 3-tier. Now that all builds are 3-tier, it can be removed. Differential Revision: https://phabricator.services.mozilla.com/D56112

Bug 1557788 - Remove NEED_XVFB from Linux instrumented and rusttest builds; r=firefox-build-system-reviewers,chmanchester xvfb was used to create a virtual framebuffer for running Firefox during build jobs to support PGO profile generation. That now runs in a separate task, so we don't need this flag for builds anymore. Note that other Linux builds still need xvfb in order to run xpcshell in 'make check'. Differential Revision: https://phabricator.services.mozilla.com/D56111

Bug 1601872 - Handle the case when the mozconfig has no configure_args r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D56207

Bug 1601704 - Work around sccache/rust confusion about mozglue-static. r=chmanchester See https://github.com/rust-lang/rust/issues/58393#issuecomment-562410696 Differential Revision: https://phabricator.services.mozilla.com/D56129

Bug 1563403 - Use 3-tier PGO for local MOZ_PGO=1 builds; r=firefox-build-system-reviewers,dmajor,chmanchester Local PGO builds now use 3-tier machinery under the hood. Instead of a single object directory that gets cleaned in between the instrumented and profile-use builds, now the instrumented build uses a separate '${objdir}/instrumented' directory. This makes it easier to handle within mach since we can drive the two builds with environment variables and keep all build artifacts separate, without needing to do manual cleanup in between. Differential Revision: https://phabricator.services.mozilla.com/D50098

Bug 1563403 - disable LTO on --enable-profile-generate; r=firefox-build-system-reviewers,chmanchester In automation this is normally handled by mozconfigs, but we should disable it automatically during the profile-generate stage for users if they have LTO enabled in their mozconfig while doing a 3-tier PGO build. Differential Revision: https://phabricator.services.mozilla.com/D50097

Bug 1563925 - Update crates that have new versions that use rand 0.7. r=firefox-build-system-reviewers,chmanchester As well as a in-tree crate. Differential Revision: https://phabricator.services.mozilla.com/D55916

Bug 1601150 - Update cargo workspace hack. r=chmanchester Bug 1601233 made cranelift bump its syn dependency to 1.0, breaking the workspace hack. Some of the features were also stale from presumably other updates. Differential Revision: https://phabricator.services.mozilla.com/D55897

Bug 1600664 - Download the wasi sysroot during bootstrap for Linux r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D55481

Bug 1593465 - Upload minidumps from PGO-run task crashes; r=firefox-build-system-reviewers,chmanchester The minidumps are stored in the temporary profile directory used by profileserver.py. Before the temporary directory goes away, we use mozcrash & minidump_stackwalk to process any crashreports and copy them to the artifacts directory. Differential Revision: https://phabricator.services.mozilla.com/D52394

Bug 1593465 - Dump symbols during instrumented builds for crashlogs in the run task; r=firefox-build-system-reviewers,chmanchester To get an accurate backtrace if the run task crashes, we need the symbols from the instrumented build. Depends on D53716 Differential Revision: https://phabricator.services.mozilla.com/D53717

Bug 1593465 - Add minidump_stackwalk dependencies to generate-profile tasks; r=firefox-build-system-reviewers,chmanchester minidump_stackwalk is used to process any crash reports generated by the PGO run task. Depends on D52393 Differential Revision: https://phabricator.services.mozilla.com/D53716

Bug 1448349 - Warn about autoconf $PATH problems on win32 r=firefox-build-system-reviewers,chmanchester Detect if Unix utilities on win32 are being picked up from a foreign installation of MinGW, such as the tools packaged with Git for Windows. If autoconf dies during `./mach configure` and foreign tools are found in $PATH then warn the user that their $PATH may need to change to fix the problem. Differential Revision: https://phabricator.services.mozilla.com/D52960

Bug 1595696 - Shift artifact namespace transformation to the job class definition. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D52631

Bug 1596431 - [python] Add an --ipython flag to |mach python|, r=firefox-build-system-reviewers,chmanchester This will install ipython into the default virtualenv if it doesn't exist. Unless --no-virtualenv is specified in which case an error will be printed. Differential Revision: https://phabricator.services.mozilla.com/D53030