Bug 1579455 - [mozbuild] Create distinct 'init' virtualenvs for Python 2 vs 3 r=nalexander,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D45419

Bug 1580899 - Remove PGO settings from mozharness; r=firefox-build-system-reviewers,chmanchester Mozharness no longer drives building with PGO; it is all handled in Taskcluster and the build system. Depends on D46070 Differential Revision: https://phabricator.services.mozilla.com/D46071

Bug 1580899 - Remove mozconfig.pgo; r=firefox-build-system-reviewers,chmanchester This file is no longer required, since taskcluster handles scheduling PGO tasks. Differential Revision: https://phabricator.services.mozilla.com/D46069

Bug 1567642 - [mozbuild] Fix flake8 under Python 3 lint issues r=firefox-build-system-reviewers,chmanchester Depends on D45412 Differential Revision: https://phabricator.services.mozilla.com/D45413

Bug 1567642 - [mach] Fix flake8 under Python 3 lint issues r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D45412

Bug 1580028 - Always merge PGO profile data in the run task; r=firefox-build-system-reviewers,chmanchester If the run task generates bad profile data, the merge step in the profile-use task will fail. However, retrying the profile-use task doesn't fix the problem, and there isn't a straightforward way to retry the run task in this situation. Instead we can add a clang toolchain to all the run tasks, and perform the merge there. This means the output from the run task will always be a successfully merged file called 'merged.profdata', and we no longer need to perform the merge as part of the profile-use build as a GENERATED_FILES step. Depends on D45262 Differential Revision: https://phabricator.services.mozilla.com/D45263

Bug 1580028 - Move pgo_flags to later in lto-pgo.configure; r=firefox-build-system-reviewers,chmanchester This is needed in the following patch, but the diff is more clear with this split out. Differential Revision: https://phabricator.services.mozilla.com/D45262

Bug 1575680 - Move node invocations to misc tier; r=firefox-build-system-reviewers,chmanchester These were originally in exports because having it in the same tier as the install rule interacted poorly with VPATH. Now that we no longer have a generic VPATH rule, make can handle the rules properly with everything in misc. Differential Revision: https://phabricator.services.mozilla.com/D42969

Bug 1579156 - Fix binding tests after bug 1573566. r=chmanchester The build system has no clue that there is something to build in dom/bindings/test. It's currently all handled via make rules generated by the backend, but ideally, this would all be handled by the frontend emitting appropriate GeneratedFiles and Sources objects. In the meanwhile, we just force the make backend to recurse through dom/bindings/test. Differential Revision: https://phabricator.services.mozilla.com/D45124

Bug 1572236: Support sccache in Google Storage r=chmanchester,dustin With the migration from AWS to GCP, we also need to migrate sccache buckets from S3 to Google Storage. The problem is how we deal with regions, since there isn't an exact correspondence on the region names between the two cloud providers. To make the transition smoother, docker-worker (and soon generic-worker) provides a new environment variable called TASKCLUSTER_WORKER_LOCATION, with information about the cloud provider the task is running on. Using this new variable, we configure sccache to use the corresponding storage service of the cloud provider where the task runs. The bucket names in Google Storage are shorter because GCS imposes a limit of 30 characteres for the names. Ref: https://github.com/taskcluster/taskcluster-rfcs/pull/148/files Differential Revision: https://phabricator.services.mozilla.com/D44845

Bug 1577411 - Don't build breakpad host tools on Windows host when building GeckoView. r=firefox-build-system-reviewers,chmanchester Bug 1558372 allows to build crash reporter for Android even if host isn't Linux. But when using Windows host, it is still broken. Host libraries and tools in `toolkit/crashreporter/google-breakpad/src/common` is only for Linux host and macOS host. So if using Windows host, it isn't buildable. So I would like to exclude this if host is windows. Also, `toolkit/crashreporter/google-breakpad/src/tools/linux/dump_syms` is only for Linux host. Differential Revision: https://phabricator.services.mozilla.com/D43923

Bug 1579131 - Enable Leanplum/MMA in Android aarch64 builds. r=chmanchester This is not the minimal patch for this issue. The minimal patch is a one-liner that adds `MOZ_ANDORID_MMA=1` to mobile/android/config/mozconfigs/android-aarch64/nightly. However, the underlying issue is that the mozconfigs have not stayed internally consistent. To reduce the likelihood of a repeat of this inconsistency, it would be best to extract the shared flags and reduce duplication and the risk for inconsistent updates. There's two ways to do that: teach `moz.configure` to have the correct defaults or share some small stanza across mozconfigs. The former is non-trivial and not worth the risk at this time. The latter is trivial but more effort than this is worth. Here's something in the middle: making all the mozconfigs have minimal differences with the mainline mozconfig for `android-api-16`. This at least allows easy auditing of the mozconfigs with `diff`. Differential Revision: https://phabricator.services.mozilla.com/D44865

Bug 1534533 - Add geckodriver toolchain tasks. r=chmanchester There was quite a bit of discussion of this in `#build` on IRC, and the consensus was that geckodriver should be built as a stand-alone Rust crate and not as part of Firefox/Gecko (say, as a new --enable-project target). This follows that approach, and the expression, modeled off of cbindgen but updated to cross compile from a Linux host to all targets, is pretty straight-forward. A sparse profile would be nice, but the way that the Gecko Cargo workspace works means that the profile must accumulate Rust code from many locations. If we want to, eventually testing/geckodriver can be removed from the top-level Rust workspace, the geckodriver-signing tasks migrated to these toolchain tasks, consumers migrated to the signing tasks, and geckodriver removed from the "common" test archive. Differential Revision: https://phabricator.services.mozilla.com/D43646

Bug 1534533 - Pre: Remove bzip2 dependency from geckodriver. r=chmanchester,whimboo The `zip` dependency doesn't appear to require `bzip2`, and `bzip2` is tricky to cross-compile, so the feature set is restricted to ease the build task. Differential Revision: https://phabricator.services.mozilla.com/D43645

Bug 1577046 - Tie extra-bindgen-flags file creation to cranelift. r=chmanchester We only recurse into js/src/rust when jsrust is built, which it may not be in Gecko builds. But cranelift, which may be enabled either way, needs the extra-bindgen-flags file. Differential Revision: https://phabricator.services.mozilla.com/D43699

Bug 1575804 - Don't decode the result from get_cmd_output. r=chmanchester Also don't pass universal_newlines to check_cmd_output. That was added to make python 3 happier when check_cmd_output still returned bytes, to hint subprocess to return a unicode string, but now that check_cmd_output does the decoding, that's not needed anymore. Differential Revision: https://phabricator.services.mozilla.com/D43205

Bug 1575760 - Generate .cargo/config as part of the build rather than as a configure subst file. r=firefox-build-system-reviewers,chmanchester We'll require preprocessing that configure subst files don't allow in the next change, so prepare for that. Differential Revision: https://phabricator.services.mozilla.com/D43011

Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D41454

Bug 1574056 - Make `mach vendor rust` use now native `cargo vendor`. r=chmanchester Note that cargo vendor always uses the native version even when cargo-vendor was already installed, so there is no concern wrt that. Differential Revision: https://phabricator.services.mozilla.com/D43007

Bug 1560700 - Remove now unused allow_unknown argument to split_triplet. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D41482

Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D41454

Bug 1550146 - Part 1: Do less (no mozharness.zip) in local `mach package`. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D41449

Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D41454

bug 1540655: mozbuild: add puppeteer test flavor; r=firefox-build-system-reviewers,chmanchester The Puppeteer test flavour are functional tests for the CDP-based Puppeteer library from Google, that we want to run against our implementation of CDP for Firefox. They are distinct from the Firefox Puppeteer tests based on Marionette. Differential Revision: https://phabricator.services.mozilla.com/D37012

bug 1540655: build, remote: add mach command for vendoring Puppeteer; r=firefox-build-system-reviewers,chmanchester Introduces "./mach remote vendor-puppeteer" for vendoring the Puppeteer client without dependencies into remote/test/puppeteer/. The particular checkout of Puppeteer is https://github.com/andreastt/puppeteer/tree/firefox, which contains a couple of hotfixes we need for the client to work with the Firefox implementation of CDP. The remote agent targets a specific version of Puppeteer, so it is not suitable for this to be vendored under third_party/. We also wouldn't want other code in central to accidentally use a patched fork. The vendoring process is not part of "./mach vendor" because it does not yet have Node.js support, and implementing that for mach is outside the scope of getting the Puppeteer tests running with the remote agent. Differential Revision: https://phabricator.services.mozilla.com/D37007

Bug 1570411 - Add --without-fennec to make `mach package` not produce Fennec APK. r=chmanchester When working on GeckoView, there's no need to produce a Fennec APK. This commit avoids doing that work at `mach package` time. There are many other things we'd like to stop doing as we remove Fennec from the tree, so we add a general flag to guard such things. Depends on D41447 Differential Revision: https://phabricator.services.mozilla.com/D41448

bug 1540655: mozbuild: add puppeteer test flavor; r=firefox-build-system-reviewers,chmanchester The Puppeteer test flavour are functional tests for the CDP-based Puppeteer library from Google, that we want to run against our implementation of CDP for Firefox. They are distinct from the Firefox Puppeteer tests based on Marionette. Differential Revision: https://phabricator.services.mozilla.com/D37012

bug 1540655: build, remote: add mach command for vendoring Puppeteer; r=firefox-build-system-reviewers,chmanchester Introduces "./mach remote vendor-puppeteer" for vendoring the Puppeteer client without dependencies into remote/test/puppeteer/. The particular checkout of Puppeteer is https://github.com/andreastt/puppeteer/tree/firefox, which contains a couple of hotfixes we need for the client to work with the Firefox implementation of CDP. The remote agent targets a specific version of Puppeteer, so it is not suitable for this to be vendored under third_party/. We also wouldn't want other code in central to accidentally use a patched fork. The vendoring process is not part of "./mach vendor" because it does not yet have Node.js support, and implementing that for mach is outside the scope of getting the Puppeteer tests running with the remote agent. Differential Revision: https://phabricator.services.mozilla.com/D37007

Bug 1569887 - Use a fetch task artifact to get llvm-dsymutil source. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D39848

Bug 1569075 - Rename FileAvoidWrite mode argument. r=chmanchester It causes confusion because it's the mode used to _read_ the overwritten file. Make that more obvious by renaming to `readmode`. Differential Revision: https://phabricator.services.mozilla.com/D39445

Bug 1561146 - Enable PGO for Windows aarch64 builds; r=firefox-build-system-reviewers,chmanchester This build re-uses the PGO profile from the win64 build in the win64-aarch64-shippable-no-eme part of the aarch64 build. Even though the profile isn't generated on the smae platform, we still get enough of a performance win to make this worthwhile. Note that the pgo_flags() in configure need to be tweaked slightly since we don't supprt the -fprofile-generate flag for aarch64 (we don't build the clang_rt.profile lib there). So we always want to return the flags namespace to make sure we get the use_* versions of flags, which we do need. Differential Revision: https://phabricator.services.mozilla.com/D38928

Bug 1561774 - Copy win64-aarch64-opt test artifacts into the shippable build; r=firefox-build-system-reviewers,chmanchester In order to run tests against the win64-aarch64-shippable build, we need the test packages from the aarch64-no-eme build since the tests aren't compiled during the -shippable stage. Since packages like cppunittest and the target.test_packages.json file won't be generated correctly in the -shippable stage, we disable the package-step tests there by setting MOZ_AUTOMATION_PACKAGE_TESTS to 0 in the environment. Differential Revision: https://phabricator.services.mozilla.com/D37895

Bug 1566960 - Remove unused Windows stuff from rules.mk r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D38396

Bug 1566687 - Use mac-cross clang for all macosx builds. r=chmanchester We've been lucky that non-sanitizer cross-builds for macosx have not required the clang runtime so far, but they soon will. And it's only available in the mac-cross clang toolchain, so we need to use that on all macosx builds. Differential Revision: https://phabricator.services.mozilla.com/D38260

Bug 1564909 - Test binaries do not need to manually include libatomic. r=chmanchester These configurations are already taken care of in js/src/build/moz.build and do not need to duplicated. Depends on D37567 Differential Revision: https://phabricator.services.mozilla.com/D37568

Bug 1564909 - Add js-standalone.mozbuild. r=bbouvier,chmanchester Consolidate build settings required by standalone builds into a single mozbuild script. A full Gecko build does not use these settings and thus we cannot simply put them in js/src/build/moz.build Differential Revision: https://phabricator.services.mozilla.com/D37567

Bug 1473498 - [mach] Fix Py3 compatibility issues in telemetry.py r=firefox-build-system-reviewers,chmanchester Depends on D36104 Differential Revision: https://phabricator.services.mozilla.com/D38193

Bug 1473498 - [mach] Fix Python 3 decorator compatibility r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D36104

Bug 1566309 - Upgrade mac builds to rust 1.36. r=chmanchester Differential Revision: https://phabricator.services.mozilla.com/D38156

Bug 1561774 - Copy win64-aarch64-opt test artifacts into the shippable build; r=firefox-build-system-reviewers,chmanchester In order to run tests against the win64-aarch64-shippable build, we need the test packages from the aarch64-no-eme build since the tests aren't compiled during the -shippable stage. Since packages like cppunittest and the target.test_packages.json file won't be generated correctly in the -shippable stage, we disable the package-step tests there by setting MOZ_AUTOMATION_PACKAGE_TESTS to 0 in the environment. Differential Revision: https://phabricator.services.mozilla.com/D37895

Bug 1561474 - Use sparse-profile on Windows generate tasks; r=firefox-build-system-reviewers,chmanchester Depends on D37807 Differential Revision: https://phabricator.services.mozilla.com/D37808

Bug 1563797 - Use 'backports.shutil_which' instead of 'which' in the build system r=firefox-build-system-reviewers,chmanchester Credit: Callek for figuring out an issue in 'make check' making the binary absolute in mozbuild.base. Differential Revision: https://phabricator.services.mozilla.com/D37319

Bug 1563053 - Wildcard profraw files for Android PGO; r=firefox-build-system-reviewers,chmanchester This adds support for the Android process writing out multiple profraw files and pulling them all from the device. We currently only generate a single profraw file, but if that changes in the future we should be able to get a PGO build using the full set of files now. Depends on D36840 Differential Revision: https://phabricator.services.mozilla.com/D36841

Bug 1563053 - Put all Android PGO outputs in a separate directory; r=firefox-build-system-reviewers,chmanchester If we end up generating multiple profraw files in the future, it will help to have them all in a separate directory so that we can just 'adb pull' the whole directory at once. Depends on D36839 Differential Revision: https://phabricator.services.mozilla.com/D36840

Bug 1563158 - Properly handle imply_option dependency loops when `when` is involved. r=chmanchester This is an extension of the relaxation added in bug 1529799. Differential Revision: https://phabricator.services.mozilla.com/D36716

Bug 1473498 - [mach] Enable test_logger with Python 3 r=firefox-build-system-reviewers,chmanchester Depends on D36101 Differential Revision: https://phabricator.services.mozilla.com/D36102

Bug 1473498 - [mach] Enable test_config with Python 3 r=firefox-build-system-reviewers,chmanchester Depends on D36100 Differential Revision: https://phabricator.services.mozilla.com/D36101

Bug 1473498 - [mach] Enable test_entry_points with Python 3 r=firefox-build-system-reviewers,chmanchester Differential Revision: https://phabricator.services.mozilla.com/D36100

Bug 1473498 - [mach] Enable test_dispatcher with python 3 r=firefox-build-system-reviewers,chmanchester This test is working with minimal effort. Let's get it running to prevent future regressions. Differential Revision: https://phabricator.services.mozilla.com/D36098

Bug 1560755 - Merge Android PGO profile data in run task; r=firefox-build-system-reviewers,chmanchester Android profile runs don't always fully write out the profile data. In this case, the corrupted profile data is successfully uploaded, but future profile-use PGO builds try to merge the data and fail. Retrying the profile-use builds doesn't help, since they all pull from the same job that published the corrupt data. We can detect this in the run task by using llvm-profdata merge, and if the merge fails the task can automatically be retried. Note that the data gets redundantly merged in the profile-use build, but it may not be possible to run the merge in the run task on all platforms (eg: OSX), so we have to keep the merge there as well. Differential Revision: https://phabricator.services.mozilla.com/D36294