searching for reviewer(ahal)
9a18000e5ead7a2979e48c7bd7238b6f68b427dd: Bug 1661513: Fix reporting of expected failures in fail-if tests. r=ahal
Kris Maglione <maglione.k@gmail.com> - Fri, 18 Sep 2020 21:00:23 +0000 - rev 549346
Push 37795 by ccoroiu@mozilla.com at Sat, 19 Sep 2020 09:34:53 +0000
Bug 1661513: Fix reporting of expected failures in fail-if tests. r=ahal Without this change, failures in tests marked with `fail-if` are reported as passes, though with messages and stacks that describe the failure. That works as far as the automation machinery is concerned, but it is extremely confusing to people who are trying to reproduce and fix the failures locally. This change fixes the reporting so that they show up as expected failures in logs, but have the same effect on final test status. Differential Revision: https://phabricator.services.mozilla.com/D90616
3b0e057ff3c395a2c0e8f48083e359fb77f49c72: Bug 1661513: Fix reporting of expected failures in fail-if tests. r=ahal
Kris Maglione <maglione.k@gmail.com> - Fri, 18 Sep 2020 01:38:59 +0000 - rev 549335
Push 37795 by ccoroiu@mozilla.com at Sat, 19 Sep 2020 09:34:53 +0000
Bug 1661513: Fix reporting of expected failures in fail-if tests. r=ahal Without this change, failures in tests marked with `fail-if` are reported as passes, though with messages and stacks that describe the failure. That works as far as the automation machinery is concerned, but it is extremely confusing to people who are trying to reproduce and fix the failures locally. This change fixes the reporting so that they show up as expected failures in logs, but have the same effect on final test status. Differential Revision: https://phabricator.services.mozilla.com/D90616
5b1c53389338b49abb62d891989f8a4223db99b1: Bug 1665494 - In `mach python-test`, don't install requirements.txt in parallel r=ahal
Ricky Stewart <rstewart@mozilla.com> - Fri, 18 Sep 2020 01:38:46 +0000 - rev 549314
Push 37794 by btara@mozilla.com at Fri, 18 Sep 2020 21:14:11 +0000
Bug 1665494 - In `mach python-test`, don't install requirements.txt in parallel r=ahal If you set a `DEFAULT` `requirements` file in a Python test manifest, the current implementation of `python-test` will try (and usually fail) to install the file once for every test file in parallel. Instead, initializing the environment should be done once when the command starts up. Differential Revision: https://phabricator.services.mozilla.com/D90475
09b2feb3809fbb47b77cc005c2b61f1a137015d5: Bug 1663159 [Linux] Add Wayland config to test-linux.sh, r=ahal
stransky <stransky@redhat.com> - Wed, 16 Sep 2020 09:48:15 +0000 - rev 548875
Push 37788 by ccoroiu@mozilla.com at Wed, 16 Sep 2020 15:37:38 +0000
Bug 1663159 [Linux] Add Wayland config to test-linux.sh, r=ahal Differential Revision: https://phabricator.services.mozilla.com/D89287
9565a2e21ca0d0ca5f47326fef7b3ab758ed0816: Bug 1661493 - Support extending OptimizationSchema in taskgraph projects. r=taskgraph-reviewers,ahal
Rob Lemley <rob@thunderbird.net> - Tue, 15 Sep 2020 23:08:52 +0000 - rev 548833
Push 37787 by malexandru@mozilla.com at Wed, 16 Sep 2020 09:56:56 +0000
Bug 1661493 - Support extending OptimizationSchema in taskgraph projects. r=taskgraph-reviewers,ahal In order for Thunderbird to effectively cut down on excess builds, there needs to be a way to define taskgraph optimization strategies beyond what is made available in the mozilla repository. taskgraph.optimize.register_strategy gets half of the job done. The other piece is task schema validation which checks optimizations against OptimizationSchema. OptimizationSchema gets moved to taskgraph.optimize.schema so that it is not set too early by taskgraph.util.schema. Projects then call set_optimization_schema() to replace the default schema object. set_optimization_schema() needs to be called in the project's taskgraph:register function before any transform code is loaded. Differential Revision: https://phabricator.services.mozilla.com/D88429
ba57887126d22db912cdf5c4acb182f40059bc9a: Bug 1663786 - Remove "copy" support from `virtualenv` handling r=ahal
Ricky Stewart <rstewart@mozilla.com> - Thu, 10 Sep 2020 12:54:49 +0000 - rev 548757
Push 37787 by malexandru@mozilla.com at Wed, 16 Sep 2020 09:56:56 +0000
Bug 1663786 - Remove "copy" support from `virtualenv` handling r=ahal This is unused. Differential Revision: https://phabricator.services.mozilla.com/D89509
4627a7a4713dbef81fb169254d08fefc52cbb235: Bug 1663755 - Remove `inherit-from-parent-environment` implementation r=ahal
Ricky Stewart <rstewart@mozilla.com> - Fri, 11 Sep 2020 22:14:41 +0000 - rev 548756
Push 37787 by malexandru@mozilla.com at Wed, 16 Sep 2020 09:56:56 +0000
Bug 1663755 - Remove `inherit-from-parent-environment` implementation r=ahal This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it. Differential Revision: https://phabricator.services.mozilla.com/D89492
fd52a2aa2b2af774618afa71a65fe27a846def98: Bug 1664409 - Ensure fission disabled for mochitest-chrome and a11y tests; r=ahal
Geoff Brown <whole.grains@protonmail.com> - Fri, 11 Sep 2020 15:12:13 +0000 - rev 548331
Push 37778 by ccoroiu@mozilla.com at Sat, 12 Sep 2020 09:26:23 +0000
Bug 1664409 - Ensure fission disabled for mochitest-chrome and a11y tests; r=ahal When mochitest-chrome (or a11y) tests are run, e10s is disabled for the suite; that conflicts with the fission option, if enabled on the command line. This patch disables fission along with e10s for non-e10s test suites. Differential Revision: https://phabricator.services.mozilla.com/D89931
4123a3bc61731cf965e0973263467534238a9393: Bug 1664300 - double the frequency of windows perf tests temporarily. r=ahal
Joel Maher <jmaher@mozilla.com> - Fri, 11 Sep 2020 12:24:14 +0000 - rev 548320
Push 37778 by ccoroiu@mozilla.com at Sat, 12 Sep 2020 09:26:23 +0000
Bug 1664300 - double the frequency of windows perf tests temporarily. r=ahal temporarily double the frequency of windows perf tests Differential Revision: https://phabricator.services.mozilla.com/D89832
972df2adcffa91fbcc885a64c10f3ae25917b1fc: Bug 1663786 - Remove "copy" support from `virtualenv` handling r=ahal
Ricky Stewart <rstewart@mozilla.com> - Wed, 09 Sep 2020 18:15:46 +0000 - rev 548026
Push 37770 by apavel@mozilla.com at Thu, 10 Sep 2020 04:03:55 +0000
Bug 1663786 - Remove "copy" support from `virtualenv` handling r=ahal This is unused. Differential Revision: https://phabricator.services.mozilla.com/D89509
f97f4ae8e721b9070e1c4e19068084d0324b4518: Bug 1663755 - Remove `inherit-from-parent-environment` implementation r=ahal
Ricky Stewart <rstewart@mozilla.com> - Wed, 09 Sep 2020 18:15:28 +0000 - rev 548025
Push 37770 by apavel@mozilla.com at Thu, 10 Sep 2020 04:03:55 +0000
Bug 1663755 - Remove `inherit-from-parent-environment` implementation r=ahal This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it. Differential Revision: https://phabricator.services.mozilla.com/D89492
5349493670b901a0d8295f45c3bd605764d6b0b6: Bug 1663159 [Linux] Add Wayland config to test-linux.sh, r=ahal
stransky <stransky@redhat.com> - Wed, 09 Sep 2020 13:34:02 +0000 - rev 547975
Push 37769 by apavel@mozilla.com at Wed, 09 Sep 2020 21:39:59 +0000
Bug 1663159 [Linux] Add Wayland config to test-linux.sh, r=ahal Differential Revision: https://phabricator.services.mozilla.com/D89287
637126b84cee525c4bb40589bfbbe789102bb62c: Bug 1660559 - In `mach bootstrap`, initialize the `mach` `virtualenv`s early and run `mach artifact` with them r=mhentges,ahal
Ricky Stewart <rstewart@mozilla.com> - Wed, 02 Sep 2020 17:02:12 +0000 - rev 547544
Push 37750 by nerli@mozilla.com at Wed, 02 Sep 2020 21:57:21 +0000
Bug 1660559 - In `mach bootstrap`, initialize the `mach` `virtualenv`s early and run `mach artifact` with them r=mhentges,ahal `mach artifact` has a dependency on `zstandard`, which is installed in the `mach` `virtualenv`s, so we have to run `mach artifact` with the correct `virtualenv`. Also create the `virtualenv`s earlier in the process to account for this. This all has a dependency on the existence of a checkout (which has the `mach` script with all its dependencies on everything else), but after bug 1647792 that's not a concern. Differential Revision: https://phabricator.services.mozilla.com/D87920
fffa3fcb18810e5e35837c5136f00b1dd0c4d969: Bug 1659539 - [python-test] Remove '--python' and stop using pipenv to manage virtualenvs r=ahal
Ricky Stewart <rstewart@mozilla.com> - Wed, 02 Sep 2020 17:05:24 +0000 - rev 547542
Push 37750 by nerli@mozilla.com at Wed, 02 Sep 2020 21:57:21 +0000
Bug 1659539 - [python-test] Remove '--python' and stop using pipenv to manage virtualenvs r=ahal Pipenv is heavy weight and overkill for the purposes it is being used. We'd like to remove it from the tree and |mach python-test| was one of the last remanining use cases. Remove the `--python` command-line argument as a result. Users who wish to run unit tests with Python 2 can do `MACH_PY2=1 ./mach python-test ...` or `python2 ./mach python-test ...`. Also update a few unit tests that would break otherwise in the presence of this change. There were a couple lines in the `setup.py` for `mozlog` that were problematic for tests and was resulting in errors due to the `mozlog` plugin being loaded by `pytest` more than once. We just delete those lines and bump up the major version number of the package to fix it. Differential Revision: https://phabricator.services.mozilla.com/D88296
3ac7571a94983003726c19f3f6d68bd61a4232aa: Bug 1662130 - Walk back new `inherit-from-parent-environment` logic in `virtualenv` handling r=ahal
Ricky Stewart <rstewart@mozilla.com> - Tue, 01 Sep 2020 16:54:27 +0000 - rev 547296
Push 37748 by apavel@mozilla.com at Wed, 02 Sep 2020 03:31:14 +0000
Bug 1662130 - Walk back new `inherit-from-parent-environment` logic in `virtualenv` handling r=ahal This logic is meant to expose packages from a globally-installed Python to be used by the in-`objdir` `virtualenv`s, so for example we don't have to figure out how to install `zstandard` (or other Python packages with native code that may or may not have prebuilt wheels for any given platform) in those `virtualenv`s. Bug 1660351 augmented that logic to work within the requirements of bug 1660353. This worked mostly, but is causing builds to unconditionally break on Arch Linux, caused a couple test failures, and in general is just introducing other weird behaviors downstream, and issues with the resultant `PYTHONPATH`s are hard to diagnose and fix. In the long-term we'll have to permanently solve the `zstandard` problem and pave the way for other Python packages with native code as well, but that's not an urgent need. The ultimate goal is to completely remove `inherit-from-parent-environment`, but we can't do that until bug 1659539 is solved. Partially reverts bugs 1660351. Entirely reverts bug 1660353, restoring that file to as it was before that patch. Differential Revision: https://phabricator.services.mozilla.com/D89001
f50e4bee2efb3afbd6016fa7286078500599775b: Bug 1660351 - Tweak `inherit-from-parent-environment` implementation in `virtualenv.py` r=ahal
Ricky Stewart <rstewart@mozilla.com> - Thu, 27 Aug 2020 21:48:06 +0000 - rev 546899
Push 37737 by csabou@mozilla.com at Sat, 29 Aug 2020 09:12:26 +0000
Bug 1660351 - Tweak `inherit-from-parent-environment` implementation in `virtualenv.py` r=ahal The intended behavior of `inherit-from-parent-environment` is that the packages from the parent Python environment are available to the sub-`virtualenv`. The implementation of that behavior thus far has been around "site directories", the idea being that custom (non-stdlib) packages are likely to be installed in the "site directory". The limitation of this approach is that there's no one location, in practice, where packages are installed, and it's hard to enumerate a static list of all those possible locations across all platforms. This patch circumvents the issue by ignoring the "site directory" question entirely and just looking at `sys.path`. If we're inheriting from the parent environment when creating a `virtualenv`, we just ask the parent Python what its `sys.path` is and configure the `virtualenv`'s `sys.path` on startup. Differential Revision: https://phabricator.services.mozilla.com/D87808
9d3523d1354f50535a6c84afcbfa5865bb69ab0c: Bug 1660548 - Don't globally install Python packages for linting r=ahal
Ricky Stewart <rstewart@mozilla.com> - Fri, 21 Aug 2020 21:03:28 +0000 - rev 546898
Push 37737 by csabou@mozilla.com at Sat, 29 Aug 2020 09:12:26 +0000
Bug 1660548 - Don't globally install Python packages for linting r=ahal `flake8`, `black`, and `pylint` are all installed in `virtualenv`s when running `mach lint`, so installing these packages globally on image setup isn't necessary. Differential Revision: https://phabricator.services.mozilla.com/D87904
a47ed8d910807ce7ee066707d510e9b8cd349cb6: Bug 1661783 - Disable `test_rst.py` on Linux r=ahal
Ricky Stewart <rstewart@mozilla.com> - Fri, 28 Aug 2020 18:57:20 +0000 - rev 546879
Push 37737 by csabou@mozilla.com at Sat, 29 Aug 2020 09:12:26 +0000
Bug 1661783 - Disable `test_rst.py` on Linux r=ahal This test begins to fail due to a `pip` installation issue after bug 1660351. It's already disabled on macOS (also due to what the comment says is a `pip` installation issue), so it's very possible the custom `requirements.txt` that this test installs is deficient in some way that surfaces in certain `virtualenv`s under certain circumstances. I can't diagnose the failure, but what I have seen is that bug 1659539 will fix that problem entirely with no extra intervention required, so we should be able to re-enable this for Linux (and maybe macOS as well?) when that patch lands. Differential Revision: https://phabricator.services.mozilla.com/D88653
11ff00d15299aaee9ca7bb51436ce09d60d308b4: Bug 1661659 - Reimplement optionally_keyed_by as a validator function rather than in terms of nested Any. r=taskgraph-reviewers,ahal
Mike Hommey <mh+mozilla@glandium.org> - Fri, 28 Aug 2020 12:47:19 +0000 - rev 546775
Push 37737 by csabou@mozilla.com at Sat, 29 Aug 2020 09:12:26 +0000
Bug 1661659 - Reimplement optionally_keyed_by as a validator function rather than in terms of nested Any. r=taskgraph-reviewers,ahal Using nested Any generates deeply nested non-trivial schemas because we also end up nesting optionally_keyed_by's, and the addition of one field in one optionally_keyed_by in bug 1657769 made the whole schema compilation step an order of magnitude slower. Even before bug 1657769, the schema compilation step was slow for tests, and with this change, it's virtually instantaneous. Differential Revision: https://phabricator.services.mozilla.com/D88543
fb502b2ec61194e274cb3c910d0af19bafdad3a2: Bug 1657121 - exclude uuid from s3_delete_missing. r=ahal
Emil Farisan <emil.farisan@gmail.com> - Thu, 27 Aug 2020 20:02:59 +0000 - rev 546650
Push 37736 by apavel@mozilla.com at Fri, 28 Aug 2020 15:31:26 +0000
Bug 1657121 - exclude uuid from s3_delete_missing. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D88282
b58242b90c5148a11eb691b92205d1b429489654: Bug 1660124 - Don't trample existing `mach` `virtualenv`s if they're already up-to-date r=mhentges,ahal
Ricky Stewart <rstewart@mozilla.com> - Tue, 25 Aug 2020 21:21:19 +0000 - rev 546191
Push 37731 by apavel@mozilla.com at Wed, 26 Aug 2020 03:24:37 +0000
Bug 1660124 - Don't trample existing `mach` `virtualenv`s if they're already up-to-date r=mhentges,ahal On some platforms re-creating the `virtualenv`s can be very time-consuming so we'd like to avoid deleting these `virtualenv`s unnecessarily. Preserve the existing behavior behind a `-f` flag in case unconditionally wiping the `virtualenv`s is what's needed for any reason. Differential Revision: https://phabricator.services.mozilla.com/D87668
4b9deff4865e84354715b2d801c556a19320b5e2: Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal
Tom Prince <mozilla@hocat.ca> - Tue, 25 Aug 2020 01:35:29 +0000 - rev 545897
Push 37729 by ccoroiu@mozilla.com at Tue, 25 Aug 2020 09:46:22 +0000
Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86671
def563fc3dea010e67b5121e550a0adf7a436a3f: Bug 1654339 - Add Glean metrics autodocs to moztreedocs r=janerik,ahal
Chris H-C <chutten@mozilla.com> - Fri, 21 Aug 2020 16:30:45 +0000 - rev 545823
Push 37727 by csabou@mozilla.com at Mon, 24 Aug 2020 21:50:21 +0000
Bug 1654339 - Add Glean metrics autodocs to moztreedocs r=janerik,ahal Also define a scheme for storing the index of Glean definitions files in a file separate from the build system for consumption by * mach build * mach doc * (future) mozilla/probe-scraper Differential Revision: https://phabricator.services.mozilla.com/D87600
9fb870ca7b1e5f9bea92a67ea9ced7f0772d2eee: Bug 1658812 - sphinx/autodoc should also ignore tests directory r#?firefox-source-docs-reviewers r=firefox-source-docs-reviewers,ahal
Sylvestre Ledru <sledru@mozilla.com> - Sat, 22 Aug 2020 14:23:47 +0000 - rev 545716
Push 37721 by apavel@mozilla.com at Sat, 22 Aug 2020 21:38:08 +0000
Bug 1658812 - sphinx/autodoc should also ignore tests directory r#?firefox-source-docs-reviewers r=firefox-source-docs-reviewers,ahal Differential Revision: https://phabricator.services.mozilla.com/D86854
5f006849e5c5746dc9da8e9dfed0e1cebd2ce41d: Bug 1660288 - fix python3 errors in the linter's hooks.py tripped when using python3-based mercurial, r=ahal
Gijs Kruitbosch <gijskruitbosch@gmail.com> - Thu, 20 Aug 2020 20:35:26 +0000 - rev 545570
Push 37716 by apavel@mozilla.com at Fri, 21 Aug 2020 03:37:46 +0000
Bug 1660288 - fix python3 errors in the linter's hooks.py tripped when using python3-based mercurial, r=ahal Differential Revision: https://phabricator.services.mozilla.com/D87800
1e676ded3d07f60818fce30d7ac27ae047e35df4: Bug 1660128 - Set `PYTHONPATH` appropriately in `testing/mochitest/runtests.py` r=ahal
Ricky Stewart <rstewart@mozilla.com> - Thu, 20 Aug 2020 17:12:04 +0000 - rev 545515
Push 37715 by apavel@mozilla.com at Thu, 20 Aug 2020 21:21:07 +0000
Bug 1660128 - Set `PYTHONPATH` appropriately in `testing/mochitest/runtests.py` r=ahal This stamps out an unspoken dependency on `six` in the `mach` `virtualenv`; adding this fixes a bunch of broken tests, namely all those that depend on `pywebsocket_wrapper.py`. Differential Revision: https://phabricator.services.mozilla.com/D87774
ac0fdd0c661ef098e5b7f9c35c0263b2c358ae77: Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal
Tom Prince <mozilla@hocat.ca> - Wed, 19 Aug 2020 01:05:39 +0000 - rev 545269
Push 37712 by btara@mozilla.com at Wed, 19 Aug 2020 21:28:29 +0000
Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86671
8e6cf79c70f71bb3a85cf5c6540697fc71bd7ba6: Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal
Tom Prince <mozilla@hocat.ca> - Tue, 18 Aug 2020 20:41:09 +0000 - rev 545221
Push 37711 by nbeleuzu@mozilla.com at Wed, 19 Aug 2020 10:01:16 +0000
Bug 1492362: [mozharness] Set reasonable `avds_dir` values; r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86671
7953ad9a7130a87b30ee704b33600f8ce4e03075: Bug 1659575 - Delete `mach python-safety` r=ahal
Ricky Stewart <rstewart@mozilla.com> - Tue, 18 Aug 2020 14:18:24 +0000 - rev 545094
Push 37710 by ncsoregi@mozilla.com at Tue, 18 Aug 2020 21:40:31 +0000
Bug 1659575 - Delete `mach python-safety` r=ahal There are zero uses of this `mach` command over the past 90 days according to our telemetry. There are no external references to `mach python-safety` in-tree, and indeed if you track the history of the originating bug 1468394, it appears that once the `mach` command was created, none of the follow-up work that was discussed (i.e. running this in CI and triaging failures to appropriate owners) was done over the following 2 years. If this ever does appear to be useful in the future, we can just resurrect this code from source control. Differential Revision: https://phabricator.services.mozilla.com/D87351
d3d9b27cdf360546b520e31c1d0513a8c98e7f62: Bug 1657806 - Add --verify-fission to run tests once with fission and once without r=ahal
Julian Descottes <jdescottes@mozilla.com> - Mon, 17 Aug 2020 17:08:54 +0000 - rev 544952
Push 37706 by csabou@mozilla.com at Mon, 17 Aug 2020 21:46:02 +0000
Bug 1657806 - Add --verify-fission to run tests once with fission and once without r=ahal Differential Revision: https://phabricator.services.mozilla.com/D87305
eff0a199fae6727caebd03b687824a398fe132ba: Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal
Ricky Stewart <rstewart@mozilla.com> - Mon, 17 Aug 2020 17:21:02 +0000 - rev 544951
Push 37706 by csabou@mozilla.com at Mon, 17 Aug 2020 21:46:02 +0000
Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916
98fddfef58ceb784e742575e9c010d04e6f82843: Bug 1636797 - In `hash.py`, enumerate files from the VCS rather than searching the filesystem directly r=ahal
Ricky Stewart <rstewart@mozilla.com> - Mon, 17 Aug 2020 15:19:34 +0000 - rev 544942
Push 37706 by csabou@mozilla.com at Mon, 17 Aug 2020 21:46:02 +0000
Bug 1636797 - In `hash.py`, enumerate files from the VCS rather than searching the filesystem directly r=ahal This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong. This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash. Differential Revision: https://phabricator.services.mozilla.com/D86780
6fb2a7336e05d62d934382ee1a220c456448bdc7: Bug 1658651 - When determining whether `virtualenv`s are up to date, don't take file size into account r=mhentges,ahal
Ricky Stewart <rstewart@mozilla.com> - Thu, 13 Aug 2020 19:29:54 +0000 - rev 544558
Push 37698 by abutkovits@mozilla.com at Fri, 14 Aug 2020 09:45:24 +0000
Bug 1658651 - When determining whether `virtualenv`s are up to date, don't take file size into account r=mhentges,ahal This check is unsound; `virtualenv` binaries are apparently not guaranteed to have the same file size as the `python` binaries used to create those `virtualenv`s, at least not with our current vendored version of the `virtualenv` library on macOS. This is trivially reproducible on my own Macbook: ``` rickystewart-a5lvdq:mozilla-unified rickystewart$ rm -rf obj-x86_64-apple-darwin19.5.0/ rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach configure ... rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("obj-x86_64-apple-darwin19.5.0/_virtualenvs/init_py3/bin/python"))' 16644 # <- ACTUAL VIRTUALENV SIZE rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("/usr/local/opt/python/bin/python3.7"))' 17704 # <- SIZE OF THE PYTHON USED TO CREATE THE VIRTUALENV ``` Concretely, this was causing unit tests to be very aggressive about deleting the parent `init_py3` `virtualenv` repeatedly in unit tests, resulting in failures. The removal of this check fixes the issue. Differential Revision: https://phabricator.services.mozilla.com/D86872
a0af59011c52208d6b93430af74d897f8912f119: Bug 1607619: only run rusttests when rust files are modified. r=ahal
Ben Hearsum <bhearsum@mozilla.com> - Wed, 12 Aug 2020 14:36:12 +0000 - rev 544410
Push 37694 by dluca@mozilla.com at Wed, 12 Aug 2020 21:49:48 +0000
Bug 1607619: only run rusttests when rust files are modified. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86434
951942b47734ce3d95f7f71ed9cb298a0b4099e8: Bug 1656614 - Delete existing `virtualenv` before re-creating it r=mhentges,ahal
Ricky Stewart <rstewart@mozilla.com> - Fri, 07 Aug 2020 15:35:27 +0000 - rev 544299
Push 37692 by cbrindusan@mozilla.com at Wed, 12 Aug 2020 03:44:18 +0000
Bug 1656614 - Delete existing `virtualenv` before re-creating it r=mhentges,ahal I wrote this patch because I noticed that the `.pth` files in my `objdir` `virtualenv`s were extremely repetitive, containing multiple references to the same directories. This happens because we [append](https://searchfox.org/mozilla-central/rev/03794edd6edcc3fc1e222de966cb27256ce08998/python/mozbuild/mozbuild/virtualenv.py#366) to the in-`virtualenv` `.pth` files when calling `populate()`, but we don't ever clean up the old ones, meaning that whenever we determine that the `virtualenv`s are out of date and need to be recreated, we actually leave A LOT of state lying around on-disk that is going to go on to impact further uses of the `virtualenv`. Concretely, how this manifested is that when I erroneously removed an entry from `virtualenv_packages.txt`, the build actually succeeded because that entry was still in the `.pth` file in the `virtualenv`; instead of "creating" a new `virtualenv` with the correct `.pth` files, it just appended the new `.pth` data to the old, stale data. I've chosen to address this by completely deleting the entire `virtualenv` when we try to re-create it. Another way you might solve this problem is by doing a `find $VIRTUALENV -name '*.pth' | xargs rm` before doing the `virtualenv` re-creation, but I'm suggesting we do it this way because we have had a long history of difficulty with `virtualenv` persistence. Bug 1628498 is an obvious example; note that we would never have encountered that bug if we always unconditionally deleted the `virtualenv` before creating a new one, as in this patch. A patch that is laser-targeted at handling the issue with `.pth` files might be fine for now but this is more foolproof and future-proof. Differential Revision: https://phabricator.services.mozilla.com/D85636
cb8dd1b13ddf323c456f468edd51665086bde17c: Bug 1658539 - Disable tab replacement in mochitests for Thunderbird. r=ahal
Mark Banner <standard8@mozilla.com> - Tue, 11 Aug 2020 15:43:56 +0000 - rev 544286
Push 37692 by cbrindusan@mozilla.com at Wed, 12 Aug 2020 03:44:18 +0000
Bug 1658539 - Disable tab replacement in mochitests for Thunderbird. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86697
9251996ce8c93142d6e87e3e3aa9585f4ab0e570: Bug 1657939 - Output an artifact that identifies replaced taskID's. r=ahal
Justin Wood <Callek@gmail.com> - Mon, 10 Aug 2020 19:20:28 +0000 - rev 544144
Push 37688 by apavel@mozilla.com at Tue, 11 Aug 2020 03:16:35 +0000
Bug 1657939 - Output an artifact that identifies replaced taskID's. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86379
2b47950ed3a1b06ed5378d50a054cee6e4105ae5: Bug 1640105 - Add socketprocess_networking variant for mochitest r=ahal,jmaher
Kershaw Chang <kershaw@mozilla.com> - Mon, 10 Aug 2020 09:21:03 +0000 - rev 544062
Push 37687 by apavel@mozilla.com at Mon, 10 Aug 2020 21:36:34 +0000
Bug 1640105 - Add socketprocess_networking variant for mochitest r=ahal,jmaher Differential Revision: https://phabricator.services.mozilla.com/D77302
ad286e404800c48b6abe082f8ccf39d505c0d68f: Bug 1657768 - Set temp_file_paths earlier. r=ahal
Mike Hommey <mh+mozilla@glandium.org> - Fri, 07 Aug 2020 14:54:55 +0000 - rev 544019
Push 37684 by btara@mozilla.com at Sun, 09 Aug 2020 21:39:40 +0000
Bug 1657768 - Set temp_file_paths earlier. r=ahal The finally where it's used may be reached before it is set currently, leading to more important error messages being hidden. Differential Revision: https://phabricator.services.mozilla.com/D86281
d4f23cd40606e84666eed4d6393460f95feeb463: Bug 1656811 - Run linux64/debug reftests with software webrender; r=ahal
Geoff Brown <gbrown@mozilla.com> - Fri, 07 Aug 2020 19:59:57 +0000 - rev 543826
Push 37682 by ccoroiu@mozilla.com at Sat, 08 Aug 2020 09:35:45 +0000
Bug 1656811 - Run linux64/debug reftests with software webrender; r=ahal Differential Revision: https://phabricator.services.mozilla.com/D86284
cd425862bc50658fb60bd3f4cc00b3551acfd767: Bug 1656465 - Run fuzzing builds on backstop pushes. r=ahal DONTBUILD
Marco Castelluccio <mcastelluccio@mozilla.com> - Fri, 07 Aug 2020 15:08:53 +0000 - rev 543794
Push 37681 by abutkovits@mozilla.com at Fri, 07 Aug 2020 21:36:18 +0000
Bug 1656465 - Run fuzzing builds on backstop pushes. r=ahal DONTBUILD Differential Revision: https://phabricator.services.mozilla.com/D86357
d25219ad617a1aac0ff1dcc892219fa76f4584ea: Bug 1408051 - Remove mozharness' copy of virtualenv and use the one under /third_party/python instead r=ahal,rail
Hamzah <hamzah18051@iiitd.ac.in> - Wed, 05 Aug 2020 19:52:24 +0000 - rev 543757
Push 37681 by abutkovits@mozilla.com at Fri, 07 Aug 2020 21:36:18 +0000
Bug 1408051 - Remove mozharness' copy of virtualenv and use the one under /third_party/python instead r=ahal,rail Differential Revision: https://phabricator.services.mozilla.com/D83181
3011872aa773ac1d9d76db7ea1cf4512c1987eec: Bug 1657097 - Stop running all builds on autoland, only run builds selected by bugbug and builds that are dependencies of test tasks. r=ahal,jmaher
Marco Castelluccio <mcastelluccio@mozilla.com> - Thu, 06 Aug 2020 14:25:17 +0000 - rev 543588
Push 37676 by cbrindusan@mozilla.com at Thu, 06 Aug 2020 21:54:39 +0000
Bug 1657097 - Stop running all builds on autoland, only run builds selected by bugbug and builds that are dependencies of test tasks. r=ahal,jmaher Differential Revision: https://phabricator.services.mozilla.com/D85976
952d83dcfb11865077c1007edbacb54410922758: Bug 1656412 - moznetwork isn't compatible with Python 3. r=ahal
Makoto Kato <m_kato@ga2.so-net.ne.jp> - Wed, 05 Aug 2020 14:54:09 +0000 - rev 543406
Push 37673 by apavel@mozilla.com at Wed, 05 Aug 2020 21:44:54 +0000
Bug 1656412 - moznetwork isn't compatible with Python 3. r=ahal After landing bug 1638963, geckoview-junit is Python 3 compatible. But moznetwork isn't still compatible with Python 3 although bug 1428711 is fixed. According to https://docs.python.org/3/library/array.html, - `names = array.array('B', '\0' * bytes)` isn't compatible with Python 3. - `names.tostring()` returns byte array on Python 3 and this is deprecated. Differential Revision: https://phabricator.services.mozilla.com/D85526
d87e870c08fbc2bfc11de05fdf0187fcbe13238b: Bug 1657192 - Add a 'tests_grouped' tag to tasks for suites that are chunked in the taskgraph. r=ahal DONTBUILD
Marco Castelluccio <mcastelluccio@mozilla.com> - Wed, 05 Aug 2020 14:27:31 +0000 - rev 543405
Push 37673 by apavel@mozilla.com at Wed, 05 Aug 2020 21:44:54 +0000
Bug 1657192 - Add a 'tests_grouped' tag to tasks for suites that are chunked in the taskgraph. r=ahal DONTBUILD Differential Revision: https://phabricator.services.mozilla.com/D85908
f43f5cbd2206614787a0570417633d960db8b077: Bug 1656372 - Add a test-info task to identify tests disabled on xorigin; r=ahal
Geoff Brown <gbrown@mozilla.com> - Wed, 05 Aug 2020 13:34:00 +0000 - rev 543384
Push 37672 by ccoroiu@mozilla.com at Wed, 05 Aug 2020 16:49:48 +0000
Bug 1656372 - Add a test-info task to identify tests disabled on xorigin; r=ahal Differential Revision: https://phabricator.services.mozilla.com/D85913
17a6cf8777b1340453047b1e853879715ff72a12: Bug 1648723 - Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal,jmaher
Marco Castelluccio <mcastelluccio@mozilla.com> - Wed, 05 Aug 2020 11:25:20 +0000 - rev 543372
Push 37672 by ccoroiu@mozilla.com at Wed, 05 Aug 2020 16:49:48 +0000
Bug 1648723 - Run full backstops every 20 pushes, and backstops optimized by bugbug every 10 pushes. r=ahal,jmaher Differential Revision: https://phabricator.services.mozilla.com/D81874
dfc9b2ea97c76338aaae5626189f4a576c4ad706: Bug 1656315 - Update json-pushes mock after the changes from bug 1648723. r=ahal
Marco Castelluccio <mcastelluccio@mozilla.com> - Tue, 04 Aug 2020 14:50:48 +0000 - rev 543243
Push 37668 by btara@mozilla.com at Wed, 05 Aug 2020 03:14:17 +0000
Bug 1656315 - Update json-pushes mock after the changes from bug 1648723. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D85743
0026f4c69875a597c2a92e33fd24c30fa9d44e05: Bug 1648723 - Define a new push-interval-20 strategy that optimizes everything except every 20th push. r=ahal
Marco Castelluccio <mcastelluccio@mozilla.com> - Thu, 30 Jul 2020 17:14:21 +0000 - rev 542645
Push 37653 by btara@mozilla.com at Thu, 30 Jul 2020 21:54:52 +0000
Bug 1648723 - Define a new push-interval-20 strategy that optimizes everything except every 20th push. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D85277
55bcd608e8340f26d69309ae72d52df4f9f3225b: Bug 1648723 - Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal
Marco Castelluccio <mcastelluccio@mozilla.com> - Thu, 30 Jul 2020 17:14:09 +0000 - rev 542644
Push 37653 by btara@mozilla.com at Thu, 30 Jul 2020 21:54:52 +0000
Bug 1648723 - Use the util/hg.py module in the backstop module to avoid code duplication. r=ahal Differential Revision: https://phabricator.services.mozilla.com/D85276