994ecbf21498e7a9a055438552be5b15794fe85d: Bug 1717051: Automatically create and activate Mach virtualenv draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:18 +0000 - rev 3893333
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Automatically create and activate Mach virtualenv Summary: Consolidate Mach virtualenv management to the front of the Mach process. This obsoletes `./mach create-mach-environment` and simplifies the `sh` portion of the top-level `./mach` script. This helps ensure that the Mach virtualenv doesn't become out-of-sync and simplifies the mental model of the Mach virtualenv situation. ----- After activating the Mach virtualenv, `sys.executable` is set to the Mach virtualenv. However, this `sys.executable` tweaking won't happen when command virtualenvs are activated. At the moment, this appears to be the best set of tradeoffs: Glean creates a child Python process that needs to be able to `import glean`. So, it creates that process from `sys.executable` with the assumption that "`sys.executable` has the current Glean module imported, so it must be able to do it again." However, `sys.executable` is not changed automatically when dynamically activating virtualenvs from an existing Python process. So, without setting `sys.executable` ourselves, it continues to point to the system Python, which isn't guaranteed to have Glean. By updating `sys.executable` to point to the Mach virtualenv, we ensure that the Glean subprocess can import `glean` properly. Unfortunately, this solution only works because we currently only have one dependency using `sys.executable`. If we have this case in the future with a dependency in a *command virtualenv*, then this solution won't work anymore. This leads us to an alternative solution (which isn't currently adopted): we could install all Mach-level dependencies into every command virtualenv. This has some benefits: * Any dependency of Mach *or* any command can start subprocesses from `sys.executable`, and it'll always run from a virtualenv with that dependency installed. * We're already going to have to validate that the dependencies of commands don't conflict with Mach dependencies. The current planned solution is to do that validation only in CI, but this would simplify that situation. * The `virtualenv` developers have stated that dynamically activating a virtualenv from an existing Python process isn't "switching" virtualenvs, but rather adding its packages to the current environment. Installing Mach dependencies into all command virtualenvs would align with the additive nature here. However, there's one big downside: performance. Installing modules, _especially_ native modules, can be slow. Having to set up `glean`, `zstandard` and `psutil` into all command virtualenvs is inefficient and will waste developer time. So, until it's needed, the manual `sys.executable` management appears to be the more effective solution. FWIW, this is very similar to our pre-1717051 behaviour of using the Mach virtualenv as `sys.executable` and not updating it when later virtualenvs are activated. To a degree, this change just maintains the status quo. Differential Revision: https://phabricator.services.mozilla.com/D120401 Depends on D120404 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-ckjnwn32gjtwgmkts2tp
4d766eb041d669611a9bab11355000b7c06bd776: Bug 1717051: Update `pkg_resources` state when activating venvs draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:14 +0000 - rev 3893332
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Update `pkg_resources` state when activating venvs Summary: `pkg_resources` parses the environment state when it's first imported. However, when we activate an existing virtualenv, `pkg_resources` isn't automatically updated. The current symptom of this is that the PyCharm debugger causes failures when a virtualenv boundary is passed, because it imports `pkg_resources` very early in the debugging lifecycle. Calling an internal `pkg_resources` function is risky, but it seems to be the most stable way of updating `pkg_resources` to the new virtualenv's state. This solution may need to be investigated further in the future if issues are encountered. Differential Revision: https://phabricator.services.mozilla.com/D120404 Depends on D120400 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-cwsc5arjq76lupj3wsjd
35677426b36bd3e49d877b2b2daf3fb736f2dd79: Bug 1717051: Consolidate state_dir creation draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:11 +0000 - rev 3893331
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Consolidate state_dir creation Summary: We had logic in both `mach_bootstrap` and the Mach Bootstrapper to create the state_dir. This joins them and has the added benefit of creating the state dir earlier in the Mach lifecycle (as will be needed for early instantiation of the Mach virtualenv). Differential Revision: https://phabricator.services.mozilla.com/D120400 Depends on D120399 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-hbue3ybje4mqczo7d3rh
d53444fcdf85ef469136fdc27601ef4bd74cf586: Bug 1717051: Move arm64 rosetta check to bootstrap draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:08 +0000 - rev 3893330
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Move arm64 rosetta check to bootstrap Summary: It was originally added to "create-mach-environment" because that would eventually be called by "./mach bootstrap". However, as "create-mach-environment" is going away, this moves it closer to the command that it's meant to impact. Differential Revision: https://phabricator.services.mozilla.com/D120399 Depends on D120410 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-642criyndjaissmvws5b
47c996f5476b0606e075b31ed5eba5facf672885: Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py" draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:04 +0000 - rev 3893329
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py" Summary: We've overloaded "bootstrap" to mean three different things: * The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`. This is to freshly clone a new repo, then run `./mach bootstrap`. * `./mach bootstrap`: Install necessary dependencies and set up the system for development. * "Mach bootstrap": do the in-process initialization work Mach needs before it can run commands. By using the term "initialize" instead, perhaps we can remove ambiguity when discussing Mach. I'm not attached to the name (or this change at all), but I'm interested in reviewer thoughts :) Differential Revision: https://phabricator.services.mozilla.com/D120410 Depends on D120398 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-3qkyajzqttvtumxgm3te
db1b3077334a4cf0ed488fbe498f92b36a7cc4e5: Bug 1708260: Remove "imp" usage when loading Mach draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:01 +0000 - rev 3893328
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1708260: Remove "imp" usage when loading Mach Summary: We always run on Python 3.6+, so the `imp` usage was dead code. Differential Revision: https://phabricator.services.mozilla.com/D120398 Depends on D120397 Test Plan: Reviewers: Subscribers: Bug #: 1708260 Differential Diff: PHID-DIFF-35km64bpmswdableljd7
f07723dfe082c4e28e6a33d0a840faf4c55b92d7: Bug 1717051: Remove "install-moz-phab" from nativecmds draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:37:59 +0000 - rev 3893327
Push 721744 by reviewbot at Fri, 20 Aug 2021 17:39:59 +0000
Bug 1717051: Remove "install-moz-phab" from nativecmds Summary: It was in `nativecmds` to make finding `pip3` easier. However, since we're removing the distinction between "regular" and "native" cmds (Mach will create and activate its venv naturally during initialization), we need to start pruning the `nativecmds` list. The new behaviour continues to support the following use cases: * Using Mach with system Python in the `$PATH`. * Using Mach from within a human-managed virtualenv. * Either of the above^ in combination with `MACH_USE_SYSTEM_PYTHON`. To do this, I had to pipe through the `ENTRY_SHELL_PYTHON_EXECUTABLE` (the python that's on the `$PATH` when Mach is called) to ensure that we installed `moz-phab` into a human-managed virtualenv if it was active. Differential Revision: https://phabricator.services.mozilla.com/D120397 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-y3hpshvery4jau3e2b4e
e060fe72cdf05a15a1567d30679a268735cef2de: try_task_config for https://phabricator.services.mozilla.com/D122900 draft
libmozevent <release-mgmt-analysis@mozilla.com> - Fri, 20 Aug 2021 17:38:52 +0000 - rev 3893326
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
try_task_config for https://phabricator.services.mozilla.com/D122900 Differential Diff: PHID-DIFF-eourgmlqpb5hdchsubqs
ffe9e84ef6210ab97149c73592e90fa1a4a8a4ba: Bug 1712151: Add test to verify virtualenv compatibility draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:49 +0000 - rev 3893325
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1712151: Add test to verify virtualenv compatibility Summary: This adds two main compatibility guarantees: 1. Vendored dependencies <=> Pypi-downloaded dependencies 2. Global Mach dependencies <=> command-specific dependencies As part of this, a new `vendored:` action was added to the virtualenv definition format. Otherwise similar to `pth:` paths, `vendored:` packages are assumed to be "pip install"-able. Some validation (the `.dist-info` check) was added to `requirements.py` to verify that `pth:` and `vendored:` are correctly used. However, I couldn't add a `setup.py` check because some of our un-installable internal modules (`mozbase`, etc) have out-of-date and potentially inaccurate `setup.py` files. Though it would be nice to leverage each internal module as an independent package with its own scoped set of dependencies, that's a project for another time. Differential Revision: https://phabricator.services.mozilla.com/D122900 Depends on D122899 Test Plan: Reviewers: Subscribers: Bug #: 1712151 Differential Diff: PHID-DIFF-eourgmlqpb5hdchsubqs
c596d58e997a5822cc8eb230c328b830eb925ea4: Bug 1712151: Bump glean-sdk version to 40.0.0 draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:46 +0000 - rev 3893324
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1712151: Bump glean-sdk version to 40.0.0 Summary: We vendor `glean_parser==3.6.0`, and that was incompatible with `glean-sdk==36.0.0`'s requirement of `glean_parser==2.5.0`. `glean-sdk==40.0.0` expects `glean_parser==3.6.0`, which is perfect. Differential Revision: https://phabricator.services.mozilla.com/D122899 Depends on D122898 Test Plan: Reviewers: Subscribers: Bug #: 1712151 Differential Diff: PHID-DIFF-5wuepkvzm72chnkyrbwn
f55a5b177235954e738a3e7aea2b77e939626d3e: Bug 1712151: Vendor pystache automatically draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:43 +0000 - rev 3893323
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1712151: Vendor pystache automatically Summary: Add pystache to vendor `requirements.in` so that it's vendored according to `./mach vendor python` "ignore" rules. This ensures that sufficient files are vendored such that installing the package from it's `setup.py` file is possible. Differential Revision: https://phabricator.services.mozilla.com/D122898 Depends on D122897 Test Plan: Reviewers: Subscribers: Bug #: 1712151 Differential Diff: PHID-DIFF-perjawzkdgcpqnlss2by
9030b46893ce9d6933f464dc49a7047e5a3557d7: Bug 1712151: Use compatible version of pyasn1-modules draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:39 +0000 - rev 3893322
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1712151: Use compatible version of pyasn1-modules Summary: The existing version of `pyasn1-modules` (`0.1.5`) is incompatible with our version of `pyasn1` (`0.4.8`). By bumping `pyasn1-modules` to `0.2.8`, we now meet its compatibility requirements. Differential Revision: https://phabricator.services.mozilla.com/D122897 Depends on D122895 Test Plan: Reviewers: Subscribers: Bug #: 1712151 Differential Diff: PHID-DIFF-7bjrbemdg4twilcg4vql
c1b481bf8ad35e09627a25568b471ea79fff8a23: Bug 1723237: Allow empty lines in python requirements files draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:33 +0000 - rev 3893321
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1723237: Allow empty lines in python requirements files Summary: This allows strategic grouping of dependencies, which can be convenient. Differential Revision: https://phabricator.services.mozilla.com/D122895 Depends on D122894 Test Plan: Reviewers: Subscribers: Bug #: 1723237 Differential Diff: PHID-DIFF-bq4tafn42n44jjghjzdx
24f83138a45d25e42e1653272bfa2921c764d355: Bug 1712133: Add missing space to pypi-optional error message draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:31 +0000 - rev 3893320
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1712133: Add missing space to pypi-optional error message Summary: Differential Revision: https://phabricator.services.mozilla.com/D122894 Depends on D122893 Test Plan: Reviewers: Subscribers: Bug #: 1712133 Differential Diff: PHID-DIFF-itlv4pds7v5sqbmaespt
4e6f206deb866e8801b1d03fe7942ee7d490d363: Bug 1723237: Warn if virtualenv req definition is missing draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:29 +0000 - rev 3893319
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1723237: Warn if virtualenv req definition is missing Summary: When porting commands to have their own virtualenv, it's useful to have helpful warnings when the associated requirements definition file is missing. Differential Revision: https://phabricator.services.mozilla.com/D122893 Depends on D122892 Test Plan: Reviewers: Subscribers: Bug #: 1723237 Differential Diff: PHID-DIFF-3kh2cby2jsb7lhwsibdz
a90fce5769f9b2c00b0eb5bda00fe311e5987c5a: Bug 1717104: Activate virtualenv before running command draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:26 +0000 - rev 3893318
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1717104: Activate virtualenv before running command Summary: All commands now have their virtualenv activated before they're invoked. If they haven't explicitly specified a virtualenv, then they use the one named "common". Removes all now-redundant manual activations of virtualenvs. Tweaks some tests that include Mach command invocations to skip virtualenv activation to avoid parallel process race conditions. Differential Revision: https://phabricator.services.mozilla.com/D122892 Depends on D122891 Test Plan: Reviewers: Subscribers: Bug #: 1717104 Differential Diff: PHID-DIFF-2oow3nmvppntzi4xti6q
8e7e349b03bf09b3459b4b437d61d6bcfe2db2b5: Bug 1717104: Align virtualenv_name with requirements definition draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:24 +0000 - rev 3893317
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1717104: Align virtualenv_name with requirements definition Summary: Maps virtualenvs one-to-one with their associated requirements definition. For example: ``` Name: "docs" Virtualenv location: <snip>/_virtualenvs/docs Requirements location: $topsrcdir/build/docs_virtualenv_packages.txt ``` An issue to be resolved in the future is that it's tricky to know that, when you define a new virtualenv, you have to *know* that a requirements definition needs to exist in `build/`. As part of this change, the default virtualenv changed from `build` to `common`. Since some python-tests depend on `glean_parser`, the `python-test` virtualenv was modified to still include it. This addition to the `python-test` virtualenv is temporary and will be removed when bug 1724273 is resolved. Differential Revision: https://phabricator.services.mozilla.com/D122891 Depends on D122890 Test Plan: Reviewers: Subscribers: Bug #: 1717104 Differential Diff: PHID-DIFF-ht35vmfmu477zilwpzy2
0c3734bf861d7c8a437a61dd3125bc7768f5b94d: Bug 1723031: Assert package dependencies with system Python draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:21 +0000 - rev 3893316
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1723031: Assert package dependencies with system Python Summary: When using `MOZ_AUTOMATION` or `MACH_USE_SYSTEM_PYTHON`, validate that required dependencies are installed. Differential Revision: https://phabricator.services.mozilla.com/D122890 Depends on D122889 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-occm2lkfvqkruamz2nrr
afc2677a53859382cd9b02e9dd768359b255a709: Bug 1723031: Allow flexible dependency-specification in the Mach venv draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:19 +0000 - rev 3893315
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1723031: Allow flexible dependency-specification in the Mach venv Summary: There's some trade-offs in play here: the major issue is that we can't pin `psutil`'s because it's pre-installed on some CI workers with a different version (5.4.2). Additionally, we can't "just" bump that version, because CI workers jump between different revisions to do work, so a specific pinned version won't work when we try to update such a package. One option is that we could avoid validating package versions in CI, but then that will cause surprises (heck, I didn't know we were still using `psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we make it more explicit and avoid accidentally depending on behaviour of too new of such a package. However, in most cases, we manage the installation environment and can pin dependencies. So, I've made the top-level Mach virtualenv the _only_ one that is able to use requirement specifiers other than "==". Differential Revision: https://phabricator.services.mozilla.com/D122889 Depends on D122888 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-7gggm543aabalgmoxljw
ab4818a4521cb2aab06a4205f0eacceeee56f8fe: Bug 1723031: Vendors `packaging`, `pyparsing` library draft
Mitchell Hentges <mhentges@mozilla.com> - Fri, 20 Aug 2021 17:38:16 +0000 - rev 3893314
Push 721743 by reviewbot at Fri, 20 Aug 2021 17:39:13 +0000
Bug 1723031: Vendors `packaging`, `pyparsing` library Summary: This will allow us to parse and compare pip package versions the same way that `pip` does. `pyparsing` was added because it's needed by `packaging`. Differential Revision: https://phabricator.services.mozilla.com/D122888 Depends on D122887 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-6t2jwvwbctxq2i2m2znf
(0) -3000000 -1000000 -300000 -100000 -30000 -10000 -3000 -1000 -300 -100 -50 -20 +20 +50 +100 +300 +1000 +3000 +10000 +30000 +100000 +300000 tip