a3f2137a434b8f23b73577736e49fcc275fd3498: Bug 1712151: Use compatible version of pyasn1-modules draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:34 +0000 - rev 3890013
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1712151: Use compatible version of pyasn1-modules Summary: The existing version of `pyasn1-modules` (`0.1.5`) is incompatible with our version of `pyasn1` (`0.4.8`). By bumping `pyasn1-modules` to `0.2.8`, we now meet its compatibility requirements. Differential Revision: https://phabricator.services.mozilla.com/D122897 Depends on D122895 Test Plan: Reviewers: Subscribers: Bug #: 1712151 Differential Diff: PHID-DIFF-33c6mfiw5ntaok2aixkv
fdafce87a0eff5322e44b08f1211d09c5071790c: Bug 1723237: Allow empty lines in python requirements files draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:27 +0000 - rev 3890012
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1723237: Allow empty lines in python requirements files Summary: This allows strategic grouping of dependencies, which can be convenient. Differential Revision: https://phabricator.services.mozilla.com/D122895 Depends on D122894 Test Plan: Reviewers: Subscribers: Bug #: 1723237 Differential Diff: PHID-DIFF-2s4bokutr2np2edyjkaq
afc081000a4049968bc42386fe3c12030cf251de: Bug 1712133: Add missing space to pypi-optional error message draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:24 +0000 - rev 3890011
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1712133: Add missing space to pypi-optional error message Summary: Differential Revision: https://phabricator.services.mozilla.com/D122894 Depends on D122893 Test Plan: Reviewers: Subscribers: Bug #: 1712133 Differential Diff: PHID-DIFF-65p5aukpxq3crwbe6rdd
195451d4650fe3894dfe1eea742327882f71457f: Bug 1723237: Warn if virtualenv req definition is missing draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:20 +0000 - rev 3890010
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1723237: Warn if virtualenv req definition is missing Summary: When porting commands to have their own virtualenv, it's useful to have helpful warnings when the associated requirements definition file is missing. Differential Revision: https://phabricator.services.mozilla.com/D122893 Depends on D122892 Test Plan: Reviewers: Subscribers: Bug #: 1723237 Differential Diff: PHID-DIFF-okr32efnsqabctlltpwm
c9d7190d5155cf1fa826934ce534fbfe584a05b3: Bug 1717104: Activate virtualenv before running command draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:17 +0000 - rev 3890009
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717104: Activate virtualenv before running command Summary: All commands now have their virtualenv activated before they're invoked. If they haven't explicitly specified a virtualenv, then they use the one named "common". Removes all now-redundant manual activations of virtualenvs. Tweaks some tests that include Mach command invocations to skip virtualenv activation to avoid parallel process race conditions. Differential Revision: https://phabricator.services.mozilla.com/D122892 Depends on D122891 Test Plan: Reviewers: Subscribers: Bug #: 1717104 Differential Diff: PHID-DIFF-y533gwetupdfxl3cyrgg
6ead01b56838d66c14cc0475c4af0071ff13a9f1: Bug 1717104: Align virtualenv_name with requirements definition draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:14 +0000 - rev 3890008
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717104: Align virtualenv_name with requirements definition Summary: Maps virtualenvs one-to-one with their associated requirements definition. For example: ``` Name: "docs" Virtualenv location: <snip>/_virtualenvs/docs Requirements location: $topsrcdir/build/docs_virtualenv_packages.txt ``` An issue to be resolved in the future is that it's tricky to know that, when you define a new virtualenv, you have to *know* that a requirements definition needs to exist in `build/`. As part of this change, the default virtualenv changed from `build` to `common`. Since some python-tests depend on `glean_parser`, the `python-test` virtualenv was modified to still include it. This addition to the `python-test` virtualenv is temporary and will be removed when bug 1724273 is resolved. Differential Revision: https://phabricator.services.mozilla.com/D122891 Depends on D122890 Test Plan: Reviewers: Subscribers: Bug #: 1717104 Differential Diff: PHID-DIFF-dqzjtl6vwdfqd6kuwyde
3649a3528bdf64f6d5572186f84adc344e9e20af: Bug 1723031: Assert package dependencies with system Python draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:10 +0000 - rev 3890007
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1723031: Assert package dependencies with system Python Summary: When using `MOZ_AUTOMATION` or `MACH_USE_SYSTEM_PYTHON`, validate that required dependencies are installed. Differential Revision: https://phabricator.services.mozilla.com/D122890 Depends on D122889 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-rfymkum5yzip53one4wg
0e88f9c742403cb3df7d48755f2161dbe5ff2527: Bug 1723031: Allow flexible dependency-specification in the Mach venv draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:07 +0000 - rev 3890006
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1723031: Allow flexible dependency-specification in the Mach venv Summary: There's some trade-offs in play here: the major issue is that we can't pin `psutil`'s because it's pre-installed on some CI workers with a different version (5.4.2). Additionally, we can't "just" bump that version, because CI workers jump between different revisions to do work, so a specific pinned version won't work when we try to update such a package. One option is that we could avoid validating package versions in CI, but then that will cause surprises (heck, I didn't know we were still using `psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we make it more explicit and avoid accidentally depending on behaviour of too new of such a package. However, in most cases, we manage the installation environment and can pin dependencies. So, I've made the top-level Mach virtualenv the _only_ one that is able to use requirement specifiers other than "==". Differential Revision: https://phabricator.services.mozilla.com/D122889 Depends on D122888 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-7gggm543aabalgmoxljw
9fd9b970de1543368e91875f87f3f4aa591fe66b: Bug 1723031: Vendors `packaging`, `pyparsing` library draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:47:04 +0000 - rev 3890005
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1723031: Vendors `packaging`, `pyparsing` library Summary: This will allow us to parse and compare pip package versions the same way that `pip` does. `pyparsing` was added because it's needed by `packaging`. Differential Revision: https://phabricator.services.mozilla.com/D122888 Depends on D122887 Test Plan: Reviewers: Subscribers: Bug #: 1723031 Differential Diff: PHID-DIFF-6t2jwvwbctxq2i2m2znf
82ebd240d3d0a7b28827c15206238c565236956b: Bug 1724279: Remove `zstandard` from Mach virtualenv draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:59 +0000 - rev 3890004
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1724279: Remove `zstandard` from Mach virtualenv Summary: `zstandard` is only needed in commands (and some ad-hoc usages in build), so it can be safely removed from the Mach environment and installed in the contexts that it's needed. This will make Mach virtualenv setup faster, since `zstandard` won't need to be installed. Differential Revision: https://phabricator.services.mozilla.com/D122887 Depends on D122886 Test Plan: Reviewers: Subscribers: Bug #: 1724279 Differential Diff: PHID-DIFF-3st3vmkb2e6icnei4dgu
54cce95e2d6f226265fb7b760da3410aa76d0e0f: Bug 1717051: Validate if optional dependencies are out-of-date draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:54 +0000 - rev 3890003
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Validate if optional dependencies are out-of-date Summary: Though it's acceptable for an optional dependency to not be installed, it *is* a problem if it's installed to the wrong version. Differential Revision: https://phabricator.services.mozilla.com/D122886 Depends on D122885 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-p6aouciercjgg4dheal3
dee8b1aba94a6f9eac008cd8d9b83eabf2614063: Bug 1717051: Remove obsolete virtualenv code draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:51 +0000 - rev 3890002
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Remove obsolete virtualenv code Summary: The "six"-related comment is no longer relevant, and `here` isn't used. Depends on D120402 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-652a76ngbc3dyovpcquc
66fbdf0186af82a40202b26c432d8f7fd9df684a: Bug 1717051: Remove redundant populate_local_paths parameter draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:48 +0000 - rev 3890001
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Remove redundant populate_local_paths parameter Summary: Now that we no longer patch sys.path when using the Mach virtualenv, the `populate_local_paths` option is no longer ever `False`. So, it can be inlined as `True`. Differential Revision: https://phabricator.services.mozilla.com/D120402 Depends on D120401 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-ajv25kz4zpcksh4regaz
16d666728d69ea820cb5252ce1ccef742a79fedc: Bug 1717051: Automatically create and activate Mach virtualenv draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:45 +0000 - rev 3890000
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Automatically create and activate Mach virtualenv Summary: Consolidate Mach virtualenv management to the front of the Mach process. This obsoletes `./mach create-mach-environment` and simplifies the `sh` portion of the top-level `./mach` script. This helps ensure that the Mach virtualenv doesn't become out-of-sync and simplifies the mental model of the Mach virtualenv situation. ----- After activating the Mach virtualenv, `sys.executable` is set to the Mach virtualenv. However, this `sys.executable` tweaking won't happen when command virtualenvs are activated. At the moment, this appears to be the best set of tradeoffs: Glean creates a child Python process that needs to be able to `import glean`. So, it creates that process from `sys.executable` with the assumption that "`sys.executable` has the current Glean module imported, so it must be able to do it again." However, `sys.executable` is not changed automatically when dynamically activating virtualenvs from an existing Python process. So, without setting `sys.executable` ourselves, it continues to point to the system Python, which isn't guaranteed to have Glean. By updating `sys.executable` to point to the Mach virtualenv, we ensure that the Glean subprocess can import `glean` properly. Unfortunately, this solution only works because we currently only have one dependency using `sys.executable`. If we have this case in the future with a dependency in a *command virtualenv*, then this solution won't work anymore. This leads us to an alternative solution (which isn't currently adopted): we could install all Mach-level dependencies into every command virtualenv. This has some benefits: * Any dependency of Mach *or* any command can start subprocesses from `sys.executable`, and it'll always run from a virtualenv with that dependency installed. * We're already going to have to validate that the dependencies of commands don't conflict with Mach dependencies. The current planned solution is to do that validation only in CI, but this would simplify that situation. * The `virtualenv` developers have stated that dynamically activating a virtualenv from an existing Python process isn't "switching" virtualenvs, but rather adding its packages to the current environment. Installing Mach dependencies into all command virtualenvs would align with the additive nature here. However, there's one big downside: performance. Installing modules, _especially_ native modules, can be slow. Having to set up `glean`, `zstandard` and `psutil` into all command virtualenvs is inefficient and will waste developer time. So, until it's needed, the manual `sys.executable` management appears to be the more effective solution. FWIW, this is very similar to our pre-1717051 behaviour of using the Mach virtualenv as `sys.executable` and not updating it when later virtualenvs are activated. To a degree, this change just maintains the status quo. Differential Revision: https://phabricator.services.mozilla.com/D120401 Depends on D120404 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-ckjnwn32gjtwgmkts2tp
b8c6d011e4ecb10b58066a4b6216b9e9ad324429: Bug 1717051: Update `pkg_resources` state when activating venvs draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:42 +0000 - rev 3889999
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Update `pkg_resources` state when activating venvs Summary: `pkg_resources` parses the environment state when it's first imported. However, when we activate an existing virtualenv, `pkg_resources` isn't automatically updated. The current symptom of this is that the PyCharm debugger causes failures when a virtualenv boundary is passed, because it imports `pkg_resources` very early in the debugging lifecycle. Calling an internal `pkg_resources` function is risky, but it seems to be the most stable way of updating `pkg_resources` to the new virtualenv's state. This solution may need to be investigated further in the future if issues are encountered. Differential Revision: https://phabricator.services.mozilla.com/D120404 Depends on D120400 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-cwsc5arjq76lupj3wsjd
186eddc0baaba4be87fbe6f24409a2759e848391: Bug 1717051: Consolidate state_dir creation draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:38 +0000 - rev 3889998
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Consolidate state_dir creation Summary: We had logic in both `mach_bootstrap` and the Mach Bootstrapper to create the state_dir. This joins them and has the added benefit of creating the state dir earlier in the Mach lifecycle (as will be needed for early instantiation of the Mach virtualenv). Differential Revision: https://phabricator.services.mozilla.com/D120400 Depends on D120399 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-hbue3ybje4mqczo7d3rh
da7496f39275c87f4d5a4cdfcae192c7469aad90: Bug 1717051: Move arm64 rosetta check to bootstrap draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:36 +0000 - rev 3889997
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Move arm64 rosetta check to bootstrap Summary: It was originally added to "create-mach-environment" because that would eventually be called by "./mach bootstrap". However, as "create-mach-environment" is going away, this moves it closer to the command that it's meant to impact. Differential Revision: https://phabricator.services.mozilla.com/D120399 Depends on D120410 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-642criyndjaissmvws5b
8b1b99b112f7c35c72df02c05225a4a416287660: Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py" draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:33 +0000 - rev 3889996
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py" Summary: We've overloaded "bootstrap" to mean three different things: * The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`. This is to freshly clone a new repo, then run `./mach bootstrap`. * `./mach bootstrap`: Install necessary dependencies and set up the system for development. * "Mach bootstrap": do the in-process initialization work Mach needs before it can run commands. By using the term "initialize" instead, perhaps we can remove ambiguity when discussing Mach. I'm not attached to the name (or this change at all), but I'm interested in reviewer thoughts :) Differential Revision: https://phabricator.services.mozilla.com/D120410 Depends on D120398 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-3qkyajzqttvtumxgm3te
751694af7af5c06c9804227bafc67bac19f5170c: Bug 1708260: Remove "imp" usage when loading Mach draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:30 +0000 - rev 3889995
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1708260: Remove "imp" usage when loading Mach Summary: We always run on Python 3.6+, so the `imp` usage was dead code. Differential Revision: https://phabricator.services.mozilla.com/D120398 Depends on D120397 Test Plan: Reviewers: Subscribers: Bug #: 1708260 Differential Diff: PHID-DIFF-35km64bpmswdableljd7
86b8e375ae9b4e10c29834ec42fccebf7d3cf35b: Bug 1717051: Remove "install-moz-phab" from nativecmds draft
Mitchell Hentges <mhentges@mozilla.com> - Thu, 19 Aug 2021 15:46:27 +0000 - rev 3889994
Push 721268 by reviewbot at Thu, 19 Aug 2021 15:48:11 +0000
Bug 1717051: Remove "install-moz-phab" from nativecmds Summary: It was in `nativecmds` to make finding `pip3` easier. However, since we're removing the distinction between "regular" and "native" cmds (Mach will create and activate its venv naturally during initialization), we need to start pruning the `nativecmds` list. The new behaviour continues to support the following use cases: * Using Mach with system Python in the `$PATH`. * Using Mach from within a human-managed virtualenv. * Either of the above^ in combination with `MACH_USE_SYSTEM_PYTHON`. To do this, I had to pipe through the `ENTRY_SHELL_PYTHON_EXECUTABLE` (the python that's on the `$PATH` when Mach is called) to ensure that we installed `moz-phab` into a human-managed virtualenv if it was active. Differential Revision: https://phabricator.services.mozilla.com/D120397 Test Plan: Reviewers: Subscribers: Bug #: 1717051 Differential Diff: PHID-DIFF-y3hpshvery4jau3e2b4e
(0) -3000000 -1000000 -300000 -100000 -30000 -10000 -3000 -1000 -300 -100 -50 -20 +20 +50 +100 +300 +1000 +3000 +10000 +30000 +100000 +300000 tip