mozautomation: match built URIs exactly in `resolve_uri_to_tree` (Bug 1768982) r=zeid `resolve_uri_to_tree` takes a URI (ie `https://hg.mozilla.org/mozilla-central`) and attempts to resolve it to a known Firefox tree shortname (ie `central`). It does so by iterating through the list of known Firefox tree repo names and building out each potential URI, and then checking if each built URI matches the passed URI. The matching condition used is `uri.startswith`, assumedly so something like `https://hg.mozilla.org/mozilla-centra` or some other minor typo results in a match still being found. This has worked without any issue to date, until `esr102` was created. Since the built URIs for esr10 would match the `startswith` metric on `esr102`, passing the URI for esr102 is returning `esr10` as the resolved tree. Switch `resolve_uri_to_tree` to return the resolved tree only when the built URI exactly matches the passed URI. Add a few comments clarifying what the function is doing and use more descriptive variable names. Differential Revision: https://phabricator.services.mozilla.com/D146210

pycompat: account for `pycompat.bytesio` being removed in 6.2+ hg (Bug 1762417) r=zeid Use `io.BytesIO` in testing specific modules for Py3 only code (hooks in this case) and add a function to paper over the differences on `configwizard` where we may still need to run on Py2 in some cases. Differential Revision: https://phabricator.services.mozilla.com/D142686

vcsreplicator: send heads message as part of `replicatesync` (Bug 1757183) r=zeid Adds a call to `sendheadsmessage` as part of `replicatesync`. Adds a flag to control whether the heads message is sent. Differential Revision: https://phabricator.services.mozilla.com/D142101

pushlog: move db commit and close to different stages of Mercurial transaction (Bug 1735633) r=zeid Moves sqlite `commit` and `close` calls to different Mercurial transaction hooks. Moving `commmit` to the `finalize` step should cause pushlog database failures to reject the push without committing the Mercurial transaction. Interestingly there is a Mercurial extension in the core hg repo that implements sqlite storage for Mercurial and this extension uses `addfinalize` as the transaction commit callback. We also add `rollback` to the `abort` Mercurial transaction step as we simply close the connection at the moment. Differential Revision: https://phabricator.services.mozilla.com/D140157

testing: remove builder-ansible-ubuntu (Bug 1740049) r=zeid We no longer have any test images built from ubuntu in this repo. Depends on D130647 Differential Revision: https://phabricator.services.mozilla.com/D130648

testing: remove builder-hgweb-chroot directory (Bug 1740049) r=zeid It's been unused for some time. Differential Revision: https://phabricator.services.mozilla.com/D130647

testing: rename variable p to builder_path in `docker.py` (Bug 1740049) r=zeid Single letter variables are bad. Differential Revision: https://phabricator.services.mozilla.com/D130646

testing: build test images from base centos (Bug 1740049) r=zeid Deprecates the `secure image` builder functionality. We remove the custom build context code and instead use the v-c-t root as the build context, passing the path to the Dockerfile to specify which image to build. Since the custom builder was responsible for creating a path to `ansible-playbook` in `/usr/bin`, we instead use the `ansible-playbook` from `PATH` to run the image creation process. Differential Revision: https://phabricator.services.mozilla.com/D130645

robustcheckout: add `parentchange` context before calling `_updateconfigandrefreshwdir` (Bug 1738958) r=zeid Without this there are test failures on Mercurial 5.9. This seems to be necessary on newer versions but mostly a no-op on older versions of hg. Differential Revision: https://phabricator.services.mozilla.com/D130202

configwizard: print a warning about an un-managed evolve extension (Bug 1728194) r=mhentges,zeid Currently when configwizard detects an evolve extension that is not in the managed `~/.mozbuild` directory, we take no action to update or otherwise touch the extension for the user. This commit changes this behaviour to instead print a small warning message, indicating that we have detected an externally managed evolve that will require manual intervention to update. We also leave instructions on how to switch to an evolve extension managed by the setup wizard. Differential Revision: https://phabricator.services.mozilla.com/D123984

hgssh: update Mercurial to 5.9 in all virtualenvs (Bug 1721228) r=zeid This changeset should be deployed after upgrading hgweb to Mercurial 5.9. Depends on D123484 Differential Revision: https://phabricator.services.mozilla.com/D123485

hgweb: update Mercurial to 5.9 (Bug 1721228) r=zeid Updates Mercurial to version 5.9 in all virtualenvs on hgweb. This changeset should be deployed first in an incremental fashion to all hgweb hosts. Depends on D123483 Differential Revision: https://phabricator.services.mozilla.com/D123484

clonebundles: remove `replicatesync` logging message from `test-clonebundles.t` (Bug 1727344) r=zeid This was removed after bug 1714463. Depends on D123482 Differential Revision: https://phabricator.services.mozilla.com/D123483

vcsreplicator: use `phaseroots` integer keys directly instead of implicit enumeration (Bug 1727344) r=zeid Mercurial internals formerly defined phases via an expansion of `range(3)`, mapping the phases to each integer. Internally the integers are used so the phases have an order (ie `draft < public`), but these phases are mapped to names that we use in the Mercurial command line. At some point new phases were added with their own unique values that break the `range(3)` standard. In `vcsreplicator` we would read the `phaseroots` dict and enumerate the keys to determine which phases are present on a given repo, however the presence of the new phases breaks this assumption. Since the `phaseroots` object is aleady a mapping of the phase integer representation to revision hash, we should use the keys of the `phaseroots` mapping directly to determine which phases are found on a given repo. Depends on D123481 Differential Revision: https://phabricator.services.mozilla.com/D123482

vcsreplicator: add missing message logging output (Bug 1695029) r=zeid Leftover from Bug 1695029. Depends on D123480 Differential Revision: https://phabricator.services.mozilla.com/D123481

hgmolib: open file in regular mode when parsing `requires` files (Bug 1727342) r=zeid The value for `--requirement` was being passed as a `str`, while the was being read and each line in the file was passed as a `bytes`, leading to comparisons like `"store" == b"store"`, which is `False`. Opening the file in regular read mode makes both values `str` and the comparison is correct again. Depends on D123479 Differential Revision: https://phabricator.services.mozilla.com/D123480

testing: remove exclamation mark from test output (Bug 1721228) r=zeid This is removed in newer versions of Mercurial. Depends on D123478 Differential Revision: https://phabricator.services.mozilla.com/D123479

testing: glob out number of bytes sent in Apache logs in tests (Bug 1727339) r=zeid These values frequently go up/down by a few bytes when updating versions of Mercurial, causing us to update them to their new values each time. Sending a few extra bytes isn't a huge deal, and as long as the wire protocol commnands are taking place in the same order we don't care about the slight bandwidth increases. Depends on D123477 Differential Revision: https://phabricator.services.mozilla.com/D123478

hghooks: fix encoding errors in `test_check_bug_references_helpers.py` (Bug 1727337) r=zeid Depends on D123475 Differential Revision: https://phabricator.services.mozilla.com/D123477

hghooks: update logging output in hooks tests (Bug 1721228) r=zeid Some logging output has changed since Mercurial 5.5.1 in the hooks framework. Depends on D123474 Differential Revision: https://phabricator.services.mozilla.com/D123475

hghooks: add updated output to merge day hooks test (Bug 1720188) r=zeid Adding this output to the tests was missed in D121668. Depends on D123473 Differential Revision: https://phabricator.services.mozilla.com/D123474

hgext: mark some extensions as compatible with Mercurial 5.9 (Bug 1721228) r=zeid These extensions require no changes for 5.9 compatibility. Depends on D123472 Differential Revision: https://phabricator.services.mozilla.com/D123473

ansible/hg-ssh: add `python-ldap` to `requirements-tools.*` explicitly (Bug 1727335) r=zeid Previously this was being added as a transitive dependency of another included requirements file. Add it explicitly as `test-daemon.t` for `unifyrepo` needs it to pass. Depends on D123471 Differential Revision: https://phabricator.services.mozilla.com/D123472

configwizard: update `wip` setup text in `test-wip.t` (Bug 1727331) r=zeid Depends on D123470 Differential Revision: https://phabricator.services.mozilla.com/D123471

hgext: update rebase extension output (Bug 1721228) r=zeid The `rebase` extension changed how some of it's output is formatted to the screen when a tag is present on a rebased revision. Update the tests to reflect this and mark a few extensions as compatible with Mercurial 5.9 as a result. Depends on D123469 Differential Revision: https://phabricator.services.mozilla.com/D123470

firefoxtree: use `utils.urlutil` functions instead of `ui.expandpath` (Bug 1721228) r=zeid Use the newer functions in `utils.urlutil` instead of `ui.expandpath` in `firefoxtree`, making it compatible with Mercurial 5.9. Also mark the extension as compatible with the latest version. Depends on D123468 Differential Revision: https://phabricator.services.mozilla.com/D123469

testing: install Mercurial 5.9 in test environment (Bug 1721228) r=zeid Depends on D123467 Differential Revision: https://phabricator.services.mozilla.com/D123468

testing: install Mercurials by cloning `hg-committed` repository (Bug 1727330) r=zeid The `hg` repo is supposed to have all the work included in `hg-committed`, but in reality `hg-committed` is the source of truth and is kept more up to date. At the moment `hg` is missing the `5.9rc1` release tag, so switch to using it to install Mercurial. Differential Revision: https://phabricator.services.mozilla.com/D123467

mozautomation: add comm-esr91 to repository list (Bug 1719569) r=zeid Depends on D119417 Differential Revision: https://phabricator.services.mozilla.com/D119418

hgwsgi: add comm-esr91 to hgweb wsgi config (Bug 1719569) r=zeid Depends on D119416 Differential Revision: https://phabricator.services.mozilla.com/D119417

ansible/hg-web: add comm-esr91 repo to mirror filters (Bug 1719569) r=zeid Differential Revision: https://phabricator.services.mozilla.com/D119416

ansible/hg-ssh: re-enable bug references check on try (Bug 1607549) r=zeid This was never re-enabled after being fixed in 002cae39b87d. Differential Revision: https://phabricator.services.mozilla.com/D101057

hgwsgi: remove `comm-moz-esr68` repo (Bug 1676914) r=zeid Since this repo was removed, attempting to load `https://hg.mozilla.org/releases` fails since the repo is missing. Differential Revision: https://phabricator.services.mozilla.com/D97218

hgext/manualoverlay: remove directory (Bug 1676933) r=zeid More dead code from the Servo syncing days. Depends on D97013 Differential Revision: https://phabricator.services.mozilla.com/D97014

hgext/overlay: remove directory (Bug 1676933) r=zeid This is a dead extension from the Servo vcs-syncing days. Depends on D97012 Differential Revision: https://phabricator.services.mozilla.com/D97013

vcssync: remove directory (Bug 1676933) r=zeid The actual code used for vcs-sync (ie hgmo -> GitHub) lives in a different repo. Differential Revision: https://phabricator.services.mozilla.com/D97012

hgmo: backout information in `json-automationrelevance` behind a flag (Bug 1676170) r=zeid Calls to `json-automationrelevance` on `mozilla-release` can take up to 30s since deploying changeset 78a0d7c424fc18d. This commit hides the expensive backout information behind a `backouts` query string parameter so only the relevant calls will perform the expensive calculation. Differential Revision: https://phabricator.services.mozilla.com/D96431

hgext/hgmo: add `backedoutbynode` information `json-automationrelevance` calls (Bug 1673985) r=marco,zeid The `backedoutbynode` data is relevant to callers of `json-automationrelevance`. Add the `backedoutbynode` information to the push, if it is available. This mimics the behaviour of other calls that have the `backedoutbynode` information, such as `json-rev`. Differential Revision: https://phabricator.services.mozilla.com/D95341

ansible: use stream clones in project repo reset playbook (Bug 1672877) r=zeid Currently when we reset a twig (project branch) on hgmo by running the Ansible playbook `hgmo-reset-twig.yml`, the repo is reset by creating a new empty repo on hgssh and locally pulling from `mozilla-central`. This creates an `hg-repo-init-2` message in vcsreplicator so the repo is created on hgweb, after which we run `hg replicatesync` to trigger a full sync of the repo. This full sync causes all hgweb nodes to pull from hgssh to gather the new data. This process is slow, and causes the vcsreplicator queue for the project repos to lag far behind. This triggers many alerts in #vcs on Slack, and can cause issues for others working on project branches. This commit updates the playbook to instead create a fresh clone of the repo using stream clones on both hgssh and hgweb, speeding up the process and working around the vcsreplicator queue lag. As a result of this change, we also run the `repo-permissions` script on hgweb and earlier in the reset process on hgssh. I used this script to resolve bug 1672465 without running into the aforementioned problems. Differential Revision: https://phabricator.services.mozilla.com/D94523

docs: update standalone doc with SSL cert info (Bug 1670031) r=dhouse,zeid Depends on D92970 Differential Revision: https://phabricator.services.mozilla.com/D92971

configwizard: update the SHA1 fingerprint (Bug 1670031) r=dhouse,zeid This change will need to be landed post-swap since older clients can't pin multiple SHA1 certs. Bug 1670034 tracks removal of this feature altogether, but for now it stays as-is. Differential Revision: https://phabricator.services.mozilla.com/D92970

ansible/hg-web: update pinned SSL cert in hgrc template (Bug 1670031) r=dhouse,zeid Updates the fingerprints in the hgweb hgrc file. Differential Revision: https://phabricator.services.mozilla.com/D92969

configwizard: add new sha256 fingerprint (Bug 1670031) r=dhouse,zeid Add the new certificate to `configwizard`, so users can run `mach vcs-setup` and have the new cert pinned in their hgrc. Differential Revision: https://phabricator.services.mozilla.com/D92968

ansible/test-hg-ssh: change moto hostname to 0.0.0.0 from localhost r=zeid Since the hostname is set to `localhost`, the service is only accessible from the `hgssh` docker container. Switching to `0.0.0.0` allows all containers on the Docker network to access the mock S3 service. Differential Revision: https://phabricator.services.mozilla.com/D91822

testing: make hgcluster runnable with docker env r?zeid Adds a command to `./run` to allow interacting with a local non-test hgcluster. Usage follows the same format as `./hgmo` (ie `./run hgmo start`). Differential Revision: https://phabricator.services.mozilla.com/D91820

hgwsgi: remove dead entries from `hgwsgi/releases/hgweb.config` (Bug 1667731) r=zeid Removes some entries for repositories that no longer exist on hgmo. The existance of these entries is causing an exception in core Mercurial code and causes https://hg.mozilla.org/releases to fail to render. The upstream issue will need to be addressed too, but this is a quick fix that we should address in the meantime. Differential Revision: https://phabricator.services.mozilla.com/D91607

firefoxtree: add compatibility branch so older versions of Mercurial still work (Bug 1666520) r=zeid `missingheads` was renamed to `ancestorsof` in newer versions of Mercurial, but the patch from 60d9824ff108c6 does not take the older variable name into account. Add a check for the attribute so both variable names work in the `prepushoutgoinghook`. Differential Revision: https://phabricator.services.mozilla.com/D91023

hgssh: upgrade Mercurial to 5.5.1 (Bug 1661734) r=zeid Depends on D89984 Differential Revision: https://phabricator.services.mozilla.com/D89985

hgweb: upgrade Mercurial to 5.5.1 (Bug 1661734) r=zeid Depends on D89983 Differential Revision: https://phabricator.services.mozilla.com/D89984

hgtemplates: update path to `python` to reflect Docker environment and update Mercurial version (Bug 1661734) r=zeid Fix the path to Python so it works in the Docker test environment, and update the Mercurial version to 5.5.1 to verify there are no outstanding changes our template patcher missed. Depends on D89982 Differential Revision: https://phabricator.services.mozilla.com/D89983