Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal
authorRicky Stewart <rstewart@mozilla.com>
Mon, 17 Aug 2020 17:21:02 +0000
changeset 544951 eff0a199fae6727caebd03b687824a398fe132ba
parent 544950 9fc8b1c30c66498706fe37e225146f4f19f2558b
child 544952 d3d9b27cdf360546b520e31c1d0513a8c98e7f62
push id37706
push usercsabou@mozilla.com
push dateMon, 17 Aug 2020 21:46:02 +0000
treeherdermozilla-central@508a0cc2f6d4 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersmhentges, ahal
bugs1656993, 1651424, 1654607, 1655781, 1654994
milestone81.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916
build/build_virtualenv_packages.txt
build/common_virtualenv_packages.txt
build/mach_bootstrap.py
build/mach_virtualenv_packages.txt
build/moz.configure/init.configure
build/sparse-profiles/mach
build/virtualenv_packages.txt
mach
moz.configure
python/mach/docs/driver.rst
python/mach/mach/main.py
python/mach/mach/test/test_commands.py
python/mozboot/mozboot/base.py
python/mozboot/mozboot/bootstrap.py
python/mozbuild/mozbuild/action/test_archive.py
python/mozbuild/mozbuild/base.py
python/mozbuild/mozbuild/configure/__init__.py
python/mozbuild/mozbuild/controller/building.py
python/mozbuild/mozbuild/mach_commands.py
python/mozbuild/mozbuild/sphinx.py
python/mozbuild/mozbuild/virtualenv.py
new file mode 100644
--- /dev/null
+++ b/build/build_virtualenv_packages.txt
@@ -0,0 +1,4 @@
+inherit-from-parent-environment
+packages.txt:build/common_virtualenv_packages.txt
+python3:mozilla.pth:third_party/python/glean_parser
+set-variable MOZBUILD_VIRTUALENV=1
rename from build/virtualenv_packages.txt
rename to build/common_virtualenv_packages.txt
--- a/build/virtualenv_packages.txt
+++ b/build/common_virtualenv_packages.txt
@@ -23,17 +23,16 @@ mozilla.pth:third_party/python/distro
 mozilla.pth:third_party/python/dlmanager
 mozilla.pth:third_party/python/ecdsa/src
 python2:mozilla.pth:third_party/python/enum34
 mozilla.pth:third_party/python/esprima
 mozilla.pth:third_party/python/fluent.migrate
 mozilla.pth:third_party/python/fluent.syntax
 mozilla.pth:third_party/python/funcsigs
 python2:mozilla.pth:third_party/python/futures
-in-virtualenv:python3:mozilla.pth:third_party/python/glean_parser
 mozilla.pth:third_party/python/importlib_metadata
 mozilla.pth:third_party/python/iso8601
 mozilla.pth:third_party/python/Jinja2/src
 mozilla.pth:third_party/python/jsonschema
 mozilla.pth:third_party/python/MarkupSafe/src
 mozilla.pth:third_party/python/mohawk
 mozilla.pth:third_party/python/more-itertools
 mozilla.pth:third_party/python/mozilla-version
--- a/build/mach_bootstrap.py
+++ b/build/mach_bootstrap.py
@@ -136,19 +136,16 @@ CATEGORIES = {
 }
 
 
 def search_path(mozilla_dir, packages_txt):
     with open(os.path.join(mozilla_dir, packages_txt)) as f:
         packages = [line.rstrip().split(':') for line in f]
 
     def handle_package(package):
-        if package[0] == 'in-virtualenv':
-            return
-
         if package[0] == 'optional':
             try:
                 for path in handle_package(package[1:]):
                     yield path
             except Exception:
                 pass
 
         if package[0] in ('windows', '!windows'):
@@ -195,19 +192,20 @@ def bootstrap(topsrcdir, mozilla_dir=Non
     # Global build system and mach state is stored in a central directory. By
     # default, this is ~/.mozbuild. However, it can be defined via an
     # environment variable. We detect first run (by lack of this directory
     # existing) and notify the user that it will be created. The logic for
     # creation is much simpler for the "advanced" environment variable use
     # case. For default behavior, we educate users and give them an opportunity
     # to react. We always exit after creating the directory because users don't
     # like surprises.
-    sys.path[0:0] = [os.path.join(mozilla_dir, path)
-                     for path in search_path(mozilla_dir,
-                                             'build/virtualenv_packages.txt')]
+    sys.path[0:0] = [
+        os.path.join(mozilla_dir, path)
+        for path in search_path(mozilla_dir,
+                                'build/mach_virtualenv_packages.txt')]
     import mach.base
     import mach.main
     from mach.util import setenv
     from mozboot.util import get_state_dir
 
     # Set a reasonable limit to the number of open files.
     #
     # Some linux systems set `ulimit -n` to a very high number, which works
new file mode 100644
--- /dev/null
+++ b/build/mach_virtualenv_packages.txt
@@ -0,0 +1,2 @@
+packages.txt:build/common_virtualenv_packages.txt
+set-variable MACH_VIRTUALENV=1
--- a/build/moz.configure/init.configure
+++ b/build/moz.configure/init.configure
@@ -301,27 +301,30 @@ def virtualenv_python3(env_python, build
     if topobjdir.endswith('/js/src'):
         topobjdir = topobjdir[:-7]
 
     virtualenvs_root = os.path.join(topobjdir, '_virtualenvs')
     with LineIO(lambda l: log.info(l), 'replace') as out:
         manager = VirtualenvManager(
             topsrcdir,
             os.path.join(virtualenvs_root, 'init_py3'), out,
-            os.path.join(topsrcdir, 'build', 'virtualenv_packages.txt'))
+            os.path.join(topsrcdir, 'build', 'build_virtualenv_packages.txt'))
 
     # If we're not in the virtualenv, we need to update the path to include some
     # necessary modules for find_program.
-    if normsep(sys.executable) != normsep(manager.python_path):
+    try:
+        import mozfile
+    except ImportError:
         sys.path.insert(
             0, os.path.join(topsrcdir, 'testing', 'mozbase', 'mozfile'))
         sys.path.insert(
             0, os.path.join(topsrcdir, 'third_party', 'python', 'backports'))
     else:
-        python = sys.executable
+        if 'MOZBUILD_VIRTUALENV' in os.environ or not python:
+            python = sys.executable
 
     # If we know the Python executable the caller is asking for then verify its
     # version. If the caller did not ask for a specific executable then find
     # a reasonable default.
     if python:
         found_python = find_program(python)
         if not found_python:
             die('The PYTHON3 environment variable does not contain '
--- a/build/sparse-profiles/mach
+++ b/build/sparse-profiles/mach
@@ -2,17 +2,19 @@
 # Various mach commands call config.guess to resolve the default objdir name.
 path:build/autoconf/config.guess
 path:build/autoconf/config.sub
 path:build/moz.configure/checks.configure
 path:build/moz.configure/init.configure
 path:build/moz.configure/util.configure
 # Used for bootstrapping the mach driver.
 path:build/mach_bootstrap.py
-path:build/virtualenv_packages.txt
+path:build/build_virtualenv_packages.txt
+path:build/common_virtualenv_packages.txt
+path:build/mach_virtualenv_packages.txt
 path:mach
 # Various dependencies. There is room to trim fat, especially in
 # third_party/python.
 path:python/
 path:testing/mozbase/
 path:third_party/python/
 # certifi is needed for Sentry
 path:testing/web-platform/tests/tools/third_party/certifi
--- a/mach
+++ b/mach
@@ -5,16 +5,17 @@
 
 # The beginning of this script is both valid POSIX shell and valid Python,
 # such that the script starts with the shell and is reexecuted with
 # the right Python.
 
 # Embeds a shell script inside a Python triple quote. This pattern is valid
 # shell because `''':'`, `':'` and `:` are all equivalent, and `:` is a no-op.
 ''':'
+# Commands that are to be run with Python 2.
 py2commands="
     android
     awsy-test
     check-spidermonkey
     cramtest
     crashtest
     devtools-css-db
     firefox-ui-functional
@@ -45,78 +46,122 @@ py2commands="
     wpt-metadata-merge
     wpt-metadata-summary
     wpt-serve
     wpt-test-paths
     wpt-unittest
     wpt-update
 "
 
+# Commands that are to be run with the system Python 3 instead of the
+# virtualenv.
+nativecmds="
+    bootstrap
+    create-mach-environment
+"
+
 run_py() {
-    # Try to run a specific Python interpreter. Fall back to the system
-    # default Python if the specific interpreter couldn't be found.
+    # Try to run a specific Python interpreter.
     py_executable="$1"
     shift
     if which "$py_executable" > /dev/null
     then
         exec "$py_executable" "$0" "$@"
-    elif [ "$py_executable" = "python2.7" ]; then
-        exec python "$0" "$@"
     else
         echo "This mach command requires $py_executable, which wasn't found on the system!"
+        case "$py_executable" in
+            python2.7|python3) ;;
+            *)
+                echo "Consider running 'mach bootstrap' or 'mach create-mach-environment' to create the mach virtualenvs, or set MACH_USE_SYSTEM_PYTHON to use the system Python installation over a virtualenv."
+                ;;
+        esac
         exit 1
     fi
 }
 
-first_arg=$1
-if [ "$first_arg" = "help" ]; then
-    # When running `./mach help <command>`, the correct Python for <command>
-    # needs to be used.
-    first_arg=$2
-elif [ "$first_arg" = "mach-completion" ]; then
-    # When running `./mach mach-completion /path/to/mach <command>`, the
-    # correct Python for <command> needs to be used.
-    first_arg=$3
+get_command() {
+    # Parse the name of the mach command out of the arguments. This is necessary
+    # in the presence of global mach arguments that come before the name of the
+    # command, e.g. `mach -v build`. We dispatch to the correct Python
+    # interpreter depending on the command.
+    while true; do
+    case $1 in
+        -v|--verbose) shift;;
+        -l|--log-file)
+            if [ "$#" -lt 2 ]
+            then
+                echo
+                break
+            else
+                shift 2
+            fi
+            ;;
+        --log-interval) shift;;
+        --log-no-times) shift;;
+        -h) shift;;
+        --debug-command) shift;;
+        --settings)
+            if [ "$#" -lt 2 ]
+            then
+                echo
+                break
+            else
+                shift 2
+            fi
+            ;;
+        # When running `./mach help <command>`, the correct Python for <command>
+        # needs to be used.
+        help) echo $2; break;;
+        # When running `./mach mach-completion /path/to/mach <command>`, the
+        # correct Python for <command> needs to be used.
+        mach-completion) echo $3; break;;
+        "") echo; break;;
+        *) echo $1; break;;
+    esac
+    done
+}
+
+state_dir=${MOZBUILD_STATE_PATH:-~/.mozbuild}
+command=$(get_command "$@")
+
+# If MACH_USE_SYSTEM_PYTHON or MOZ_AUTOMATION are set, always use the
+# python{2.7,3} executables and not the virtualenv locations.
+if [ -z ${MACH_USE_SYSTEM_PYTHON} ] && [ -z ${MOZ_AUTOMATION} ]
+then
+    case "$OSTYPE" in
+        cygwin|msys|win32) bin_path=Scripts;;
+        *) bin_path=bin;;
+    esac
+    py2executable=$state_dir/_virtualenvs/mach_py2/$bin_path/python
+    py3executable=$state_dir/_virtualenvs/mach/$bin_path/python
+else
+    py2executable=python2.7
+    py3executable=python3
 fi
 
-if [ -z "$first_arg" ]; then
-    # User ran `./mach` or `./mach help`, use Python 3.
-    run_py python3 "$@"
-fi
-
-case "${first_arg}" in
-    "-"*)
-        # We have global arguments which are tricky to parse from this shell
-        # script. So invoke `mach` with a special --print-command argument to
-        # return the name of the command. This adds extra overhead when using
-        # global arguments, but global arguments are an edge case and this hack
-        # is only needed temporarily for the Python 3 migration. We use Python
-        # 2.7 because using Python 3 hits this error in build tasks:
-        # https://searchfox.org/mozilla-central/rev/c7e8bc4996f9/build/moz.configure/init.configure#319
-        command=`run_py python2.7 --print-command "$@" | tail -n1`
-        ;;
-    *)
-        # In the common case, the first argument is the command.
-        command=${first_arg};
+# Check whether we need to run with the native Python 3 interpreter.
+case " $(echo $nativecmds) " in
+    *\ $command\ *)
+        run_py python3 "$@"
         ;;
 esac
 
 # Check for the mach subcommand in the Python 2 commands list and run it
 # with the correct interpreter.
 case " $(echo $py2commands) " in
     *\ $command\ *)
-        run_py python2.7 "$@"
+        run_py "$py2executable" "$@"
         ;;
     *)
-        run_py python3 "$@"
+        run_py "$py3executable" "$@"
         ;;
 esac
 
 # Run Python 3 for everything else.
-run_py python3 "$@"
+run_py "$py3executable" "$@"
 '''
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 import os
 import sys
 
 def ancestors(path):
--- a/moz.configure
+++ b/moz.configure
@@ -764,17 +764,19 @@ def config_status_deps(build_env, build_
         os.path.join(topsrcdir, 'configure'),
         os.path.join(topsrcdir, 'js', 'src', 'configure'),
         os.path.join(topsrcdir, 'configure.in'),
         os.path.join(topsrcdir, 'js', 'src', 'configure.in'),
         os.path.join(topsrcdir, 'nsprpub', 'configure'),
         os.path.join(topsrcdir, 'config', 'milestone.txt'),
         os.path.join(topsrcdir, 'browser', 'config', 'version.txt'),
         os.path.join(topsrcdir, 'browser', 'config', 'version_display.txt'),
-        os.path.join(topsrcdir, 'build', 'virtualenv_packages.txt'),
+        os.path.join(topsrcdir, 'build', 'build_virtualenv_packages.txt'),
+        os.path.join(topsrcdir, 'build', 'common_virtualenv_packages.txt'),
+        os.path.join(topsrcdir, 'build', 'mach_virtualenv_packages.txt'),
         os.path.join(topsrcdir, 'python', 'mozbuild', 'mozbuild', 'virtualenv.py'),
         os.path.join(topsrcdir, 'testing', 'mozbase', 'packages.txt'),
         os.path.join(topsrcdir, 'aclocal.m4'),
         os.path.join(topsrcdir, 'old-configure.in'),
         os.path.join(topsrcdir, 'js', 'src', 'aclocal.m4'),
         os.path.join(topsrcdir, 'js', 'src', 'old-configure.in'),
     ] + glob.glob(os.path.join(topsrcdir, 'build', 'autoconf', '*.m4'))
 
--- a/python/mach/docs/driver.rst
+++ b/python/mach/docs/driver.rst
@@ -25,27 +25,8 @@ providers. e.g.:
 See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
 for more information on creating an entry point. To search for entry
 point plugins, you can call
 :py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.:
 
 .. code-block:: python
 
    mach.load_commands_from_entry_point("mach.external.providers")
-
-Adding Global Arguments
-=======================
-
-Arguments to mach commands are usually command-specific. However,
-mach ships with a handful of global arguments that apply to all
-commands.
-
-It is possible to extend the list of global arguments. In your
-*mach driver*, simply call
-:py:meth:`mach.main.Mach.add_global_argument`. e.g.:
-
-.. code-block:: python
-
-   mach = mach.main.Mach(os.getcwd())
-
-   # Will allow --example to be specified on every mach command.
-   mach.add_global_argument('--example', action='store_true',
-       help='Demonstrate an example global argument.')
--- a/python/mach/mach/main.py
+++ b/python/mach/mach/main.py
@@ -214,27 +214,18 @@ To see more help for a specific command,
         self.logger = logging.getLogger(__name__)
         self.settings = ConfigSettings()
         self.settings_paths = []
 
         if 'MACHRC' in os.environ:
             self.settings_paths.append(os.environ['MACHRC'])
 
         self.log_manager.register_structured_logger(self.logger)
-        self.global_arguments = []
         self.populate_context_handler = None
 
-    def add_global_argument(self, *args, **kwargs):
-        """Register a global argument with the argument parser.
-
-        Arguments are proxied to ArgumentParser.add_argument()
-        """
-
-        self.global_arguments.append((args, kwargs))
-
     def load_commands_from_directory(self, path):
         """Scan for mach commands from modules in a directory.
 
         This takes a path to a directory, loads the .py files in it, and
         registers and found mach command providers with this mach instance.
         """
         for f in sorted(os.listdir(path)):
             if not f.endswith('.py') or f == '__init__.py':
@@ -412,28 +403,17 @@ To see more help for a specific command,
             # We don't register the usage until here because if it is globally
             # registered, argparse always prints it. This is not desired when
             # running with --help.
             parser.usage = Mach.USAGE
             parser.print_usage()
             return 0
 
         try:
-            try:
-                args = parser.parse_args(argv)
-            except NoCommandError as e:
-                if e.namespace.print_command:
-                    context.get_command = True
-                    args = parser.parse_args(e.namespace.print_command)
-                    if args.command == 'mach-completion':
-                        args = parser.parse_args(e.namespace.print_command[2:])
-                    print(args.command)
-                    return 0
-                else:
-                    raise
+            args = parser.parse_args(argv)
         except NoCommandError:
             print(NO_COMMAND_ERROR)
             return 1
         except UnknownCommandError as e:
             suggestion_message = SUGGESTED_COMMANDS_MESSAGE % (
                 e.verb, ', '.join(e.suggested_commands)) if e.suggested_commands else ''
             print(UNKNOWN_COMMAND_ERROR %
                   (e.verb, e.command, suggestion_message))
@@ -590,16 +570,18 @@ To see more help for a specific command,
 
     def get_argument_parser(self, context):
         """Returns an argument parser for the command-line interface."""
 
         parser = ArgumentParser(add_help=False,
                                 usage='%(prog)s [global arguments] '
                                 'command [command arguments]')
 
+        # WARNING!!! If you add a global argument here, also add it to the
+        # global argument handling in the top-level `mach` script.
         # Order is important here as it dictates the order the auto-generated
         # help messages are printed.
         global_group = parser.add_argument_group('Global Arguments')
 
         global_group.add_argument('-v', '--verbose', dest='verbose',
                                   action='store_true', default=False,
                                   help='Print verbose output.')
         global_group.add_argument('-l', '--log-file', dest='logfile',
@@ -621,20 +603,15 @@ To see more help for a specific command,
         global_group.add_argument('-h', '--help', dest='help',
                                   action='store_true', default=False,
                                   help='Show this help message.')
         global_group.add_argument('--debug-command', action='store_true',
                                   help='Start a Python debugger when command is dispatched.')
         global_group.add_argument('--settings', dest='settings_file',
                                   metavar='FILENAME', default=None,
                                   help='Path to settings file.')
-        global_group.add_argument('--print-command', nargs=argparse.REMAINDER,
-                                  help=argparse.SUPPRESS)
-
-        for args, kwargs in self.global_arguments:
-            global_group.add_argument(*args, **kwargs)
 
         # We need to be last because CommandAction swallows all remaining
         # arguments and argparse parses arguments in the order they were added.
         parser.add_argument('command', action=CommandAction,
                             registrar=Registrar, context=context)
 
         return parser
--- a/python/mach/mach/test/test_commands.py
+++ b/python/mach/mach/test/test_commands.py
@@ -51,30 +51,11 @@ class TestCommands(TestBase):
         # 'cmd_f' as a prefix, the completion script will handle this case
         # properly.
         assert stdout == self.format(self.all_commands)
 
         result, stdout, stderr = self._run_mach(['mach-completion', 'cmd_foo'])
         assert result == 0
         assert stdout == self.format(['help', '--arg'])
 
-    def test_print_command(self):
-        result, stdout, stderr = self._run_mach(['--print-command', 'cmd_foo', '-flag'])
-        assert result == 0
-        assert stdout == 'cmd_foo\n'
-
-        result, stdout, stderr = self._run_mach(['--print-command', '-v', 'cmd_foo', '-flag'])
-        assert result == 0
-        assert stdout == 'cmd_foo\n'
-
-        result, stdout, stderr = self._run_mach(
-            ['--print-command', 'mach-completion', 'mach', 'cmd_foo', '-flag'])
-        assert result == 0
-        assert stdout == 'cmd_foo\n'
-
-        result, stdout, stderr = self._run_mach(
-            ['--print-command', 'mach-completion', 'mach', '-v', 'cmd_foo', '-flag'])
-        assert result == 0
-        assert stdout == 'cmd_foo\n'
-
 
 if __name__ == '__main__':
     main()
--- a/python/mozboot/mozboot/base.py
+++ b/python/mozboot/mozboot/base.py
@@ -248,16 +248,24 @@ class BaseBootstrapper(object):
 
         GeckoView/Firefox for Android Artifact Mode needs an application and an ABI set,
         and it needs paths to the Android SDK.
         '''
         raise NotImplementedError(
             '%s does not yet implement generate_mobile_android_artifact_mode_mozconfig()'
             % __name__)
 
+    def ensure_mach_environment(self, checkout_root):
+        if checkout_root:
+            mach_binary = os.path.abspath(os.path.join(checkout_root, 'mach'))
+            if not os.path.exists(mach_binary):
+                raise ValueError('mach not found at %s' % mach_binary)
+            cmd = [sys.executable, mach_binary, 'create-mach-environment']
+            subprocess.check_call(cmd, cwd=checkout_root)
+
     def ensure_clang_static_analysis_package(self, state_dir, checkout_root):
         '''
         Install the clang static analysis package
         '''
         raise NotImplementedError(
             '%s does not yet implement ensure_clang_static_analysis_package()'
             % __name__)
 
--- a/python/mozboot/mozboot/bootstrap.py
+++ b/python/mozboot/mozboot/bootstrap.py
@@ -12,18 +12,16 @@ import re
 import sys
 import subprocess
 import time
 from distutils.version import LooseVersion
 
 # NOTE: This script is intended to be run with a vanilla Python install.  We
 # have to rely on the standard library instead of Python 2+3 helpers like
 # the six module.
-from subprocess import CalledProcessError
-
 if sys.version_info < (3,):
     from ConfigParser import (
         Error as ConfigParserError,
         RawConfigParser,
     )
     input = raw_input  # noqa
 else:
     from configparser import (
@@ -486,16 +484,17 @@ class Bootstrapper(object):
             have_clone = bool(checkout_type)
 
             if state_dir_available:
                 self.check_telemetry_opt_in(state_dir)
             self.maybe_install_private_packages_or_exit(state_dir,
                                                         state_dir_available,
                                                         have_clone,
                                                         checkout_root)
+            self.instance.ensure_mach_environment(checkout_root)
             self._output_mozconfig(application)
             sys.exit(0)
 
         self.instance.install_system_packages()
 
         # Like 'install_browser_packages' or 'install_mobile_android_packages'.
         getattr(self.instance, 'install_%s_packages' % application)()
 
@@ -568,25 +567,21 @@ class Bootstrapper(object):
                 git = self.instance.which('git')
                 watchman = self.instance.which('watchman')
                 have_clone = git_clone_firefox(git, dest, watchman)
                 checkout_root = dest
 
         if not have_clone:
             print(SOURCE_ADVERTISE)
 
-        if state_dir_available:
-            is_telemetry_enabled = self.check_telemetry_opt_in(state_dir)
-            if is_telemetry_enabled:
-                _install_glean()
-
         self.maybe_install_private_packages_or_exit(state_dir,
                                                     state_dir_available,
                                                     have_clone,
                                                     checkout_root)
+        self.instance.ensure_mach_environment(checkout_root)
 
         print(self.finished % name)
         if not (self.instance.which('rustc') and self.instance._parse_version('rustc')
                 >= MODERN_RUST_VERSION):
             print("To build %s, please restart the shell (Start a new terminal window)" % name)
 
         if not self.instance.which("moz-phab"):
             print(MOZ_PHAB_ADVERTISE)
@@ -877,38 +872,16 @@ def git_clone_firefox(git, dest, watchma
     except Exception as e:
         print(e)
         return False
 
     print('Firefox source code available at %s' % dest)
     return True
 
 
-def _install_glean():
-    """Installs glean to the current python environment.
-
-    If the current python instance is a virtualenv, then glean is installed
-    directly.
-    If not, then glean is installed to the Python user install directory.
-    Upgrades glean if it's out-of-date.
-    """
-    pip_call = [sys.executable, '-m', 'pip', 'install', 'glean_sdk~=31.5.0']
-    if not os.environ.get('VIRTUAL_ENV'):
-        # If the user is already using a virtual environment before they invoked
-        # `mach bootstrap`, then we shouldn't add the "--user" flag. This is because
-        # virtual environments don't support the flags since they don't have a
-        # separate "user install directory".
-        pip_call.append('--user')
-
-    try:
-        subprocess.check_output(pip_call)
-    except CalledProcessError:
-        print("Failed to install glean, telemetry will not be gathered")
-
-
 def _warn_if_risky_revision(path):
     # Warn the user if they're trying to bootstrap from an obviously old
     # version of tree as reported by the version control system (a month in
     # this case). This is an approximate calculation but is probably good
     # enough for our purposes.
     NUM_SECONDS_IN_MONTH = 60 * 60 * 24 * 30
     from mozversioncontrol import get_repository_object
     repo = get_repository_object(path)
--- a/python/mozbuild/mozbuild/action/test_archive.py
+++ b/python/mozbuild/mozbuild/action/test_archive.py
@@ -501,17 +501,25 @@ ARCHIVE_FILES = {
             'pattern': 'python/**'
         },
         {
             'source': buildconfig.topsrcdir,
             'pattern': 'build/mach_bootstrap.py'
         },
         {
             'source': buildconfig.topsrcdir,
-            'pattern': 'build/virtualenv_packages.txt'
+            'pattern': 'build/build_virtualenv_packages.txt'
+        },
+        {
+            'source': buildconfig.topsrcdir,
+            'pattern': 'build/common_virtualenv_packages.txt'
+        },
+        {
+            'source': buildconfig.topsrcdir,
+            'pattern': 'build/mach_virtualenv_packages.txt'
         },
         {
             'source': buildconfig.topsrcdir,
             'pattern': 'mach/**'
         },
         {
             'source': buildconfig.topsrcdir,
             'pattern': 'testing/web-platform/tests/tools/third_party/certifi/**'
--- a/python/mozbuild/mozbuild/base.py
+++ b/python/mozbuild/mozbuild/base.py
@@ -277,17 +277,18 @@ class MozbuildObject(ProcessExecutionMix
         from .virtualenv import VirtualenvManager
 
         if self._virtualenv_manager is None:
             self._virtualenv_manager = VirtualenvManager(
                 self.topsrcdir,
                 os.path.join(self.topobjdir, '_virtualenvs',
                              self._virtualenv_name),
                 sys.stdout,
-                os.path.join(self.topsrcdir, 'build', 'virtualenv_packages.txt')
+                os.path.join(self.topsrcdir, 'build',
+                             'build_virtualenv_packages.txt')
                 )
 
         return self._virtualenv_manager
 
     @staticmethod
     @memoize
     def get_mozconfig_and_target(topsrcdir, path, env_mozconfig):
         # env_mozconfig is only useful for unittests, which change the value of
--- a/python/mozbuild/mozbuild/configure/__init__.py
+++ b/python/mozbuild/mozbuild/configure/__init__.py
@@ -287,17 +287,17 @@ class ConfigureSandbox(dict):
 
     # The default set of builtins. We expose unicode as str to make sandboxed
     # files more python3-ready.
     BUILTINS = ReadOnlyDict({
         b: getattr(__builtin__, b, None)
         for b in ('None', 'False', 'True', 'int', 'bool', 'any', 'all', 'len',
                   'list', 'tuple', 'set', 'dict', 'isinstance', 'getattr',
                   'hasattr', 'enumerate', 'range', 'zip', 'AssertionError',
-                  '__build_class__',  # will be None on py2
+                  'ImportError', '__build_class__',  # will be None on py2
                   )
     }, __import__=forbidden_import, str=six.text_type)
 
     # Expose a limited set of functions from os.path
     OS = ReadOnlyNamespace(path=ReadOnlyNamespace(**{
         k: getattr(mozpath, k, getattr(os.path, k))
         for k in ('abspath', 'basename', 'dirname', 'isabs', 'join',
                   'normcase', 'normpath', 'realpath', 'relpath')
--- a/python/mozbuild/mozbuild/controller/building.py
+++ b/python/mozbuild/mozbuild/controller/building.py
@@ -1506,16 +1506,22 @@ class BuildDriver(MozbuildObject):
 
         if mozconfig_make_lines:
             self.log(logging.WARNING, 'mozconfig_content', {
                 'path': mozconfig['path'],
                 'content': '\n    '.join(mozconfig_make_lines),
             }, 'Adding make options from {path}\n    {content}')
 
         append_env['OBJDIR'] = mozpath.normsep(self.topobjdir)
+        if (mozpath.normpath(os.path.dirname(sys.executable)) not in
+            [mozpath.normpath(s) for s in
+             os.environ['PATH'].split(os.pathsep)]):
+            append_env['PATH'] = (
+                os.path.dirname(sys.executable) + os.pathsep +
+                os.environ['PATH'])
 
         return self._run_make(srcdir=True,
                               filename='client.mk',
                               allow_parallel=False,
                               ensure_exit_code=False,
                               print_directory=False,
                               target=target,
                               line_handler=line_handler,
--- a/python/mozbuild/mozbuild/mach_commands.py
+++ b/python/mozbuild/mozbuild/mach_commands.py
@@ -1478,8 +1478,60 @@ class L10NCommands(MachCommandBase):
                 [mozpath.join(self.topsrcdir, 'mach'), 'android',
                  'archive-geckoview'],
                 append_env=append_env,
                 pass_thru=True,
                 ensure_exit_code=True,
                 cwd=mozpath.join(self.topsrcdir))
 
         return 0
+
+
+@CommandProvider
+class CreateMachEnvironment(MachCommandBase):
+    """Create the mach virtualenvs."""
+
+    @Command('create-mach-environment', category='devenv',
+             description=(
+                 'Create the `mach` virtualenvs. If executed with python3 (the '
+                 'default when entering from `mach`), create both a python3 '
+                 'and python2.7 virtualenv. If executed with python2, only '
+                 'create the python2.7 virtualenv.'))
+    def create_mach_environment(self):
+        from mozboot.util import get_state_dir
+        from mozbuild.pythonutil import find_python2_executable
+        from mozbuild.virtualenv import VirtualenvManager
+        from six import PY3
+
+        state_dir = get_state_dir()
+        virtualenv_path = os.path.join(state_dir, '_virtualenvs',
+                                       'mach' if PY3 else 'mach_py2')
+        if sys.executable.startswith(virtualenv_path):
+            print('You can only create a mach environment with the system '
+                  'Python. Re-run this `mach` command with the system Python.',
+                  file=sys.stderr)
+            return 1
+
+        manager = VirtualenvManager(
+            self.topsrcdir, virtualenv_path, sys.stdout,
+            os.path.join(self.topsrcdir, 'build',
+                         'mach_virtualenv_packages.txt'),
+            populate_local_paths=False)
+        manager.build(sys.executable)
+
+        manager.install_pip_package('zstandard>=0.9.0,<=0.13.0')
+
+        if PY3:
+            manager.install_pip_package('glean_sdk~=31.5.0')
+            print('Python 3 mach environment created.')
+            python2, _ = find_python2_executable()
+            if not python2:
+                print('WARNING! Could not find a Python 2 executable to create '
+                      'a Python 2 virtualenv', file=sys.stderr)
+                return 0
+            ret = subprocess.call([
+                python2, os.path.join(self.topsrcdir, 'mach'),
+                'create-mach-environment'])
+            if ret:
+                print('WARNING! Failed to create a Python 2 mach environment.',
+                      file=sys.stderr)
+        else:
+            print('Python 2 mach environment created.')
--- a/python/mozbuild/mozbuild/sphinx.py
+++ b/python/mozbuild/mozbuild/sphinx.py
@@ -188,14 +188,13 @@ def setup(app):
     # Here, we invoke our custom code for staging/generating all our
     # documentation.
     manager.generate_docs(app)
     app.srcdir = manager.staging_dir
 
     # We need to adjust sys.path in order for Python API docs to get generated
     # properly. We leverage the in-tree virtualenv for this.
     topsrcdir = manager.topsrcdir
-    ve = VirtualenvManager(topsrcdir,
-                           os.path.join(app.outdir, '_venv'),
-                           sys.stderr,
-                           os.path.join(topsrcdir, 'build', 'virtualenv_packages.txt'))
+    ve = VirtualenvManager(
+        topsrcdir, os.path.join(app.outdir, '_venv'), sys.stderr,
+        os.path.join(topsrcdir, 'build', 'build_virtualenv_packages.txt'))
     ve.ensure()
     ve.activate()
--- a/python/mozbuild/mozbuild/virtualenv.py
+++ b/python/mozbuild/mozbuild/virtualenv.py
@@ -2,16 +2,17 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 # This file contains code for populating the virtualenv environment for
 # Mozilla's build system. It is typically called as part of configure.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
+import argparse
 import os
 import platform
 import shutil
 import subprocess
 import sys
 
 IS_NATIVE_WIN = (sys.platform == 'win32' and os.sep == '\\')
 IS_CYGWIN = (sys.platform == 'cygwin')
@@ -30,20 +31,51 @@ Run |mach bootstrap| to ensure your syst
 If you still receive this error, your shell environment is likely detecting
 another Python version. Ensure a modern Python can be found in the paths
 defined by the $PATH environment variable and try again.
 '''.lstrip()
 
 here = os.path.abspath(os.path.dirname(__file__))
 
 
+# We can't import six.ensure_binary() or six.ensure_text() because this module
+# has to run stand-alone.  Instead we'll implement an abbreviated version of the
+# checks it does.
+if PY3:
+    text_type = str
+    binary_type = bytes
+else:
+    text_type = unicode
+    binary_type = str
+
+
+def ensure_binary(s, encoding='utf-8'):
+    if isinstance(s, text_type):
+        return s.encode(encoding, errors='strict')
+    elif isinstance(s, binary_type):
+        return s
+    else:
+        raise TypeError("not expecting type '%s'" % type(s))
+
+
+def ensure_text(s, encoding='utf-8'):
+    if isinstance(s, binary_type):
+        return s.decode(encoding, errors='strict')
+    elif isinstance(s, text_type):
+        return s
+    else:
+        raise TypeError("not expecting type '%s'" % type(s))
+
+
 class VirtualenvManager(object):
     """Contains logic for managing virtualenvs for building the tree."""
 
-    def __init__(self, topsrcdir, virtualenv_path, log_handle, manifest_path):
+    def __init__(
+            self, topsrcdir, virtualenv_path, log_handle, manifest_path,
+            parent_site_dir=None, populate_local_paths=True):
         """Create a new manager.
 
         Each manager is associated with a source directory, a path where you
         want the virtualenv to be created, and a handle to write output to.
         """
         assert os.path.isabs(
             manifest_path), "manifest_path must be an absolute path: %s" % (manifest_path)
         self.topsrcdir = topsrcdir
@@ -52,16 +84,21 @@ class VirtualenvManager(object):
         # Record the Python executable that was used to create the Virtualenv
         # so we can check this against sys.executable when verifying the
         # integrity of the virtualenv.
         self.exe_info_path = os.path.join(self.virtualenv_root,
                                           'python_exe.txt')
 
         self.log_handle = log_handle
         self.manifest_path = manifest_path
+        self.parent_site_dir = parent_site_dir
+        if not self.parent_site_dir:
+            import distutils.sysconfig
+            self.parent_site_dir = distutils.sysconfig.get_python_lib()
+        self.populate_local_paths = populate_local_paths
 
     @property
     def virtualenv_script_path(self):
         """Path to virtualenv's own populator script."""
         return os.path.join(self.topsrcdir, 'third_party', 'python',
                             'virtualenv', 'virtualenv.py')
 
     @property
@@ -272,35 +309,57 @@ class VirtualenvManager(object):
             on non-Windows systems.
 
         python3 -- This denotes that the action should only be taken when run
             on Python 3.
 
         python2 -- This denotes that the action should only be taken when run
             on python 2.
 
-        in-virtualenv -- This denotes that the action should only be taken when
-            constructing a virtualenv (and not when bootstrapping a `mach`
-            action).
+        inherit-from-parent-environment -- This denotes that we should add the
+            configured site directory of the "parent" to the virtualenv's list
+            of site directories. This can be specified on the command line as
+            --parent-site-dir or passed in the constructor of this class. This
+            defaults to the site-packages directory of the current Python
+            interpreter if not provided.
+
+        set-variable -- Set the given environment variable; e.g.
+            `set-variable FOO=1`.
 
         Note that the Python interpreter running this function should be the
         one from the virtualenv. If it is the system Python or if the
         environment is not configured properly, packages could be installed
         into the wrong place. This is how virtualenv's work.
         """
         import distutils.sysconfig
 
         packages = self.packages()
         python_lib = distutils.sysconfig.get_python_lib()
+        sitecustomize = open(
+            os.path.join(os.path.dirname(os.__file__), 'sitecustomize.py'),
+            mode='w')
 
         def handle_package(package):
-            if package[0] == 'in-virtualenv':
-                assert len(package) >= 2
-                package = package[1:]
-                # Continue processing normally.
+            if package[0] == 'inherit-from-parent-environment':
+                assert len(package) == 1
+                sitecustomize.write(
+                    'import site\n'
+                    "site.addsitedir(%s)\n" % repr(self.parent_site_dir))
+                return True
+
+            if package[0].startswith('set-variable '):
+                assert len(package) == 1
+                assignment = package[0][len('set-variable '):].strip()
+                var, val = assignment.split('=', 1)
+                var = var if PY3 else ensure_binary(var)
+                val = val if PY3 else ensure_binary(val)
+                sitecustomize.write(
+                    'import os\n'
+                    "os.environ[%s] = %s\n" % (repr(var), repr(val)))
+                return True
 
             if package[0] == 'setup.py':
                 assert len(package) >= 2
 
                 self.call_setup(os.path.join(self.topsrcdir, package[1]),
                                 package[2:])
 
                 return True
@@ -315,27 +374,30 @@ class VirtualenvManager(object):
 
                 return True
 
             if package[0] == 'packages.txt':
                 assert len(package) == 2
 
                 src = os.path.join(self.topsrcdir, package[1])
                 assert os.path.isfile(src), "'%s' does not exist" % src
-                submanager = VirtualenvManager(self.topsrcdir,
-                                               self.virtualenv_root,
-                                               self.log_handle,
-                                               src)
+                submanager = VirtualenvManager(
+                    self.topsrcdir, self.virtualenv_root, self.log_handle, src,
+                    parent_site_dir=self.parent_site_dir,
+                    populate_local_paths=self.populate_local_paths)
                 submanager.populate()
 
                 return True
 
             if package[0].endswith('.pth'):
                 assert len(package) == 2
 
+                if not self.populate_local_paths:
+                    return True
+
                 path = os.path.join(self.topsrcdir, package[1])
 
                 with open(os.path.join(python_lib, package[0]), 'a') as f:
                     # This path is relative to the .pth file.  Using a
                     # relative path allows the srcdir/objdir combination
                     # to be moved around (as long as the paths relative to
                     # each other remain the same).
                     f.write("%s\n" % os.path.relpath(path, python_lib))
@@ -397,26 +459,25 @@ class VirtualenvManager(object):
                     continue
 
                 old_env_variables[k] = os.environ[k]
                 del os.environ[k]
 
             for package in packages:
                 handle_package(package)
 
-            sitecustomize = os.path.join(
-                os.path.dirname(os.__file__), 'sitecustomize.py')
-            with open(sitecustomize, 'w') as f:
-                f.write(
-                    '# Importing mach_bootstrap has the side effect of\n'
-                    '# installing an import hook\n'
-                    'import mach_bootstrap\n'
-                )
+            sitecustomize.write(
+                '# Importing mach_bootstrap has the side effect of\n'
+                '# installing an import hook\n'
+                'import mach_bootstrap\n'
+            )
 
         finally:
+            sitecustomize.close()
+
             os.environ.pop('MACOSX_DEPLOYMENT_TARGET', None)
 
             if old_target is not None:
                 os.environ['MACOSX_DEPLOYMENT_TARGET'] = old_target
 
             os.environ.update(old_env_variables)
 
     def call_setup(self, directory, arguments):
@@ -442,16 +503,17 @@ class VirtualenvManager(object):
 
             raise Exception('Error installing package: %s' % directory)
 
     def build(self, python):
         """Build a virtualenv per tree conventions.
 
         This returns the path of the created virtualenv.
         """
+        import distutils
 
         self.create(python)
 
         # We need to populate the virtualenv using the Python executable in
         # the virtualenv for paths to be proper.
 
         # If this module was run from Python 2 then the __file__ attribute may
         # point to a Python 2 .pyc file. If we are generating a Python 3
@@ -461,17 +523,20 @@ class VirtualenvManager(object):
             thismodule = __file__[:-1]
         else:
             thismodule = __file__
 
         # __PYVENV_LAUNCHER__ confuses pip about the python interpreter
         # See https://bugzilla.mozilla.org/show_bug.cgi?id=1635481
         os.environ.pop('__PYVENV_LAUNCHER__', None)
         args = [self.python_path, thismodule, 'populate', self.topsrcdir,
-                self.virtualenv_root, self.manifest_path]
+                self.virtualenv_root, self.manifest_path, '--parent-site-dir',
+                distutils.sysconfig.get_python_lib()]
+        if self.populate_local_paths:
+            args.append('--populate-local-paths')
 
         result = self._log_process_output(args, cwd=self.topsrcdir)
 
         if result != 0:
             raise Exception('Error populating virtualenv.')
 
         os.utime(self.activate_path, None)
 
@@ -481,36 +546,42 @@ class VirtualenvManager(object):
         """Activate the virtualenv in this Python context.
 
         If you run a random Python script and wish to "activate" the
         virtualenv, you can simply instantiate an instance of this class
         and call .ensure() and .activate() to make the virtualenv active.
         """
 
         exec(open(self.activate_path).read(), dict(__file__=self.activate_path))
-        if PY2 and isinstance(os.environ['PATH'], unicode):
-            os.environ['PATH'] = os.environ['PATH'].encode('utf-8')
+        # Activating the virtualenv can make `os.environ` a little janky under
+        # Python 2.
+        env = ensure_subprocess_env(os.environ)
+        os.environ.clear()
+        os.environ.update(env)
 
     def install_pip_package(self, package, vendored=False):
         """Install a package via pip.
 
         The supplied package is specified using a pip requirement specifier.
         e.g. 'foo' or 'foo==1.0'.
 
         If the package is already installed, this is a no-op.
 
         If vendored is True, no package index will be used and no dependencies
         will be installed.
         """
-        from pip._internal.req.constructors import install_req_from_line
+        if sys.executable.startswith(self.bin_path):
+            # If we're already running in this interpreter, we can optimize in
+            # the case that the package requirement is already satisfied.
+            from pip._internal.req.constructors import install_req_from_line
 
-        req = install_req_from_line(package)
-        req.check_if_exists(use_user_site=False)
-        if req.satisfied_by is not None:
-            return
+            req = install_req_from_line(package)
+            req.check_if_exists(use_user_site=False)
+            if req.satisfied_by is not None:
+                return
 
         args = [
             'install',
             package,
         ]
 
         if vendored:
             args.extend([
@@ -581,16 +652,18 @@ class VirtualenvManager(object):
         """Activate a virtual environment managed by pipenv
 
         If ``pipfile`` is not ``None`` then the Pipfile located at the path
         provided will be used to create the virtual environment. If
         ``populate`` is ``True`` then the virtual environment will be
         populated from the manifest file. The optional ``python`` argument
         indicates the version of Python for pipenv to use.
         """
+
+        import distutils.sysconfig
         from distutils.version import LooseVersion
 
         pipenv = os.path.join(self.bin_path, 'pipenv')
         env = ensure_subprocess_env(os.environ.copy())
         env.update(ensure_subprocess_env({
             'PIPENV_IGNORE_VIRTUALENVS': '1',
             'WORKON_HOME': str(os.path.normpath(workon_home)),
         }))
@@ -663,20 +736,23 @@ class VirtualenvManager(object):
                 'PIPENV_PIPFILE': str(pipfile)
             }))
             subprocess.check_call([pipenv, 'install'], stderr=subprocess.STDOUT, env=env)
 
         self.virtualenv_root = ensure_venv()
 
         if populate:
             # Populate from the manifest
-            subprocess.check_call([
+            args = [
                 pipenv, 'run', 'python', os.path.join(here, 'virtualenv.py'), 'populate',
-                self.topsrcdir, self.virtualenv_root, self.manifest_path],
-                stderr=subprocess.STDOUT, env=env)
+                self.topsrcdir, self.virtualenv_root, self.manifest_path,
+                '--parent-site-dir', distutils.sysconfig.get_python_lib()]
+            if self.populate_local_paths:
+                args.append('--populate-local-paths')
+            subprocess.check_call(args, stderr=subprocess.STDOUT, env=env)
 
         self.activate()
 
 
 def verify_python_version(log_handle):
     """Ensure the current version of Python is sufficient."""
     from distutils.version import LooseVersion
 
@@ -712,67 +788,53 @@ def ensure_subprocess_env(env, encoding=
     This will convert all keys and values to bytes on Python 2, and text on
     Python 3.
 
     Args:
         env (dict): Environment to ensure.
         encoding (str): Encoding to use when converting to/from bytes/text
                         (default: utf-8).
     """
-    # We can't import six.ensure_binary() or six.ensure_text() because this module
-    # has to run stand-alone.  Instead we'll implement an abbreviated version of the
-    # checks it does.
-
-    if PY3:
-        text_type = str
-        binary_type = bytes
-    else:
-        text_type = unicode
-        binary_type = str
-
-    def ensure_binary(s):
-        if isinstance(s, text_type):
-            return s.encode(encoding, errors='strict')
-        elif isinstance(s, binary_type):
-            return s
-        else:
-            raise TypeError("not expecting type '%s'" % type(s))
-
-    def ensure_text(s):
-        if isinstance(s, binary_type):
-            return s.decode(encoding, errors='strict')
-        elif isinstance(s, text_type):
-            return s
-        else:
-            raise TypeError("not expecting type '%s'" % type(s))
-
     ensure = ensure_binary if PY2 else ensure_text
 
     try:
-        return {ensure(k): ensure(v) for k, v in env.iteritems()}
+        return {
+            ensure(k, encoding=encoding): ensure(v, encoding=encoding)
+            for k, v in env.iteritems()
+        }
     except AttributeError:
-        return {ensure(k): ensure(v) for k, v in env.items()}
+        return {
+            ensure(k, encoding=encoding): ensure(v, encoding=encoding)
+            for k, v in env.items()
+        }
 
 
 if __name__ == '__main__':
-    if len(sys.argv) < 4:
-        print(
-            'Usage: virtualenv.py /path/to/topsrcdir '
-            '/path/to/virtualenv /path/to/virtualenv_manifest')
-        sys.exit(1)
-
     verify_python_version(sys.stdout)
 
-    topsrcdir, virtualenv_path, manifest_path = sys.argv[1:4]
-    populate = False
+    if len(sys.argv) < 2:
+        print('Too few arguments', file=sys.stderr)
+        sys.exit(1)
 
-    # This should only be called internally.
+    parser = argparse.ArgumentParser()
+    parser.add_argument('topsrcdir')
+    parser.add_argument('virtualenv_path')
+    parser.add_argument('manifest_path')
+    parser.add_argument('--parent-site-dir', default=None)
+    parser.add_argument('--populate-local-paths', action='store_true')
+
     if sys.argv[1] == 'populate':
+        # This should only be called internally.
         populate = True
-        topsrcdir, virtualenv_path, manifest_path = sys.argv[2:]
+        opts = parser.parse_args(sys.argv[2:])
+    else:
+        populate = False
+        opts = parser.parse_args(sys.argv[1:])
 
-    manager = VirtualenvManager(topsrcdir, virtualenv_path,
-                                sys.stdout, manifest_path)
+    manager = VirtualenvManager(
+        opts.topsrcdir, opts.virtualenv_path, sys.stdout, opts.manifest_path,
+        parent_site_dir=opts.parent_site_dir,
+        populate_local_paths=opts.populate_local_paths)
 
     if populate:
         manager.populate()
     else:
         manager.ensure()