Bug 1321847 - Allow to override the mozharness tooltool manifest from the environment. r=mshal
authorMike Hommey <mh+mozilla@glandium.org>
Fri, 02 Jun 2017 11:28:26 +0900
changeset 412593 4db381159f27a7d1a1b5d112be7441290dc12550
parent 412592 2c0e0f3062aeb2e1bca2b743ab0d82ca19619c9e
child 412594 d9b7971a270cde0cff0adbe068707fd6f92f5028
push id1490
push usermtabara@mozilla.com
push dateMon, 31 Jul 2017 14:08:16 +0000
treeherdermozilla-release@70e32e6bf15e [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersmshal
bugs1321847, 1313111, 1356529
milestone55.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1321847 - Allow to override the mozharness tooltool manifest from the environment. r=mshal The main motivation behind this change is that going towards toolchain tasks hooked up in the task graph (bug 1313111), we're going to end up with jobs using both taskcluster toolchain job and tooltool artifacts for their toolchain needs. With the current setup, this means the toolchain dependencies will be spread between taskcluster task graph definition and mozharness configuration. It also makes things more complex to provide a command that pulls the right toolchains from both taskcluster and tooltool (bug 1356529), because one needs to find and parse the mozharness config (which also happens to be python code that uses platform-specific things, so e.g. reading windows mozharness config fails on other platforms). All in all, moving the tooltool manifest path to the taskcluster task definitions would make things simpler, and would also allow make patches switching from tooltool to taskcluster artifacts more straightforward to validate. But since some build types still run on buildbot, we'll have to keep part of the current setup using mozharness configs. So we allow to override the tooltool manifest path from the environment, and we'll rely on taskcluster task definitions being able to set environment variables. Actually moving the relevant tooltool manifest paths from mozharness config to taskcluster task definitions is left for a followup. Another followup is to move the tooltool manifest paths declared in some ad-hoc build scripts to taskcluster task definitions as well. The immediate need for this, though, is to allow to have duplicated jobs that only differ in their tooltool manifest, without duplicating a complete mozharness config that will end up stale (the goal being that really only the tooltool manifest differs, even when the original jobs change).
testing/mozharness/mozharness/mozilla/building/buildbase.py
testing/mozharness/scripts/desktop_l10n.py
testing/mozharness/scripts/repackage.py
--- a/testing/mozharness/mozharness/mozilla/building/buildbase.py
+++ b/testing/mozharness/mozharness/mozilla/building/buildbase.py
@@ -1110,20 +1110,23 @@ or run without that action (ie: --no-{ac
         env.update(self.query_mach_build_env())
 
         self._assert_cfg_valid_for_action(
             ['tooltool_script', 'tooltool_url'],
             'build'
         )
         c = self.config
         dirs = self.query_abs_dirs()
-        if not c.get('tooltool_manifest_src'):
+        manifest_src = os.environ.get('TOOLTOOL_MANIFEST')
+        if not manifest_src:
+            manifest_src = c.get('tooltool_manifest_src')
+        if not manifest_src:
             return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
         tooltool_manifest_path = os.path.join(dirs['abs_src_dir'],
-                                              c['tooltool_manifest_src'])
+                                              manifest_src)
         cmd = [
             sys.executable, '-u',
             os.path.join(dirs['abs_src_dir'], 'mach'),
             'artifact',
             'toolchain',
             '-v',
             '--retry', '4',
             '--tooltool-manifest',
--- a/testing/mozharness/scripts/desktop_l10n.py
+++ b/testing/mozharness/scripts/desktop_l10n.py
@@ -1030,20 +1030,23 @@ class DesktopSingleLocale(LocalesMixin, 
         # necessary.
         if os.path.exists(fn):
             return fn
 
     def _run_tooltool(self):
         env = self.query_bootstrap_env()
         config = self.config
         dirs = self.query_abs_dirs()
-        if not config.get('tooltool_manifest_src'):
+        manifest_src = os.environ.get('TOOLTOOL_MANIFEST')
+        if not manifest_src:
+            manifest_src = config.get('tooltool_manifest_src')
+        if not manifest_src:
             return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
         tooltool_manifest_path = os.path.join(dirs['abs_mozilla_dir'],
-                                              config['tooltool_manifest_src'])
+                                              manifest_src)
         cmd = [
             sys.executable, '-u',
             os.path.join(dirs['abs_mozilla_dir'], 'mach'),
             'artifact',
             'toolchain',
             '-v',
             '--retry', '4',
             '--tooltool-manifest',
--- a/testing/mozharness/scripts/repackage.py
+++ b/testing/mozharness/scripts/repackage.py
@@ -87,22 +87,25 @@ class Repackage(BaseScript):
             command=command,
             cwd=dirs['abs_mozilla_dir'],
             halt_on_failure=True,
         )
 
     def _run_tooltool(self):
         config = self.config
         dirs = self.query_abs_dirs()
-        if not config.get('tooltool_manifest_src'):
+        manifest_src = os.environ.get('TOOLTOOL_MANIFEST')
+        if not manifest_src:
+            manifest_src = config.get('tooltool_manifest_src')
+        if not manifest_src:
             return self.warning(ERROR_MSGS['tooltool_manifest_undetermined'])
         fetch_script_path = os.path.join(dirs['abs_tools_dir'],
                                          'scripts/tooltool/tooltool_wrapper.sh')
         tooltool_manifest_path = os.path.join(dirs['abs_mozilla_dir'],
-                                              config['tooltool_manifest_src'])
+                                              manifest_src)
         cmd = [
             'sh',
             fetch_script_path,
             tooltool_manifest_path,
             config['tooltool_url'],
             config['tooltool_bootstrap'],
         ]
         cmd.extend(config['tooltool_script'])