Bug 1501878 - add mar-signing{,-l10n} kinds. r=tomprince
☠☠ backed out by 94debc6ca20b ☠ ☠
authorAki Sasaki <aki@escapewindow.com>
Fri, 16 Nov 2018 21:01:38 +0000
changeset 503277 4e2dc6c23463e976ba90cf2113eea9dbfcb89fcb
parent 503276 050c4bba5d710059175026e9f0185e795fcdfc28
child 503278 01311f87d2875954fd7cfb347cf7d123974c3a1f
push id10290
push userffxbld-merge
push dateMon, 03 Dec 2018 16:23:23 +0000
treeherdermozilla-beta@700bed2445e6 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstomprince
bugs1501878
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1501878 - add mar-signing{,-l10n} kinds. r=tomprince also rename the partials_signing transform to mar_signing. Differential Revision: https://phabricator.services.mozilla.com/D11731
taskcluster/ci/config.yml
taskcluster/ci/mar-signing-l10n/kind.yml
taskcluster/ci/mar-signing/kind.yml
taskcluster/ci/partials-signing/kind.yml
taskcluster/docs/kinds.rst
taskcluster/taskgraph/transforms/mar_signing.py
taskcluster/taskgraph/transforms/partials_signing.py
--- a/taskcluster/ci/config.yml
+++ b/taskcluster/ci/config.yml
@@ -72,16 +72,17 @@ treeherder:
         'TW64': 'Toolchain builds for Windows 64-bits'
         'WMC32': 'MinGW-Clang builds for Windows 32-bits'
         'WMC64': 'MinGW-Clang builds for Windows 64-bits'
         'Searchfox': 'Searchfox builds'
         'SM': 'Spidermonkey builds'
         'pub': 'APK publishing'
         'p': 'Partial generation'
         'ps': 'Partials signing'
+        'ms': 'Complete MAR signing'
         'Rel': 'Release promotion'
         'Snap': 'Snap image generation'
         'langpack': 'Langpack sigatures and uploads'
         'TPS': 'Sync tests'
         'UV': 'Update verify'
         'pipfu': 'pipfile update'
 
 index:
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/mar-signing-l10n/kind.yml
@@ -0,0 +1,31 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+loader: taskgraph.loader.single_dep:loader
+
+transforms:
+    - taskgraph.transforms.name_sanity:transforms
+    - taskgraph.transforms.mar_signing:transforms
+    - taskgraph.transforms.task:transforms
+
+kind-dependencies:
+    - repackage-l10n
+
+only-for-build-platforms:
+    - linux-nightly/opt
+    - linux-devedition-nightly/opt
+    - linux64-nightly/opt
+    - linux64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - macosx64-nightly/opt
+    - macosx64-devedition-nightly/opt
+    - win32-nightly/opt
+    - win32-devedition-nightly/opt
+    - win64-nightly/opt
+    - win64-devedition-nightly/opt
+
+job-template:
+    shipping-phase: promote
+    treeherder-group: ms
+    description-suffix: 'mar signing'
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/mar-signing/kind.yml
@@ -0,0 +1,33 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+loader: taskgraph.loader.single_dep:loader
+
+transforms:
+    - taskgraph.transforms.name_sanity:transforms
+    - taskgraph.transforms.mar_signing:transforms
+    - taskgraph.transforms.task:transforms
+
+kind-dependencies:
+    - repackage
+
+only-for-build-platforms:
+    - linux-nightly/opt
+    - linux-devedition-nightly/opt
+    - linux64-nightly/opt
+    - linux64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - macosx64-nightly/opt
+    - macosx64-devedition-nightly/opt
+    - win32-nightly/opt
+    - win32-devedition-nightly/opt
+    - win64-nightly/opt
+    - win64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - win64-asan-reporter-nightly/opt
+
+job-template:
+    shipping-phase: promote
+    treeherder-group: ms
+    description-suffix: 'mar signing'
--- a/taskcluster/ci/partials-signing/kind.yml
+++ b/taskcluster/ci/partials-signing/kind.yml
@@ -1,16 +1,18 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 loader: taskgraph.loader.single_dep:loader
 
 transforms:
   - taskgraph.transforms.name_sanity:transforms
-  - taskgraph.transforms.partials_signing:transforms
+  - taskgraph.transforms.mar_signing:transforms
   - taskgraph.transforms.task:transforms
 
 kind-dependencies:
   - partials
 
 job-template:
   shipping-phase: promote
+  treeherder-group: ps
+  description-suffix: 'partial signing'
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -443,23 +443,29 @@ and this task would package that up as a
 
 repackage-l10n
 --------------
 Repackage-L10n is a ```Repackage``` task split up to be suitable for use after l10n repacks.
 
 
 repackage-signing
 -----------------
-Repackage-signing take the repackaged installers (windows) and update packaging (with
-the signed internal bits) and signs them.
+Repackage-signing take the repackaged installers (windows) and signs them.
 
 repackage-signing-l10n
 ----------------------
-Repackage-signing take the repackaged installers (windows) and update packaging (with
-the signed internal bits) and signs them for localized versions.
+Repackage-signing-l10n take the repackaged installers (windows) and signs them for localized versions.
+
+mar-signing
+-----------
+Mar-signing takes the complete update MARs and signs them.
+
+mar-signing-l10n
+----------------
+Mar-signing-l10n takes the complete update MARs and signs them for localized versions.
 
 repackage-msi
 -------------
 Repackage-msi takes the signed full installer and produces an msi installer (that wraps the full installer)
 Using the ```./mach repackage``` command
 
 repackage-signing-msi
 ---------------------
rename from taskcluster/taskgraph/transforms/partials_signing.py
rename to taskcluster/taskgraph/transforms/mar_signing.py
--- a/taskcluster/taskgraph/transforms/partials_signing.py
+++ b/taskcluster/taskgraph/transforms/mar_signing.py
@@ -1,32 +1,33 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 """
-Transform the partials task into an actual task description.
+Transform the {partials,mar}-signing task into an actual task description.
 """
 from __future__ import absolute_import, print_function, unicode_literals
 
 from taskgraph.transforms.base import TransformSequence
 from taskgraph.util.attributes import copy_attributes_from_dependent_job
 from taskgraph.util.scriptworker import (
     get_signing_cert_scope_per_platform,
     get_worker_type_for_scope,
 )
 from taskgraph.util.partials import get_balrog_platform_name, get_partials_artifacts
 from taskgraph.util.taskcluster import get_artifact_prefix
+from taskgraph.util.treeherder import join_symbol
 
 import logging
 logger = logging.getLogger(__name__)
 
 transforms = TransformSequence()
 
 
-def generate_upstream_artifacts(job, release_history, platform, locale=None):
+def generate_partials_artifacts(job, release_history, platform, locale=None):
     artifact_prefix = get_artifact_prefix(job)
     if locale:
         artifact_prefix = '{}/{}'.format(artifact_prefix, locale)
     else:
         locale = 'en-US'
 
     artifacts = get_partials_artifacts(release_history, platform, locale)
 
@@ -57,65 +58,89 @@ def generate_upstream_artifacts(job, rel
     }
 
     if old_mar_upstream_artifacts["paths"]:
         upstream_artifacts.append(old_mar_upstream_artifacts)
 
     return upstream_artifacts
 
 
+def generate_complete_artifacts(job, platform, locale=None):
+    artifact_prefix = get_artifact_prefix(job)
+    if locale:
+        artifact_prefix = '{}/{}'.format(artifact_prefix, locale)
+
+    upstream_artifacts = [{
+        "taskId": {"task-reference": '<{}>'.format(job.kind)},
+        "taskType": 'build',
+        "paths": [
+            "{}/target.complete.mar".format(artifact_prefix)
+        ],
+        "formats": ["autograph_hash_only_mar384"],
+    }]
+
+    return upstream_artifacts
+
+
 @transforms.add
 def make_task_description(config, jobs):
     for job in jobs:
         dep_job = job['primary-dependency']
+        locale = dep_job.attributes.get('locale')
 
         treeherder = job.get('treeherder', {})
-        treeherder.setdefault('symbol', 'ps(N)')
+        treeherder['symbol'] = join_symbol(
+            job.get('treeherder-group', 'ms'),
+            locale or 'N'
+        )
 
         dep_th_platform = dep_job.task.get('extra', {}).get(
             'treeherder', {}).get('machine', {}).get('platform', '')
-        label = job.get('label', "partials-signing-{}".format(dep_job.label))
+        label = job.get('label', "{}-{}".format(config.kind, dep_job.label))
         dep_th_platform = dep_job.task.get('extra', {}).get(
             'treeherder', {}).get('machine', {}).get('platform', '')
         treeherder.setdefault('platform',
                               "{}/opt".format(dep_th_platform))
         treeherder.setdefault('kind', 'build')
         treeherder.setdefault('tier', 1)
 
         dependent_kind = str(dep_job.kind)
         dependencies = {dependent_kind: dep_job.label}
         signing_dependencies = dep_job.dependencies
         # This is so we get the build task etc in our dependencies to
         # have better beetmover support.
         dependencies.update(signing_dependencies)
 
         attributes = copy_attributes_from_dependent_job(dep_job)
-        locale = dep_job.attributes.get('locale')
+        attributes['shipping_phase'] = job['shipping-phase']
         if locale:
             attributes['locale'] = locale
-            treeherder['symbol'] = 'ps({})'.format(locale)
 
         balrog_platform = get_balrog_platform_name(dep_th_platform)
-        upstream_artifacts = generate_upstream_artifacts(
-            dep_job, config.params['release_history'], balrog_platform, locale)
+        if config.kind == 'partials-signing':
+            upstream_artifacts = generate_partials_artifacts(
+                dep_job, config.params['release_history'], balrog_platform, locale)
+        else:
+            upstream_artifacts = generate_complete_artifacts(
+                dep_job, balrog_platform, locale)
 
         build_platform = dep_job.attributes.get('build_platform')
         is_nightly = dep_job.attributes.get('nightly')
         signing_cert_scope = get_signing_cert_scope_per_platform(
             build_platform, is_nightly, config
         )
 
         scopes = [signing_cert_scope, 'project:releng:signing:format:autograph_hash_only_mar384']
         if any("mar" in upstream_details["formats"] for upstream_details in upstream_artifacts):
             scopes.append('project:releng:signing:format:mar')
 
         task = {
             'label': label,
-            'description': "{} Partials".format(
-                dep_job.task["metadata"]["description"]),
+            'description': "{} {}".format(
+                dep_job.task["metadata"]["description"], job['description-suffix']),
             'worker-type': get_worker_type_for_scope(config, signing_cert_scope),
             'worker': {'implementation': 'scriptworker-signing',
                        'upstream-artifacts': upstream_artifacts,
                        'max-run-time': 3600},
             'dependencies': dependencies,
             'attributes': attributes,
             'scopes': scopes,
             'run-on-projects': dep_job.attributes.get('run_on_projects'),