Bug 1501878 - add mar-signing{,-l10n} kinds. r=tomprince,Callek
authorAki Sasaki <aki@escapewindow.com>
Wed, 21 Nov 2018 19:04:01 +0000
changeset 506856 15d3e389b233bbebf60e7e45e69672f348db0c3f
parent 506855 d4b468d560b23e6530ce4441120bb2f06bd9161b
child 506857 cf994c5954587af5c7956e99bacc944673f6dc47
push id1905
push userffxbld-merge
push dateMon, 21 Jan 2019 12:33:13 +0000
treeherdermozilla-release@c2fca1944d8c [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstomprince, Callek
bugs1501878
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1501878 - add mar-signing{,-l10n} kinds. r=tomprince,Callek also rename the partials_signing transform to mar_signing. Differential Revision: https://phabricator.services.mozilla.com/D11731
taskcluster/ci/config.yml
taskcluster/ci/mar-signing-l10n/kind.yml
taskcluster/ci/mar-signing/kind.yml
taskcluster/ci/partials-signing/kind.yml
taskcluster/docs/kinds.rst
taskcluster/taskgraph/transforms/mar_signing.py
taskcluster/taskgraph/transforms/partials_signing.py
--- a/taskcluster/ci/config.yml
+++ b/taskcluster/ci/config.yml
@@ -72,16 +72,17 @@ treeherder:
         'TW64': 'Toolchain builds for Windows 64-bits'
         'WMC32': 'MinGW-Clang builds for Windows 32-bits'
         'WMC64': 'MinGW-Clang builds for Windows 64-bits'
         'Searchfox': 'Searchfox builds'
         'SM': 'Spidermonkey builds'
         'pub': 'APK publishing'
         'p': 'Partial generation'
         'ps': 'Partials signing'
+        'ms': 'Complete MAR signing'
         'Rel': 'Release promotion'
         'Snap': 'Snap image generation'
         'langpack': 'Langpack sigatures and uploads'
         'TPS': 'Sync tests'
         'UV': 'Update verify'
         'pipfu': 'pipfile update'
 
 index:
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/mar-signing-l10n/kind.yml
@@ -0,0 +1,31 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+loader: taskgraph.loader.single_dep:loader
+
+transforms:
+    - taskgraph.transforms.name_sanity:transforms
+    - taskgraph.transforms.mar_signing:transforms
+    - taskgraph.transforms.task:transforms
+
+kind-dependencies:
+    - repackage-l10n
+
+only-for-build-platforms:
+    - linux-nightly/opt
+    - linux-devedition-nightly/opt
+    - linux64-nightly/opt
+    - linux64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - macosx64-nightly/opt
+    - macosx64-devedition-nightly/opt
+    - win32-nightly/opt
+    - win32-devedition-nightly/opt
+    - win64-nightly/opt
+    - win64-devedition-nightly/opt
+
+job-template:
+    shipping-phase: promote
+    treeherder-group: ms
+    description-suffix: 'mar signing'
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/mar-signing/kind.yml
@@ -0,0 +1,33 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+loader: taskgraph.loader.single_dep:loader
+
+transforms:
+    - taskgraph.transforms.name_sanity:transforms
+    - taskgraph.transforms.mar_signing:transforms
+    - taskgraph.transforms.task:transforms
+
+kind-dependencies:
+    - repackage
+
+only-for-build-platforms:
+    - linux-nightly/opt
+    - linux-devedition-nightly/opt
+    - linux64-nightly/opt
+    - linux64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - macosx64-nightly/opt
+    - macosx64-devedition-nightly/opt
+    - win32-nightly/opt
+    - win32-devedition-nightly/opt
+    - win64-nightly/opt
+    - win64-devedition-nightly/opt
+    - linux64-asan-reporter-nightly/opt
+    - win64-asan-reporter-nightly/opt
+
+job-template:
+    shipping-phase: promote
+    treeherder-group: ms
+    description-suffix: 'mar signing'
--- a/taskcluster/ci/partials-signing/kind.yml
+++ b/taskcluster/ci/partials-signing/kind.yml
@@ -1,16 +1,18 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 loader: taskgraph.loader.single_dep:loader
 
 transforms:
   - taskgraph.transforms.name_sanity:transforms
-  - taskgraph.transforms.partials_signing:transforms
+  - taskgraph.transforms.mar_signing:transforms
   - taskgraph.transforms.task:transforms
 
 kind-dependencies:
   - partials
 
 job-template:
   shipping-phase: promote
+  treeherder-group: ps
+  description-suffix: 'partial signing'
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -443,23 +443,29 @@ and this task would package that up as a
 
 repackage-l10n
 --------------
 Repackage-L10n is a ```Repackage``` task split up to be suitable for use after l10n repacks.
 
 
 repackage-signing
 -----------------
-Repackage-signing take the repackaged installers (windows) and update packaging (with
-the signed internal bits) and signs them.
+Repackage-signing take the repackaged installers (windows) and signs them.
 
 repackage-signing-l10n
 ----------------------
-Repackage-signing take the repackaged installers (windows) and update packaging (with
-the signed internal bits) and signs them for localized versions.
+Repackage-signing-l10n take the repackaged installers (windows) and signs them for localized versions.
+
+mar-signing
+-----------
+Mar-signing takes the complete update MARs and signs them.
+
+mar-signing-l10n
+----------------
+Mar-signing-l10n takes the complete update MARs and signs them for localized versions.
 
 repackage-msi
 -------------
 Repackage-msi takes the signed full installer and produces an msi installer (that wraps the full installer)
 Using the ```./mach repackage``` command
 
 repackage-signing-msi
 ---------------------
rename from taskcluster/taskgraph/transforms/partials_signing.py
rename to taskcluster/taskgraph/transforms/mar_signing.py
--- a/taskcluster/taskgraph/transforms/partials_signing.py
+++ b/taskcluster/taskgraph/transforms/mar_signing.py
@@ -1,32 +1,35 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 """
-Transform the partials task into an actual task description.
+Transform the {partials,mar}-signing task into an actual task description.
 """
 from __future__ import absolute_import, print_function, unicode_literals
 
+import os
+
 from taskgraph.transforms.base import TransformSequence
 from taskgraph.util.attributes import copy_attributes_from_dependent_job
 from taskgraph.util.scriptworker import (
     get_signing_cert_scope_per_platform,
     get_worker_type_for_scope,
 )
 from taskgraph.util.partials import get_balrog_platform_name, get_partials_artifacts
 from taskgraph.util.taskcluster import get_artifact_prefix
+from taskgraph.util.treeherder import join_symbol
 
 import logging
 logger = logging.getLogger(__name__)
 
 transforms = TransformSequence()
 
 
-def generate_upstream_artifacts(job, release_history, platform, locale=None):
+def generate_partials_artifacts(job, release_history, platform, locale=None):
     artifact_prefix = get_artifact_prefix(job)
     if locale:
         artifact_prefix = '{}/{}'.format(artifact_prefix, locale)
     else:
         locale = 'en-US'
 
     artifacts = get_partials_artifacts(release_history, platform, locale)
 
@@ -57,65 +60,86 @@ def generate_upstream_artifacts(job, rel
     }
 
     if old_mar_upstream_artifacts["paths"]:
         upstream_artifacts.append(old_mar_upstream_artifacts)
 
     return upstream_artifacts
 
 
+def generate_complete_artifacts(job):
+    upstream_artifacts = []
+    for artifact in job.release_artifacts:
+        basename = os.path.basename(artifact)
+        if basename.endswith('.complete.mar'):
+            upstream_artifacts.append({
+                "taskId": {"task-reference": '<{}>'.format(job.kind)},
+                "taskType": 'build',
+                "paths": [artifact],
+                "formats": ["autograph_hash_only_mar384"],
+            })
+
+    return upstream_artifacts
+
+
 @transforms.add
 def make_task_description(config, jobs):
     for job in jobs:
         dep_job = job['primary-dependency']
+        locale = dep_job.attributes.get('locale')
 
         treeherder = job.get('treeherder', {})
-        treeherder.setdefault('symbol', 'ps(N)')
+        treeherder['symbol'] = join_symbol(
+            job.get('treeherder-group', 'ms'),
+            locale or 'N'
+        )
 
         dep_th_platform = dep_job.task.get('extra', {}).get(
             'treeherder', {}).get('machine', {}).get('platform', '')
-        label = job.get('label', "partials-signing-{}".format(dep_job.label))
+        label = job.get('label', "{}-{}".format(config.kind, dep_job.label))
         dep_th_platform = dep_job.task.get('extra', {}).get(
             'treeherder', {}).get('machine', {}).get('platform', '')
         treeherder.setdefault('platform',
                               "{}/opt".format(dep_th_platform))
         treeherder.setdefault('kind', 'build')
         treeherder.setdefault('tier', 1)
 
         dependent_kind = str(dep_job.kind)
         dependencies = {dependent_kind: dep_job.label}
         signing_dependencies = dep_job.dependencies
         # This is so we get the build task etc in our dependencies to
         # have better beetmover support.
         dependencies.update(signing_dependencies)
 
         attributes = copy_attributes_from_dependent_job(dep_job)
-        locale = dep_job.attributes.get('locale')
+        attributes['shipping_phase'] = job['shipping-phase']
         if locale:
             attributes['locale'] = locale
-            treeherder['symbol'] = 'ps({})'.format(locale)
 
         balrog_platform = get_balrog_platform_name(dep_th_platform)
-        upstream_artifacts = generate_upstream_artifacts(
-            dep_job, config.params['release_history'], balrog_platform, locale)
+        if config.kind == 'partials-signing':
+            upstream_artifacts = generate_partials_artifacts(
+                dep_job, config.params['release_history'], balrog_platform, locale)
+        else:
+            upstream_artifacts = generate_complete_artifacts(dep_job)
 
         build_platform = dep_job.attributes.get('build_platform')
         is_nightly = dep_job.attributes.get('nightly')
         signing_cert_scope = get_signing_cert_scope_per_platform(
             build_platform, is_nightly, config
         )
 
         scopes = [signing_cert_scope, 'project:releng:signing:format:autograph_hash_only_mar384']
         if any("mar" in upstream_details["formats"] for upstream_details in upstream_artifacts):
             scopes.append('project:releng:signing:format:mar')
 
         task = {
             'label': label,
-            'description': "{} Partials".format(
-                dep_job.task["metadata"]["description"]),
+            'description': "{} {}".format(
+                dep_job.task["metadata"]["description"], job['description-suffix']),
             'worker-type': get_worker_type_for_scope(config, signing_cert_scope),
             'worker': {'implementation': 'scriptworker-signing',
                        'upstream-artifacts': upstream_artifacts,
                        'max-run-time': 3600},
             'dependencies': dependencies,
             'attributes': attributes,
             'scopes': scopes,
             'run-on-projects': dep_job.attributes.get('run_on_projects'),