Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r=dustin
authorAndrew Halberstadt <ahalberstadt@mozilla.com>
Tue, 15 Aug 2017 11:36:29 -0400
changeset 375483 a43cb4709418ca9b44a9328d79fa635fb6418c4b
parent 375482 859e657f26dc808e922938350ffa7e6c4fc67583
child 375484 2fcca872720152569f78220fe5c4e1200ab12f76
push id32357
push userkwierso@gmail.com
push dateFri, 18 Aug 2017 20:11:21 +0000
treeherdermozilla-central@8febdfacc716 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersdustin
bugs1387135
milestone57.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r=dustin This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json looks something like: { "tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"], "templates": { "artifact": {"enabled": 1} } } This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based .yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template against every task definition. The templates themselves can then use JSONe condition statements to filter out which tasks they should or shouldn't apply to. MozReview-Commit-ID: J8HVZzOt4mX
taskcluster/docs/how-tos.rst
taskcluster/docs/parameters.rst
taskcluster/taskgraph/decision.py
taskcluster/taskgraph/generator.py
taskcluster/taskgraph/morph.py
taskcluster/taskgraph/parameters.py
taskcluster/taskgraph/target_tasks.py
taskcluster/taskgraph/test/test_target_tasks.py
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -273,33 +273,91 @@ a list of matching task labels. For more
 
 The second method uses a checked-in file called ``try_task_config.json`` which
 lives at the root of the source dir. The format of this file is either a list
 of task labels, or a JSON object where task labels make up the keys. For
 example, the ``try_task_config.json`` file might look like:
 
 .. parsed-literal::
 
-    [
-      "test-windows10-64/opt-web-platform-tests-12",
-      "test-windows7-32/opt-reftest-1",
-      "test-windows7-32/opt-reftest-2",
-      "test-windows7-32/opt-reftest-3",
-      "build-linux64/debug",
-      "source-test-mozlint-eslint"
-    ]
+    {
+      "tasks": [
+        "test-windows10-64/opt-web-platform-tests-12",
+        "test-windows7-32/opt-reftest-1",
+        "test-windows7-32/opt-reftest-2",
+        "test-windows7-32/opt-reftest-3",
+        "build-linux64/debug",
+        "source-test-mozlint-eslint"
+      ]
+    }
 
 Very simply, this will run any task label that gets passed in as well as their
 dependencies. While it is possible to manually commit this file and push to
-try, it is mainly meant to be a generation target for various trychooser tools.
+try, it is mainly meant to be a generation target for various `tryselect`_
+choosers.
 
 A list of all possible task labels can be obtained by running:
 
 .. parsed-literal::
 
     $ ./mach taskgraph tasks
 
 A list of task labels relevant to a tree (defaults to mozilla-central) can be
 obtained with:
 
 .. parsed-literal::
 
     $ ./mach taskgraph target
+
+Modifying Task Behavior on Try
+``````````````````````````````
+
+It's possible to alter the definition of a task with templates. Templates are
+`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
+from the ``try_task_config.json`` like this:
+
+.. parsed-literal::
+
+    {
+      "tasks": [...],
+      "templates": {
+        artifact: {"enabled": 1}
+      }
+    }
+
+Each key in the templates object denotes a new template to apply, and the value
+denotes extra context to use while rendering. When specified, a template will
+be applied to every task no matter what. If the template should only be applied
+to certain kinds of tasks, this needs to be specified in the template itself
+using JSON-e `condition statements`_.
+
+The context available to the JSON-e render aims to match that of ``actions``.
+It looks like this:
+
+.. parsed-literal::
+
+    {
+      "task": {
+        "payload": {
+          "env": { ... },
+          ...
+        }
+        "extra": {
+          "treeherder": { ... },
+          ...
+        },
+        "tags": { "kind": "<kind>", ... },
+        ...
+      },
+      "input": {
+        "enabled": 1,
+        ...
+      },
+      "taskId": "<task id>"
+    }
+
+See the `existing templates`_ for examples.
+
+.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
+.. _JSON-e: https://taskcluster.github.io/json-e/
+.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
+.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
+.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
--- a/taskcluster/docs/parameters.rst
+++ b/taskcluster/docs/parameters.rst
@@ -88,17 +88,30 @@ those in the target set, recursively.  I
 specified programmatically using one of a variety of methods (e.g., parsing try
 syntax or reading a project-specific configuration file).
 
 ``filters``
     List of filter functions (from ``taskcluster/taskgraph/filter_tasks.py``) to
     apply. This is usually defined internally, as filters are typically
     global.
 
+``target_task_labels``
+    List of task labels to select. Labels not listed will be filtered out.
+    Enabled on try only.
+
 ``target_tasks_method``
     The method to use to determine the target task set.  This is the suffix of
     one of the functions in ``taskcluster/taskgraph/target_tasks.py``.
 
 ``optimize_target_tasks``
-   If true, then target tasks are eligible for optimization.
+    If true, then target tasks are eligible for optimization.
 
 ``include_nightly``
-   If true, then nightly tasks are eligible for optimization.
+    If true, then nightly tasks are eligible for optimization.
+
+Morphed Set
+-----------
+
+``morph_templates``
+    Dict of JSON-e templates to apply to each task, keyed by template name.
+    Values are extra context that will be available to the template under the
+    ``input.<template>`` key. Available templates live in
+    ``taskcluster/taskgraph/templates``. Enabled on try only.
--- a/taskcluster/taskgraph/decision.py
+++ b/taskcluster/taskgraph/decision.py
@@ -161,16 +161,18 @@ def get_decision_parameters(options):
     ] if n in options}
 
     # Define default filter list, as most configurations shouldn't need
     # custom filters.
     parameters['filters'] = [
         'check_servo',
         'target_tasks_method',
     ]
+    parameters['target_task_labels'] = []
+    parameters['morph_templates'] = {}
 
     # owner must be an email, but sometimes (e.g., for ffxbld) it is not, in which
     # case, fake it
     if '@' not in parameters['owner']:
         parameters['owner'] += '@noreply.mozilla.org'
 
     # use the pushdate as build_date if given, else use current time
     parameters['build_date'] = parameters['pushdate'] or int(time.time())
@@ -182,16 +184,25 @@ def get_decision_parameters(options):
     try:
         parameters.update(PER_PROJECT_PARAMETERS[project])
     except KeyError:
         logger.warning("using default project parameters; add {} to "
                        "PER_PROJECT_PARAMETERS in {} to customize behavior "
                        "for this project".format(project, __file__))
         parameters.update(PER_PROJECT_PARAMETERS['default'])
 
+    # morph_templates and target_task_labels are only used on try, so don't
+    # bother loading them elsewhere
+    task_config_file = os.path.join(GECKO, 'try_task_config.json')
+    if project == 'try' and os.path.isfile(task_config_file):
+        with open(task_config_file, 'r') as fh:
+            task_config = json.load(fh)
+        parameters['morph_templates'] = task_config.get('templates', {})
+        parameters['target_task_labels'] = task_config.get('tasks')
+
     # `target_tasks_method` has higher precedence than `project` parameters
     if options.get('target_tasks_method'):
         parameters['target_tasks_method'] = options['target_tasks_method']
 
     return Parameters(parameters)
 
 
 def write_artifact(filename, data):
--- a/taskcluster/taskgraph/generator.py
+++ b/taskcluster/taskgraph/generator.py
@@ -275,17 +275,18 @@ class TaskGraphGenerator(object):
         if not self.parameters.get('optimize_target_tasks', True):
             do_not_optimize = target_task_set.graph.nodes
         optimized_task_graph, label_to_taskid = optimize_task_graph(target_task_graph,
                                                                     self.parameters,
                                                                     do_not_optimize)
 
         yield 'optimized_task_graph', optimized_task_graph
 
-        morphed_task_graph, label_to_taskid = morph(optimized_task_graph, label_to_taskid)
+        morphed_task_graph, label_to_taskid = morph(
+            optimized_task_graph, label_to_taskid, self.parameters)
 
         yield 'label_to_taskid', label_to_taskid
         yield 'morphed_task_graph', morphed_task_graph
 
     def _run_until(self, name):
         while name not in self._run_results:
             try:
                 k, v = self._run.next()
--- a/taskcluster/taskgraph/morph.py
+++ b/taskcluster/taskgraph/morph.py
@@ -14,23 +14,27 @@ the graph.
 # Note that the translation of `{'task-reference': '..'}` is handled in the
 # optimization phase (since optimization involves dealing with taskIds
 # directly).  Similarly, `{'relative-datestamp': '..'}` is handled at the last
 # possible moment during task creation.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 import logging
+import os
 import re
 
+import jsone
+import yaml
 from slugid import nice as slugid
 from .task import Task
 from .graph import Graph
 from .taskgraph import TaskGraph
 
+here = os.path.abspath(os.path.dirname(__file__))
 logger = logging.getLogger(__name__)
 MAX_ROUTES = 10
 
 
 def amend_taskgraph(taskgraph, label_to_taskid, to_add):
     """Add the given tasks to the taskgraph, returning a new taskgraph"""
     new_tasks = taskgraph.tasks.copy()
     new_edges = taskgraph.graph.edges.copy()
@@ -236,17 +240,51 @@ def add_s3_uploader_task(taskgraph, labe
             added = make_s3_uploader_task(task)
             taskgraph, label_to_taskid = amend_taskgraph(
                 taskgraph, label_to_taskid, [added])
             update_test_tasks(added.task_id, task.task_id, taskgraph)
             logger.info('Added s3-uploader task')
     return taskgraph, label_to_taskid
 
 
-def morph(taskgraph, label_to_taskid):
+class apply_jsone_templates(object):
+    """Apply a set of JSON-e templates to each task's `task` attribute.
+
+    :param templates: A dict with the template name as the key, and extra context
+                      to use (in addition to task.to_json()) as the value.
+    """
+    template_dir = os.path.join(here, 'templates')
+
+    def __init__(self, templates):
+        self.templates = templates
+
+    def __call__(self, taskgraph, label_to_taskid):
+        if not self.templates:
+            return taskgraph, label_to_taskid
+
+        for task in taskgraph.tasks.itervalues():
+            for template in sorted(self.templates):
+                context = {
+                    'task': task.task,
+                    'taskGroup': None,
+                    'taskId': task.task_id,
+                    'kind': task.kind,
+                    'input': self.templates[template],
+                }
+
+                template_path = os.path.join(self.template_dir, template + '.yml')
+                with open(template_path) as f:
+                    template = yaml.load(f)
+                task.task = jsone.render(template, context)
+
+        return taskgraph, label_to_taskid
+
+
+def morph(taskgraph, label_to_taskid, parameters):
     """Apply all morphs"""
     morphs = [
         add_index_tasks,
         add_s3_uploader_task,
+        apply_jsone_templates(parameters.get('morph_templates')),
     ]
     for m in morphs:
         taskgraph, label_to_taskid = m(taskgraph, label_to_taskid)
     return taskgraph, label_to_taskid
--- a/taskcluster/taskgraph/parameters.py
+++ b/taskcluster/taskgraph/parameters.py
@@ -16,33 +16,40 @@ PARAMETER_NAMES = set([
     'build_date',
     'filters',
     'head_ref',
     'head_repository',
     'head_rev',
     'include_nightly',
     'level',
     'message',
+    'morph_templates',
     'moz_build_date',
     'optimize_target_tasks',
     'owner',
     'project',
     'pushdate',
     'pushlog_id',
+    'target_task_labels',
     'target_tasks_method',
 ])
 
+TRY_ONLY_PARAMETERS = set([
+    'morph_templates',
+    'target_task_labels',
+])
+
 
 class Parameters(ReadOnlyDict):
     """An immutable dictionary with nicer KeyError messages on failure"""
     def check(self):
         names = set(self)
         msg = []
 
-        missing = PARAMETER_NAMES - names
+        missing = PARAMETER_NAMES - TRY_ONLY_PARAMETERS - names
         if missing:
             msg.append("missing parameters: " + ", ".join(missing))
 
         extra = names - PARAMETER_NAMES
         if extra:
             msg.append("extra parameters: " + ", ".join(extra))
 
         if msg:
--- a/taskcluster/taskgraph/target_tasks.py
+++ b/taskcluster/taskgraph/target_tasks.py
@@ -2,21 +2,21 @@
 
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 import os
-import json
 
 from taskgraph import try_option_syntax
 from taskgraph.util.attributes import match_run_on_projects
 
+here = os.path.abspath(os.path.dirname(__file__))
 _target_task_methods = {}
 
 
 def _target_task(name):
     def wrap(func):
         _target_task_methods[name] = func
         return func
     return wrap
@@ -48,30 +48,21 @@ def filter_upload_symbols(task, paramete
 def standard_filter(task, parameters):
     return all(
         filter_func(task, parameters) for filter_func in
         (filter_on_nightly, filter_for_project, filter_upload_symbols)
     )
 
 
 def _try_task_config(full_task_graph, parameters):
-    task_config_file = os.path.join(os.getcwd(), 'try_task_config.json')
-
-    if not os.path.isfile(task_config_file):
+    if not parameters.get('target_task_labels'):
         return []
 
-    with open(task_config_file, 'r') as fh:
-        task_config = json.load(fh)
-
-    target_task_labels = []
-    for task in full_task_graph.tasks.itervalues():
-        if task.label in task_config:
-            target_task_labels.append(task.label)
-
-    return target_task_labels
+    return [t.label for t in full_task_graph.tasks.itervalues()
+            if t.label in parameters['target_task_labels']]
 
 
 def _try_option_syntax(full_task_graph, parameters):
     """Generate a list of target tasks based on try syntax in
     parameters['message'] and, for context, the full task graph."""
     options = try_option_syntax.TryOptionSyntax(parameters['message'], full_task_graph)
     target_tasks_labels = [t.label for t in full_task_graph.tasks.itervalues()
                            if options.task_matches(t)]
--- a/taskcluster/taskgraph/test/test_target_tasks.py
+++ b/taskcluster/taskgraph/test/test_target_tasks.py
@@ -1,15 +1,14 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
-import os
 import unittest
 
 from taskgraph import target_tasks
 from taskgraph import try_option_syntax
 from taskgraph.graph import Graph
 from taskgraph.taskgraph import TaskGraph
 from taskgraph.task import Task
 from mozunit import main
@@ -71,39 +70,38 @@ class TestTargetTasks(unittest.TestCase)
             'a': Task(kind=None, label='a', attributes={}, task={}),
             'b': Task(kind=None, label='b', attributes={'at-at': 'yep'}, task={}),
             'c': Task(kind=None, label='c', attributes={}, task={}),
         }
         graph = Graph(nodes=set('abc'), edges=set())
         tg = TaskGraph(tasks, graph)
 
         method = target_tasks.get_method('try_tasks')
-        config = os.path.join(os.getcwd(), 'try_task_config.json')
+        params = {
+            'message': '',
+            'target_task_labels': [],
+        }
 
         orig_TryOptionSyntax = try_option_syntax.TryOptionSyntax
         try:
             try_option_syntax.TryOptionSyntax = FakeTryOptionSyntax
 
             # no try specifier
-            self.assertEqual(method(tg, {'message': ''}), ['b'])
+            self.assertEqual(method(tg, params), ['b'])
 
             # try syntax only
-            self.assertEqual(method(tg, {'message': 'try: me'}), ['b'])
+            params['message'] = 'try: me'
+            self.assertEqual(method(tg, params), ['b'])
 
             # try task config only
-            with open(config, 'w') as fh:
-                fh.write('["c"]')
-            self.assertEqual(method(tg, {'message': ''}), ['c'])
-
-            with open(config, 'w') as fh:
-                fh.write('{"c": {}}')
-            self.assertEqual(method(tg, {'message': ''}), ['c'])
+            params['message'] = ''
+            params['target_task_labels'] = ['c']
+            self.assertEqual(method(tg, params), ['c'])
 
             # both syntax and config
-            self.assertEqual(set(method(tg, {'message': 'try: me'})), set(['b', 'c']))
+            params['message'] = 'try: me'
+            self.assertEqual(set(method(tg, params)), set(['b', 'c']))
         finally:
             try_option_syntax.TryOptionSyntax = orig_TryOptionSyntax
-            if os.path.isfile(config):
-                os.remove(config)
 
 
 if __name__ == '__main__':
     main()