Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r?dustin draft
authorAndrew Halberstadt <ahalberstadt@mozilla.com>
Tue, 15 Aug 2017 11:36:29 -0400
changeset 647707 e0cb147a5e7a2321e92531ddf05f681b4196c123
parent 647624 07ce8c96222d533fc89c02802143d35d7c351f9c
child 647708 2db3d6007b9dbbd32b9965e54fd591ccf0424c8d
push id74520
push userahalberstadt@mozilla.com
push dateWed, 16 Aug 2017 20:33:54 +0000
reviewersdustin
bugs1387135
milestone57.0a1
Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r?dustin This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json looks something like: { "tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"], "templates": { "artifact": {"enabled": 1} } } This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based .yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template against every task definition. The templates themselves can then use JSONe condition statements to filter out which tasks they should or shouldn't apply to. MozReview-Commit-ID: J8HVZzOt4mX
taskcluster/docs/how-tos.rst
taskcluster/docs/parameters.rst
taskcluster/taskgraph/decision.py
taskcluster/taskgraph/generator.py
taskcluster/taskgraph/morph.py
taskcluster/taskgraph/parameters.py
taskcluster/taskgraph/target_tasks.py
taskcluster/taskgraph/test/test_target_tasks.py
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -273,33 +273,89 @@ a list of matching task labels. For more
 
 The second method uses a checked-in file called ``try_task_config.json`` which
 lives at the root of the source dir. The format of this file is either a list
 of task labels, or a JSON object where task labels make up the keys. For
 example, the ``try_task_config.json`` file might look like:
 
 .. parsed-literal::
 
-    [
-      "test-windows10-64/opt-web-platform-tests-12",
-      "test-windows7-32/opt-reftest-1",
-      "test-windows7-32/opt-reftest-2",
-      "test-windows7-32/opt-reftest-3",
-      "build-linux64/debug",
-      "source-test-mozlint-eslint"
+    {
+      "tasks": [
+        "test-windows10-64/opt-web-platform-tests-12",
+        "test-windows7-32/opt-reftest-1",
+        "test-windows7-32/opt-reftest-2",
+        "test-windows7-32/opt-reftest-3",
+        "build-linux64/debug",
+        "source-test-mozlint-eslint"
+      ]
     ]
 
 Very simply, this will run any task label that gets passed in as well as their
 dependencies. While it is possible to manually commit this file and push to
-try, it is mainly meant to be a generation target for various trychooser tools.
+try, it is mainly meant to be a generation target for various `tryselect`_
+choosers.
 
 A list of all possible task labels can be obtained by running:
 
 .. parsed-literal::
 
     $ ./mach taskgraph tasks
 
 A list of task labels relevant to a tree (defaults to mozilla-central) can be
 obtained with:
 
 .. parsed-literal::
 
     $ ./mach taskgraph target
+
+Modifying Task Behavior on Try
+``````````````````````````````
+
+It's possible to alter the definition of a task with templates. Templates are
+`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
+from the ``try_task_config.json`` like this:
+
+.. parsed-literal::
+
+    {
+      "tasks": [...],
+      "templates": {
+        artifact: {"enabled": 1}
+      }
+    }
+
+Each key in the templates object denotes a new template to apply, and the value
+denotes extra context to use while rendering. When specified, a template will
+be applied to every task no matter what. If the template should only be applied
+to certain kinds of tasks, this needs to be specified in the template itself
+using JSON-e `condition statements`_.
+
+The context available to the JSON-e render aims to match that of ``actions``.
+It looks like this:
+
+.. parsed-literal::
+
+    {
+      "task": {
+        "payload": { ... },
+        "env": { ... },
+        "extra": {
+          "treeherder": { ... },
+          ...
+        },
+        "tags": { "kind": "<kind>", ... },
+        ...
+      },
+      "input": {
+        "artifact": {"enabled": 1},
+        ...
+      },
+      "taskId": "<task id>"
+    }
+
+See the `existing templates`_ for examples.
+
+.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
+.. _JSON-e: https://taskcluster.github.io/json-e/
+.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
+.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
+.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
--- a/taskcluster/docs/parameters.rst
+++ b/taskcluster/docs/parameters.rst
@@ -88,17 +88,30 @@ those in the target set, recursively.  I
 specified programmatically using one of a variety of methods (e.g., parsing try
 syntax or reading a project-specific configuration file).
 
 ``filters``
     List of filter functions (from ``taskcluster/taskgraph/filter_tasks.py``) to
     apply. This is usually defined internally, as filters are typically
     global.
 
+``target_task_labels``
+    List of task labels to select. Labels not listed will be filtered out.
+    Enabled on try only.
+
 ``target_tasks_method``
     The method to use to determine the target task set.  This is the suffix of
     one of the functions in ``taskcluster/taskgraph/target_tasks.py``.
 
 ``optimize_target_tasks``
-   If true, then target tasks are eligible for optimization.
+    If true, then target tasks are eligible for optimization.
 
 ``include_nightly``
-   If true, then nightly tasks are eligible for optimization.
+    If true, then nightly tasks are eligible for optimization.
+
+Morphed Set
+-----------
+
+``morph_templates``
+    Dict of JSON-e templates to apply to each task, keyed by template name.
+    Values are extra context that will be available to the template under the
+    ``input.<template>`` key. Available templates live in
+    ``taskcluster/taskgraph/templates``. Enabled on try only.
--- a/taskcluster/taskgraph/decision.py
+++ b/taskcluster/taskgraph/decision.py
@@ -161,16 +161,18 @@ def get_decision_parameters(options):
     ] if n in options}
 
     # Define default filter list, as most configurations shouldn't need
     # custom filters.
     parameters['filters'] = [
         'check_servo',
         'target_tasks_method',
     ]
+    parameters['target_task_labels'] = []
+    parameters['morph_templates'] = {}
 
     # owner must be an email, but sometimes (e.g., for ffxbld) it is not, in which
     # case, fake it
     if '@' not in parameters['owner']:
         parameters['owner'] += '@noreply.mozilla.org'
 
     # use the pushdate as build_date if given, else use current time
     parameters['build_date'] = parameters['pushdate'] or int(time.time())
@@ -182,16 +184,25 @@ def get_decision_parameters(options):
     try:
         parameters.update(PER_PROJECT_PARAMETERS[project])
     except KeyError:
         logger.warning("using default project parameters; add {} to "
                        "PER_PROJECT_PARAMETERS in {} to customize behavior "
                        "for this project".format(project, __file__))
         parameters.update(PER_PROJECT_PARAMETERS['default'])
 
+    # morph_templates and target_task_labels are only used on try, so don't
+    # bother loading them elsewhere
+    task_config_file = os.path.join(GECKO, 'try_task_config.json')
+    if project == 'try' and os.path.isfile(task_config_file):
+        with open(task_config_file, 'r') as fh:
+            task_config = json.load(fh)
+        parameters['morph_templates'] = task_config.get('templates')
+        parameters['target_task_labels'] = task_config.get('tasks')
+
     # `target_tasks_method` has higher precedence than `project` parameters
     if options.get('target_tasks_method'):
         parameters['target_tasks_method'] = options['target_tasks_method']
 
     return Parameters(parameters)
 
 
 def write_artifact(filename, data):
--- a/taskcluster/taskgraph/generator.py
+++ b/taskcluster/taskgraph/generator.py
@@ -275,17 +275,18 @@ class TaskGraphGenerator(object):
         if not self.parameters.get('optimize_target_tasks', True):
             do_not_optimize = target_task_set.graph.nodes
         optimized_task_graph, label_to_taskid = optimize_task_graph(target_task_graph,
                                                                     self.parameters,
                                                                     do_not_optimize)
 
         yield 'optimized_task_graph', optimized_task_graph
 
-        morphed_task_graph, label_to_taskid = morph(optimized_task_graph, label_to_taskid)
+        morphed_task_graph, label_to_taskid = morph(
+            optimized_task_graph, label_to_taskid, self.parameters)
 
         yield 'label_to_taskid', label_to_taskid
         yield 'morphed_task_graph', morphed_task_graph
 
     def _run_until(self, name):
         while name not in self._run_results:
             try:
                 k, v = self._run.next()
--- a/taskcluster/taskgraph/morph.py
+++ b/taskcluster/taskgraph/morph.py
@@ -14,23 +14,27 @@ the graph.
 # Note that the translation of `{'task-reference': '..'}` is handled in the
 # optimization phase (since optimization involves dealing with taskIds
 # directly).  Similarly, `{'relative-datestamp': '..'}` is handled at the last
 # possible moment during task creation.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 import logging
+import os
 import re
 
+import jsone
+import yaml
 from slugid import nice as slugid
 from .task import Task
 from .graph import Graph
 from .taskgraph import TaskGraph
 
+here = os.path.abspath(os.path.dirname(__file__))
 logger = logging.getLogger(__name__)
 MAX_ROUTES = 10
 
 
 def amend_taskgraph(taskgraph, label_to_taskid, to_add):
     """Add the given tasks to the taskgraph, returning a new taskgraph"""
     new_tasks = taskgraph.tasks.copy()
     new_edges = taskgraph.graph.edges.copy()
@@ -236,17 +240,51 @@ def add_s3_uploader_task(taskgraph, labe
             added = make_s3_uploader_task(task)
             taskgraph, label_to_taskid = amend_taskgraph(
                 taskgraph, label_to_taskid, [added])
             update_test_tasks(added.task_id, task.task_id, taskgraph)
             logger.info('Added s3-uploader task')
     return taskgraph, label_to_taskid
 
 
-def morph(taskgraph, label_to_taskid):
+class apply_jsone_templates(object):
+    """Apply a set of JSON-e templates to each task's `task` attribute.
+
+    :param templates: A dict with the template name as the key, and extra context
+                      to use (in addition to task.to_json()) as the value.
+    """
+    template_dir = os.path.join(here, 'templates')
+
+    def __init__(self, templates):
+        self.templates = templates
+
+    def __call__(self, taskgraph, label_to_taskid):
+        if not self.templates:
+            return taskgraph, label_to_taskid
+
+        for task in taskgraph.tasks.itervalues():
+            for template in sorted(self.templates):
+                context = {
+                    'task': task.task,
+                    'taskGroup': None,
+                    'taskId': task.task_id,
+                    'kind': task.kind,
+                    'input': self.templates,
+                }
+
+                template_path = os.path.join(self.template_dir, template + '.yml')
+                with open(template_path) as f:
+                    template = yaml.load(f)
+                task.task = jsone.render(template, context)
+
+        return taskgraph, label_to_taskid
+
+
+def morph(taskgraph, label_to_taskid, parameters):
     """Apply all morphs"""
     morphs = [
         add_index_tasks,
         add_s3_uploader_task,
+        apply_jsone_templates(parameters.get('morph_templates')),
     ]
     for m in morphs:
         taskgraph, label_to_taskid = m(taskgraph, label_to_taskid)
     return taskgraph, label_to_taskid
--- a/taskcluster/taskgraph/parameters.py
+++ b/taskcluster/taskgraph/parameters.py
@@ -16,33 +16,40 @@ PARAMETER_NAMES = set([
     'build_date',
     'filters',
     'head_ref',
     'head_repository',
     'head_rev',
     'include_nightly',
     'level',
     'message',
+    'morph_templates',
     'moz_build_date',
     'optimize_target_tasks',
     'owner',
     'project',
     'pushdate',
     'pushlog_id',
+    'target_task_labels',
     'target_tasks_method',
 ])
 
+TRY_ONLY_PARAMETERS = set([
+    'morph_templates',
+    'target_task_labels',
+])
+
 
 class Parameters(ReadOnlyDict):
     """An immutable dictionary with nicer KeyError messages on failure"""
     def check(self):
         names = set(self)
         msg = []
 
-        missing = PARAMETER_NAMES - names
+        missing = PARAMETER_NAMES - TRY_ONLY_PARAMETERS - names
         if missing:
             msg.append("missing parameters: " + ", ".join(missing))
 
         extra = names - PARAMETER_NAMES
         if extra:
             msg.append("extra parameters: " + ", ".join(extra))
 
         if msg:
--- a/taskcluster/taskgraph/target_tasks.py
+++ b/taskcluster/taskgraph/target_tasks.py
@@ -2,21 +2,21 @@
 
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 import os
-import json
 
 from taskgraph import try_option_syntax
 from taskgraph.util.attributes import match_run_on_projects
 
+here = os.path.abspath(os.path.dirname(__file__))
 _target_task_methods = {}
 
 
 def _target_task(name):
     def wrap(func):
         _target_task_methods[name] = func
         return func
     return wrap
@@ -48,30 +48,21 @@ def filter_upload_symbols(task, paramete
 def standard_filter(task, parameters):
     return all(
         filter_func(task, parameters) for filter_func in
         (filter_on_nightly, filter_for_project, filter_upload_symbols)
     )
 
 
 def _try_task_config(full_task_graph, parameters):
-    task_config_file = os.path.join(os.getcwd(), 'try_task_config.json')
-
-    if not os.path.isfile(task_config_file):
+    if not parameters.get('target_task_labels'):
         return []
 
-    with open(task_config_file, 'r') as fh:
-        task_config = json.load(fh)
-
-    target_task_labels = []
-    for task in full_task_graph.tasks.itervalues():
-        if task.label in task_config:
-            target_task_labels.append(task.label)
-
-    return target_task_labels
+    return [t.label for t in full_task_graph.tasks.itervalues()
+            if t.label in parameters['target_task_labels']]
 
 
 def _try_option_syntax(full_task_graph, parameters):
     """Generate a list of target tasks based on try syntax in
     parameters['message'] and, for context, the full task graph."""
     options = try_option_syntax.TryOptionSyntax(parameters['message'], full_task_graph)
     target_tasks_labels = [t.label for t in full_task_graph.tasks.itervalues()
                            if options.task_matches(t)]
--- a/taskcluster/taskgraph/test/test_target_tasks.py
+++ b/taskcluster/taskgraph/test/test_target_tasks.py
@@ -1,15 +1,14 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function, unicode_literals
 
-import os
 import unittest
 
 from taskgraph import target_tasks
 from taskgraph import try_option_syntax
 from taskgraph.graph import Graph
 from taskgraph.taskgraph import TaskGraph
 from taskgraph.task import Task
 from mozunit import main
@@ -71,39 +70,38 @@ class TestTargetTasks(unittest.TestCase)
             'a': Task(kind=None, label='a', attributes={}, task={}),
             'b': Task(kind=None, label='b', attributes={'at-at': 'yep'}, task={}),
             'c': Task(kind=None, label='c', attributes={}, task={}),
         }
         graph = Graph(nodes=set('abc'), edges=set())
         tg = TaskGraph(tasks, graph)
 
         method = target_tasks.get_method('try_tasks')
-        config = os.path.join(os.getcwd(), 'try_task_config.json')
+        params = {
+            'message': '',
+            'target_task_labels': [],
+        }
 
         orig_TryOptionSyntax = try_option_syntax.TryOptionSyntax
         try:
             try_option_syntax.TryOptionSyntax = FakeTryOptionSyntax
 
             # no try specifier
-            self.assertEqual(method(tg, {'message': ''}), ['b'])
+            self.assertEqual(method(tg, params), ['b'])
 
             # try syntax only
-            self.assertEqual(method(tg, {'message': 'try: me'}), ['b'])
+            params['message'] = 'try: me'
+            self.assertEqual(method(tg, params), ['b'])
 
             # try task config only
-            with open(config, 'w') as fh:
-                fh.write('["c"]')
-            self.assertEqual(method(tg, {'message': ''}), ['c'])
-
-            with open(config, 'w') as fh:
-                fh.write('{"c": {}}')
-            self.assertEqual(method(tg, {'message': ''}), ['c'])
+            params['message'] = ''
+            params['target_task_labels'] = ['c']
+            self.assertEqual(method(tg, params), ['c'])
 
             # both syntax and config
-            self.assertEqual(set(method(tg, {'message': 'try: me'})), set(['b', 'c']))
+            params['message'] = 'try: me'
+            self.assertEqual(set(method(tg, params)), set(['b', 'c']))
         finally:
             try_option_syntax.TryOptionSyntax = orig_TryOptionSyntax
-            if os.path.isfile(config):
-                os.remove(config)
 
 
 if __name__ == '__main__':
     main()