Bug 1274611: Implement task-graph optimization; r=gps
authorDustin J. Mitchell <dustin@mozilla.com>
Sun, 05 Jun 2016 19:49:41 +0000
changeset 300751 051b7c10b6175d6aba58af013f8e4e289c6d05d7
parent 300750 aa8cdba912d328070bf56330a3abe5b86e9cd136
child 300752 def3b07ccd145e92a8b5d2062dc437427028ad94
push id30321
push usercbook@mozilla.com
push dateTue, 07 Jun 2016 13:29:08 +0000
treeherderautoland@7f7c7d24700e [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersgps
bugs1274611
milestone49.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1274611: Implement task-graph optimization; r=gps * Implement & document optimization (although legacy kind doesn't do much of it) * Introduce `optimize_target_tasks` parameter to control whether tasks in the target set can be optimized (no for try, yes for most other branches) * Refactor to include resolved taskIds in the optimized task graph * Include a `label-to-taskid.json` artifact. * Introduce {'task-reference': '... <dependency-name> ...'} for referring to parent tasks' taskId. MozReview-Commit-ID: LWvlWNz49U5
taskcluster/docs/parameters.rst
taskcluster/docs/taskgraph.rst
taskcluster/mach_commands.py
taskcluster/taskgraph/create.py
taskcluster/taskgraph/decision.py
taskcluster/taskgraph/generator.py
taskcluster/taskgraph/graph.py
taskcluster/taskgraph/kind/base.py
taskcluster/taskgraph/kind/legacy.py
taskcluster/taskgraph/optimize.py
taskcluster/taskgraph/target_tasks.py
taskcluster/taskgraph/test/test_create.py
taskcluster/taskgraph/test/test_decision.py
taskcluster/taskgraph/test/test_generator.py
taskcluster/taskgraph/test/test_graph.py
taskcluster/taskgraph/test/test_kind_legacy.py
taskcluster/taskgraph/test/test_optimize.py
taskcluster/taskgraph/test/test_target_tasks.py
taskcluster/taskgraph/types.py
testing/taskcluster/tasks/post-builds/mulet_simulator.yml
testing/taskcluster/tasks/test.yml
testing/taskcluster/tasks/tests/fx_linux64_cppunit.yml
testing/taskcluster/tasks/tests/fx_linux64_crashtest.yml
testing/taskcluster/tasks/tests/fx_linux64_crashtest_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_external_media_tests.yml
testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional.yml
testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_gtest.yml
testing/taskcluster/tasks/tests/fx_linux64_jittests.yml
testing/taskcluster/tasks/tests/fx_linux64_jsreftest.yml
testing/taskcluster/tasks/tests/fx_linux64_jsreftest_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_luciddream.yml
testing/taskcluster/tasks/tests/fx_linux64_marionette.yml
testing/taskcluster/tasks/tests/fx_linux64_marionette_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_a11y.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_chrome.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_jetpack.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_media.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_media_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_mochitest_vg.yml
testing/taskcluster/tasks/tests/fx_linux64_reftest.yml
testing/taskcluster/tasks/tests/fx_linux64_reftest_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated.yml
testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests.yml
testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests.yml
testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests_e10s.yml
testing/taskcluster/tasks/tests/fx_linux64_xpcshell.yml
testing/taskcluster/tasks/tests/mulet_build_test.yml
testing/taskcluster/tasks/tests/mulet_build_unit.yml
testing/taskcluster/tasks/tests/mulet_gaia_js_integration_tests.yml
testing/taskcluster/tasks/tests/mulet_gaia_unit.yml
testing/taskcluster/tasks/tests/mulet_gaia_unit_oop.yml
testing/taskcluster/tasks/tests/mulet_linter.yml
testing/taskcluster/tasks/tests/mulet_mochitests.yml
testing/taskcluster/tasks/tests/mulet_reftests.yml
--- a/taskcluster/docs/parameters.rst
+++ b/taskcluster/docs/parameters.rst
@@ -78,8 +78,12 @@ with other ``mach taskgraph`` commands.
 ``target_tasks_method``
    (optional) The method to use to determine the target task set.  This is the
    suffix of one of the functions in ``tascluster/taskgraph/target_tasks.py``.
    If omitted, all tasks are targeted.
 
 ``target_tasks``
    (optional) The target set method ``from_parameters`` reads the target set, as
    a list of task labels, from this parameter.
+
+``optimize_target_tasks``
+   (optional; default True) If true, then target tasks are eligible for
+   optimization.
--- a/taskcluster/docs/taskgraph.rst
+++ b/taskcluster/docs/taskgraph.rst
@@ -82,16 +82,43 @@ Graph generation, as run via ``mach task
 #. Based on the full task graph, calculate the transitive closure of the target
    task set.  That is, the target tasks and all requirements of those tasks.
    The result is the "target task graph".
 #. Optimize the target task graph based on kind-specific optimization methods.
    The result is the "optimized task graph" with fewer nodes than the target
    task graph.
 #. Create tasks for all tasks in the optimized task graph.
 
+Optimization
+------------
+
+The objective of optimization to remove as many tasks from the graph as
+possible, as efficiently as possible, thereby delivering useful results as
+quickly as possible.  For example, ideally if only a test script is modified in
+a push, then the resulting graph contains only the corresponding test suite
+task.
+
+A task is said to be "optimized" when it is either replaced with an equivalent,
+already-existing task, or dropped from the graph entirely.
+
+A task can be optimized if all of its dependencies can be optimized and none of
+its inputs have changed.  For a task on which no other tasks depend (a "leaf
+task"), the optimizer can determine what has changed by looking at the
+version-control history of the push: if the relevant files are not modified in
+the push, then it considers the inputs unchanged.  For tasks on which other
+tasks depend ("non-leaf tasks"), the optimizer must replace the task with
+another, equivalent task, so it generates a hash of all of the inputs and uses
+that to search for a matching, existing task.
+
+In some cases, such as try pushes, tasks in the target task set have been
+explicitly requested and are thus excluded from optimization. In other cases,
+the target task set is almost the entire task graph, so targetted tasks are
+considered for optimization.  This behavior is controlled with the
+``optimize_target_tasks`` parameter.
+
 Mach commands
 -------------
 
 A number of mach subcommands are available aside from ``mach taskgraph
 decision`` to make this complex system more accesssible to those trying to
 understand or modify it.  They allow you to run portions of the
 graph-generation process and output the results.
 
@@ -115,8 +142,46 @@ parameters to guide the graph generation
 such a file on every run, and that is generally the easiest way to get a
 parameter file.  The parameter keys and values are described in
 :doc:`parameters`.
 
 Finally, the ``mach taskgraph decision`` subcommand performs the entire
 task-graph generation process, then creates the tasks.  This command should
 only be used within a decision task, as it assumes it is running in that
 context.
+
+Taskgraph JSON Format
+---------------------
+
+Each task graph artifact is represented as a JSON object.  The object's
+properties are the task labels or taskIds (see below), and the value of each
+property describes a task in an object with the following attributes:
+
+``label``
+   The task's label (never a taskId).
+
+``attributes``
+   The task's attributes
+
+``dependencies``
+   The task's in-graph dependencies, each represented as a pair ``[name, label]``
+   giving the dependency name and the label for the required task.
+
+``task``
+   The task's TaskCluster task definition.
+
+The task definition may contain "task references" of the form
+``{"task-reference": "string containing <task-label>"}``.  These will be
+replaced during the optimization step, with the appropriate taskId substituted
+for ``<task-label>`` in the string.  Multiple labels may be substituted in a
+single string, and ``<<>`` can be used to escape a literal ``<``.
+
+The results from each command are in the same format, but with some differences
+in the content:
+
+* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
+  That is, just collections of tasks without any dependencies indicated.
+
+* The ``optimized`` subcommand returns a graph keyed by taskId rather than
+  label.  The dependencies array, too, contains taskIds instead of labels.
+  Dependencies on optimized tasks are omitted.  However, the
+  ``task.dependencies`` array is populated with the full list of dependency
+  taskIds.  All task references are resolved in the optimized graph.
--- a/taskcluster/mach_commands.py
+++ b/taskcluster/mach_commands.py
@@ -124,20 +124,16 @@ class MachCommands(MachCommandBase):
         required=True,
         default=0)
     @CommandArgument('--owner',
         required=True,
         help='email address of who owns this graph')
     @CommandArgument('--level',
         required=True,
         help='SCM level of this repository')
-    @CommandArgument('--target-tasks-method',
-        required=False,
-        help='Method to use to determine the target task (e.g., `try_option_syntax`); '
-             'default is to run the full task graph')
     def taskgraph_decision(self, **options):
         """Run the decision task: generate a task graph and submit to
         TaskCluster.  This is only meant to be called within decision tasks,
         and requires a great many arguments.  Commands like `mach taskgraph
         optimized` are better suited to use on the command line, and can take
         the parameters file generated by a decision task.  """
 
         import taskgraph.decision
--- a/taskcluster/taskgraph/create.py
+++ b/taskcluster/taskgraph/create.py
@@ -8,35 +8,27 @@ import requests
 import json
 import collections
 import logging
 
 from slugid import nice as slugid
 
 logger = logging.getLogger(__name__)
 
-def create_tasks(taskgraph):
+def create_tasks(taskgraph, label_to_taskid):
     # TODO: use the taskGroupId of the decision task
     task_group_id = slugid()
-    label_to_taskid = collections.defaultdict(slugid)
+    taskid_to_label = {t: l for l, t in label_to_taskid.iteritems()}
 
     session = requests.Session()
 
-    for label in taskgraph.graph.visit_postorder():
-        task = taskgraph.tasks[label]
-        deps_by_name = {
-            n: label_to_taskid[r]
-            for (l, r, n) in taskgraph.graph.edges
-            if l == label}
-        task_def = task.kind.get_task_definition(task, deps_by_name)
+    for task_id in taskgraph.graph.visit_postorder():
+        task_def = taskgraph.tasks[task_id].task
         task_def['taskGroupId'] = task_group_id
-        task_def['dependencies'] = deps_by_name.values()
-        task_def['requires'] = 'all-completed'
-
-        _create_task(session, label_to_taskid[label], label, task_def)
+        _create_task(session, task_id, taskid_to_label[task_id], task_def)
 
 def _create_task(session, task_id, label, task_def):
     # create the task using 'http://taskcluster/queue', which is proxied to the queue service
     # with credentials appropriate to this job.
     logger.debug("Creating task with taskId {} for {}".format(task_id, label))
     res = session.put('http://taskcluster/queue/v1/task/{}'.format(task_id), data=json.dumps(task_def))
     if res.status_code != 200:
         try:
--- a/taskcluster/taskgraph/decision.py
+++ b/taskcluster/taskgraph/decision.py
@@ -21,21 +21,25 @@ ARTIFACTS_DIR = 'artifacts'
 
 logger = logging.getLogger(__name__)
 
 # For each project, this gives a set of parameters specific to the project.
 # See `taskcluster/docs/parameters.rst` for information on parameters.
 PER_PROJECT_PARAMETERS = {
     'try': {
         'target_tasks_method': 'try_option_syntax',
+        # for try, if a task was specified as a target, it should
+        # not be optimized away
+        'optimize_target_tasks': False,
     },
 
     # the default parameters are used for projects that do not match above.
     'default': {
-        'target_tasks_method': 'all_tasks',
+        'target_tasks_method': 'all_builds_and_tests',
+        'optimize_target_tasks': True,
     }
 }
 
 
 def taskgraph_decision(options):
     """
     Run the decision task.  This function implements `mach taskgraph decision`,
     and is responsible for
@@ -63,22 +67,24 @@ def taskgraph_decision(options):
     # write out the full graph for reference
     write_artifact('full-task-graph.json',
                    taskgraph_to_json(tgg.full_task_graph))
 
     # write out the target task set to allow reproducing this as input
     write_artifact('target_tasks.json',
                    tgg.target_task_set.tasks.keys())
 
-    # write out the optimized task graph to describe what will happen
+    # write out the optimized task graph to describe what will actually happen,
+    # and the map of labels to taskids
     write_artifact('task-graph.json',
                    taskgraph_to_json(tgg.optimized_task_graph))
+    write_artifact('label-to-taskid.json', tgg.label_to_taskid)
 
     # actually create the graph
-    create_tasks(tgg.optimized_task_graph)
+    create_tasks(tgg.optimized_task_graph, tgg.label_to_taskid)
 
 
 def get_decision_parameters(options):
     """
     Load parameters from the command-line options for 'taskgraph decision'.
     This also applies per-project parameters, based on the given project.
 
     """
@@ -108,16 +114,17 @@ def get_decision_parameters(options):
     return Parameters(parameters)
 
 
 def taskgraph_to_json(taskgraph):
     tasks = taskgraph.tasks
 
     def tojson(task):
         return {
+            'label': task.label,
             'task': task.task,
             'attributes': task.attributes,
             'dependencies': []
         }
     rv = {label: tojson(tasks[label]) for label in taskgraph.graph.nodes}
 
     # add dependencies with one trip through the graph edges
     for (left, right, name) in taskgraph.graph.edges:
--- a/taskcluster/taskgraph/generator.py
+++ b/taskcluster/taskgraph/generator.py
@@ -1,19 +1,21 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function, unicode_literals
 import logging
 import os
+import re
 import yaml
 
 from .graph import Graph
 from .types import TaskGraph
+from .optimize import optimize_task_graph
 
 logger = logging.getLogger(__name__)
 
 class TaskGraphGenerator(object):
     """
     The central controller for taskgraph.  This handles all phases of graph
     generation.  The task is generated from all of the kinds defined in
     subdirectories of the generator's root directory.
@@ -94,16 +96,26 @@ class TaskGraphGenerator(object):
         The set of targetted tasks and all of their dependencies; tasks that
         have been optimized out are either omitted or replaced with a Task
         instance containing only a task_id.
 
         @type: TaskGraph
         """
         return self._run_until('optimized_task_graph')
 
+    @property
+    def label_to_taskid(self):
+        """
+        A dictionary mapping task label to assigned taskId.  This property helps
+        in interpreting `optimized_task_graph`.
+
+        @type: dictionary
+        """
+        return self._run_until('label_to_taskid')
+
     def _load_kinds(self):
         for path in os.listdir(self.root_dir):
             path = os.path.join(self.root_dir, path)
             if not os.path.isdir(path):
                 continue
             name = os.path.basename(path)
             logger.debug("loading kind `{}` from `{}`".format(name, path))
 
@@ -160,20 +172,23 @@ class TaskGraphGenerator(object):
 
         logger.info("Generating target task graph")
         target_graph = full_task_graph.graph.transitive_closure(target_tasks)
         target_task_graph = TaskGraph(
             {l: all_tasks[l] for l in target_graph.nodes},
             target_graph)
         yield 'target_task_graph', target_task_graph
 
-        # optimization is not yet implemented
-
         logger.info("Generating optimized task graph")
-        yield 'optimized_task_graph', target_task_graph
+        do_not_optimize = set()
+        if not self.parameters.get('optimize_target_tasks', True):
+            do_not_optimize = target_task_set.graph.nodes
+        optimized_task_graph, label_to_taskid = optimize_task_graph(target_task_graph, do_not_optimize)
+        yield 'label_to_taskid', label_to_taskid
+        yield 'optimized_task_graph', optimized_task_graph
 
     def _run_until(self, name):
         while name not in self._run_results:
             try:
                 k, v = self._run.next()
             except StopIteration:
                 raise AttributeError("No such run result {}".format(name))
             self._run_results[k] = v
--- a/taskcluster/taskgraph/graph.py
+++ b/taskcluster/taskgraph/graph.py
@@ -80,25 +80,35 @@ class Graph(object):
                 seen.add(node)
                 yield node
             else:
                 queue.extend(n for n in links if n not in seen)
                 queue.append(node)
 
     def links_dict(self):
         """
-        Return a dictionary mapping each node to a set of its downstream
-        nodes (omitting edge names)
+        Return a dictionary mapping each node to a set of the nodes it links to
+        (omitting edge names)
         """
         links = collections.defaultdict(set)
         for left, right, _ in self.edges:
             links[left].add(right)
         return links
 
+    def named_links_dict(self):
+        """
+        Return a two-level dictionary mapping each node to a dictionary mapping
+        edge names to labels.
+        """
+        links = collections.defaultdict(dict)
+        for left, right, name in self.edges:
+            links[left][name] = right
+        return links
+
     def reverse_links_dict(self):
         """
-        Return a dictionary mapping each node to a set of its upstream
-        nodes (omitting edge names)
+        Return a dictionary mapping each node to a set of the nodes linking to
+        it (omitting edge names)
         """
         links = collections.defaultdict(set)
         for left, right, _ in self.edges:
             links[right].add(left)
         return links
--- a/taskcluster/taskgraph/kind/base.py
+++ b/taskcluster/taskgraph/kind/base.py
@@ -36,40 +36,22 @@ class Kind(object):
     def get_task_dependencies(self, task, taskgraph):
         """
         Get the set of task labels this task depends on, by querying the task graph.
 
         Returns a list of (task_label, dependency_name) pairs describing the
         dependencies.
         """
 
-    @abc.abstractmethod
-    def get_task_optimization_key(self, task, taskgraph):
-        """
-        Get the *optimization key* for the given task.  When called, all
-        dependencies of this task will already have their `optimization_key`
-        attribute set.
-
-        The optimization key is a unique identifier covering all inputs to this
-        task.  If another task with the same optimization key has already been
-        performed, it will be used directly instead of executing the task
-        again.
-
-        Returns a string suitable for inclusion in a TaskCluster index
-        namespace (generally of the form `<optimizationName>.<hash>`), or None
-        if this task cannot be optimized.
+    def optimize_task(self, task):
         """
+        Determine whether this task can be optimized, and if it can, what taskId
+        it should be replaced with.
 
-    @abc.abstractmethod
-    def get_task_definition(self, task, dependent_taskids):
+        The return value is a tuple `(optimized, taskId)`.  If `optimized` is
+        true, then the task will be optimized (in other words, not included in
+        the task graph).  If the second argument is a taskid, then any
+        dependencies on this task will isntead depend on that taskId.  It is an
+        error to return no taskId for a task on which other tasks depend.
+
+        The default never optimizes.
         """
-        Get the final task definition for the given task.  This is the time to
-        substitute actual taskIds for dependent tasks into the task definition.
-        Note that this method is only used in the decision tasks, so it should
-        not perform any processing that users might want to test or see in
-        other `mach taskgraph` commands.
-
-        The `dependent_taskids` parameter is a dictionary mapping dependency
-        name to assigned taskId.
-
-        The returned task definition will be modified before being passed to
-        `queue.createTask`.
-        """
+        return False, None
--- a/taskcluster/taskgraph/kind/legacy.py
+++ b/taskcluster/taskgraph/kind/legacy.py
@@ -39,17 +39,16 @@ from taskcluster_graph.from_now import (
 )
 from taskcluster_graph.templates import Templates
 import taskcluster_graph.build_task
 
 # TASKID_PLACEHOLDER is the "internal" form of a taskid; it is substituted with
 # actual taskIds at the very last minute, in get_task_definition
 TASKID_PLACEHOLDER = 'TaskLabel=={}'
 
-ARTIFACT_URL = 'https://queue.taskcluster.net/v1/task/{}/artifacts/{}'
 DEFINE_TASK = 'queue:define-task:aws-provisioner-v1/{}'
 DEFAULT_TRY = 'try: -b do -p all -u all -t all'
 DEFAULT_JOB_PATH = os.path.join(
     'tasks', 'branches', 'base_jobs.yml'
 )
 
 # time after which a try build's results will expire
 TRY_EXPIRATION = "14 days"
@@ -263,20 +262,17 @@ class LegacyKind(base.Kind):
                 attributes['build_type'] = {'dbg': 'debug'}.get(task_extra['build_type'],
                                                                 task_extra['build_type'])
             if build.get('is_job'):
                 attributes['job'] = build['build_name']
                 attributes['legacy_kind'] = 'job'
             graph['tasks'].append(build_task)
 
             for location in build_task['task']['extra'].get('locations', {}):
-                build_parameters['{}_url'.format(location)] = ARTIFACT_URL.format(
-                    build_parameters['build_slugid'],
-                    build_task['task']['extra']['locations'][location]
-                )
+                build_parameters['{}_location'.format(location)] = build_task['task']['extra']['locations'][location]
 
             for url in build_task['task']['extra'].get('url', {}):
                 build_parameters['{}_url'.format(url)] = \
                     build_task['task']['extra']['url'][url]
 
             define_task = DEFINE_TASK.format(build_task['task']['workerType'])
 
             for route in build_task['task'].get('routes', []):
@@ -434,26 +430,12 @@ class LegacyKind(base.Kind):
         return [Task(self, t['taskId'], task=t['task'], attributes=t['attributes'])
                 for t in self.graph['tasks']]
 
     def get_task_dependencies(self, task, taskgraph):
         # fetch dependency information from the cached graph
         taskdict = self.tasks_by_label[task.label]
         return [(label, label) for label in taskdict.get('requires', [])]
 
-    def get_task_optimization_key(self, task, taskgraph):
-        pass
+    def optimize_task(self, task, taskgraph):
+        # no optimization for the moment
+        return False, None
 
-    def get_task_definition(self, task, dependent_taskids):
-        # Note that the keys for `dependent_taskids` are task labels in this
-        # case, since that's how get_task_dependencies set it up.
-        placeholder_pattern = re.compile(r'TaskLabel==[a-zA-Z0-9-_]{22}')
-        def repl(mo):
-            return dependent_taskids[mo.group(0)]
-
-        # this is a cheap but easy way to replace all placeholders with
-        # actual real taskIds now that they are known.  The placeholder
-        # may be embedded in a longer string, so traversing the data structure
-        # would still require regexp matching each string and not be
-        # appreciably faster.
-        task_def = json.dumps(task.task)
-        task_def = placeholder_pattern.sub(repl, task_def)
-        return json.loads(task_def)
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/optimize.py
@@ -0,0 +1,143 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import absolute_import, print_function, unicode_literals
+import logging
+import re
+
+from .graph import Graph
+from .types import TaskGraph
+from slugid import nice as slugid
+
+logger = logging.getLogger(__name__)
+TASK_REFERENCE_PATTERN = re.compile('<([^>]+)>')
+
+
+def optimize_task_graph(target_task_graph, do_not_optimize):
+    """
+    Perform task optimization, without optimizing tasks named in
+    do_not_optimize.
+    """
+    named_links_dict = target_task_graph.graph.named_links_dict()
+    label_to_taskid = {}
+
+    # This proceeds in two phases.  First, mark all optimized tasks (those
+    # which will be removed from the graph) as such, including a replacement
+    # taskId where applicable.  Second, generate a new task graph containing
+    # only the non-optimized tasks, with all task labels resolved to taskIds
+    # and with task['dependencies'] populated.
+    annotate_task_graph(target_task_graph, do_not_optimize, named_links_dict, label_to_taskid)
+    return get_subgraph(target_task_graph, named_links_dict, label_to_taskid), label_to_taskid
+
+
+def resolve_task_references(label, task_def, taskid_for_edge_name):
+    def repl(match):
+        key = match.group(1)
+        try:
+            return taskid_for_edge_name[key]
+        except KeyError:
+            # handle escaping '<'
+            if key == '<':
+                return key
+            raise KeyError("task '{}' has no dependency with label '{}'".format(label, key))
+
+    def recurse(val):
+        if isinstance(val, list):
+            return [recurse(v) for v in val]
+        elif isinstance(val, dict):
+            if val.keys() == ['task-reference']:
+                return TASK_REFERENCE_PATTERN.sub(repl, val['task-reference'])
+            else:
+                return {k: recurse(v) for k, v in val.iteritems()}
+        else:
+            return val
+    return recurse(task_def)
+
+
+def annotate_task_graph(target_task_graph, do_not_optimize, named_links_dict, label_to_taskid):
+    """
+    Annotate each task in the graph with .optimized (boolean) and .task_id
+    (possibly None), following the rules for optimization and calling the task
+    kinds' `optimize_task` method.
+
+    As a side effect, label_to_taskid is updated with labels for all optimized
+    tasks that are replaced with existing tasks.
+    """
+
+    # set .optimized for all tasks, and .task_id for optimized tasks
+    # with replacements
+    for label in target_task_graph.graph.visit_postorder():
+        task = target_task_graph.tasks[label]
+        named_task_dependencies = named_links_dict.get(label, {})
+
+        # check whether any dependencies have been optimized away
+        dependencies = [target_task_graph.tasks[l] for l in named_task_dependencies.itervalues()]
+        for t in dependencies:
+            if t.optimized and not t.task_id:
+                raise Exception("task {} was optimized away, but {} depends on it".format(t.label, label))
+
+        # if this task is blacklisted, don't even consider optimizing
+        replacement_task_id = None
+        if label in do_not_optimize:
+            optimized = False
+        # if any dependencies can't be optimized, this task can't, either
+        elif any(not t.optimized for t in dependencies):
+            optimized = False
+        # otherwise, examine the task itself (which may be an expensive operation)
+        else:
+            optimized, replacement_task_id = task.kind.optimize_task(task, named_task_dependencies)
+
+        task.optimized = optimized
+        task.task_id = replacement_task_id
+        if replacement_task_id:
+            label_to_taskid[label] = replacement_task_id
+
+        if optimized:
+            if replacement_task_id:
+                logger.debug("optimizing `{}`, replacing with task `{}`"
+                             .format(label, replacement_task_id))
+            else:
+                logger.debug("optimizing `{}` away".format(label))
+                # note: any dependent tasks will fail when they see this
+        else:
+            if replacement_task_id:
+                raise Exception("{}: optimize_task returned False with a taskId".format(label))
+
+
+def get_subgraph(annotated_task_graph, named_links_dict, label_to_taskid):
+    """
+    Return the subgraph of annotated_task_graph consisting only of
+    non-optimized tasks and edges between them.
+
+    To avoid losing track of taskIds for tasks optimized away, this method
+    simultaneously substitutes real taskIds for task labels in the graph, and
+    populates each task definition's `dependencies` key with the appropriate
+    taskIds.  Task references are resolved in the process.
+    """
+
+    # resolve labels to taskIds and populate task['dependencies']
+    tasks_by_taskid = {}
+    for label in annotated_task_graph.graph.visit_postorder():
+        task = annotated_task_graph.tasks[label]
+        if task.optimized:
+            continue
+        task.task_id = label_to_taskid[label] = slugid()
+        named_task_dependencies = {
+                name: label_to_taskid[label]
+                for name, label in named_links_dict.get(label, {}).iteritems()}
+        task.task = resolve_task_references(task.label, task.task, named_task_dependencies)
+        task.task.setdefault('dependencies', []).extend(named_task_dependencies.itervalues())
+        tasks_by_taskid[task.task_id] = task
+
+    # resolve edges to taskIds
+    edges_by_taskid = ((label_to_taskid.get(left), label_to_taskid.get(right), name)
+        for (left, right, name) in annotated_task_graph.graph.edges)
+    # ..and drop edges that are no longer in the task graph
+    edges_by_taskid = set((left, right, name)
+        for (left, right, name) in edges_by_taskid
+        if left in tasks_by_taskid and right in tasks_by_taskid)
+
+    return TaskGraph(
+        tasks_by_taskid,
+        Graph(set(tasks_by_taskid), edges_by_taskid))
--- a/taskcluster/taskgraph/target_tasks.py
+++ b/taskcluster/taskgraph/target_tasks.py
@@ -28,13 +28,15 @@ def target_tasks_from_parameters(full_ta
 @_target_task('try_option_syntax')
 def target_tasks_try_option_syntax(full_task_graph, parameters):
     """Generate a list of target tasks based on try syntax in
     parameters['message'] and, for context, the full task graph."""
     options = try_option_syntax.TryOptionSyntax(parameters['message'], full_task_graph)
     return [t.label for t in full_task_graph.tasks.itervalues()
             if options.task_matches(t.attributes)]
 
-@_target_task('all_tasks')
-def target_tasks_all_tasks(full_task_graph, parameters):
-    """Trivially target all tasks."""
-    return full_task_graph.tasks.keys()
-
+@_target_task('all_builds_and_tests')
+def target_tasks_all_builds_and_tests(full_task_graph, parameters):
+    """Trivially target all build and test tasks.  This is used for
+    branches where we want to build "everyting", but "everything"
+    does not include uninteresting things like docker images"""
+    return [t.label for t in full_task_graph.tasks.itervalues()
+            if t.attributes.get('kind') == 'legacy']
--- a/taskcluster/taskgraph/test/test_create.py
+++ b/taskcluster/taskgraph/test/test_create.py
@@ -32,26 +32,27 @@ class TestCreate(unittest.TestCase):
         create._create_task = self.old_create_task
 
     def fake_create_task(self, session, task_id, label, task_def):
         self.created_tasks[task_id] = task_def
 
     def test_create_tasks(self):
         kind = FakeKind()
         tasks = {
-            'a': Task(kind=kind, label='a'),
-            'b': Task(kind=kind, label='b'),
+            'tid-a': Task(kind=kind, label='a', task={'payload': 'hello world'}),
+            'tid-b': Task(kind=kind, label='b', task={'payload': 'hello world'}),
         }
-        graph = Graph(nodes=set('ab'), edges={('a', 'b', 'edge')})
+        label_to_taskid = {'a': 'tid-a', 'b': 'tid-b'}
+        graph = Graph(nodes={'tid-a', 'tid-b'}, edges={('tid-a', 'tid-b', 'edge')})
         taskgraph = TaskGraph(tasks, graph)
 
-        create.create_tasks(taskgraph)
+        create.create_tasks(taskgraph, label_to_taskid)
 
         for tid, task in self.created_tasks.iteritems():
             self.assertEqual(task['payload'], 'hello world')
             # make sure the dependencies exist, at least
-            for depid in task['dependencies']:
+            for depid in task.get('dependencies', []):
                 self.assertIn(depid, self.created_tasks)
 
 
 if __name__ == '__main__':
     main()
 
--- a/taskcluster/taskgraph/test/test_decision.py
+++ b/taskcluster/taskgraph/test/test_decision.py
@@ -25,21 +25,23 @@ class TestDecision(unittest.TestCase):
         }
         graph = Graph(nodes=set('ab'), edges={('a', 'b', 'edgelabel')})
         taskgraph = TaskGraph(tasks, graph)
 
         res = decision.taskgraph_to_json(taskgraph)
 
         self.assertEqual(res, {
             'a': {
+                'label': 'a',
                 'attributes': {'attr': 'a-task'},
                 'task': {},
                 'dependencies': [('edgelabel', 'b')],
             },
             'b': {
+                'label': 'b',
                 'attributes': {},
                 'task': {'task': 'def'},
                 'dependencies': [],
             }
         })
 
 
     def test_write_artifact_json(self):
--- a/taskcluster/taskgraph/test/test_generator.py
+++ b/taskcluster/taskgraph/test/test_generator.py
@@ -28,16 +28,19 @@ class FakeKind(object):
 
     def get_task_dependencies(self, task, full_task_set):
         i = task.extra['i']
         if i > 0:
             return [('t-{}'.format(i - 1), 'prev')]
         else:
             return []
 
+    def optimize_task(self, task, dependencies):
+        return False, None
+
 
 class WithFakeKind(TaskGraphGenerator):
 
     def _load_kinds(self):
         yield FakeKind()
 
 
 class TestGenerator(unittest.TestCase):
@@ -80,15 +83,19 @@ class TestGenerator(unittest.TestCase):
         self.target_tasks = ['t-1']
         self.assertEqual(self.tgg.target_task_graph.graph,
                          graph.Graph({'t-0', 't-1'},
                                      {('t-1', 't-0', 'prev')}))
         self.assertEqual(sorted(self.tgg.target_task_graph.tasks.keys()),
                          sorted(['t-0', 't-1']))
 
     def test_optimized_task_graph(self):
-        "The optimized task graph is the target task graph (for now)"
-        self.target_tasks = ['t-1']
+        "The optimized task graph contains task ids"
+        self.target_tasks = ['t-2']
+        tid = self.tgg.label_to_taskid
         self.assertEqual(self.tgg.optimized_task_graph.graph,
-                         self.tgg.target_task_graph.graph)
+             graph.Graph({tid['t-0'], tid['t-1'], tid['t-2']}, {
+                 (tid['t-1'], tid['t-0'], 'prev'),
+                 (tid['t-2'], tid['t-1'], 'prev'),
+             }))
 
 if __name__ == '__main__':
     main()
--- a/taskcluster/taskgraph/test/test_graph.py
+++ b/taskcluster/taskgraph/test/test_graph.py
@@ -132,16 +132,24 @@ class TestGraph(unittest.TestCase):
     def test_links_dict(self):
         "link dict for a graph with multiple edges is correct"
         self.assertEqual(self.multi_edges.links_dict(), {
             '2': set(['1']),
             '3': set(['1', '2']),
             '4': set(['3']),
         })
 
+    def test_named_links_dict(self):
+        "named link dict for a graph with multiple edges is correct"
+        self.assertEqual(self.multi_edges.named_links_dict(), {
+            '2': dict(red='1', blue='1'),
+            '3': dict(red='1', blue='2', green='2'),
+            '4': dict(green='3'),
+        })
+
     def test_reverse_links_dict(self):
         "reverse link dict for a graph with multiple edges is correct"
         self.assertEqual(self.multi_edges.reverse_links_dict(), {
             '1': set(['2', '3']),
             '2': set(['3']),
             '3': set(['4']),
         })
 
--- a/taskcluster/taskgraph/test/test_kind_legacy.py
+++ b/taskcluster/taskgraph/test/test_kind_legacy.py
@@ -14,26 +14,11 @@ from mozunit import main
 class TestLegacyKind(unittest.TestCase):
     # NOTE: much of LegacyKind is copy-pasted from the old legacy code, which
     # is emphatically *not* designed for testing, so this test class does not
     # attempt to test the entire class.
 
     def setUp(self):
         self.kind = LegacyKind('/root', {})
 
-    def test_get_task_definition_artifact_sub(self):
-        "get_task_definition correctly substiatutes artifact URLs"
-        task_def = {
-            'input_file': TASKID_PLACEHOLDER.format("G5BoWlCBTqOIhn3K3HyvWg"),
-            'embedded': 'TASK={} FETCH=lazy'.format(
-                TASKID_PLACEHOLDER.format('G5BoWlCBTqOIhn3K3HyvWg')),
-        }
-        task = Task(self.kind, 'label', task=task_def)
-        dep_taskids = {TASKID_PLACEHOLDER.format('G5BoWlCBTqOIhn3K3HyvWg'): 'parent-taskid'}
-        task_def = self.kind.get_task_definition(task, dep_taskids)
-        self.assertEqual(task_def, {
-            'input_file': 'parent-taskid',
-            'embedded': 'TASK=parent-taskid FETCH=lazy',
-        })
-
 
 if __name__ == '__main__':
     main()
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/test/test_optimize.py
@@ -0,0 +1,240 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+import unittest
+
+from ..optimize import optimize_task_graph, resolve_task_references
+from ..optimize import annotate_task_graph, get_subgraph
+from .. import optimize
+from .. import types
+from .. import graph
+from mozunit import main
+
+
+class TestResolveTaskReferences(unittest.TestCase):
+
+    def do(self, input, output):
+        taskid_for_edge_name = {'edge%d' % n: 'tid%d' % n for n in range(1, 4)}
+        self.assertEqual(resolve_task_references('subject', input, taskid_for_edge_name), output)
+
+    def test_in_list(self):
+        "resolve_task_references resolves task references in a list"
+        self.do({'in-a-list': ['stuff', {'task-reference': '<edge1>'}]},
+                {'in-a-list': ['stuff', 'tid1']})
+
+    def test_in_dict(self):
+        "resolve_task_references resolves task references in a dict"
+        self.do({'in-a-dict': {'stuff': {'task-reference': '<edge2>'}}},
+                {'in-a-dict': {'stuff': 'tid2'}})
+
+    def test_multiple(self):
+        "resolve_task_references resolves multiple references in the same string"
+        self.do({'multiple': {'task-reference': 'stuff <edge1> stuff <edge2> after'}},
+                {'multiple': 'stuff tid1 stuff tid2 after'})
+
+    def test_embedded(self):
+        "resolve_task_references resolves ebmedded references"
+        self.do({'embedded': {'task-reference': 'stuff before <edge3> stuff after'}},
+                {'embedded': 'stuff before tid3 stuff after'})
+
+    def test_escaping(self):
+        "resolve_task_references resolves escapes in task references"
+        self.do({'escape': {'task-reference': '<<><edge3>>'}},
+                {'escape': '<tid3>'})
+
+    def test_invalid(self):
+        "resolve_task_references raises a KeyError on reference to an invalid task"
+        self.assertRaisesRegexp(KeyError, "task 'subject' has no dependency with label 'no-such'", lambda:
+            resolve_task_references('subject', {'task-reference': '<no-such>'}, {}))
+
+
+class FakeKind(object):
+
+    def __init__(self, optimize_task):
+        self.optimize_task = optimize_task
+
+
+class TestOptimize(unittest.TestCase):
+
+    kind = None
+
+    def make_kind(self, optimize_task):
+        self.kind = FakeKind(optimize_task)
+
+    def make_task(self, label, task_def=None, optimized=None, task_id=None):
+        task_def = task_def or {'sample': 'task-def'}
+        task = types.Task(self.kind, label=label, task=task_def)
+        task.optimized = optimized
+        task.task_id = task_id
+        return task
+
+    def make_graph(self, *tasks_and_edges):
+        tasks = {t.label: t for t in tasks_and_edges if isinstance(t, types.Task)}
+        edges = {e for e in tasks_and_edges if not isinstance(e, types.Task)}
+        return types.TaskGraph(tasks, graph.Graph(set(tasks), edges))
+
+    def assert_annotations(self, graph, **annotations):
+        def repl(task_id):
+            return 'SLUGID' if task_id and len(task_id) == 22 else task_id
+        got_annotations = {t.label: (t.optimized, repl(t.task_id)) for t in graph.tasks.itervalues()}
+        self.assertEqual(got_annotations, annotations)
+
+    def test_annotate_task_graph_no_optimize(self):
+        "annotating marks everything as un-optimized if the kind returns that"
+        self.make_kind(lambda task, deps: (False, None))
+        graph = self.make_graph(
+            self.make_task('task1'),
+            self.make_task('task2'),
+            self.make_task('task3'),
+            ('task2', 'task1', 'build'),
+            ('task2', 'task3', 'image'),
+        )
+        opt = annotate_task_graph(graph, set(),
+                graph.graph.named_links_dict(), {})
+        self.assert_annotations(graph,
+            task1=(False, None),
+            task2=(False, None),
+            task3=(False, None))
+
+    def test_annotate_task_graph_taskid_without_optimize(self):
+        "raises exception if kind returns a taskid without optimizing"
+        self.make_kind(lambda task, deps: (False, 'some-taskid'))
+        graph = self.make_graph(self.make_task('task1'))
+        self.assertRaises(Exception, lambda:
+            annotate_task_graph(graph, set(), graph.graph.named_links_dict(), {}))
+
+    def test_annotate_task_graph_optimize_away_dependency(self):
+        "raises exception if kind optimizes away a task on which another depends"
+        self.make_kind(lambda task, deps: (True, None) if task.label == 'task1' else (False, None))
+        graph = self.make_graph(
+            self.make_task('task1'),
+            self.make_task('task2'),
+            ('task2', 'task1', 'build'),
+        )
+        self.assertRaises(Exception, lambda:
+            annotate_task_graph(graph, set(), graph.graph.named_links_dict(), {}))
+
+    def test_annotate_task_graph_do_not_optimize(self):
+        "annotating marks everything as un-optimized if in do_not_optimize"
+        self.make_kind(lambda task, deps: (True, 'taskid'))
+        graph = self.make_graph(
+            self.make_task('task1'),
+            self.make_task('task2'),
+            ('task2', 'task1', 'build'),
+        )
+        label_to_taskid = {}
+        opt = annotate_task_graph(graph, {'task1', 'task2'},
+                graph.graph.named_links_dict(), label_to_taskid)
+        self.assert_annotations(graph,
+            task1=(False, None),
+            task2=(False, None))
+        self.assertEqual
+
+    def test_annotate_task_graph_nos_propagate(self):
+        "annotating marks a task with a non-optimized dependency as non-optimized"
+        self.make_kind(lambda task, deps: (False, None) if task.label == 'task1' else (True, 'taskid'))
+        graph = self.make_graph(
+            self.make_task('task1'),
+            self.make_task('task2'),
+            self.make_task('task3'),
+            ('task2', 'task1', 'build'),
+            ('task2', 'task3', 'image'),
+        )
+        opt = annotate_task_graph(graph, set(),
+                graph.graph.named_links_dict(), {})
+        self.assert_annotations(graph,
+            task1=(False, None),
+            task2=(False, None),  # kind would have returned (True, 'taskid') here
+            task3=(True, 'taskid'))
+
+    def test_get_subgraph_single_dep(self):
+        "when a single dependency is optimized, it is omitted from the graph"
+        graph = self.make_graph(
+            self.make_task('task1', optimized=True, task_id='dep1'),
+            self.make_task('task2', optimized=False),
+            self.make_task('task3', optimized=False),
+            ('task2', 'task1', 'build'),
+            ('task2', 'task3', 'image'),
+        )
+        label_to_taskid = {'task1': 'dep1'}
+        sub = get_subgraph(graph, graph.graph.named_links_dict(), label_to_taskid)
+        task2 = label_to_taskid['task2']
+        task3 = label_to_taskid['task3']
+        self.assertEqual(sub.graph.nodes, {task2, task3})
+        self.assertEqual(sub.graph.edges, {(task2, task3, 'image')})
+        self.assertEqual(sub.tasks[task2].task_id, task2)
+        self.assertEqual(sorted(sub.tasks[task2].task['dependencies']),
+                         sorted([task3, 'dep1']))
+        self.assertEqual(sub.tasks[task3].task_id, task3)
+        self.assertEqual(sorted(sub.tasks[task3].task['dependencies']), [])
+
+    def test_get_subgraph_dep_chain(self):
+        "when a dependency chain is optimized, it is omitted from the graph"
+        graph = self.make_graph(
+            self.make_task('task1', optimized=True, task_id='dep1'),
+            self.make_task('task2', optimized=True, task_id='dep2'),
+            self.make_task('task3', optimized=False),
+            ('task2', 'task1', 'build'),
+            ('task3', 'task2', 'image'),
+        )
+        label_to_taskid = {'task1': 'dep1', 'task2': 'dep2'}
+        sub = get_subgraph(graph, graph.graph.named_links_dict(), label_to_taskid)
+        task3 = label_to_taskid['task3']
+        self.assertEqual(sub.graph.nodes, {task3})
+        self.assertEqual(sub.graph.edges, set())
+        self.assertEqual(sub.tasks[task3].task_id, task3)
+        self.assertEqual(sorted(sub.tasks[task3].task['dependencies']), ['dep2'])
+
+    def test_get_subgraph_opt_away(self):
+        "when a leaf task is optimized away, it is omitted from the graph"
+        graph = self.make_graph(
+            self.make_task('task1', optimized=False),
+            self.make_task('task2', optimized=True),
+            ('task2', 'task1', 'build'),
+        )
+        label_to_taskid = {'task2': 'dep2'}
+        sub = get_subgraph(graph, graph.graph.named_links_dict(), label_to_taskid)
+        task1 = label_to_taskid['task1']
+        self.assertEqual(sub.graph.nodes, {task1})
+        self.assertEqual(sub.graph.edges, set())
+        self.assertEqual(sub.tasks[task1].task_id, task1)
+        self.assertEqual(sorted(sub.tasks[task1].task['dependencies']), [])
+
+    def test_get_subgraph_refs_resolved(self):
+        "get_subgraph resolves task references"
+        graph = self.make_graph(
+            self.make_task('task1', optimized=True, task_id='dep1'),
+            self.make_task('task2', optimized=False,
+                task_def={'payload': {'task-reference': 'http://<build>/<test>'}}),
+            ('task2', 'task1', 'build'),
+            ('task2', 'task3', 'test'),
+            self.make_task('task3', optimized=False),
+        )
+        label_to_taskid = {'task1': 'dep1'}
+        sub = get_subgraph(graph, graph.graph.named_links_dict(), label_to_taskid)
+        task2 = label_to_taskid['task2']
+        task3 = label_to_taskid['task3']
+        self.assertEqual(sub.graph.nodes, {task2, task3})
+        self.assertEqual(sub.graph.edges, {(task2, task3, 'test')})
+        self.assertEqual(sub.tasks[task2].task_id, task2)
+        self.assertEqual(sorted(sub.tasks[task2].task['dependencies']), sorted([task3, 'dep1']))
+        self.assertEqual(sub.tasks[task2].task['payload'], 'http://dep1/' + task3)
+        self.assertEqual(sub.tasks[task3].task_id, task3)
+
+    def test_optimize(self):
+        "optimize_task_graph annotates and extracts the subgraph from a simple graph"
+        self.make_kind(lambda task, deps: (True, 'dep1') if task.label == 'task1' else (False, None))
+        input = self.make_graph(
+            self.make_task('task1'),
+            self.make_task('task2'),
+            self.make_task('task3'),
+            ('task2', 'task1', 'build'),
+            ('task2', 'task3', 'image'),
+        )
+        opt, label_to_taskid = optimize_task_graph(input, set())
+        self.assertEqual(opt.graph, graph.Graph(
+            {label_to_taskid['task2'], label_to_taskid['task3']},
+            {(label_to_taskid['task2'], label_to_taskid['task3'], 'image')}))
--- a/taskcluster/taskgraph/test/test_target_tasks.py
+++ b/taskcluster/taskgraph/test/test_target_tasks.py
@@ -24,21 +24,24 @@ class FakeTryOptionSyntax(object):
 
 class TestTargetTasks(unittest.TestCase):
 
     def test_from_parameters(self):
         method = target_tasks.get_method('from_parameters')
         self.assertEqual(method(None, {'target_tasks': ['a', 'b']}),
                          ['a', 'b'])
 
-    def test_all_tasks(self):
-        method = target_tasks.get_method('all_tasks')
-        graph = TaskGraph(tasks={'a': Task(kind=None, label='a')},
-                          graph=Graph(nodes={'a'}, edges=set()))
-        self.assertEqual(method(graph, {}), ['a'])
+    def test_all_builds_and_tests(self):
+        method = target_tasks.get_method('all_builds_and_tests')
+        graph = TaskGraph(tasks={
+            'a': Task(kind=None, label='a', attributes={'kind': 'legacy'}),
+            'b': Task(kind=None, label='b', attributes={'kind': 'legacy'}),
+            'boring': Task(kind=None, label='boring', attributes={'kind': 'docker-image'}),
+        }, graph=Graph(nodes={'a', 'b', 'boring'}, edges=set()))
+        self.assertEqual(sorted(method(graph, {})), sorted(['a', 'b']))
 
     def test_try_option_syntax(self):
         tasks = {
             'a': Task(kind=None, label='a'),
             'b': Task(kind=None, label='b', attributes={'at-at': 'yep'}),
         }
         graph = Graph(nodes=set('ab'), edges=set())
         tg = TaskGraph(tasks, graph)
--- a/taskcluster/taskgraph/types.py
+++ b/taskcluster/taskgraph/types.py
@@ -24,17 +24,16 @@ class Task(object):
 
     def __init__(self, kind, label, attributes=None, task=None, **extra):
         self.kind = kind
         self.label = label
         self.attributes = attributes or {}
         self.task = task or {}
         self.extra = extra
 
-        self.optimization_key = None
         self.task_id = None
 
         if not (all(isinstance(x, basestring) for x in self.attributes.iterkeys()) and
                 all(isinstance(x, basestring) for x in self.attributes.itervalues())):
             raise TypeError("attribute names and values must be strings")
 
     def __str__(self):
         return "{} ({})".format(self.task_id or self.label,
--- a/testing/taskcluster/tasks/post-builds/mulet_simulator.yml
+++ b/testing/taskcluster/tasks/post-builds/mulet_simulator.yml
@@ -29,20 +29,21 @@ task:
       path: 'public/image.tar'
       taskId: '{{#task_id_for_image}}builder{{/task_id_for_image}}'
 
     maxRunTime: 600
 
     command:
       - /bin/bash
       - -exc
-      - >
+      - task-reference: >
         tc-vcs checkout ./gecko {{base_repository}} {{head_repository}} {{head_rev}} {{head_ref}} &&
         ./gecko/testing/taskcluster/scripts/builder/pull-gaia.sh ./gecko ./gaia gaia_props.py &&
-        ./gecko/testing/taskcluster/scripts/builder/build-simulator.sh ./gecko ./gaia {{build_url}}
+        ./gecko/testing/taskcluster/scripts/builder/build-simulator.sh ./gecko ./gaia
+          https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}
 
     artifacts:
       'public/build':
         type: directory
         path: '/home/worker/artifacts/'
         expires: '{{#from_now}}1 year{{/from_now}}'
 
     env:
--- a/testing/taskcluster/tasks/test.yml
+++ b/testing/taskcluster/tasks/test.yml
@@ -13,18 +13,18 @@ task:
 
   scopes:
     - 'docker-worker:feature:allowPtrace'
 
   payload:
     image: '{{#docker_image}}tester{{/docker_image}}'
     maxRunTime: 3600
     env:
-      MOZILLA_BUILD_URL: '{{build_url}}'
-      MOZHARNESS_URL: '{{mozharness_url}}'
+      MOZILLA_BUILD_URL: {"task-reference": "https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      MOZHARNESS_URL: {"task-reference": "https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{mozharness_location}}"}
 
     # All builds share a common artifact directory for ease of uploading.
     artifacts:
       'public/logs/':
         type: directory
         path: '/home/worker/workspace/build/upload/logs/'
         expires: '{{#from_now}}1 year{{/from_now}}'
       'public/test_info/':
--- a/testing/taskcluster/tasks/tests/fx_linux64_cppunit.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_cppunit.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --cppunittest-suite=cppunittest
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
           mozharness/configs/remove_executables.py
   metadata:
     name: '[TC] Linux64 cppunit'
--- a/testing/taskcluster/tasks/tests/fx_linux64_crashtest.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_crashtest.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_crashtest_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_crashtest_e10s.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_external_media_tests.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_external_media_tests.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/firefox_media_tests_buildbot.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/mediatests/buildbot_posix_config.py
           mozharness/configs/remove_executables.py
     maxRunTime: 5400
   metadata:
     name: '[TC] Linux64 external-media-tests'
--- a/testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional.yml
@@ -1,16 +1,16 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/firefox_ui_tests/functional.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/firefox_ui_tests/taskcluster.py
           mozharness/configs/remove_executables.py
     maxRunTime: 5400
   metadata:
     name: '[TC] Linux64 firefox-ui-tests functional'
--- a/testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_firefox_ui_functional_e10s.yml
@@ -1,16 +1,16 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/firefox_ui_tests/functional.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/firefox_ui_tests/taskcluster.py
           mozharness/configs/remove_executables.py
     maxRunTime: 5400
   metadata:
--- a/testing/taskcluster/tasks/tests/fx_linux64_gtest.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_gtest.yml
@@ -1,18 +1,18 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   workerType: desktop-test-xlarge
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
           mozharness/configs/remove_executables.py
   metadata:
     name: '[TC] Linux64 gtest'
--- a/testing/taskcluster/tasks/tests/fx_linux64_jittests.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_jittests.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_jsreftest.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_jsreftest.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_jsreftest_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_jsreftest_e10s.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_luciddream.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_luciddream.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
     env:
       MOZHARNESS_SCRIPT: 'mozharness/scripts/luciddream_unittest.py'
       MOZHARNESS_CONFIG: >
         mozharness/configs/luciddream/linux_config.py
   metadata:
     name: '[TC] Linux64 luciddream'
     description: Luciddream run
   extra:
--- a/testing/taskcluster/tasks/tests/fx_linux64_marionette.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_marionette.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/marionette.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/marionette/prod_config.py
           mozharness/configs/remove_executables.py
     maxRunTime: 5400
   metadata:
     name: '[TC] Linux64 marionette'
--- a/testing/taskcluster/tasks/tests/fx_linux64_marionette_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_marionette_e10s.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/marionette.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/marionette/prod_config.py
           mozharness/configs/remove_executables.py
     maxRunTime: 5400
   metadata:
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_a11y.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_a11y.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_bc_e10s.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_chrome.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_chrome.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_clipboard_e10s.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt.yml
@@ -5,18 +5,18 @@ task:
   payload:
     # these tests take longer than most to complete
     maxRunTime: 5400
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_dt_e10s.yml
@@ -5,18 +5,18 @@ task:
   payload:
     # these tests take longer than most to complete
     maxRunTime: 5400
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gl_e10s.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu.yml
@@ -2,18 +2,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_gpu_e10s.yml
@@ -2,18 +2,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_jetpack.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_jetpack.yml
@@ -4,18 +4,18 @@
 task:
   payload:
     maxRunTime: 5400
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_media.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_media.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_media_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_media_e10s.yml
@@ -3,18 +3,18 @@
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml
@@ -4,18 +4,18 @@
 task:
   payload:
     maxRunTime: 5400
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain_e10s.yml
@@ -4,18 +4,18 @@
 task:
   payload:
     maxRunTime: 5400
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_mochitest_vg.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_mochitest_vg.yml
@@ -6,18 +6,18 @@ task:
     - 'docker-worker:capability:device:loopbackVideo'
   payload:
     maxRunTime: 28800
     capabilities:
       devices:
         loopbackVideo: true
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --mochitest-suite=valgrind-plain
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
       MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
       MOZHARNESS_CONFIG: >
         mozharness/configs/unittests/linux_unittest.py
         mozharness/configs/remove_executables.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_reftest.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_reftest.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_reftest_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_reftest_e10s.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_reftest_not_accelerated_e10s.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
--- a/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests.yml
@@ -2,18 +2,18 @@
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   workerType: desktop-test-xlarge
   payload:
     maxRunTime: 7200
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --test-type=testharness
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/web_platform_tests.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/web_platform_tests/prod_config.py
           mozharness/configs/remove_executables.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_e10s.yml
@@ -2,18 +2,18 @@
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   workerType: desktop-test-xlarge
   payload:
     maxRunTime: 7200
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --test-type=testharness
       - --e10s
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/web_platform_tests.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/web_platform_tests/prod_config.py
--- a/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests.yml
@@ -2,18 +2,18 @@
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   workerType: desktop-test-xlarge
   payload:
     maxRunTime: 5400
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --test-type=reftest
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/web_platform_tests.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/web_platform_tests/prod_config.py
           mozharness/configs/remove_executables.py
   metadata:
     name: '[TC] Linux64 web-platform-tests-reftests'
--- a/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests_e10s.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_web_platform_tests_reftests_e10s.yml
@@ -1,18 +1,18 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     maxRunTime: 5400
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --test-type=reftest
       - --e10s
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/web_platform_tests.py'
         MOZHARNESS_CONFIG: >
           mozharness/configs/web_platform_tests/prod_config.py
           mozharness/configs/remove_executables.py
   metadata:
--- a/testing/taskcluster/tasks/tests/fx_linux64_xpcshell.yml
+++ b/testing/taskcluster/tasks/tests/fx_linux64_xpcshell.yml
@@ -1,17 +1,17 @@
 ---
 $inherits:
   from: 'tasks/tests/fx_desktop_generic.yml'
 task:
   payload:
     command:
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --{{suite}}-suite={{flavor}}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
     env:
         MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
         # TODO move linux_unittest.py to a platform specific config
         MOZHARNESS_CONFIG: >
           mozharness/configs/unittests/linux_unittest.py
--- a/testing/taskcluster/tasks/tests/mulet_build_test.yml
+++ b/testing/taskcluster/tasks/tests/mulet_build_test.yml
@@ -10,18 +10,18 @@ task:
     env:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_build_integration.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_integration.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --xre-url=https://api.pub.build.mozilla.org/tooltool/sha512/cefa8c00db04969d3a50e2a5509bd4ea1dc17d256a651a9518cb28dad72e87a1dbbcd3c88ef770be0edf0ab73d2d73925140df93618ffb7fab81b789d312f547
 
   extra:
     treeherderEnv:
       - production
       - staging
     treeherder:
       groupSymbol: "?"
--- a/testing/taskcluster/tasks/tests/mulet_build_unit.yml
+++ b/testing/taskcluster/tasks/tests/mulet_build_unit.yml
@@ -10,18 +10,18 @@ task:
     env:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_build_unit.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_integration.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
 
   extra:
     treeherderEnv:
       - production
       - staging
     treeherder:
       groupSymbol: "?"
       symbol: 'Gbu'
--- a/testing/taskcluster/tasks/tests/mulet_gaia_js_integration_tests.yml
+++ b/testing/taskcluster/tasks/tests/mulet_gaia_js_integration_tests.yml
@@ -11,18 +11,18 @@ task:
     env:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_integration.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_integration.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
 
   extra:
     chunks:
       total: 40
     treeherderEnv:
       - production
--- a/testing/taskcluster/tasks/tests/mulet_gaia_unit.yml
+++ b/testing/taskcluster/tasks/tests/mulet_gaia_unit.yml
@@ -10,18 +10,18 @@ task:
     env:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_unit.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_unit_production.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --xre-url=https://api.pub.build.mozilla.org/tooltool/sha512/cefa8c00db04969d3a50e2a5509bd4ea1dc17d256a651a9518cb28dad72e87a1dbbcd3c88ef770be0edf0ab73d2d73925140df93618ffb7fab81b789d312f547
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
 
   extra:
     treeherderEnv:
       - production
       - staging
--- a/testing/taskcluster/tasks/tests/mulet_gaia_unit_oop.yml
+++ b/testing/taskcluster/tasks/tests/mulet_gaia_unit_oop.yml
@@ -11,18 +11,18 @@ task:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_unit.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_unit_production.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
       - --browser-arg=-oop
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --xre-url=https://api.pub.build.mozilla.org/tooltool/sha512/cefa8c00db04969d3a50e2a5509bd4ea1dc17d256a651a9518cb28dad72e87a1dbbcd3c88ef770be0edf0ab73d2d73925140df93618ffb7fab81b789d312f547
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
 
   extra:
     treeherderEnv:
       - staging
     chunks:
--- a/testing/taskcluster/tasks/tests/mulet_linter.yml
+++ b/testing/taskcluster/tasks/tests/mulet_linter.yml
@@ -10,18 +10,18 @@ task:
     env:
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/gaia_linter.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_gaia_integration.py'
     command:
       - bash
       - /home/worker/bin/test.sh
       - --application=firefox
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
 
   extra:
     treeherderEnv:
       - production
       - staging
     treeherder:
       groupSymbol: "?"
       symbol: 'Li'
--- a/testing/taskcluster/tasks/tests/mulet_mochitests.yml
+++ b/testing/taskcluster/tasks/tests/mulet_mochitests.yml
@@ -12,18 +12,18 @@ task:
       MOZHARNESS_SCRIPT: 'mozharness/scripts/desktop_unittest.py'
       MOZHARNESS_CONFIG: >
         mozharness/configs/unittests/linux_unittest.py
         mozharness/configs/remove_executables.py
     command:
       - bash
       - /home/worker/bin/test.sh
       - --no-read-buildbot-config
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --download-symbols=ondemand
       - --mochitest-suite=plain-chunked
       - --total-chunk={{total_chunks}}
       - --this-chunk={{chunk}}
 
   extra:
     treeherderEnv:
       - production
--- a/testing/taskcluster/tasks/tests/mulet_reftests.yml
+++ b/testing/taskcluster/tasks/tests/mulet_reftests.yml
@@ -10,18 +10,18 @@ task:
     env:
       MOZ_DISABLE_NONLOCAL_CONNECTIONS: 0
       NEED_PULL_GAIA: true
       MOZHARNESS_SCRIPT: 'mozharness/scripts/mulet_unittest.py'
       MOZHARNESS_CONFIG: 'mozharness/configs/b2g/taskcluster_mulet_automation.py'
     command:
       - bash
       - /home/worker/bin/test.sh
-      - --installer-url={{build_url}}
-      - --test-packages-url={{test_packages_url}}
+      - {"task-reference": "--installer-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{build_location}}"}
+      - {"task-reference": "--test-packages-url=https://queue.taskcluster.net/v1/task/<{{build_slugid}}>/artifacts/{{test_packages_location}}"}
       - --test-suite=reftest
       - --test-manifest=tests/layout/reftests/reftest.list
       - --this-chunk={{chunk}}
       - --total-chunk={{total_chunks}}
 
   extra:
     treeherderEnv:
       - production