Bug 1377875: update taskgraph how-to documentation; r=rforbes
authorDustin J. Mitchell <dustin@mozilla.com>
Sat, 01 Jul 2017 16:40:39 -0400
changeset 368780 185b16ee154d039a2d22b387642d173ccb3dad00
parent 368779 6c8692cc9c497492ee36bfcc42362db16a434f69
child 368781 30aea103f6b4f19cb9c5a6dd0f1e4b0d033faa47
push id32171
push userkwierso@gmail.com
push dateThu, 13 Jul 2017 22:51:35 +0000
treeherdermozilla-central@1afceff864d5 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersrforbes
bugs1377875
milestone56.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1377875: update taskgraph how-to documentation; r=rforbes Minor modificatinons based on conversations at the SF all-hands. MozReview-Commit-ID: AnkwXppUPJd
taskcluster/docs/how-tos.rst
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -150,66 +150,80 @@ Existing Kind
 The simplest option is to add your task to an existing kind.  This is most
 practical when the task "makes sense" as part of that kind -- for example, if
 your task is building an installer for a new platform using mozharness scripts
 similar to the existing build tasks, it makes most sense to add your task to
 the ``build`` kind.  If you need some additional functionality in the kind,
 it's OK to modify the implementation as necessary, as long as the modification
 is complete and useful to the next developer to come along.
 
+Tasks in the ``build`` kind generate Firefox installers, and the ``test`` kind
+will add a full set of Firefox tests for each ``build`` task.
+
 New Kind
 ````````
 
 The next option to consider is adding a new kind.  A distinct kind gives you
 some isolation from other task types, which can be nice if you are adding an
 experimental kind of task.
 
-Kinds can range in complexity.  The simplest sort of kind uses the
-``TransformTask`` implementation to read a list of jobs from the ``jobs`` key,
-and applies the standard ``job`` and ``task`` transforms:
+Kinds can range in complexity.  The simplest sort of kind uses the transform
+loader to read a list of jobs from the ``jobs`` key, and applies the standard
+``job`` and ``task`` transforms:
 
 .. code-block:: yaml
 
     implementation: taskgraph.task.transform:TransformTask
     transforms:
        - taskgraph.transforms.job:transforms
        - taskgraph.transforms.task:transforms
     jobs:
        - ..your job description here..
 
-Custom Kind Implementation
-``````````````````````````
+Job descriptions are defined and documented in
+``taskcluster/taskgraph/transforms/job/__init__.py``.
+
+Custom Kind Loader
+``````````````````
 
 If your task depends on other tasks, then the decision of which tasks to create
-may require some code.  For example, the ``upload-symbols`` kind iterates over
-the builds in the graph, generating a task for each one.  This specific
-post-build behavior is implemented in the general
-``taskgraph.task.post_build:PostBuildTask`` kind implementation.  If your task
-needs something more purpose-specific, then it may be time to write a new kind
-implementation.
+may require some code.  For example, the ``test`` kind iterates over
+the builds in the graph, generating a full set of test tasks for each one.  This specific
+post-build behavior is implemented as a loader defined in ``taskcluster/taskgraph/loader/test.py``.
+
+A custom loader is useful when the set of tasks you want to create is not
+static but based on something else (such as the available builds) or when the
+dependency relationships for your tasks are complex.
 
 Custom Transforms
 `````````````````
 
-If your task needs to create many tasks from a single description, for example
-to implement chunking, it is time to implement some custom transforms.  Ideally
-those transforms will produce job descriptions, so you can use the existing ``job``
-and ``task`` transforms:
+Most loaders apply a series of ":doc:`transforms <transforms>`" that start with
+an initial human-friendly description of a task and end with a task definition
+suitable for insertion into a Taskcluster queue.
+
+Custom transforms can be useful to apply defaults, simplifying the YAML files
+in your kind. They can also apply business logic that is more easily expressed
+in code than in YAML.
+
+Transforms need not be one-to-one: a transform can produce zero or more outputs
+for each input. For example, the test transforms perform chunking by producing
+an output for each chunk of a given input.
+
+Ideally those transforms will produce job descriptions, so you can use the
+existing ``job`` and ``task`` transforms:
 
 .. code-block:: yaml
 
     transforms:
        - taskgraph.transforms.my_stuff:transforms
        - taskgraph.transforms.job:transforms
        - taskgraph.transforms.task:transforms
 
-Similarly, if you need to include dynamic task defaults -- perhaps some feature
-is only available in level-3 repositories, or on specific projects -- then
-custom transforms are the appropriate tool.  Try to keep transforms simple,
-single-purpose and well-documented!
+Try to keep transforms simple, single-purpose and well-documented!
 
 Custom Run-Using
 ````````````````
 
 If the way your task is executed is unique (so, not a mach command or
 mozharness invocation), you can add a new implementation of the job
 description's "run" section.  Before you do this, consider that it might be a
 better investment to modify your task to support invocation via mozharness or