Bug 1258497: Document task-graph generation draft
authorDustin J. Mitchell <dustin@mozilla.com>
Fri, 29 Apr 2016 19:39:11 +0000
changeset 358384 689b83866aad24d2d5827f0c58cd9ad2d391c0fe
parent 358383 39d0e3d8071e9286bf13446974e9992928f67eb7
child 358385 b7a281510f328a68cdb65e2f7dd52d3a9caf762c
push id16995
push userdmitchell@mozilla.com
push dateMon, 02 May 2016 18:47:33 +0000
bugs1258497
milestone49.0a1
Bug 1258497: Document task-graph generation MozReview-Commit-ID: IP0LaZkEPKI
moz.build
taskcluster/PARAMETERS.md
taskcluster/TASK_ATTRIBUTES.md
taskcluster/docs/attributes.rst
taskcluster/docs/index.rst
taskcluster/docs/old.rst
taskcluster/docs/parameters.rst
taskcluster/docs/taskgraph.rst
taskcluster/mach_commands.py
taskcluster/moz.build
testing/moz.build
testing/taskcluster/docs/index.rst
--- a/moz.build
+++ b/moz.build
@@ -16,17 +16,16 @@ CONFIGURE_SUBST_FILES += [
 ]
 
 if CONFIG['ENABLE_CLANG_PLUGIN']:
     DIRS += ['build/clang-plugin']
 
 DIRS += [
     'config',
     'python',
-    'testing',
     'taskcluster',
 ]
 
 if not CONFIG['JS_STANDALONE']:
     CONFIGURE_SUBST_FILES += [
         'tools/update-packaging/Makefile',
     ]
     CONFIGURE_DEFINE_FILES += [
deleted file mode 100644
--- a/taskcluster/PARAMETERS.md
+++ /dev/null
@@ -1,76 +0,0 @@
-Task-Graph Generation Parameters
-================================
-
-Task-graph generation takes a collection of parameters as input, in the form of
-a JSON object.
-
-During decision-task processing, some of these parameters are supplied on the
-command line or by environment variables.  It helpfully produces a full
-parameters file as one of its output artifacts.  The other `mach taskgraph`
-commands can take this file as input.  This can be very helpful when working on
-a change to the task graph.
-
-The properties of the parameters object are described here, divided rougly by
-topic.
-
-Push Information
-----------------
-
-* `base_repository` - the repository from which to do an initial clone, utilizing
-  any available caching.
-
-* `head_repository` - the repository containing the changeset to be built.  This
-  may differ from `base_repository` in cases where `base_repository` is likely
-  to be cached and only a few additional commits are needed from
-  `head_repository`.
-
-* `head_rev` - the revision to check out; this can be a short revision string
-
-* `head_ref` - for Mercurial repositories, this is the same as `head_rev`.  For
-  git repositories, which do not allow pulling explicit revisions, this gives
-  the symbolic ref containing `head_rev` that should be pulled from
-  `head_repository`.
-
-* `revision_hash` - the full-length revision string
-
-* `revision_hash` - same thing?? XXX
-
-* `owner` - email address indicating the person who made the push.  Note that this
-  value may be forged and *must not* be relied on for authentication.
-
-* `message` (optional) - the commit message
-
-* `pushlog_id` (optional) - the ID from the `hg.mozilla.org` pushlog
-
-Tree Information
-----------------
-
-* `project` - another name for what may otherwise be called tree or branch or
-  repository.  This is the unqualified name, such as `mozilla-central` or
-  `cedar`.
-
-* `level` - the SCM level associated with this tree.  This dictates the names
-  of resources used in the generated tasks, and those tasks will fail if it
-  is incorrect.
-
-Target Set
-----------
-
-The "target set" is the set of task labels which must be included in a task
-graph.  The task graph generation process will include any tasks required by
-those in the target set, recursively.  In a decision task, this set can be
-specified programmatically using one of a variety of methods (e.g., parsing try
-syntax or reading a project-specific configuration file).
-
-The decision task writes its task set to the `task_set.json` artifact, and this
-can be copied into `parameters['task_set']` for debugging with other `mach
-taskgraph` commands.
-
-* `target_set_method` (optional) - the method to use to determine the target
-  task set.  This is the suffix of one of the `task_set_xxx` methods in
-  `tascluster/mach_commands:DecisionTask`.  If omitted, all tasks are targeted.
-
-* `target_set` (optional) - the target set method `from_parameters` reads the
-  target set, as a list of task labels, from this parameter.  The `mach
-  taskgraph` commands other than `decision` read from this value automatically,
-  if it is set.
deleted file mode 100644
--- a/taskcluster/TASK_ATTRIBUTES.md
+++ /dev/null
@@ -1,83 +0,0 @@
-Tasks can be filtered, for example to support "try" pushes which only perform a
-subset of the task graph or to link dependent tasks.  This filtering is the
-difference between a full task graph and a target task graph.
-
-Filtering takes place on the basis of attributes.  Each task has a dictionary
-of attributes (all strings), and a filter is an arbitrary expression over those
-attributes.  A task will not have a value for every attribute.
-
-The attributes, and acceptable values, are defined here.  In general, attribute
-names and values are the short, lower-case form, with underscores.
-
-kind
-====
-
-A task's `kind` attribute gives the name of the kind that generated it, e.g.,
-`build` or `legacy`.
-
-build_platform
-==============
-
-The build platform defines the platform for which the binary was built.  It is
-set for both build and test jobs, although test jobs may have a different
-`test_platform`.
-
-test_platform
-=============
-
-The test platform defines the platform on which tests are run.  It is only
-defined for test jobs and may differ from `build_platform` when the same binary
-is tested on several platforms (for example, on several versions of Windows).
-This applies for both talos and unit tests.
-
-build_type
-==========
-
-The type of build being performed.  This is a subdivision of `build_platform`,
-used for different kinds of builds that target the same platform.  Values are
-
- * debug
- * opt
-
-unittest_suite
-==============
-
-This is the unit test suite being run in a unit test task.  This matches the
-value passed to treeherder.
-
-unittest_flavor
-===============
-
-If a unittest suite has subdivisions, those are represented as flavors.  Not
-all suites have flavors, in which case this attribute should be omitted (but is
-sometimes set to match the suite).  This matches the value passed to treeherder.
-
-unittest_try_name
-=================
-
-This is the name used to refer to a unit test via try syntax.  It may not match
-either of `unittest_suite` or `unittest_flavor`.
-
-test_chunk
-==========
-
-This is the chunk number of a chunked test suite (talos or unittest).  Note
-that this is a string!
-
-legacy_kind
-===========
-
-(deprecated) The kind of task as created by the legacy kind.  This is valid
-only for the `legacy` kind.  One of `build`, `unittest,` `post_build`, or `job`.
-
-job
-===
-
-(deprecated) The name of the job (corresponding to a `-j` option or the name
-of a post-build job).  This is valid only for the `legacy` kind.
-
-post_build
-==========
-
-(deprecated) The name of the post-build activity.  This is valid only for the
-`legacy` kind.
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/attributes.rst
@@ -0,0 +1,89 @@
+===============
+Task Attributes
+===============
+
+Tasks can be filtered, for example to support "try" pushes which only perform a
+subset of the task graph or to link dependent tasks.  This filtering is the
+difference between a full task graph and a target task graph.
+
+Filtering takes place on the basis of attributes.  Each task has a dictionary
+of attributes (all strings), and a filter is an arbitrary expression over those
+attributes.  A task may not have a value for every attribute.
+
+The attributes, and acceptable values, are defined here.  In general, attribute
+names and values are the short, lower-case form, with underscores.
+
+kind
+====
+
+A task's ``kind`` attribute gives the name of the kind that generated it, e.g.,
+``build`` or ``legacy``.
+
+build_platform
+==============
+
+The build platform defines the platform for which the binary was built.  It is
+set for both build and test jobs, although test jobs may have a different
+``test_platform``.
+
+test_platform
+=============
+
+The test platform defines the platform on which tests are run.  It is only
+defined for test jobs and may differ from ``build_platform`` when the same binary
+is tested on several platforms (for example, on several versions of Windows).
+This applies for both talos and unit tests.
+
+build_type
+==========
+
+The type of build being performed.  This is a subdivision of ``build_platform``,
+used for different kinds of builds that target the same platform.  Values are
+
+ * ``debug``
+ * ``opt``
+
+unittest_suite
+==============
+
+This is the unit test suite being run in a unit test task.  This matches the
+value passed to treeherder.
+
+unittest_flavor
+===============
+
+If a unittest suite has subdivisions, those are represented as flavors.  Not
+all suites have flavors, in which case this attribute should be omitted (but is
+sometimes set to match the suite).  This matches the value passed to treeherder.
+
+unittest_try_name
+=================
+
+This is the name used to refer to a unit test via try syntax.  It may not match
+either of ``unittest_suite`` or ``unittest_flavor``.
+
+test_chunk
+==========
+
+This is the chunk number of a chunked test suite (talos or unittest).  Note
+that this is a string!
+
+legacy_kind
+===========
+
+(deprecated) The kind of task as created by the legacy kind.  This is valid
+only for the ``legacy`` kind.  One of ``build``, ``unittest,`` ``post_build``,
+or ``job``.
+
+job
+===
+
+(deprecated) The name of the job (corresponding to a ``-j`` option or the name
+of a post-build job).  This is valid only for the ``legacy`` kind.
+
+post_build
+==========
+
+(deprecated) The name of the post-build activity.  This is valid only for the
+``legacy`` kind.
+
rename from testing/taskcluster/docs/index.rst
rename to taskcluster/docs/index.rst
--- a/testing/taskcluster/docs/index.rst
+++ b/taskcluster/docs/index.rst
@@ -1,232 +1,20 @@
 .. taskcluster_index:
 
-======================
-TaskCluster Automation
-======================
-
-Directory structure
-===================
-
-tasks/
-   All task definitions
-
-tests/
-   Tests for the mach target internals related to task graph
-   generation
-
-scripts/
-   Various scripts used by taskcluster docker images and
-   utilities these exist in tree primarily to avoid rebuilding
-   docker images.
-
-Task Conventions
-================
-
-In order to properly enable task reuse there are a small number of
-conventions and parameters that are specialized for build tasks vs test
-tasks. The goal here should be to provide as much of the power as
-taskcluster but not at the cost of making it easy to support the current
-model of build/test.
-
-All tasks are in the YAML format and are also processed via mustache to
-allow for greater customizations. All tasks have the following
-templates variables:
-
-``docker_image``
-----------------
-Helper for always using the latest version of a docker image that exist
-in tree::
-
-   {{#docker_image}}base{{/docker_image}}
-
-Will produce something like (see the docker folder):
-
-   quay.io/mozilla.com/base:0.11
-
-
-``from_now``
-------------
-
-Helper for crafting a JSON date in the future.::
-
-
-   {{#from_now}}1 year{{/from_now}}
-
-Will produce::
-
-   2014-10-19T22:45:45.655Z
-
+TaskCluster Task-Graph Generation
+=================================
 
-``now``
--------
-
-Current time as a json formatted date.
-
-Build tasks
-===========
-
-By convention build tasks are stored in ``tasks/builds/`` the location of
-each particular type of build is specified in ``job_flags.yml`` (and more
-locations in the future), which is located in the appropriate subdirectory
-of ``branches/``.
-
-Task format
------------
-
-To facilitate better reuse of tasks there are some expectations of the
-build tasks. These are required for the test tasks to interact with the
-builds correctly but may not effect the builds or indexing services.
-
-.. code-block:: yaml
-
-    # This is an example of just the special fields. Other fields that are
-    # required by taskcluster are omitted and documented on http://docs.taskcluster.net/
-    task:
-
-      payload:
-        # Builders usually create at least two important artifacts the build
-        # and the tests these can be anywhere in the task and also may have
-        # different path names to include things like arch and extension
-        artifacts:
-          # The build this can be anything as long as its referenced in
-          # locations.
-          'public/name_i_made_up.tar.gz': '/path/to/build'
-          'public/some_tests.zip': '/path/to/tests'
-
-      extra:
-        # Build tasks may put their artifacts anywhere but there are common
-        # resources that test tasks need to do their job correctly so we
-        # need to provide an easy way to lookup the correct aritfact path.
-        locations:
-          build: 'public/name_i_made_up.tar.gz'
-          tests: 'public/some_tests.zip' or test_packages: 'public/target.test_packages.json'
-
-
-Templates properties
---------------------
-
-``repository``
-   Target HG repository (e.g.: ``https://hg.mozilla.org/mozilla-central``)
-
-``revision``
-   Target HG revision for gecko
-
-``owner``
-   Email address of the committer
+The ``taskcluster`` directory contains support for defining the graph of tasks
+that must be executed to build and test the Gecko tree.  This is more complex
+than you might suppose!  This implementation supports:
 
-Test Tasks
-==========
-
-By convention test tasks are stored in ``tasks/tests/`` the location of
-each particular type of build is specified in ``job_flags.yml`` (and more
-locations in the future)
-
-Template properties
--------------------
-
-repository
-   Target HG repository (e.g.: ``https://hg.mozilla.org/mozilla-central``)
-
-revision
-   Target HG revision for gecko
-
-owner
-   Email address of the committer
-
-build_url
-   Location of the build
-
-tests_url
-   Location of the tests.zip package
-
-chunk
-   Current chunk
-
-total_chunks
-   Total number of chunks
-
-Generic Tasks
-=============
-
-Generic tasks are neither build tasks nor test tasks. They are intended for
-tasks that don't fit into either category.
-
-.. important::
-
-   Generic tasks are a new feature and still under development. The
-   conventions will likely change significantly.
-
-Generic tasks are defined under a top-level ``tasks`` dictionary in the
-YAML. Keys in the dictionary are the unique task name. Values are
-dictionaries of task attributes. The following attributes can be defined:
-
-task
-   *required* Path to the YAML file declaring the task.
-
-root
-   *optional* Boolean indicating whether this is a *root* task. Root
-   tasks are scheduled immediately, if scheduled to run.
-
-additional-parameters
-   *optional* Dictionary of additional parameters to pass to template
-   expansion.
+ * A huge array of tasks
+ * Different behavior for different repositories
+ * "Try" pushes, with special means to select a subset of the graph for execution
+ * Optimization -- skipping tasks that have already been performed
 
-when
-   *optional* Dictionary of conditions that must be met for this task
-   to run. See the section below for more details.
-
-tags
-   *optional* List of string labels attached to the task. Multiple tasks
-   with the same tag can all be scheduled at once by specifying the tag
-   with the ``-j <tag>`` try syntax.
-
-Conditional Execution
----------------------
-
-The ``when`` generic task dictionary entry can declare conditions that
-must be true for a task to run. Valid entries in this dictionary are
-described below.
-
-file_patterns
-   List of path patterns that will be matched against all files changed.
-
-   The set of changed files is obtained from version control. If the changed
-   files could not be determined, this condition is ignored and no filtering
-   occurs.
-
-   Values use the ``mozpack`` matching code. ``*`` is a wildcard for
-   all path characters except ``/``. ``**`` matches all directories. To
-   e.g. match against all ``.js`` files, one would use ``**/*.js``.
-
-   If a single pattern matches a single changed file, the task will be
-   scheduled.
+.. toctree::
 
-Developing
-==========
-
-Running commands via mach is the best way to invoke commands testing
-works a little differently (I have not figured out how to invoke
-python-test without running install steps first)::
-
-   mach python-test tests/
-
-Examples
---------
-
-Requires `taskcluster-cli <https://github.com/taskcluster/taskcluster-cli>`_::
-
-    mach taskcluster-trygraph --message 'try: -b do -p all' \
-     --head-rev=33c0181c4a25 \
-     --head-repository=http://hg.mozilla.org/mozilla-central \
-     --owner=jlal@mozilla.com | taskcluster run-graph
-
-Creating only a build task and submitting to taskcluster::
-
-    mach taskcluster-build \
-      --head-revision=33c0181c4a25 \
-      --head-repository=http://hg.mozilla.org/mozilla-central \
-      --owner=user@domain.com tasks/builds/b2g_desktop.yml | taskcluster run-task --verbose
-
-    mach taskcluster-tests --task-id=Mcnvz7wUR_SEMhmWb7cGdQ  \
-     --owner=user@domain.com tasks/tests/b2g_mochitest.yml | taskcluster run-task --verbose
-
+    taskgraph
+    parameters
+    attributes
+    old
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/old.rst
@@ -0,0 +1,234 @@
+==================================
+Legacy TaskCluster Task Definition
+==================================
+
+The "legacy" task definitions are in ``testing/taskcluster``.
+
+These are being replaced by a more flexible system in ``taskcluster``.
+
+Directory structure
+===================
+
+tasks/
+   All task definitions
+
+tests/
+   Tests for the mach target internals related to task graph
+   generation
+
+scripts/
+   Various scripts used by taskcluster docker images and
+   utilities these exist in tree primarily to avoid rebuilding
+   docker images.
+
+Task Conventions
+================
+
+In order to properly enable task reuse there are a small number of
+conventions and parameters that are specialized for build tasks vs test
+tasks. The goal here should be to provide as much of the power as
+taskcluster but not at the cost of making it easy to support the current
+model of build/test.
+
+All tasks are in the YAML format and are also processed via mustache to
+allow for greater customizations. All tasks have the following
+templates variables:
+
+``docker_image``
+----------------
+Helper for always using the latest version of a docker image that exist
+in tree::
+
+   {{#docker_image}}base{{/docker_image}}
+
+Will produce something like (see the docker folder):
+
+   quay.io/mozilla.com/base:0.11
+
+
+``from_now``
+------------
+
+Helper for crafting a JSON date in the future.::
+
+
+   {{#from_now}}1 year{{/from_now}}
+
+Will produce::
+
+   2014-10-19T22:45:45.655Z
+
+
+``now``
+-------
+
+Current time as a json formatted date.
+
+Build tasks
+===========
+
+By convention build tasks are stored in ``tasks/builds/`` the location of
+each particular type of build is specified in ``job_flags.yml`` (and more
+locations in the future), which is located in the appropriate subdirectory
+of ``branches/``.
+
+Task format
+-----------
+
+To facilitate better reuse of tasks there are some expectations of the
+build tasks. These are required for the test tasks to interact with the
+builds correctly but may not effect the builds or indexing services.
+
+.. code-block:: yaml
+
+    # This is an example of just the special fields. Other fields that are
+    # required by taskcluster are omitted and documented on http://docs.taskcluster.net/
+    task:
+
+      payload:
+        # Builders usually create at least two important artifacts the build
+        # and the tests these can be anywhere in the task and also may have
+        # different path names to include things like arch and extension
+        artifacts:
+          # The build this can be anything as long as its referenced in
+          # locations.
+          'public/name_i_made_up.tar.gz': '/path/to/build'
+          'public/some_tests.zip': '/path/to/tests'
+
+      extra:
+        # Build tasks may put their artifacts anywhere but there are common
+        # resources that test tasks need to do their job correctly so we
+        # need to provide an easy way to lookup the correct aritfact path.
+        locations:
+          build: 'public/name_i_made_up.tar.gz'
+          tests: 'public/some_tests.zip' or test_packages: 'public/target.test_packages.json'
+
+
+Templates properties
+--------------------
+
+``repository``
+   Target HG repository (e.g.: ``https://hg.mozilla.org/mozilla-central``)
+
+``revision``
+   Target HG revision for gecko
+
+``owner``
+   Email address of the committer
+
+Test Tasks
+==========
+
+By convention test tasks are stored in ``tasks/tests/`` the location of
+each particular type of build is specified in ``job_flags.yml`` (and more
+locations in the future)
+
+Template properties
+-------------------
+
+repository
+   Target HG repository (e.g.: ``https://hg.mozilla.org/mozilla-central``)
+
+revision
+   Target HG revision for gecko
+
+owner
+   Email address of the committer
+
+build_url
+   Location of the build
+
+tests_url
+   Location of the tests.zip package
+
+chunk
+   Current chunk
+
+total_chunks
+   Total number of chunks
+
+Generic Tasks
+=============
+
+Generic tasks are neither build tasks nor test tasks. They are intended for
+tasks that don't fit into either category.
+
+.. important::
+
+   Generic tasks are a new feature and still under development. The
+   conventions will likely change significantly.
+
+Generic tasks are defined under a top-level ``tasks`` dictionary in the
+YAML. Keys in the dictionary are the unique task name. Values are
+dictionaries of task attributes. The following attributes can be defined:
+
+task
+   *required* Path to the YAML file declaring the task.
+
+root
+   *optional* Boolean indicating whether this is a *root* task. Root
+   tasks are scheduled immediately, if scheduled to run.
+
+additional-parameters
+   *optional* Dictionary of additional parameters to pass to template
+   expansion.
+
+when
+   *optional* Dictionary of conditions that must be met for this task
+   to run. See the section below for more details.
+
+tags
+   *optional* List of string labels attached to the task. Multiple tasks
+   with the same tag can all be scheduled at once by specifying the tag
+   with the ``-j <tag>`` try syntax.
+
+Conditional Execution
+---------------------
+
+The ``when`` generic task dictionary entry can declare conditions that
+must be true for a task to run. Valid entries in this dictionary are
+described below.
+
+file_patterns
+   List of path patterns that will be matched against all files changed.
+
+   The set of changed files is obtained from version control. If the changed
+   files could not be determined, this condition is ignored and no filtering
+   occurs.
+
+   Values use the ``mozpack`` matching code. ``*`` is a wildcard for
+   all path characters except ``/``. ``**`` matches all directories. To
+   e.g. match against all ``.js`` files, one would use ``**/*.js``.
+
+   If a single pattern matches a single changed file, the task will be
+   scheduled.
+
+Developing
+==========
+
+Running commands via mach is the best way to invoke commands testing
+works a little differently (I have not figured out how to invoke
+python-test without running install steps first)::
+
+   mach python-test tests/
+
+Examples
+--------
+
+Requires `taskcluster-cli <https://github.com/taskcluster/taskcluster-cli>`_::
+
+    mach taskcluster-trygraph --message 'try: -b do -p all' \
+     --head-rev=33c0181c4a25 \
+     --head-repository=http://hg.mozilla.org/mozilla-central \
+     --owner=jlal@mozilla.com | taskcluster run-graph
+
+Creating only a build task and submitting to taskcluster::
+
+    mach taskcluster-build \
+      --head-revision=33c0181c4a25 \
+      --head-repository=http://hg.mozilla.org/mozilla-central \
+      --owner=user@domain.com tasks/builds/b2g_desktop.yml | taskcluster run-task --verbose
+
+    mach taskcluster-tests --task-id=Mcnvz7wUR_SEMhmWb7cGdQ  \
+     --owner=user@domain.com tasks/tests/b2g_mochitest.yml | taskcluster run-task --verbose
+
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/parameters.rst
@@ -0,0 +1,76 @@
+==========
+Parameters
+==========
+
+Task-graph generation takes a collection of parameters as input, in the form of
+a JSON or YAML file.
+
+During decision-task processing, some of these parameters are supplied on the
+command line or by environment variables.  The decision task helpfully produces
+a full parameters file as one of its output artifacts.  The other ``mach
+taskgraph`` commands can take this file as input.  This can be very helpful
+when working on a change to the task graph.
+
+The properties of the parameters object are described here, divided rougly by
+topic.
+
+Push Information
+----------------
+
+* ``base_repository`` - the repository from which to do an initial clone, utilizing
+  any available caching.
+
+* ``head_repository`` - the repository containing the changeset to be built.  This
+  may differ from ``base_repository`` in cases where ``base_repository`` is likely
+  to be cached and only a few additional commits are needed from
+  ``head_repository``.
+
+* ``head_rev`` - the revision to check out; this can be a short revision string
+
+* ``head_ref`` - for Mercurial repositories, this is the same as ``head_rev``.  For
+  git repositories, which do not allow pulling explicit revisions, this gives
+  the symbolic ref containing ``head_rev`` that should be pulled from
+  ``head_repository``.
+
+* ``revision_hash`` - the full-length revision string
+
+* ``owner`` - email address indicating the person who made the push.  Note that this
+  value may be forged and *must not* be relied on for authentication.
+
+* ``message`` (optional) - the commit message
+
+* ``pushlog_id`` (optional) - the ID from the ``hg.mozilla.org`` pushlog
+
+Tree Information
+----------------
+
+* ``project`` - another name for what may otherwise be called tree or branch or
+  repository.  This is the unqualified name, such as ``mozilla-central`` or
+  ``cedar``.
+
+* ``level`` - the SCM level associated with this tree.  This dictates the names
+  of resources used in the generated tasks, and those tasks will fail if it
+  is incorrect.
+
+Target Set
+----------
+
+The "target set" is the set of task labels which must be included in a task
+graph.  The task graph generation process will include any tasks required by
+those in the target set, recursively.  In a decision task, this set can be
+specified programmatically using one of a variety of methods (e.g., parsing try
+syntax or reading a project-specific configuration file).
+
+The decision task writes its task set to the ``target_tasks.json`` artifact, and this
+can be copied into ``parameters.target_tasks`` and ``parameters.target_tasks_method`` set to ``"from_parameters"`` for debugging with other ``mach
+taskgraph`` commands.
+
+* ``target_tasks_method`` (optional) - the method to use to determine the target
+  task set.  This is the suffix of one of the ``task_set_xxx`` methods in
+  ``tascluster/mach_commands:DecisionTask``.  If omitted, all tasks are targeted.
+
+* ``target_tasks`` (optional) - the target set method ``from_parameters`` reads the
+  target set, as a list of task labels, from this parameter.  The ``mach
+  taskgraph`` commands other than ``decision`` read from this value automatically,
+  if it is set.
+
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/taskgraph.rst
@@ -0,0 +1,96 @@
+======================
+TaskGraph Mach Command
+======================
+
+The task graph is built by linking different kinds of tasks together, pruning
+out tasks that are not required, then optimizing by replacing subgraphs with
+links to already-completed tasks.
+
+Concepts
+--------
+
+* *Task Kind* - Tasks are grouped by kind, where tasks of the same kind do not
+  have interdependencies but have substantial similarities, and may depend on
+  tasks of other kinds.  Kinds are the primary means of supporting diversity, in
+  that a developer can add a new kind to do just about anything without
+  impacting other kinds.
+
+* *Task Attributes* - Tasks have string attributes by which can be used for filtering.
+  Attributes are documented in :doc:`attributes`.
+
+* *Task Labels* - Each task has a unique identifier within the graph that is
+  stable across runs of the graph generation algorithm.  Labels are replaced with
+  TaskCluster TaskIds at the latest time possible, preventing spurious taskId
+  changes in diffs made before that time.
+
+* *Optimization* - replacement of a task in a graph with an equivalent,
+  already-completed task, or a null task, avoiding repetition of work.
+
+Kinds
+-----
+
+Kinds are the focal point of this system.
+They provide an interface between the large-scale graph-generation process and the small-scale task-definition needs of different kinds of tasks.
+Each kind may implement task generation differently.
+Some kinds may generate task definitions entirely internally (for example, symbol-upload tasks are all alike, and very simple), while other kinds may do little more than parse a directory of YAML files.
+
+A `kind.yml` file contains data about the kind, as well as referring to a Python class implementing the kind in its ``implementation`` key.
+That implementation may rely on lots of code shared with other kinds, or contain a completely unique implementation of some functionality.
+
+The result is a nice segmentation of implementation so that the more esoteric in-tree projects can do their crazy stuff in an isolated kind without making the bread-and-butter build and test configuration more complicated.
+
+Dependencies
+------------
+
+Dependency links between tasks are always between different kinds(*).
+At a large scale, you can think of the dependency graph as one between kinds, rather than between tasks.
+For example, the unittest kind depends on the build kind.
+The details of *which* tasks of the two kinds are linked is left to the kind definition.
+
+(*) A kind can depend on itself, though.  You can safely ignore that detail.
+Tasks can also be linked within a kind using explicit dependencies.
+
+Decision Task
+-------------
+
+The decision task is the first task created when a new graph begins.
+It is responsible for creating the rest of the task graph.
+
+The decision task for pushes is defined in-tree, currently at ``testing/taskcluster/tasks/decision``.
+The task description invokes ``mach taskcluster decision`` with some metadata about the push.
+That mach command determines the optimized task graph, then calls the TaskCluster API to create the tasks.
+
+Graph Generation
+----------------
+
+Graph generation, as run via ``mach taskgraph decision``, proceeds as follows:
+
+1. For all kinds, generate all tasks.  The result is the "full task set"
+
+1. Create links between tasks using kind-specific mechanisms.  The result is the "full task graph".
+
+1. Select the target tasks (based on try syntax or a tree-specific specification).  The result is the "target task set".
+
+1. Based on the full task graph, calculate the transitive closure of the target task set.  That is, the target tasks and all requirements of those tasks.  The result is the "target task graph".
+
+
+1. Optimize the target task graph based on kind-specific optimization methods.  The result is the "optimized task graph" with fewer nodes than the target task graph.
+
+1. Create tasks for all tasks in the optimized task graph.
+
+Mach commands
+-------------
+
+A number of mach subcommands are available aside from ``mach taskgraph decision`` to make this complex system more accesssible to those trying to understand or modify it.
+They allow you to run portions of the graph-generation process and output the results.
+
+* ``mach taskgraph tasks`` -- get the full task set
+* ``mach taskgraph full`` -- get the full task graph
+* ``mach taskgraph target`` -- get the target task set
+* ``mach taskgraph target-graph`` -- get the target task graph
+* ``mach taskgraph optimized`` -- get the optimized task graph
+* ``mach taskgraph decision`` -- run the whole task-graph generation process (expects to be run in a decision task)
+
+Each of these commands taskes a ``--parameters`` option giving a file with parameters to guide the graph generation.
+The decision task helpfully produces such a file on every run, and that is generally the easiest way to get a parameter file.
+The parameter keys and values are described in :doc:`parameters`.
--- a/taskcluster/mach_commands.py
+++ b/taskcluster/mach_commands.py
@@ -34,17 +34,17 @@ class TaskGraphSubCommand(SubCommand):
 
     def __call__(self, func):
         after = SubCommand.__call__(self, func)
         args = [
             CommandArgument('--root', '-r', default='taskcluster/ci',
                             help="root of the taskgraph definition relative to topsrcdir"),
             CommandArgument('--parameters', '-p', required=False,
                             help="parameters file (.yml or .json; see "
-                                 "`taskcluster/PARAMETERS.md)`"),
+                                 "`taskcluster/docs/parameters.rst`)`"),
             CommandArgument('--no-optimize', dest="optimize", action="store_false",
                             default="true",
                             help="do not remove tasks from the graph that are found in the "
                             "index (a.k.a. optimize the graph)"),
         ]
         for arg in args:
             after = arg(after)
         return after
new file mode 100644
--- /dev/null
+++ b/taskcluster/moz.build
@@ -0,0 +1,7 @@
+# -*- Mode: python; c-basic-offset: 4; indent-tabs-mode: nil; tab-width: 40 -*-
+# vim: set filetype=python:
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+SPHINX_TREES['taskcluster'] = 'docs'
deleted file mode 100644
--- a/testing/moz.build
+++ /dev/null
@@ -1,7 +0,0 @@
-# -*- Mode: python; c-basic-offset: 4; indent-tabs-mode: nil; tab-width: 40 -*-
-# vim: set filetype=python:
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-SPHINX_TREES['taskcluster'] = 'taskcluster/docs'
\ No newline at end of file