taskcluster/docs/mach.rst
author Nick Alexander <nalexander@mozilla.com>
Mon, 16 Oct 2017 15:09:15 -0700
changeset 386854 1c49a7a7cb5ecab97b2bd3a3384631e6718b748f
parent 383774 a6d12cead25a83c94ab48943a53bd6bcd421524f
child 424340 e3d9b7024efa256b0835b0acdd807d75121bdbdf
permissions -rw-r--r--
Bug 1366644 - Part 3a: Update Android build-tools (25.0.3), Gradle (3.4.1), Android-Gradle (2.3.3). r=maliu The goal is to use a newer Android-Gradle build plugin version (2.3.3 is latest stable). That requires a modern Gradle (anything 3.3+, but 3.4.1 is the default from my Android Studio), and also a newer build-tools (25.0.3 is latest stable). The locations of lint output changed, and we want to use the standard output location because it's difficult to accommodate variant details in custom names. We change the location of findbugs output to follow suit. This requires either: - fixing lint errors - adding to the lint whitelist - using the new lint baseline It's best to use the new lint baseline, which will happen in the next commit. MozReview-Commit-ID: D19FzIDCJrE

Mach commands
=============

A number of mach subcommands are available aside from ``mach taskgraph
decision`` to make this complex system more accessible to those trying to
understand or modify it.  They allow you to run portions of the
graph-generation process and output the results.

``mach taskgraph tasks``
   Get the full task set

``mach taskgraph full``
   Get the full task graph

``mach taskgraph target``
   Get the target task set

``mach taskgraph target-graph``
   Get the target task graph

``mach taskgraph optimized``
   Get the optimized task graph

``mach taskgraph morphed``
   Get the morhped task graph

See :doc:`how-tos` for further practical tips on debugging task-graph mechanics
locally.

Parameters
----------

Each of these commands takes an optional ``--parameters`` argument giving a file
with parameters to guide the graph generation.  The decision task helpfully
produces such a file on every run, and that is generally the easiest way to get
a parameter file.  The parameter keys and values are described in
:doc:`parameters`; using that information, you may modify an existing
``parameters.yml`` or create your own.  The ``--parameters`` option can also
take the following forms:

``project=<project>``
   Fetch the parameters from the latest push on that project
``task-id=<task-id>``
   Fetch the parameters from the given decision task id

If not specified, parameters will default to ``project=mozilla-central``.

Taskgraph JSON Format
---------------------
By default, the above commands will only output a list of tasks. Use `-J` flag
to output full task definitions. For example:

.. code-block:: shell

    $ ./mach taskgraph optimized -J


Task graphs -- both the graph artifacts produced by the decision task and those
output by the ``--json`` option to the ``mach taskgraph`` commands -- are JSON
objects, keyed by label, or for optimized task graphs, by taskId.  For
convenience, the decision task also writes out ``label-to-taskid.json``
containing a mapping from label to taskId.  Each task in the graph is
represented as a JSON object.

Each task has the following properties:

``kind``
   The name of this task's kind

``task_id``
   The task's taskId (only for optimized task graphs)

``label``
   The task's label

``attributes``
   The task's attributes

``dependencies``
   The task's in-graph dependencies, represented as an object mapping
   dependency name to label (or to taskId for optimized task graphs)

``optimizations``
   The optimizations to be applied to this task

``task``
   The task's TaskCluster task definition.

The results from each command are in the same format, but with some differences
in the content:

* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
  That is, just collections of tasks without any dependencies indicated.

* The ``optimized`` subcommand returns tasks that have been assigned taskIds.
  The dependencies array, too, contains taskIds instead of labels, with
  dependencies on optimized tasks omitted.  However, the ``task.dependencies``
  array is populated with the full list of dependency taskIds.  All task
  references are resolved in the optimized graph.

The output of the ``mach taskgraph`` commands are suitable for processing with
the `jq <https://stedolan.github.io/jq/>`_ utility.  For example, to extract all
tasks' labels and their dependencies:

.. code-block:: shell

    jq 'to_entries | map({label: .value.label, dependencies: .value.dependencies})'

An alternate way of searching the output of ``mach taskgraph`` is
`gron <https://github.com/tomnomnom/gron>`_, which converts json into a format
that's easily searched with ``grep``

.. code-block:: shell

    gron taskgraph.json | grep -E 'test.*machine.platform = "linux64";'
    ./mach taskgraph --json | gron | grep ...