taskcluster/docs/caches.rst
author Manuel Rego Casasnovas <rego@igalia.com>
Mon, 19 Nov 2018 18:46:13 +0000
changeset 506778 359ee839f1a358175a0d4d537476d32a7f4a39a4
parent 481923 e3d9b7024efa256b0835b0acdd807d75121bdbdf
permissions -rw-r--r--
Bug 1507805 [wpt PR 14092] - [css-grid] Some fixes on grid-container-scrollbar-* tests, a=testonly Automatic update from web-platform-tests[css-grid] Some fixes on grid-container-scrollbar-* tests * Use Ahem font to avoid weird small pixel differences on iOS. * Fix some typos reported by @fred-wang at https://bugs.webkit.org/show_bug.cgi?id=191656#c14 -- Merge pull request #14092 from mrego/grid-container-scrollbars [css-grid] Some fixes on grid-container-scrollbar-* tests -- wpt-commits: ef7824a8ff88c95d36b31b377fe252f7c56d2da0, 1b543a1083f48c13b25f23a5f23d14f6ade958d4 wpt-pr: 14092

.. taskcluster_caches:

Caches
======

There are various caches used by the in-tree tasks. This page attempts to
document them and their appropriate use.

Caches are essentially isolated filesystems that are persisted between
tasks. For example, if 2 tasks run on a worker - one after the other -
and both tasks request the same cache, the subsequent task will be
able to see files in the cache that were created by the first task.
It's also worth noting that TaskCluster workers ensure a cache can only
be used by 1 task at a time. If a worker is simultaneously running
multiple tasks requesting the same named cache, the worker will
have multiple caches of the same name on the worker.

Caches and ``run-task``
-----------------------

``run-task`` is our generic task wrapper script. It does common activities
like ensure a version control checkout is present.

One of the roles of ``run-task`` is to verify and sanitize caches.
It does this by storing state in a cache on its first use. If the recorded
*capabilities* of an existing cache don't match expectations for the
current task, ``run-task`` bails. This ensures that caches are only
reused by tasks with similar execution *profiles*. This prevents
accidental cache use across incompatible tasks. It also allows run-task
to make assumptions about the state of caches, which can help avoid
costly operations.

In addition, the hash of ``run-task`` is used to derive the cache name.
So any time ``run-task`` changes, a new set of caches are used. This
ensures that any backwards incompatible changes or bug fixes to
``run-task`` result in fresh caches.

Some caches are reserved for use with run-task. That property will be denoted
below.

Common Caches
-------------

Version Control Caches
::::::::::::::::::::::

``level-{{level}}-checkouts-{{hash}}``
   This cache holds version control checkouts, each in a subdirectory named
   after the repo (e.g., ``gecko``).

   Checkouts should be read-only. If a task needs to create new files from
   content of a checkout, this content should be written in a separate
   directory/cache (like a workspace).

   This cache name pattern is managed by ``run-task`` and must only be
   used with ``run-task``.

``level-{{level}}-checkouts-sparse-{{hash}}``
   This is like the above but is used when the checkout is sparse (contains
   a subset of files).

``level-{{level}}-checkouts-{{version}}`` (deprecated)
   This cache holds version control checkouts, each in a subdirectory named
   after the repo (e.g., ``gecko``).

   Checkouts should be read-only. If a task needs to create new files from
   content of a checkout, this content should be written in a separate
   directory/cache (like a workspace).

   A ``version`` parameter appears in the cache name to allow
   backwards-incompatible changes to the cache's behavior.

   The ``hg-store`` contains a `shared store <https://www.mercurial-scm.org/wiki/ShareExtension>`
   that is is used by ``hg robustcheckout``. If you are using ``run-task`` you
   should set the ``HG_STORE_PATH`` environment variable to point to this
   directory. If you are using ``hg robustcheckout``, pass this directory to the
   ``--sharebase`` option.

Workspace Caches
::::::::::::::::

``level-{{level}}-*-workspace``
   These caches (of various names typically ending with ``workspace``)
   contain state to be shared between task invocations. Use cases are
   dependent on the task.

Other
:::::

``level-{{level}}-tooltool-cache-{{hash}}``
   Tooltool invocations should use this cache. Tooltool will store files here
   indexed by their hash.

   This cache name pattern is reserved for use with ``run-task`` and must only
   be used by ``run-task``

``tooltool-cache`` (deprecated)
   Legacy location for tooltool files. Use the per-level one instead.