author Gregory Szorc <gps@mozilla.com>
Fri, 29 Jul 2016 10:09:06 -0700
changeset 394620 f351e9f3e564f2f3f025664e1f701a151c61e1bb
parent 392622 cab3a2a82ac02633e14c31e4bc62d35f16ac15b6
child 394548 6690559ecbcfbbb70e34b4b38cbf4c711c0edfa6
child 394583 25020657e91a320b83b9ccf9ed726334081e2d6c
child 395238 3d6656c3a28d4c30a348ce389d3ade800b42a451
child 395830 aeddf0b867c0c31d071158875d1b2f5a8398ca0b
permissions -rw-r--r--
Bug 1290516 - Remove {{project}} from cache path for decision and action tasks; r=dustin Currently, the gecko-decision and action tasks maintain a separate cache for each "project" (Firefox repo) source checkout. In my opinion, this is not necessary. Separate caches/source checkouts mean a lower chance for a cache hit: a lower chance that any given worker will have a source checkout readily available for the "project" being built. They also mean workers store more data and files in caches. By using a shared cache for source checkouts, workers will ideally only have a single source checkout cache and will be pretty much guaranteed to have a checkout readily available (the main exception being the first task on a freshly-provisioned worker). A downside of a shared cache is that workers may spend more time doing `hg update` operations. e.g. if the last task was from mozilla-release and the current task is from mozilla-central, thousands of files will be updated. However, I believe we will still be better off because separate caches *increase* the probability for a full checkout, which involves writing ~150,000 files! So I believe this change will lower the overall amount of VCS work done by workers while possibly sacrificing the lowest possible execution time (which is achieved by minimizing the `hg update` work). It's worth noting that `hg robustcheckout` (which is used by these tasks) does the equivalent of an `hg purge --all`. This ensures that all untracked and ignored files from previous tasks are deleted. So there should be no contamination from previous tasks. MozReview-Commit-ID: I5Ng3xFEfou

version: 0
  name: 'Taskcluster tasks for Gecko'
  description: "The taskcluster task graph for Gecko trees"
  owner: mozilla-taskcluster-maintenance@mozilla.com
  source: {{{source}}}

  # Note the below scopes are insecure however these get overriden on the server
  # side to whatever scopes are set by mozilla-taskcluster.
  - queue:*
  - docker-worker:*
  - scheduler:*

# Available mustache parameters (see the mozilla-taskcluster source):
# - owner:          push user (email address)
# - source:         URL of this YAML file
# - url:            repository URL
# - project:        alias for the destination repository (basename of
#                   the repo url)
# - level:          SCM level of the destination repository
#                   (1 = try, 3 = core)
# - revision:       (short) hg revision of the head of the push
# - revision_hash:  (long) hg revision of the head of the push
# - comment:        comment of the push
# - pushlog_id:     id in the pushlog table of the repository
# and functions:
# - as_slugid:      convert a label into a slugId
# - from_now:       generate a timestamp at a fixed offset from now

  - taskId: '{{#as_slugid}}decision task{{/as_slugid}}'
      created: '{{now}}'
      deadline: '{{#from_now}}1 day{{/from_now}}'
      expires: '{{#from_now}}14 day{{/from_now}}'
        owner: mozilla-taskcluster-maintenance@mozilla.com
        source: {{{source}}}
        name: "Gecko Decision Task"
        description: |
            The task that creates all of the other tasks in the task graph

      workerType: "gecko-decision"
      provisionerId: "aws-provisioner-v1"

        createdForUser: {{owner}}

        # Bug 1269443: cache scopes, etc. must be listed explicitly
        - "docker-worker:cache:level-{{level}}-*"
        - "docker-worker:cache:tooltool-cache"
        # mozilla-taskcluster will append the appropriate assume:repo:<repo>
        # scope here.

        - "index.gecko.v2.{{project}}.latest.firefox.decision"
        - "tc-treeherder.v2.{{project}}.{{revision}}.{{pushlog_id}}"
        - "tc-treeherder-stage.v2.{{project}}.{{revision}}.{{pushlog_id}}"

          # checkout-gecko uses these to check out the source; the inputs
          # to `mach taskgraph decision` are all on the command line.
          GECKO_BASE_REPOSITORY: 'https://hg.mozilla.org/mozilla-unified'
          GECKO_HEAD_REPOSITORY: '{{{url}}}'
          GECKO_HEAD_REF: '{{revision}}'
          GECKO_HEAD_REV: '{{revision}}'
          # Arguments passed into `mach taskgraph decision`
          # TODO use mozilla-unified for the base repository once the tc-vcs
          # tar.gz archives are created or tc-vcs isn't being used.
          DECISION_ARGS: >

          level-{{level}}-hg-shared: /home/worker/hg-shared
          level-{{level}}-gecko-decision: /home/worker/workspace

          taskclusterProxy: true

        # Note: This task is built server side without the context or tooling that
        # exist in tree so we must hard code the version
        image: 'taskcluster/decision:0.1.2'

        maxRunTime: 1800

          - /home/worker/bin/run-decision

            type: 'directory'
            path: '/home/worker/artifacts'
            expires: '{{#from_now}}7 days{{/from_now}}'

          symbol: D