Bug 1436369 periodic file updates migration, stage 1 r=callek
authorSimon Fraser <sfraser@mozilla.com>
Tue, 06 Mar 2018 15:27:58 +0000
changeset 406980 6d0dcc642e1a110400c2ad05746c0b2a398a4af5
parent 406979 a984ece92c01c6bfe45c71a9ade07228559ec04a
child 406981 7afca6992b724c6bd29b6021cf7f54b56b36b028
push id33587
push usershindli@mozilla.com
push dateThu, 08 Mar 2018 00:20:14 +0000
treeherdermozilla-central@8a7d069e9b70 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerscallek
bugs1436369, 1436469
milestone60.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1436369 periodic file updates migration, stage 1 r=callek Summary: We'll be adding the new periodic file updates task to run in parallel. This patch moves the existing one to make it clear it's running on buildbot, so we don't get confused later on. Reviewers: Callek Reviewed By: Callek Bug #: 1436369 Differential Revision: https://phabricator.services.mozilla.com/D681 *** Bug 1436369 Add taskcluster periodic file updates r=callek Reviewers: Callek Reviewed By: Callek Differential Revision: https://phabricator.services.mozilla.com/D683 *** Bug 1436469 Fix task definition r=callek *** Bug 1436469 Move periodic-file updates docker image r=callek
taskcluster/ci/docker-image/kind.yml
taskcluster/ci/repo-update-bb/kind.yml
taskcluster/ci/repo-update/kind.yml
taskcluster/docker/periodic-updates/.eslintrc.js
taskcluster/docker/periodic-updates/Dockerfile
taskcluster/docker/periodic-updates/README.md
taskcluster/docker/periodic-updates/runme.sh
taskcluster/docker/periodic-updates/scripts/genHPKPStaticPins.js
taskcluster/docker/periodic-updates/scripts/getHSTSPreloadList.js
taskcluster/docker/periodic-updates/scripts/periodic_file_updates.sh
taskcluster/docker/periodic_updates/.eslintrc.js
taskcluster/docker/periodic_updates/Dockerfile
taskcluster/docker/periodic_updates/README.md
taskcluster/docker/periodic_updates/runme.sh
taskcluster/docker/periodic_updates/scripts/genHPKPStaticPins.js
taskcluster/docker/periodic_updates/scripts/getHSTSPreloadList.js
taskcluster/docker/periodic_updates/scripts/periodic_file_updates.sh
taskcluster/docs/kinds.rst
taskcluster/taskgraph/target_tasks.py
--- a/taskcluster/ci/docker-image/kind.yml
+++ b/taskcluster/ci/docker-image/kind.yml
@@ -91,8 +91,10 @@ jobs:
   funsize-balrog-submitter:
     symbol: I(fbs)
   beet-mover:
     symbol: I(bm)
   update-verify:
     symbol: I(uv)
   diffoscope:
     symbol: I(diff)
+  periodic-updates:
+    symbol: I(file)
copy from taskcluster/ci/repo-update/kind.yml
copy to taskcluster/ci/repo-update-bb/kind.yml
--- a/taskcluster/ci/repo-update/kind.yml
+++ b/taskcluster/ci/repo-update/kind.yml
@@ -1,21 +1,48 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 loader: taskgraph.loader.transform:loader
 
 transforms:
-   - taskgraph.transforms.job:transforms
    - taskgraph.transforms.task:transforms
 
 
 jobs:
    hsts-hpkp-blocklist:
       name: periodic_file_update
       description: HSTS, HPKP, and Blocklist update
-      worker-type: buildbot-bridge/buildbot-bridge
       run-on-projects: []  # Only run via cron
-      run:
-         using: buildbot
-         product: firefox
-         buildername: Linux x86-64 {branch} periodic file update
+      treeherder:
+         kind: build
+         platform: linux64/opt
+         symbol: pfu
+         tier: 1
+      worker-type: aws-provisioner-v1/gecko-{level}-b-linux
+      worker:
+         implementation: docker-worker
+         os: linux
+         docker-image: {in-tree: periodic-updates}
+         max-run-time: 3600  # Sometimes takes ~40 minutes
+         env:
+            DO_HSTS: "1"
+            DO_HPKP: "1"
+            DO_BLOCKLIST: "1"
+            USE_MOZILLA_CENTRAL: "1"
+            BRANCH: mozilla-central
+            PRODUCT: firefox
+            REVIEWERS: sfraser
+         command:
+            - /runme.sh
+         artifacts:
+            - name: 'public/build/nsSTSPreloadList.diff'
+              path: '/home/worker/artifacts/nsSTSPreloadList.diff'
+              type: file
+            - name: 'public/build/StaticHPKPins.h.diff'
+              path: '/home/worker/artifacts/StaticHPKPins.h.diff'
+              type: file
+            - name: 'public/build/blocklist.diff'
+              path: '/home/worker/artifacts/blocklist.diff'
+              type: file
+      scopes:
+         - secrets:get:project/releng/gecko/build/level-{level}/arc-phabricator-token
rename from taskcluster/docker/periodic_updates/.eslintrc.js
rename to taskcluster/docker/periodic-updates/.eslintrc.js
rename from taskcluster/docker/periodic_updates/Dockerfile
rename to taskcluster/docker/periodic-updates/Dockerfile
--- a/taskcluster/docker/periodic_updates/Dockerfile
+++ b/taskcluster/docker/periodic-updates/Dockerfile
@@ -7,20 +7,20 @@ ENV DEBIAN_FRONTEND noninteractive
 # Chain apt-get commands with apt-get clean in a single docker RUN
 # to make sure that files are removed within a single docker layer
 
 RUN apt-get update -q && \
     apt-get install -y --no-install-recommends \
     arcanist=0~git20170812-1 \
     bzip2=1.0.6-8.1 \
     ca-certificates=20170717 \
-    curl=7.58.0-2ubuntu1 \
+    curl=7.58.0-2ubuntu2 \
     jq=1.5+dfsg-2 \
     libdbus-glib-1-2=0.110-2 \
-    libgtk-3-0=3.22.28-1ubuntu1  \
+    libgtk-3-0=3.22.28-1ubuntu3 \
     libxml2-utils=2.9.4+dfsg1-6.1ubuntu1 \
     libxt6=1:1.1.5-1 \
     mercurial=4.3.1-2 \
     python3=3.6.4-1 \
     shellcheck=0.4.6-1 \
     unzip=6.0-21ubuntu1 \
     wget=1.19.4-1ubuntu2 \
     && apt-get clean
rename from taskcluster/docker/periodic_updates/README.md
rename to taskcluster/docker/periodic-updates/README.md
rename from taskcluster/docker/periodic_updates/runme.sh
rename to taskcluster/docker/periodic-updates/runme.sh
rename from taskcluster/docker/periodic_updates/scripts/genHPKPStaticPins.js
rename to taskcluster/docker/periodic-updates/scripts/genHPKPStaticPins.js
rename from taskcluster/docker/periodic_updates/scripts/getHSTSPreloadList.js
rename to taskcluster/docker/periodic-updates/scripts/getHSTSPreloadList.js
rename from taskcluster/docker/periodic_updates/scripts/periodic_file_updates.sh
rename to taskcluster/docker/periodic-updates/scripts/periodic_file_updates.sh
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -361,16 +361,22 @@ repackage-signing
 Repackage-signing take the repackaged installers (windows) and update packaging (with
 the signed internal bits) and signs them.
 
 repo-update
 -----------
 Repo-Update tasks are tasks that perform some action on the project repo itself,
 in order to update its state in some way.
 
+repo-update-bb
+--------------
+Repo-Update tasks are tasks that perform some action on the project repo itself,
+in order to update its state in some way. This kind is the older, buildbot version.
+It will be removed after the migration to taskcluster.
+
 partials
 --------
 Partials takes the complete.mar files produced in previous tasks and generates partial
 updates between previous nightly releases and the new one. Requires a release_history
 in the parameters. See ``mach release-history`` if doing this manually.
 
 partials-signing
 ----------------
--- a/taskcluster/taskgraph/target_tasks.py
+++ b/taskcluster/taskgraph/target_tasks.py
@@ -616,10 +616,10 @@ def target_tasks_searchfox(full_task_gra
 
 
 @_target_task('file_update')
 def target_tasks_file_update(full_task_graph, parameters, graph_config):
     """Select the set of tasks required to perform nightly in-tree file updates
     """
     def filter(task):
         # For now any task in the repo-update kind is ok
-        return task.kind in ['repo-update']
+        return task.kind in ['repo-update', 'repo-update-bb']
     return [l for l, t in full_task_graph.tasks.iteritems() if filter(t)]