Bug 1525094 - run linux builds in gcp at tier 3 r=wcosta
authorRob Thijssen <rthijssen@mozilla.com>
Tue, 16 Apr 2019 16:48:52 +0000
changeset 469809 679d694324d49b4947f03937f18159d6d9d3eedb
parent 469808 671d87853621c4bb7d03116f82bedb4e4ffb0011
child 469810 5e262e96eac73a03f7b7c6b7a0a42bf90b7670af
push id35883
push userbtara@mozilla.com
push dateWed, 17 Apr 2019 21:47:29 +0000
treeherdermozilla-central@02b89c29412b [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerswcosta
bugs1525094
milestone68.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1525094 - run linux builds in gcp at tier 3 r=wcosta this change adds support for parallel gcp builds for the following linux build configurations: - linux(32) - opt - debug - shippable - linux64 - opt - debug - shippable implementation notes: - this patch mostly mirrors the equivalent windows-on-gcp patch at: https://phabricator.services.mozilla.com/D24865 - gcp builds are triggered with a treeherder tier 3 flag so that they are only displayed in the treeherder ui when the user has a tier 3 flag set. - gcp builds use a th build symbol of "Bg" to make them easy to differentiate from ec2 builds in the treeherder ui. - gcp builds use a perfherder "gcp" flag to make them easier to differentiate from ec2 builds in the perfherder ui. - gcp builds on linux for all scm levels are built on the only available gcp linux worker type (at the time of this change): gce/gecko-1-b-linux-32 Differential Revision: https://phabricator.services.mozilla.com/D26490
taskcluster/ci/build/linux.yml
taskcluster/taskgraph/util/workertypes.py
--- a/taskcluster/ci/build/linux.yml
+++ b/taskcluster/ci/build/linux.yml
@@ -1126,8 +1126,241 @@ linux64-aarch64/opt:
     toolchains:
         - linux64-binutils
         - linux64-clang
         - linux64-rust
         - linux64-rust-size
         - linux64-cbindgen
         - linux64-sccache
         - linux64-node
+
+linux-gcp/debug:
+    description: "Linux32 Debug - built on GCP"
+    index:
+        product: firefox
+        job-name: linux-gcp-debug
+    attributes:
+        enable-full-crashsymbols: true
+    treeherder:
+        platform: linux32/debug
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        docker-image: {in-tree: debian7-i386-build}
+        max-run-time: 5400
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_32_builds.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        custom-build-variant-cfg: debug
+        mozconfig-variant: debug
+        tooltool-downloads: public
+        need-xvfb: true
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-cbindgen
+        - linux64-sccache
+        - linux64-nasm
+        - linux64-node
+
+linux-gcp/opt:
+    description: "Linux32 Opt - built on GCP"
+    index:
+        product: firefox
+        job-name: linux-gcp-opt
+    attributes:
+        enable-full-crashsymbols: true
+    treeherder:
+        platform: linux32/opt
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        docker-image: {in-tree: debian7-i386-build}
+        max-run-time: 7200
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_32_builds.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        tooltool-downloads: public
+        need-xvfb: true
+    run-on-projects: ['mozilla-central', 'try']
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-cbindgen
+        - linux64-sccache
+        - linux64-nasm
+        - linux64-node
+
+linux-gcp-shippable/opt:
+    description: "Linux32 Shippable - built on GCP"
+    attributes:
+        shippable: true
+        enable-full-crashsymbols: true
+    shipping-phase: build
+    shipping-product: firefox
+    index:
+        product: firefox
+        job-name: linux-gcp-opt
+        type: shippable
+    treeherder:
+        platform: linux32-shippable/opt
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        docker-image: {in-tree: debian7-i386-build}
+        max-run-time: 7200
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_32_builds.py
+            - taskcluster_nightly.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        tooltool-downloads: public
+        need-xvfb: true
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-cbindgen
+        - linux64-nasm
+        - linux64-node
+
+linux64-gcp/debug:
+    description: "Linux64 Debug - built on GCP"
+    index:
+        product: firefox
+        job-name: linux64-gcp-debug
+    attributes:
+        enable-full-crashsymbols: true
+    treeherder:
+        platform: linux64/debug
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        max-run-time: 5400
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_64_builds.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        custom-build-variant-cfg: debug
+        mozconfig-variant: debug
+        tooltool-downloads: public
+        need-xvfb: true
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-cbindgen
+        - linux64-sccache
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-nasm
+        - linux64-node
+
+linux64-gcp/opt:
+    description: "Linux64 Opt - built on GCP"
+    index:
+        product: firefox
+        job-name: linux64-gcp-opt
+    attributes:
+        enable-full-crashsymbols: true
+    treeherder:
+        platform: linux64/opt
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        max-run-time: 7200
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_64_builds.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        tooltool-downloads: public
+        need-xvfb: true
+    run-on-projects: ['mozilla-central', 'try']
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-cbindgen
+        - linux64-sccache
+        - linux64-nasm
+        - linux64-node
+
+linux64-gcp-shippable/opt:
+    description: "Linux64 Shippable - built on GCP"
+    attributes:
+        shippable: true
+        enable-full-crashsymbols: true
+    shipping-phase: build
+    shipping-product: firefox
+    index:
+        product: firefox
+        job-name: linux64-gcp-opt
+        type: shippable
+    treeherder:
+        platform: linux64-shippable/opt
+        symbol: Bg
+        tier: 3
+    worker-type: gce/gecko-{level}-b-linux
+    worker:
+        max-run-time: 7200
+        env:
+            PERFHERDER_EXTRA_OPTIONS: gcp
+    run:
+        using: mozharness
+        actions: [get-secrets, build, check-test]
+        config:
+            - builds/releng_base_firefox.py
+            - builds/releng_base_linux_64_builds.py
+            - taskcluster_nightly.py
+        script: "mozharness/scripts/fx_desktop_build.py"
+        secrets: true
+        tooltool-downloads: public
+        need-xvfb: true
+    toolchains:
+        - linux64-binutils
+        - linux64-clang
+        - linux64-rust
+        - linux64-rust-size
+        - linux64-cbindgen
+        - linux64-nasm
+        - linux64-node
--- a/taskcluster/taskgraph/util/workertypes.py
+++ b/taskcluster/taskgraph/util/workertypes.py
@@ -7,28 +7,31 @@ from __future__ import absolute_import, 
 from mozbuild.util import memoize
 
 from .keyed_by import evaluate_keyed_by
 
 WORKER_TYPES = {
     'aws-provisioner-v1/gecko-1-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-1-b-linux-large': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-1-b-linux-xlarge': ('docker-worker', 'linux'),
+    'gce/gecko-1-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-1-b-win2012': ('generic-worker', 'windows'),
     'releng-hardware/gecko-1-b-win2012-gamma': ('generic-worker', 'windows'),
     'aws-provisioner-v1/gecko-1-images': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-2-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-2-b-linux-large': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-2-b-linux-xlarge': ('docker-worker', 'linux'),
+    'gce/gecko-2-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-2-b-win2012': ('generic-worker', 'windows'),
     'releng-hardware/gecko-2-b-win2012-gamma': ('generic-worker', 'windows'),
     'aws-provisioner-v1/gecko-2-images': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-3-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-3-b-linux-large': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-3-b-linux-xlarge': ('docker-worker', 'linux'),
+    'gce/gecko-3-b-linux': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-3-b-win2012': ('generic-worker', 'windows'),
     'releng-hardware/gecko-3-b-win2012-gamma': ('generic-worker', 'windows'),
     'aws-provisioner-v1/gecko-3-images': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-symbol-upload': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-t-linux-large': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-t-linux-medium': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-t-linux-xlarge': ('docker-worker', 'linux'),
     'aws-provisioner-v1/gecko-t-win10-64': ('generic-worker', 'windows'),