Bug 1430037 - Use sparse checkouts for docker image builds. r=dustin
authorMike Hommey <mh+mozilla@glandium.org>
Thu, 11 Jan 2018 17:03:09 +0900
changeset 399762 63351f0612b47df5f244fd99cdb7a79a109c8c3a
parent 399761 1227e2800bd87612b60a69a2cf5c2971432ceaee
child 399763 f628439e6f52a377ea79f82eb8863c5a5d9d5b4f
push id33274
push userdluca@mozilla.com
push dateThu, 18 Jan 2018 09:41:57 +0000
treeherdermozilla-central@a9f3d83fec60 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersdustin
bugs1430037
milestone59.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1430037 - Use sparse checkouts for docker image builds. r=dustin In many cases, building docker images starts on machines that don't have a cached checkout, and it often takes forever to get a full clone. It used to be worsened when 3 jobs could run at the same time because the worker would start up clean, and 3 jobs would be doing a mercurial clone at the same time, thrashing I/O, but that part is fortunately fixed. It is still, however, appreciable not to waste time in the mercurial clone part of image creation.
build/sparse-profiles/docker-image
taskcluster/docker/image_builder/build-image.sh
taskcluster/taskgraph/transforms/docker_image.py
new file mode 100644
--- /dev/null
+++ b/build/sparse-profiles/docker-image
@@ -0,0 +1,16 @@
+%include build/sparse-profiles/mach
+
+[include]
+path:taskcluster/
+
+# Result from `grep -hr %include taskcluster/docker | grep -v " taskcluster/" | sort -u`
+path:build/unix/build-gtk3/build-gtk3.sh
+path:build/valgrind/valgrind-epochs.patch
+path:python/mozbuild/mozbuild/action/tooltool.py
+path:testing/config/tooltool-manifests/linux64/releng.manifest
+path:testing/mozharness/external_tools/robustcheckout.py
+path:tools/lint/eslint/eslint-plugin-mozilla/manifest.tt
+path:tools/lint/eslint/manifest.tt
+path:tools/lint/python/flake8_requirements.txt
+path:tools/lint/tox/tox_requirements.txt
+
--- a/taskcluster/docker/image_builder/build-image.sh
+++ b/taskcluster/docker/image_builder/build-image.sh
@@ -19,16 +19,17 @@ test -n "$HASH"       || raise_error "Co
 test -n "$IMAGE_NAME" || raise_error "IMAGE_NAME must be provided."
 
 # Construct a CONTEXT_FILE
 CONTEXT_FILE=/builds/worker/workspace/context.tar
 
 # Run ./mach taskcluster-build-image with --context-only to build context
 run-task \
   --vcs-checkout "/builds/worker/checkouts/gecko" \
+  --sparse-profile build/sparse-profiles/docker-image \
   -- \
   /builds/worker/checkouts/gecko/mach taskcluster-build-image \
   --context-only "$CONTEXT_FILE" \
   "$IMAGE_NAME"
 test -f "$CONTEXT_FILE" || raise_error "Context file wasn't created"
 
 # Create artifact folder (note that this must occur after run-task)
 mkdir -p /builds/worker/workspace/artifacts
--- a/taskcluster/taskgraph/transforms/docker_image.py
+++ b/taskcluster/taskgraph/transforms/docker_image.py
@@ -163,17 +163,17 @@ def fill_template(config, tasks):
             # docker image whose digest is referenced above.
             worker['volumes'] = [
                 '/builds/worker/checkouts',
                 '/builds/worker/workspace',
             ]
             cache_name = 'imagebuilder-v1'
         else:
             worker['docker-image'] = {'in-tree': 'image_builder'}
-            cache_name = 'imagebuilder-{}'.format(_run_task_suffix())
+            cache_name = 'imagebuilder-sparse-{}'.format(_run_task_suffix())
             # Force images built against the in-tree image builder to
             # have a different digest by adding a fixed string to the
             # hashed data.
             digest_data.append('image_builder')
 
         worker['caches'] = [{
             'type': 'persistent',
             'name': 'level-{}-{}'.format(config.params['level'], cache_name),