Bug 1498640 - deploy latest image_builder image r=glandium
☠☠ backed out by 09be3daa0787 ☠ ☠
authorDustin J. Mitchell <dustin@mozilla.com>
Wed, 31 Oct 2018 23:02:42 +0000
changeset 446472 aa32ba9e6df3e20aab896a63538c8e8a432b4988
parent 446471 f975bf0a920911016f7af1460be4678d58ca4c0b
child 446473 565215cf2e5e42034860257f4ea4b5d0b0adf75d
push id73019
push usergszorc@mozilla.com
push dateThu, 15 Nov 2018 00:38:48 +0000
treeherderautoland@aa32ba9e6df3 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersglandium
bugs1498640
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1498640 - deploy latest image_builder image r=glandium This uses the latest image_builder image (on docker hub) to build even the image_builder image. The change to `docker.py` handles a new API response (`aux`) from the Docker daemon. It's unclear what this key means, but displaying it is simple. Differential Revision: https://phabricator.services.mozilla.com/D8441
taskcluster/docker/image_builder/VERSION
taskcluster/docs/docker-images.rst
taskcluster/taskgraph/transforms/docker_image.py
taskcluster/taskgraph/util/docker.py
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/image_builder/VERSION
@@ -0,0 +1,1 @@
+3.0.0
--- a/taskcluster/docs/docker-images.rst
+++ b/taskcluster/docs/docker-images.rst
@@ -89,40 +89,39 @@ following namespaces.
 
 Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace
 is meant to view the latest built image.  This functions similarly to the 'latest' tag
 for docker images that are pushed to a registry.
 
 Docker Registry Images (prebuilt)
 :::::::::::::::::::::::::::::::::
 
-***Warning: Use of prebuilt images should only be used for base images (those that other images
-will inherit from), or private images that must be stored in a private docker registry account.***
+***Warning: Registry images are only used for ``decision`` and
+``image_builder`` images.***
 
 These are images that are intended to be pushed to a docker registry and used
 by specifying the docker image name in task definitions.  They are generally
 referred to by a ``<repo>@<repodigest>`` string:
 
 Example:
 
 .. code-block:: none
 
     image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854
 
-Each image has a repo digest and a version. The repo digest is stored in the
-``HASH`` file in the image directory and used to refer to the image as above.
-The version is in ``VERSION``.
+Such images must always be referred to with both a version and a repo digest.
+For the decision image, the repo digest is stored in the ``HASH`` file in the
+image directory and used to refer to the image as above.  The version for both
+images is in ``VERSION``.
 
-The version file only serves to provide convenient names, such that old
-versions are easy to discover in the registry (and ensuring old versions aren't
-deleted by garbage-collection).
+The version file serves to help users identify which image is being used, and makes old
+versions easy to discover in the registry.
 
-Each image directory also has a ``REGISTRY``, defaulting to the ``REGISTRY`` in
-the ``taskcluster/docker`` directory, and specifying the image registry to
-which the completed image should be uploaded.
+The file ``taskcluster/docker/REGISTRY`` specifies the image registry to which
+the completed image should be uploaded.
 
 Docker Hashes and Digests
 .........................
 
 There are several hashes involved in this process:
 
  * Image Hash -- the long version of the image ID; can be seen with
    ``docker images --no-trunc`` or in the ``Id`` field in ``docker inspect``.
@@ -159,23 +158,29 @@ It's a good idea to bump the ``VERSION``
 For task images, test your image locally or push to try. This is all that is
 required.
 
 Docker Registry Images
 ::::::::::::::::::::::
 
 Landing docker registry images takes a little more care.
 
-Once a new version of the image has been built and tested locally, push it to
-the docker registry and make note of the resulting repo digest.  Put this value
-in the ``HASH`` file, and update any references to the image in the code or
-task definitions.
+Begin by bumping the ``VERSION``.  Once the new version of the image has been
+built and tested locally, push it to the docker registry and make note of the
+resulting repo digest.  Put this value in the ``HASH`` file for the
+``decision`` image and in ``taskcluster/taskgraph/transforms/docker_image.py``
+for the ``image_builder`` image.
 
 The change is now safe to use in Try pushes.
 
+Note that ``image_builder`` change can be tested directly in try pushes without
+using a registry, as the in-registry ``image_builder`` image is used to build a
+task image which is then used to build other images.  It is referenced by hash
+in ``taskcluster/taskgraph/transforms/docker_image.py``.
+
 Special Dockerfile Syntax
 -------------------------
 
 Dockerfile syntax has been extended to allow *any* file from the
 source checkout to be added to the image build *context*. (Traditionally
 you can only ``ADD`` files from the same directory as the Dockerfile.)
 
 Simply add the following syntax as a comment in a Dockerfile::
--- a/taskcluster/taskgraph/transforms/docker_image.py
+++ b/taskcluster/taskgraph/transforms/docker_image.py
@@ -184,21 +184,22 @@ def fill_template(config, tasks):
         # Retry for 'funsize-update-generator' if exit status code is -1
         if image_name in ['funsize-update-generator']:
             taskdesc['worker']['retry-exit-status'] = [-1]
 
         worker = taskdesc['worker']
 
         # We use the in-tree image_builder image to build docker images, but
         # that can't be used to build the image_builder image itself,
-        # obviously. So we fall back to the last snapshot of the image that
-        # was uploaded to docker hub.
+        # obviously. So we fall back to an image on docker hub, identified
+        # by hash.  After the image-builder image is updated, it's best to push
+        # and update this hash as well, to keep image-builder builds up to date.
         if image_name == 'image_builder':
-            worker['docker-image'] = 'taskcluster/image_builder@sha256:' + \
-                '24ce54a1602453bc93515aecd9d4ad25a22115fbc4b209ddb5541377e9a37315'
+            hash = 'sha256:c6622fd3e5794842ad83d129850330b26e6ba671e39c58ee288a616a3a1c4c73'
+            worker['docker-image'] = 'taskcluster/image_builder@' + hash
             # Keep in sync with the Dockerfile used to generate the
             # docker image whose digest is referenced above.
             worker['volumes'] = [
                 '/builds/worker/checkouts',
                 '/builds/worker/workspace',
             ]
             cache_name = 'imagebuilder-v1'
         else:
--- a/taskcluster/taskgraph/util/docker.py
+++ b/taskcluster/taskgraph/util/docker.py
@@ -98,16 +98,18 @@ def post_to_docker(tar, api_path, **kwar
                     if status != data['status']:
                         sys.stderr.write('{}: {}\n'.format(data['id'], data['status']))
                         status_line[data['id']] = data['status']
             else:
                 status_line = {}
                 sys.stderr.write('{}\n'.format(data['status']))
         elif 'stream' in data:
             sys.stderr.write(data['stream'])
+        elif 'aux' in data:
+            sys.stderr.write(repr(data['aux']))
         elif 'error' in data:
             sys.stderr.write('{}\n'.format(data['error']))
             # Sadly, docker doesn't give more than a plain string for errors,
             # so the best we can do to propagate the error code from the command
             # that failed is to parse the error message...
             errcode = 1
             m = re.search(r'returned a non-zero code: (\d+)', data['error'])
             if m: