Bug 1498640 - deploy latest image_builder image r=glandium
☠☠ backed out by a64e34971586 ☠ ☠
authorDustin J. Mitchell <dustin@mozilla.com>
Wed, 31 Oct 2018 23:02:42 +0000
changeset 443866 f33b9e6aca06b7883f048c6c068704680cbfbbb2
parent 443865 52dd8045db5a4888f4b0e8fef8a1ce1da0bdc0fc
child 443867 60082f8c23b4eb80e52bc0672da5fe6eece3f9fa
push id72123
push userdmitchell@mozilla.com
push dateThu, 01 Nov 2018 00:04:59 +0000
treeherderautoland@f33b9e6aca06 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1498640 - deploy latest image_builder image r=glandium This uses the latest image_builder image (on docker hub) to build even the image_builder image. The change to `docker.py` handles a new API response (`aux`) from the Docker daemon. It's unclear what this key means, but displaying it is simple. Differential Revision: https://phabricator.services.mozilla.com/D8441
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/image_builder/VERSION
@@ -0,0 +1,1 @@
--- a/taskcluster/docs/docker-images.rst
+++ b/taskcluster/docs/docker-images.rst
@@ -89,40 +89,39 @@ following namespaces.
 Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace
 is meant to view the latest built image.  This functions similarly to the 'latest' tag
 for docker images that are pushed to a registry.
 Docker Registry Images (prebuilt)
-***Warning: Use of prebuilt images should only be used for base images (those that other images
-will inherit from), or private images that must be stored in a private docker registry account.***
+***Warning: Registry images are only used for ``decision`` and
+``image_builder`` images.***
 These are images that are intended to be pushed to a docker registry and used
 by specifying the docker image name in task definitions.  They are generally
 referred to by a ``<repo>@<repodigest>`` string:
 .. code-block:: none
     image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854
-Each image has a repo digest and a version. The repo digest is stored in the
-``HASH`` file in the image directory and used to refer to the image as above.
-The version is in ``VERSION``.
+Such images must always be referred to with both a version and a repo digest.
+For the decision image, the repo digest is stored in the ``HASH`` file in the
+image directory and used to refer to the image as above.  The version for both
+images is in ``VERSION``.
-The version file only serves to provide convenient names, such that old
-versions are easy to discover in the registry (and ensuring old versions aren't
-deleted by garbage-collection).
+The version file serves to help users identify which image is being used, and makes old
+versions easy to discover in the registry.
-Each image directory also has a ``REGISTRY``, defaulting to the ``REGISTRY`` in
-the ``taskcluster/docker`` directory, and specifying the image registry to
-which the completed image should be uploaded.
+The file ``taskcluster/docker/REGISTRY`` specifies the image registry to which
+the completed image should be uploaded.
 Docker Hashes and Digests
 There are several hashes involved in this process:
  * Image Hash -- the long version of the image ID; can be seen with
    ``docker images --no-trunc`` or in the ``Id`` field in ``docker inspect``.
@@ -159,23 +158,29 @@ It's a good idea to bump the ``VERSION``
 For task images, test your image locally or push to try. This is all that is
 Docker Registry Images
 Landing docker registry images takes a little more care.
-Once a new version of the image has been built and tested locally, push it to
-the docker registry and make note of the resulting repo digest.  Put this value
-in the ``HASH`` file, and update any references to the image in the code or
-task definitions.
+Begin by bumping the ``VERSION``.  Once the new version of the image has been
+built and tested locally, push it to the docker registry and make note of the
+resulting repo digest.  Put this value in the ``HASH`` file for the
+``decision`` image and in ``taskcluster/taskgraph/transforms/docker_image.py``
+for the ``image_builder`` image.
 The change is now safe to use in Try pushes.
+Note that ``image_builder`` change can be tested directly in try pushes without
+using a registry, as the in-registry ``image_builder`` image is used to build a
+task image which is then used to build other images.  It is referenced by hash
+in ``taskcluster/taskgraph/transforms/docker_image.py``.
 Special Dockerfile Syntax
 Dockerfile syntax has been extended to allow *any* file from the
 source checkout to be added to the image build *context*. (Traditionally
 you can only ``ADD`` files from the same directory as the Dockerfile.)
 Simply add the following syntax as a comment in a Dockerfile::
--- a/taskcluster/taskgraph/transforms/docker_image.py
+++ b/taskcluster/taskgraph/transforms/docker_image.py
@@ -184,21 +184,22 @@ def fill_template(config, tasks):
         # Retry for 'funsize-update-generator' if exit status code is -1
         if image_name in ['funsize-update-generator']:
             taskdesc['worker']['retry-exit-status'] = [-1]
         worker = taskdesc['worker']
         # We use the in-tree image_builder image to build docker images, but
         # that can't be used to build the image_builder image itself,
-        # obviously. So we fall back to the last snapshot of the image that
-        # was uploaded to docker hub.
+        # obviously. So we fall back to an image on docker hub, identified
+        # by hash.  After the image-builder image is updated, it's best to push
+        # and update this hash as well, to keep image-builder builds up to date.
         if image_name == 'image_builder':
-            worker['docker-image'] = 'taskcluster/image_builder@sha256:' + \
-                '24ce54a1602453bc93515aecd9d4ad25a22115fbc4b209ddb5541377e9a37315'
+            hash = 'sha256:c6622fd3e5794842ad83d129850330b26e6ba671e39c58ee288a616a3a1c4c73'
+            worker['docker-image'] = 'taskcluster/image_builder@' + hash
             # Keep in sync with the Dockerfile used to generate the
             # docker image whose digest is referenced above.
             worker['volumes'] = [
             cache_name = 'imagebuilder-v1'
--- a/taskcluster/taskgraph/util/docker.py
+++ b/taskcluster/taskgraph/util/docker.py
@@ -98,16 +98,18 @@ def post_to_docker(tar, api_path, **kwar
                     if status != data['status']:
                         sys.stderr.write('{}: {}\n'.format(data['id'], data['status']))
                         status_line[data['id']] = data['status']
                 status_line = {}
         elif 'stream' in data:
+        elif 'aux' in data:
+            sys.stderr.write(repr(data['aux']))
         elif 'error' in data:
             raise Exception(data['error'])
             raise NotImplementedError(repr(data))
 def docker_image(name, by_tag=False):