Bug 1488148 - Rework CI images, r=jcj
authorMartin Thomson <martin.thomson@gmail.com>
Mon, 03 Sep 2018 16:31:31 +1000
changeset 14493 e56241dc5b61300dc426884b7c3cf50f28e9586b
parent 14492 87b0372b6a2da431630336702ae8c2d0d0fdb04e
child 14494 3d41453587e4a03dad482cea95222a4c88a5b7ad
push id3193
push usermartin.thomson@gmail.com
push dateSun, 23 Sep 2018 17:42:25 +0000
reviewersjcj
bugs1488148, 1488325
Bug 1488148 - Rework CI images, r=jcj This does some fairly major restructuring of the docker images we use for CI. The genesis of the change is that we were pulling a version of clang that didn't work for fuzzing tests. It turns out that is a use case that is not well-supported anyway, and we have open bugs on it, but this installs workarounds for all the problems I found. Firstly, our images were bloated. This slims down most of the images. The biggest gains are in the clang-format image (down to around a fifth of its previous size). The main linux image we use for building and running tests is also less than half its original size. To achieve that, I had to make two new images. One for all the esoteric builds we run (we compile with multiple gcc and clang versions, and I've added some more to that list). That's a fairly sizeable image. The other is for the interop and bogo suites, where we rely on having Rust and go available. go adds a tidy 250Mb to an image, and Rust adds 750Mb. Using an image with both of those for regular builds can't be good for performance. I didn't expect to see real performance gains here, but the Linux build (32-bit, default config) went from 4:18 down to 2:42 (roughly). The bulk of that time is accounted for by downloading the docker image, so it's clear that an optimization worth spending the time on. Secondly, we had a lot of custom configuration stuff in the builds. This removes most of that in favour of using stock Ubuntu packages from 18.04. The one exception here is - I hope - temporary. As noted in the bug comments, the current release of LLVM 6 has a bug where you can't run address sanitizer on a 32-bit machine if it has glibc 2.27 (which Ubuntu 18.04 does). That's fairly crippling because we need a newer version of LLVM than runs by default on Ubuntu 16.04, so we're stuck with installing a non-stock version for 32-bit runs. I've opted to (temporarily) run 16.04 with an LLVM from the LLVM project. The final change, which is minor, but a little odd and worth noting: the images now rely on "localhost.localdomain" being aliased to the local machine. This is something :wcosta has done for us (thanks!). Thus, we no longer have to run as root so that we can tweak /etc/hosts before we run. There is a little cleanup related to this, but nothing significant. (The scripts still include the `su worker` tweak for aarch64, but I've added a guard and we can remove that with bug 1488325.) There is still more work to be done for the HACL* and SAW builds, which use some very strange configurations. Also, all of the aarch64 images aren't built automatically, so we use images from Franziskus' dockerhub account. This is not good. After digging around a little, there's probably something to be done with QEMU, but I decided that was a project for another time.
automation/clang-format/Dockerfile
automation/clang-format/setup.sh
automation/taskcluster/docker-aarch64/Dockerfile
automation/taskcluster/docker-arm/Dockerfile
automation/taskcluster/docker-builds/Dockerfile
automation/taskcluster/docker-builds/bin/checkout.sh
automation/taskcluster/docker-clang-3.9/Dockerfile
automation/taskcluster/docker-clang-3.9/bin/checkout.sh
automation/taskcluster/docker-clang-3.9/setup.sh
automation/taskcluster/docker-clang-format/Dockerfile
automation/taskcluster/docker-clang-format/bin/checkout.sh
automation/taskcluster/docker-decision/Dockerfile
automation/taskcluster/docker-decision/setup.sh
automation/taskcluster/docker-fuzz/Dockerfile
automation/taskcluster/docker-fuzz/setup.sh
automation/taskcluster/docker-fuzz32/Dockerfile
automation/taskcluster/docker-fuzz32/bin/checkout.sh
automation/taskcluster/docker-gcc-4.4/Dockerfile
automation/taskcluster/docker-gcc-4.4/setup.sh
automation/taskcluster/docker-interop/Dockerfile
automation/taskcluster/docker-interop/bin/checkout.sh
automation/taskcluster/docker/Dockerfile
automation/taskcluster/docker/setup.sh
automation/taskcluster/graph/src/extend.js
automation/taskcluster/scripts/build_image.sh
automation/taskcluster/scripts/tools.sh
tests/chains/chains.sh
tests/common/init.sh
--- a/automation/clang-format/Dockerfile
+++ b/automation/clang-format/Dockerfile
@@ -1,26 +1,35 @@
-FROM ubuntu:16.04
-MAINTAINER Franziskus Kiefer <franziskuskiefer@gmail.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
+# Minimal image with clang-format 3.9.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    clang-format-3.9 \
+    locales \
+    mercurial \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Change user.
-USER worker
+RUN update-alternatives --install /usr/bin/clang-format \
+    clang-format $(which clang-format-3.9) 10
 
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV HOSTNAME taskcluster-worker
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
-# Entrypoint.
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+USER $USER
+
+# Entrypoint - which only works if /home/worker/nss is mounted.
 ENTRYPOINT ["/home/worker/nss/automation/clang-format/run_clang_format.sh"]
deleted file mode 100644
--- a/automation/clang-format/setup.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Install packages.
-apt_packages=()
-apt_packages+=('ca-certificates')
-apt_packages+=('curl')
-apt_packages+=('xz-utils')
-apt_packages+=('mercurial')
-apt_packages+=('git')
-apt_packages+=('locales')
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-# Download clang.
-curl -L https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz -o clang.tar.xz
-curl -L https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz.sig -o clang.tar.xz.sig
-# Verify the signature.
-gpg --keyserver pool.sks-keyservers.net --recv-keys B6C8F98282B944E3B0D5C2530FC3042E345AD05D
-gpg --verify clang.tar.xz.sig
-# Install into /usr/local/.
-tar xJvf *.tar.xz -C /usr/local --strip-components=1
-
-# Cleanup.
-function cleanup() {
-  rm -f clang.tar.xz clang.tar.xz.sig
-}
-trap cleanup ERR EXIT
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-
-# We're done. Remove this script.
-rm $0
--- a/automation/taskcluster/docker-aarch64/Dockerfile
+++ b/automation/taskcluster/docker-aarch64/Dockerfile
@@ -15,16 +15,15 @@ RUN bash /tmp/setup.sh
 # Change user.
 # USER worker # See bug 1347473.
 
 # Env variables.
 ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
 ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
 ENV LANG en_US.UTF-8
 ENV LC_ALL en_US.UTF-8
 ENV HOST localhost
 ENV DOMSUF localdomain
 
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
--- a/automation/taskcluster/docker-arm/Dockerfile
+++ b/automation/taskcluster/docker-arm/Dockerfile
@@ -12,16 +12,15 @@ RUN chmod +x /home/worker/bin/*
 ADD setup.sh /tmp/setup.sh
 RUN bash /tmp/setup.sh
 
 # Env variables.
 ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
 ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
 ENV LANG en_US.UTF-8
 ENV LC_ALL en_US.UTF-8
 ENV HOST localhost
 ENV DOMSUF localdomain
 
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-builds/Dockerfile
@@ -0,0 +1,75 @@
+# Dockerfile for building extra builds.  This includes more tools than the
+# default image, so it's a fair bit bigger.  Only use this for builds where
+# the smaller docker image is missing something.  These builds will run on
+# the leaner configuration.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
+
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    clang-4.0 \
+    clang \
+    cmake \
+    curl \
+    g++-4.8-multilib \
+    g++-5-multilib \
+    g++-6-multilib \
+    g++-multilib \
+    git \
+    gyp \
+    libelf-dev \
+    libdw-dev \
+    libssl-dev \
+    libssl-dev:i386 \
+    libxml2-utils \
+    lib32z1-dev \
+    linux-libc-dev:i386 \
+    llvm-dev \
+    locales \
+    mercurial \
+    ninja-build \
+    pkg-config \
+    valgrind \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
+
+# Latest version of abigail-tools
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends automake libtool libxml2-dev \
+ && git clone git://sourceware.org/git/libabigail.git /tmp/libabigail \
+ && cd /tmp/libabigail \
+ && autoreconf -fi \
+ && ./configure --prefix=/usr --disable-static --disable-apidoc --disable-manual \
+ && make && make install \
+ && rm -rf /tmp/libabigail \
+ && apt-get remove -y automake libtool libxml2-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
+
+ENV SHELL /bin/bash
+ENV USER worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
+ENV LANG en_US.UTF-8
+ENV LC_ALL $LANG
+ENV HOST localhost
+ENV DOMSUF localdomain
+
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
+
+# Set a default command for debugging.
+CMD ["/bin/bash", "--login"]
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-builds/bin/checkout.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -v -e -x
+
+if [ $(id -u) = 0 ]; then
+    # Drop privileges by re-running this script.
+    exec su worker $0
+fi
+
+# Default values for testing.
+REVISION=${NSS_HEAD_REVISION:-default}
+REPOSITORY=${NSS_HEAD_REPOSITORY:-https://hg.mozilla.org/projects/nss}
+
+# Clone NSS.
+for i in 0 2 5; do
+    sleep $i
+    hg clone -r $REVISION $REPOSITORY nss && exit 0
+    rm -rf nss
+done
+exit 1
deleted file mode 100644
--- a/automation/taskcluster/docker-clang-3.9/setup.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Need this to add keys for PPAs below.
-apt-get install -y --no-install-recommends apt-utils
-
-apt_packages=()
-apt_packages+=('ca-certificates')
-apt_packages+=('curl')
-apt_packages+=('locales')
-apt_packages+=('xz-utils')
-
-# Latest Mercurial.
-apt_packages+=('mercurial')
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 41BD8711B1F0EC2B0D85B91CF59CE3A8323293EE
-echo "deb http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu xenial main" > /etc/apt/sources.list.d/mercurial.list
-
-# Install packages.
-apt-get -y update
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-# Download clang.
-curl -LO https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz
-curl -LO https://releases.llvm.org/3.9.1/clang+llvm-3.9.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz.sig
-# Verify the signature.
-gpg --keyserver pool.sks-keyservers.net --recv-keys B6C8F98282B944E3B0D5C2530FC3042E345AD05D
-gpg --verify *.tar.xz.sig
-# Install into /usr/local/.
-tar xJvf *.tar.xz -C /usr/local --strip-components=1
-# Cleanup.
-rm *.tar.xz*
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-rm $0
rename from automation/taskcluster/docker-clang-3.9/Dockerfile
rename to automation/taskcluster/docker-clang-format/Dockerfile
--- a/automation/taskcluster/docker-clang-3.9/Dockerfile
+++ b/automation/taskcluster/docker-clang-format/Dockerfile
@@ -1,30 +1,38 @@
-FROM ubuntu:16.04
-MAINTAINER Tim Taubert <ttaubert@mozilla.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
-
-# Add build and test scripts.
-ADD bin /home/worker/bin
-RUN chmod +x /home/worker/bin/*
+# Minimal image with clang-format 3.9.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    clang-format-3.9 \
+    locales \
+    mercurial \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Change user.
-USER worker
+RUN update-alternatives --install /usr/bin/clang-format \
+    clang-format $(which clang-format-3.9) 10
 
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
+
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
rename from automation/taskcluster/docker-clang-3.9/bin/checkout.sh
rename to automation/taskcluster/docker-clang-format/bin/checkout.sh
--- a/automation/taskcluster/docker-decision/Dockerfile
+++ b/automation/taskcluster/docker-decision/Dockerfile
@@ -1,30 +1,37 @@
-FROM ubuntu:16.04
-MAINTAINER Tim Taubert <ttaubert@mozilla.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
-
-# Add build and test scripts.
-ADD bin /home/worker/bin
-RUN chmod +x /home/worker/bin/*
+# Minimal image for running the decision task.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    locales \
+    mercurial \
+    nodejs \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Change user.
-USER worker
-
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
+
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
deleted file mode 100644
--- a/automation/taskcluster/docker-decision/setup.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Need those to install newer packages below.
-apt-get install -y --no-install-recommends apt-utils curl ca-certificates locales
-
-# Latest Mercurial.
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 41BD8711B1F0EC2B0D85B91CF59CE3A8323293EE
-echo "deb http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu xenial main" > /etc/apt/sources.list.d/mercurial.list
-
-# Install packages.
-apt-get -y update && apt-get install -y --no-install-recommends mercurial
-
-# Latest Node.JS.
-curl -sL https://deb.nodesource.com/setup_6.x | bash -
-apt-get install -y --no-install-recommends nodejs
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-rm $0
--- a/automation/taskcluster/docker-fuzz/Dockerfile
+++ b/automation/taskcluster/docker-fuzz/Dockerfile
@@ -1,33 +1,59 @@
-FROM ubuntu:16.04
-MAINTAINER Tim Taubert <ttaubert@mozilla.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
-
-# Add build and test scripts.
-ADD bin /home/worker/bin
-RUN chmod +x /home/worker/bin/*
+# Dockerfile for running fuzzing tests.
+#
+# Note that when running this, you need to add `--cap-add SYS_PTRACE` to the
+# docker invocation or ASAN won't work.
+# On taskcluster use `features: ["allowPtrace"]`.
+# See https://github.com/google/sanitizers/issues/764#issuecomment-276700920
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    clang \
+    clang-tools \
+    curl \
+    g++-multilib \
+    git \
+    gyp \
+    libssl-dev \
+    libssl-dev:i386 \
+    libxml2-utils \
+    lib32z1-dev \
+    linux-libc-dev:i386 \
+    llvm-dev \
+    locales \
+    mercurial \
+    ninja-build \
+    pkg-config \
+    valgrind \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Change user.
-USER worker
-
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
-# LLVM 4.0
-ENV PATH "${PATH}:/home/worker/third_party/llvm-build/Release+Asserts/bin/"
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+# Change user.
+USER $USER
 
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
deleted file mode 100644
--- a/automation/taskcluster/docker-fuzz/setup.sh
+++ /dev/null
@@ -1,58 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Need this to add keys for PPAs below.
-apt-get install -y --no-install-recommends apt-utils
-
-apt_packages=()
-apt_packages+=('build-essential')
-apt_packages+=('ca-certificates')
-apt_packages+=('curl')
-apt_packages+=('git')
-apt_packages+=('gyp')
-apt_packages+=('libssl-dev')
-apt_packages+=('libxml2-utils')
-apt_packages+=('locales')
-apt_packages+=('ninja-build')
-apt_packages+=('pkg-config')
-apt_packages+=('zlib1g-dev')
-
-# 32-bit builds
-apt_packages+=('gcc-multilib')
-apt_packages+=('g++-multilib')
-
-# Latest Mercurial.
-apt_packages+=('mercurial')
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 41BD8711B1F0EC2B0D85B91CF59CE3A8323293EE
-echo "deb http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu xenial main" > /etc/apt/sources.list.d/mercurial.list
-
-# Install packages.
-apt-get -y update
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-# 32-bit builds
-dpkg --add-architecture i386
-apt-get -y update
-apt-get install -y --no-install-recommends libssl-dev:i386
-
-# Install LLVM/clang-4.0.
-mkdir clang-tmp
-git clone -n --depth 1 https://chromium.googlesource.com/chromium/src/tools/clang clang-tmp/clang
-git -C clang-tmp/clang checkout HEAD scripts/update.py
-clang-tmp/clang/scripts/update.py
-rm -fr clang-tmp
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-rm $0
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-fuzz32/Dockerfile
@@ -0,0 +1,73 @@
+# Dockerfile for running fuzzing tests on linux32.
+#
+# This is a temporary workaround for bugs in clang that make it incompatible
+# with Ubuntu 18.04 (see bug 1488148). This image can be removed once a new
+# release of LLVM includes the necessary fixes.
+
+FROM ubuntu:16.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
+
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    g++-multilib \
+    git \
+    gyp \
+    libssl-dev \
+    libssl-dev:i386 \
+    libxml2-utils \
+    lib32z1-dev \
+    linux-libc-dev:i386 \
+    locales \
+    mercurial \
+    ninja-build \
+    pkg-config \
+    software-properties-common \
+    valgrind \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
+
+# Install clang and tools from the LLVM PPA.
+RUN curl -sf https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
+ && apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends \
+    clang-6.0 \
+    clang-tools-6.0 \
+    llvm-6.0-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
+
+# Alias all the clang commands.
+RUN for i in $(dpkg -L clang-6.0 clang-tools-6.0 | grep '^/usr/bin/' | xargs -i basename {} -6.0); do \
+      update-alternatives --install "/usr/bin/$i" "$i" "/usr/bin/${i}-6.0" 10; \
+    done
+
+ENV SHELL /bin/bash
+ENV USER worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
+ENV LANG en_US.UTF-8
+ENV LC_ALL $LANG
+ENV HOST localhost
+ENV DOMSUF localdomain
+
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+# Change user.
+USER $USER
+
+# Set a default command for debugging.
+CMD ["/bin/bash", "--login"]
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-fuzz32/bin/checkout.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -v -e -x
+
+if [ $(id -u) = 0 ]; then
+    # Drop privileges by re-running this script.
+    exec su worker $0
+fi
+
+# Default values for testing.
+REVISION=${NSS_HEAD_REVISION:-default}
+REPOSITORY=${NSS_HEAD_REPOSITORY:-https://hg.mozilla.org/projects/nss}
+
+# Clone NSS.
+for i in 0 2 5; do
+    sleep $i
+    hg clone -r $REVISION $REPOSITORY nss && exit 0
+    rm -rf nss
+done
+exit 1
--- a/automation/taskcluster/docker-gcc-4.4/Dockerfile
+++ b/automation/taskcluster/docker-gcc-4.4/Dockerfile
@@ -1,30 +1,39 @@
 FROM ubuntu:14.04
-MAINTAINER Tim Taubert <ttaubert@mozilla.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
-
-# Add build and test scripts.
-ADD bin /home/worker/bin
-RUN chmod +x /home/worker/bin/*
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    g++-4.4 \
+    gcc-4.4 \
+    locales \
+    make \
+    mercurial \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Change user.
-USER worker
-
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
+
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
deleted file mode 100644
--- a/automation/taskcluster/docker-gcc-4.4/setup.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-apt_packages=()
-apt_packages+=('ca-certificates')
-apt_packages+=('g++-4.4')
-apt_packages+=('gcc-4.4')
-apt_packages+=('locales')
-apt_packages+=('make')
-apt_packages+=('mercurial')
-apt_packages+=('zlib1g-dev')
-
-# Install packages.
-apt-get -y update
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-rm $0
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-interop/Dockerfile
@@ -0,0 +1,56 @@
+# Dockerfile for running interop tests.
+# This includes Rust, golang, and nodejs.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
+
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    clang \
+    cmake \
+    curl \
+    g++-multilib \
+    git \
+    golang \
+    gyp \
+    libxml2-utils \
+    lib32z1-dev \
+    linux-libc-dev:i386 \
+    llvm-dev \
+    locales \
+    mercurial \
+    ninja-build \
+    npm \
+    pkg-config \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
+
+ENV SHELL /bin/bash
+ENV USER worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
+ENV LANG en_US.UTF-8
+ENV LC_ALL $LANG
+ENV HOST localhost
+ENV DOMSUF localdomain
+
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
+
+# Install Rust stable as $USER.
+RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
+
+# Set a default command for debugging.
+CMD ["/bin/bash", "--login"]
new file mode 100644
--- /dev/null
+++ b/automation/taskcluster/docker-interop/bin/checkout.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -v -e -x
+
+if [ $(id -u) = 0 ]; then
+    # Drop privileges by re-running this script.
+    exec su worker $0
+fi
+
+# Default values for testing.
+REVISION=${NSS_HEAD_REVISION:-default}
+REPOSITORY=${NSS_HEAD_REPOSITORY:-https://hg.mozilla.org/projects/nss}
+
+# Clone NSS.
+for i in 0 2 5; do
+    sleep $i
+    hg clone -r $REVISION $REPOSITORY nss && exit 0
+    rm -rf nss
+done
+exit 1
--- a/automation/taskcluster/docker/Dockerfile
+++ b/automation/taskcluster/docker/Dockerfile
@@ -1,30 +1,49 @@
-FROM ubuntu:16.04
-MAINTAINER Tim Taubert <ttaubert@mozilla.com>
-
-RUN useradd -d /home/worker -s /bin/bash -m worker
-WORKDIR /home/worker
+# Lean image for running the bulk of the NSS CI tests on taskcluster.
+FROM ubuntu:18.04
+LABEL maintainer="Martin Thomson <martin.thomson@gmail.com>"
 
-# Add build and test scripts.
-ADD bin /home/worker/bin
-RUN chmod +x /home/worker/bin/*
+RUN dpkg --add-architecture i386
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    clang \
+    curl \
+    g++-multilib \
+    git \
+    gyp \
+    libxml2-utils \
+    lib32z1-dev \
+    linux-libc-dev:i386 \
+    llvm-dev \
+    locales \
+    mercurial \
+    ninja-build \
+    pkg-config \
+    zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/* \
+ && apt-get autoremove -y && apt-get clean -y
 
-# Install dependencies.
-ADD setup.sh /tmp/setup.sh
-RUN bash /tmp/setup.sh
-
-# Env variables.
-ENV HOME /home/worker
 ENV SHELL /bin/bash
 ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
+ENV LOGNAME $USER
+ENV HOME /home/$USER
 ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV LC_ALL $LANG
 ENV HOST localhost
 ENV DOMSUF localdomain
 
-# Rust + Go
-ENV PATH "${PATH}:/home/worker/.cargo/bin/:/usr/lib/go-1.6/bin"
+RUN locale-gen $LANG \
+ && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
+
+RUN useradd -d $HOME -s $SHELL -m $USER
+WORKDIR $HOME
+
+# Add build and test scripts.
+ADD bin $HOME/bin
+RUN chmod +x $HOME/bin/*
+
+USER $USER
 
 # Set a default command for debugging.
 CMD ["/bin/bash", "--login"]
deleted file mode 100644
--- a/automation/taskcluster/docker/setup.sh
+++ /dev/null
@@ -1,75 +0,0 @@
-#!/usr/bin/env bash
-
-set -v -e -x
-
-# Update packages.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -y update && apt-get -y upgrade
-
-# Need this to add keys for PPAs below.
-apt-get install -y --no-install-recommends apt-utils
-
-apt_packages=()
-apt_packages+=('build-essential')
-apt_packages+=('ca-certificates')
-apt_packages+=('clang-5.0')
-apt_packages+=('curl')
-apt_packages+=('npm')
-apt_packages+=('git')
-apt_packages+=('golang-1.6')
-apt_packages+=('libxml2-utils')
-apt_packages+=('locales')
-apt_packages+=('ninja-build')
-apt_packages+=('pkg-config')
-apt_packages+=('zlib1g-dev')
-apt_packages+=('cmake')
-
-# 32-bit builds
-apt_packages+=('lib32z1-dev')
-apt_packages+=('gcc-multilib')
-apt_packages+=('g++-multilib')
-
-# ct-verif and sanitizers
-apt_packages+=('valgrind')
-
-# Latest Mercurial.
-apt_packages+=('mercurial')
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 41BD8711B1F0EC2B0D85B91CF59CE3A8323293EE
-echo "deb http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu xenial main" > /etc/apt/sources.list.d/mercurial.list
-
-# gcc 4.8 and 6
-apt_packages+=('g++-6')
-apt_packages+=('g++-4.8')
-apt_packages+=('g++-6-multilib')
-apt_packages+=('g++-4.8-multilib')
-apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 60C317803A41BA51845E371A1E9377A2BA9EF27F
-echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main" > /etc/apt/sources.list.d/toolchain.list
-
-# Install packages.
-apt-get -y update
-apt-get install -y --no-install-recommends ${apt_packages[@]}
-
-# Latest version of abigail-tools
-apt-get install -y libxml2-dev autoconf libelf-dev libdw-dev libtool
-git clone git://sourceware.org/git/libabigail.git
-cd ./libabigail
-autoreconf -fi
-./configure --prefix=/usr --disable-static --disable-apidoc --disable-manual
-make
-make install
-cd ..
-apt-get remove -y libxml2-dev autoconf libtool
-rm -rf libabigail
-
-# Install latest Rust (stable).
-su worker -c "curl https://sh.rustup.rs -sSf | sh -s -- -y"
-
-locale-gen en_US.UTF-8
-dpkg-reconfigure locales
-
-# Cleanup.
-rm -rf ~/.ccache ~/.cache
-apt-get autoremove -y
-apt-get clean
-apt-get autoclean
-rm $0
--- a/automation/taskcluster/graph/src/extend.js
+++ b/automation/taskcluster/graph/src/extend.js
@@ -5,31 +5,47 @@
 import merge from "./merge";
 import * as queue from "./queue";
 
 const LINUX_IMAGE = {
   name: "linux",
   path: "automation/taskcluster/docker"
 };
 
-const LINUX_CLANG39_IMAGE = {
-  name: "linux-clang-3.9",
-  path: "automation/taskcluster/docker-clang-3.9"
+const LINUX_BUILDS_IMAGE = {
+  name: "linux-builds",
+  path: "automation/taskcluster/docker-builds"
+};
+
+const LINUX_INTEROP_IMAGE = {
+  name: "linux-interop",
+  path: "automation/taskcluster/docker-interop"
+};
+
+const CLANG_FORMAT_IMAGE = {
+  name: "clang-format",
+  path: "automation/taskcluster/docker-clang-format"
 };
 
 const LINUX_GCC44_IMAGE = {
   name: "linux-gcc-4.4",
   path: "automation/taskcluster/docker-gcc-4.4"
 };
 
 const FUZZ_IMAGE = {
   name: "fuzz",
   path: "automation/taskcluster/docker-fuzz"
 };
 
+// Bug 1488148 - temporary image for fuzzing 32-bit builds.
+const FUZZ_IMAGE_32 = {
+  name: "fuzz32",
+  path: "automation/taskcluster/docker-fuzz32"
+};
+
 const HACL_GEN_IMAGE = {
   name: "hacl",
   path: "automation/taskcluster/docker-hacl"
 };
 
 const SAW_IMAGE = {
   name: "saw",
   path: "automation/taskcluster/docker-saw"
@@ -84,17 +100,19 @@ queue.filter(task => {
 
   // Only old make builds have -Ddisable_libpkix=0 and can run chain tests.
   if (task.tests == "chains" && task.collection != "make") {
     return false;
   }
 
   if (task.group == "Test") {
     // Don't run test builds on old make platforms, and not for fips gyp.
-    if (task.collection == "make" || task.collection == "fips") {
+    // Disable on aarch64, see bug 1488331.
+    if (task.collection == "make" || task.collection == "fips"
+        || task.platform == "aarch64") {
       return false;
     }
   }
 
   // Don't run all additional hardware tests on ARM.
   if (task.group == "Cipher" && task.platform == "aarch64" && task.env &&
       (task.env.NSS_DISABLE_PCLMUL == "1" || task.env.NSS_DISABLE_HW_AES == "1"
        || task.env.NSS_DISABLE_AVX == "1")) {
@@ -188,18 +206,18 @@ export default async function main() {
     ],
   });
 
   await scheduleLinux("Linux 64 (ASan, debug)", {
     env: {
       UBSAN_OPTIONS: "print_stacktrace=1",
       NSS_DISABLE_ARENA_FREE_LIST: "1",
       NSS_DISABLE_UNLOAD: "1",
-      CC: "clang-5.0",
-      CCC: "clang++-5.0",
+      CC: "clang",
+      CCC: "clang++",
     },
     platform: "linux64",
     collection: "asan",
     image: LINUX_IMAGE,
     features: ["allowPtrace"],
   }, "--ubsan --asan");
 
   await scheduleLinux("Linux 64 (FIPS opt)", {
@@ -246,47 +264,47 @@ export default async function main() {
     image: "franziskus/nss-aarch64-ci",
     provisioner: "localprovisioner",
     workerType: "nss-aarch64",
     platform: "aarch64",
     maxRunTime: 7200
   };
 
   await scheduleLinux("Linux AArch64 (debug)",
-    merge({
+    merge(aarch64_base, {
       command: [
         "/bin/bash",
         "-c",
         "bin/checkout.sh && nss/automation/taskcluster/scripts/build_gyp.sh"
       ],
       collection: "debug",
-    }, aarch64_base)
+    })
   );
 
   await scheduleLinux("Linux AArch64 (opt)",
-    merge({
+    merge(aarch64_base, {
       command: [
         "/bin/bash",
         "-c",
         "bin/checkout.sh && nss/automation/taskcluster/scripts/build_gyp.sh --opt"
       ],
       collection: "opt",
-    }, aarch64_base)
+    })
   );
 
   await scheduleLinux("Linux AArch64 (debug, make)",
-    merge({
+    merge(aarch64_base, {
       env: {USE_64: "1"},
       command: [
          "/bin/bash",
          "-c",
          "bin/checkout.sh && nss/automation/taskcluster/scripts/build.sh"
       ],
       collection: "make",
-    }, aarch64_base)
+    })
   );
 
   await scheduleMac("Mac (opt)", {collection: "opt"}, "--opt");
   await scheduleMac("Mac (debug)", {collection: "debug"});
 }
 
 
 async function scheduleMac(name, base, args = "") {
@@ -298,34 +316,34 @@ async function scheduleMac(name, base, a
       HOST: "localhost",
     },
     provisioner: "localprovisioner",
     workerType: "nss-macos-10-12",
     platform: "mac"
   });
 
   // Build base definition.
-  let build_base = merge({
+  let build_base = merge(mac_base, {
     command: [
       MAC_CHECKOUT_CMD,
       ["bash", "-c",
        "nss/automation/taskcluster/scripts/build_gyp.sh", args]
     ],
     provisioner: "localprovisioner",
     workerType: "nss-macos-10-12",
     platform: "mac",
     maxRunTime: 7200,
     artifacts: [{
       expires: 24 * 7,
       type: "directory",
       path: "public"
     }],
     kind: "build",
     symbol: "B"
-  }, mac_base);
+  });
 
   // The task that builds NSPR+NSS.
   let task_build = queue.scheduleTask(merge(build_base, {name}));
 
   // The task that generates certificates.
   let task_cert = queue.scheduleTask(merge(build_base, {
     name: "Certificates",
     command: [
@@ -346,34 +364,38 @@ async function scheduleMac(name, base, a
     ]
   }));
 
   return queue.submit();
 }
 
 /*****************************************************************************/
 
-async function scheduleLinux(name, base, args = "") {
-  // Build base definition.
-  let build_base = merge({
+async function scheduleLinux(name, overrides, args = "") {
+  // Construct a base definition.  This takes |overrides| second because
+  // callers expect to be able to overwrite the |command| key.
+  let base = merge({
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/build_gyp.sh " + args
     ],
+  }, overrides);
+  // The base for building.
+  let build_base = merge(base, {
     artifacts: {
       public: {
         expires: 24 * 7,
         type: "directory",
         path: "/home/worker/artifacts"
       }
     },
     kind: "build",
-    symbol: "B"
-  }, base);
+    symbol: "B",
+  });
 
   // The task that builds NSPR+NSS.
   let task_build = queue.scheduleTask(merge(build_base, {name}));
 
   // Make builds run FIPS tests, which need an extra FIPS build.
   if (base.collection == "make") {
     let extra_build = queue.scheduleTask(merge(build_base, {
       env: { NSS_FORCE_FIPS: "1" },
@@ -429,24 +451,27 @@ async function scheduleLinux(name, base,
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/run_tests.sh"
     ]
   }));
 
   // Extra builds.
-  let extra_base = merge({group: "Builds"}, build_base);
+  let extra_base = merge(build_base, {
+    group: "Builds",
+    image: LINUX_BUILDS_IMAGE,
+  });
   queue.scheduleTask(merge(extra_base, {
-    name: `${name} w/ clang-5.0`,
+    name: `${name} w/ clang-4`,
     env: {
-      CC: "clang-5.0",
-      CCC: "clang++-5.0",
+      CC: "clang-4",
+      CCC: "clang++-4",
     },
-    symbol: "clang-5.0"
+    symbol: "clang-4"
   }));
 
   queue.scheduleTask(merge(extra_base, {
     name: `${name} w/ gcc-4.4`,
     image: LINUX_GCC44_IMAGE,
     env: {
       USE_64: "1",
       CC: "gcc-4.4",
@@ -469,36 +494,46 @@ async function scheduleLinux(name, base,
     env: {
       CC: "gcc-4.8",
       CCC: "g++-4.8"
     },
     symbol: "gcc-4.8"
   }));
 
   queue.scheduleTask(merge(extra_base, {
-    name: `${name} w/ gcc-6.1`,
+    name: `${name} w/ gcc-5`,
+    env: {
+      CC: "gcc-5",
+      CCC: "g++-5"
+    },
+    symbol: "gcc-5"
+  }));
+
+  queue.scheduleTask(merge(extra_base, {
+    name: `${name} w/ gcc-6`,
     env: {
       CC: "gcc-6",
       CCC: "g++-6"
     },
-    symbol: "gcc-6.1"
+    symbol: "gcc-6"
   }));
 
   queue.scheduleTask(merge(extra_base, {
     name: `${name} w/ modular builds`,
+    image: LINUX_IMAGE,
     env: {NSS_BUILD_MODULAR: "1"},
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/build.sh",
     ],
     symbol: "modular"
   }));
 
-  await scheduleTestBuilds(merge(base, {group: "Test"}), args);
+  await scheduleTestBuilds(name + " Test", merge(base, {group: "Test"}), args);
 
   return queue.submit();
 }
 
 /*****************************************************************************/
 
 function scheduleFuzzingRun(base, name, target, max_len, symbol = null, corpus = null) {
   const MAX_FUZZ_TIME = 300;
@@ -529,33 +564,33 @@ async function scheduleFuzzing() {
     },
     features: ["allowPtrace"],
     platform: "linux64",
     collection: "fuzz",
     image: FUZZ_IMAGE
   };
 
   // Build base definition.
-  let build_base = merge({
+  let build_base = merge(base, {
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && " +
       "nss/automation/taskcluster/scripts/build_gyp.sh -g -v --fuzz"
     ],
     artifacts: {
       public: {
         expires: 24 * 7,
         type: "directory",
         path: "/home/worker/artifacts"
       }
     },
     kind: "build",
     symbol: "B"
-  }, base);
+  });
 
   // The task that builds NSPR+NSS.
   let task_build = queue.scheduleTask(merge(build_base, {
     name: "Linux x64 (debug, fuzz)"
   }));
 
   // The task that builds NSPR+NSS (TLS fuzzing mode).
   let task_build_tls = queue.scheduleTask(merge(build_base, {
@@ -630,37 +665,37 @@ async function scheduleFuzzing32() {
       NSS_DISABLE_ARENA_FREE_LIST: "1",
       NSS_DISABLE_UNLOAD: "1",
       CC: "clang",
       CCC: "clang++"
     },
     features: ["allowPtrace"],
     platform: "linux32",
     collection: "fuzz",
-    image: FUZZ_IMAGE
+    image: FUZZ_IMAGE_32
   };
 
   // Build base definition.
-  let build_base = merge({
+  let build_base = merge(base, {
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && " +
       "nss/automation/taskcluster/scripts/build_gyp.sh -g -v --fuzz -m32"
     ],
     artifacts: {
       public: {
         expires: 24 * 7,
         type: "directory",
         path: "/home/worker/artifacts"
       }
     },
     kind: "build",
     symbol: "B"
-  }, base);
+  });
 
   // The task that builds NSPR+NSS.
   let task_build = queue.scheduleTask(merge(build_base, {
     name: "Linux 32 (debug, fuzz)"
   }));
 
   // The task that builds NSPR+NSS (TLS fuzzing mode).
   let task_build_tls = queue.scheduleTask(merge(build_base, {
@@ -723,44 +758,51 @@ async function scheduleFuzzing32() {
   scheduleFuzzingRun(tls_fm_base, "DTLS Client", "dtls-client", 20000, "dtls-client");
   scheduleFuzzingRun(tls_fm_base, "DTLS Server", "dtls-server", 20000, "dtls-server");
 
   return queue.submit();
 }
 
 /*****************************************************************************/
 
-async function scheduleTestBuilds(base, args = "") {
+async function scheduleTestBuilds(name, base, args = "") {
   // Build base definition.
-  let build = merge({
+  let build = merge(base, {
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && " +
       "nss/automation/taskcluster/scripts/build_gyp.sh -g -v --test --ct-verif " + args
     ],
     artifacts: {
       public: {
         expires: 24 * 7,
         type: "directory",
         path: "/home/worker/artifacts"
       }
     },
     kind: "build",
     symbol: "B",
-    name: "Linux 64 (debug, test)"
-  }, base);
+    name: `${name} build`,
+  });
+
+  // On linux we have a specialized build image for building.
+  if (build.platform === "linux32" || build.platform === "linux64") {
+    build = merge(build, {
+      image: LINUX_BUILDS_IMAGE,
+    });
+  }
 
   // The task that builds NSPR+NSS.
   let task_build = queue.scheduleTask(build);
 
   // Schedule tests.
   queue.scheduleTask(merge(base, {
     parent: task_build,
-    name: "mpi",
+    name: `${name} mpi tests`,
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/run_tests.sh"
     ],
     tests: "mpi",
     cycle: "standard",
     symbol: "mpi",
@@ -768,17 +810,17 @@ async function scheduleTestBuilds(base, 
   }));
   queue.scheduleTask(merge(base, {
     parent: task_build,
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/run_tests.sh"
     ],
-    name: "Gtests",
+    name: `${name} gtests`,
     symbol: "Gtest",
     tests: "gtests",
     cycle: "standard",
     kind: "test"
   }));
 
   return queue.submit();
 }
@@ -876,28 +918,36 @@ async function scheduleWindows(name, bas
   }));
 
   return queue.submit();
 }
 
 /*****************************************************************************/
 
 function scheduleTests(task_build, task_cert, test_base) {
-  test_base = merge({kind: "test"}, test_base);
+  test_base = merge(test_base, {kind: "test"});
 
   // Schedule tests that do NOT need certificates.
   let no_cert_base = merge(test_base, {parent: task_build});
   queue.scheduleTask(merge(no_cert_base, {
     name: "Gtests", symbol: "Gtest", tests: "ssl_gtests gtests", cycle: "standard"
   }));
   queue.scheduleTask(merge(no_cert_base, {
-    name: "Bogo tests", symbol: "Bogo", tests: "bogo", cycle: "standard"
+    name: "Bogo tests",
+    symbol: "Bogo",
+    tests: "bogo",
+    cycle: "standard",
+    image: LINUX_INTEROP_IMAGE,
   }));
   queue.scheduleTask(merge(no_cert_base, {
-    name: "Interop tests", symbol: "Interop", tests: "interop", cycle: "standard"
+    name: "Interop tests",
+    symbol: "Interop",
+    tests: "interop",
+    cycle: "standard",
+    image: LINUX_INTEROP_IMAGE,
   }));
   queue.scheduleTask(merge(no_cert_base, {
     name: "Chains tests", symbol: "Chains", tests: "chains"
   }));
   queue.scheduleTask(merge(no_cert_base, {
     name: "Cipher tests", symbol: "Default", tests: "cipher", group: "Cipher"
   }));
   queue.scheduleTask(merge(no_cert_base, {
@@ -969,32 +1019,32 @@ function scheduleTests(task_build, task_
 /*****************************************************************************/
 
 async function scheduleTools() {
   let base = {
     platform: "nss-tools",
     kind: "test"
   };
 
-  //ABI check task
+  // ABI check task
   queue.scheduleTask(merge(base, {
     symbol: "abi",
     name: "abi",
-    image: LINUX_IMAGE,
+    image: LINUX_BUILDS_IMAGE,
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/taskcluster/scripts/check_abi.sh"
     ],
   }));
 
   queue.scheduleTask(merge(base, {
-    symbol: "clang-format-3.9",
-    name: "clang-format-3.9",
-    image: LINUX_CLANG39_IMAGE,
+    symbol: "clang-format",
+    name: "clang-format",
+    image: CLANG_FORMAT_IMAGE,
     command: [
       "/bin/bash",
       "-c",
       "bin/checkout.sh && nss/automation/clang-format/run_clang_format.sh"
     ]
   }));
 
   queue.scheduleTask(merge(base, {
--- a/automation/taskcluster/scripts/build_image.sh
+++ b/automation/taskcluster/scripts/build_image.sh
@@ -8,17 +8,17 @@ raise_error() {
    echo "[taskcluster-image-build:error] $1"
    exit 1
 }
 
 # Ensure that the PROJECT is specified so the image can be indexed
 test -n "$PROJECT" || raise_error "Project must be provided."
 test -n "$HASH" || raise_error "Context Hash must be provided."
 
-CONTEXT_PATH=/home/worker/nss/$CONTEXT_PATH
+CONTEXT_PATH="/home/worker/nss/$CONTEXT_PATH"
 
-test -d $CONTEXT_PATH || raise_error "Context Path $CONTEXT_PATH does not exist."
+test -d "$CONTEXT_PATH" || raise_error "Context Path $CONTEXT_PATH does not exist."
 test -f "$CONTEXT_PATH/Dockerfile" || raise_error "Dockerfile must be present in $CONTEXT_PATH."
 
-docker build -t $PROJECT:$HASH $CONTEXT_PATH
+docker build -t "$PROJECT:$HASH" "$CONTEXT_PATH"
 
 mkdir /artifacts
-docker save $PROJECT:$HASH > /artifacts/image.tar
+docker save "$PROJECT:$HASH" > /artifacts/image.tar
--- a/automation/taskcluster/scripts/tools.sh
+++ b/automation/taskcluster/scripts/tools.sh
@@ -1,18 +1,17 @@
 #!/usr/bin/env bash
 
 set -v -e -x
 
+# Assert that we're not running as root.
 if [[ $(id -u) -eq 0 ]]; then
-    # Stupid Docker. It works without sometimes... But not always.
-    echo "127.0.0.1 localhost.localdomain" >> /etc/hosts
-
-    # Drop privileges by re-running this script.
-    # Note: this mangles arguments, better to avoid running scripts as root.
+    # This exec is still needed until aarch64 images are updated (Bug 1488325).
+    # Remove when images are updated.  Until then, assert that things are good.
+    [[ $(uname -m) == aarch64 ]]
     exec su worker -c "$0 $*"
 fi
 
 export PATH="${PATH}:/home/worker/.cargo/bin/:/usr/lib/go-1.6/bin"
 
 # Usage: hg_clone repo dir [revision=@]
 hg_clone() {
     repo=$1
--- a/tests/chains/chains.sh
+++ b/tests/chains/chains.sh
@@ -46,23 +46,23 @@ is_httpserv_alive()
 }
 
 ########################### wait_for_httpserv ##########################
 # local shell function to wait until httpserver is running and initialized
 ########################################################################
 wait_for_httpserv()
 {
   echo "trying to connect to httpserv at `date`"
-  echo "tstclnt -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v"
-  ${BINDIR}/tstclnt -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v
+  echo "tstclnt -4 -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v"
+  ${BINDIR}/tstclnt -4 -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v
   if [ $? -ne 0 ]; then
       sleep 5
       echo "retrying to connect to httpserv at `date`"
-      echo "tstclnt -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v"
-      ${BINDIR}/tstclnt -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v
+      echo "tstclnt -4 -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v"
+      ${BINDIR}/tstclnt -4 -p ${NSS_AIA_PORT} -h ${HOSTADDR} -q -v
       if [ $? -ne 0 ]; then
           html_failed "Waiting for Server"
       fi
   fi
   is_httpserv_alive
 }
 
 ########################### kill_httpserv ##############################
@@ -969,18 +969,18 @@ check_ocsp()
         CERT_FILE=${CERT}
     fi
 
     # sample line:
     #   URI: "http://ocsp.server:2601"
     OCSP_HOST=$(${BINDIR}/pp -w -t certificate -i ${CERT_FILE} | grep URI | sed "s/.*:\/\///" | sed "s/:.*//")
     OCSP_PORT=$(${BINDIR}/pp -w -t certificate -i ${CERT_FILE} | grep URI | sed "s/^.*:.*:\/\/.*:\([0-9]*\).*$/\1/")
 
-    echo "tstclnt -h ${OCSP_HOST} -p ${OCSP_PORT} -q -t 20"
-    tstclnt -h ${OCSP_HOST} -p ${OCSP_PORT} -q -t 20
+    echo "tstclnt -4 -h ${OCSP_HOST} -p ${OCSP_PORT} -q -t 20"
+    tstclnt -4 -h ${OCSP_HOST} -p ${OCSP_PORT} -q -t 20
     return $?
 }
 
 ############################ parse_result ##############################
 # local shell function to process expected result value
 # this function was created for case that expected result depends on
 # some conditions - in our case type of cert DB
 #
--- a/tests/common/init.sh
+++ b/tests/common/init.sh
@@ -351,59 +351,53 @@ if [ -z "${INIT_SOURCED}" -o "${INIT_SOU
 
     if [ ! -d "${TESTDIR}" ]; then
         echo "$SCRIPTNAME init: Creating ${TESTDIR}"
         mkdir -p ${TESTDIR}
     fi
 
 #HOST and DOMSUF are needed for the server cert
 
-    DOMAINNAME=`which domainname`
-    if [ -z "${DOMSUF}" -a $? -eq 0 -a -n "${DOMAINNAME}" ]; then
+    if [ -z "$DOMSUF" ] && hash domainname 2>/dev/null; then
         DOMSUF=`domainname`
     fi
+    # hostname -d and domainname both return (none) if hostname doesn't
+    # include a dot.  Pretend we didn't get an answer.
+    if [ "$DOMSUF" = "(none)" ]; then
+        DOMSUF=
+    fi
 
-    case $HOST in
+    if [ -z "$HOST" ]; then
+        HOST=`uname -n`
+    fi
+    case "$HOST" in
         *\.*)
-            if [ -z "${DOMSUF}" ]; then
-                DOMSUF=`echo $HOST | sed -e "s/^[^.]*\.//"`
+            if [ -z "$DOMSUF" ]; then
+                DOMSUF="${HOST#*.}"
             fi
-            HOST=`echo $HOST | sed -e "s/\..*//"`
+            HOST="${HOST%%.*}"
             ;;
         ?*)
             ;;
         *)
-            HOST=`uname -n`
-            case $HOST in
-                *\.*)
-                    if [ -z "${DOMSUF}" ]; then
-                        DOMSUF=`echo $HOST | sed -e "s/^[^.]*\.//"`
-                    fi
-                    HOST=`echo $HOST | sed -e "s/\..*//"`
-                    ;;
-                ?*)
-                    ;;
-                *)
-                    echo "$SCRIPTNAME: Fatal HOST environment variable is not defined."
-                    exit 1 #does not need to be Exit, very early in script
-                    ;;
-            esac
+            echo "$SCRIPTNAME: Fatal HOST environment variable is not defined."
+            exit 1 #does not need to be Exit, very early in script
             ;;
     esac
 
-    if [ -z "${DOMSUF}" -a "${OS_ARCH}" != "Android" ]; then
+    if [ -z "$DOMSUF" -a "$OS_ARCH" != "Android" ]; then
         echo "$SCRIPTNAME: Fatal DOMSUF env. variable is not defined."
         exit 1 #does not need to be Exit, very early in script
     fi
 
 #HOSTADDR was a workaround for the dist. stress test, and is probably
 #not needed anymore (purpose: be able to use IP address for the server
 #cert instead of PC name which was not in the DNS because of dyn IP address
-    if [ -z "$USE_IP" -o "$USE_IP" != "TRUE" ] ; then
-        if [ -z "${DOMSUF}" ]; then
+    if [ "$USE_IP" != "TRUE" ] ; then
+        if [ -z "$DOMSUF" ]; then
             HOSTADDR=${HOST}
         else
             HOSTADDR=${HOST}.${DOMSUF}
         fi
     else
         HOSTADDR=${IP_ADDRESS}
     fi
 
@@ -590,17 +584,17 @@ if [ -z "${INIT_SOURCED}" -o "${INIT_SOU
         P_R_DAVEDIR="multiaccess:${D_DAVE}"
         P_R_EVEDIR="multiaccess:${D_EVE}"
         P_R_SERVERDIR="multiaccess:${D_SERVER}"
         P_R_CLIENTDIR="multiaccess:${D_CLIENT}"
         P_R_NOLOGINDIR="multiaccess:${D_NOLOGIN}"
         P_R_EXT_SERVERDIR="multiaccess:${D_EXT_SERVER}"
         P_R_EXT_CLIENTDIR="multiaccess:${D_EXT_CLIENT}"
         P_R_IMPLICIT_INIT_DIR="multiaccess:${D_IMPLICIT_INIT}"
-	P_R_RSAPSSDIR="multiaccess:${D_RSAPSS}"
+        P_R_RSAPSSDIR="multiaccess:${D_RSAPSS}"
     fi
 
     R_PWFILE=../tests.pw
     R_EMPTY_FILE=../tests_empty
     R_NOISE_FILE=../tests_noise
 
     R_FIPSPWFILE=../tests.fipspw
     R_FIPSBADPWFILE=../tests.fipsbadpw