docker: stop using `rsync -a` because owner/group updates are slow on overlayfs (bug 1349407)
authorGregory Szorc <gps@mozilla.com>
Mon, 15 May 2017 11:37:32 -0700
changeset 5333 63f19cfacc5c
parent 5332 69221bbea902
child 5334 83c3a80a797c
push id2460
push usergszorc@mozilla.com
push date2017-05-16 01:14 +0000
treeherderversion-control-tools@743e28b371d1 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
bugs1349407
docker: stop using `rsync -a` because owner/group updates are slow on overlayfs (bug 1349407) When I switched my Docker install to the overlay2 storage driver, incremental rebuilds of Docker images became really slow. I tracked this down to fchownat() taking ~100ms per call. This is apparently a known deficiency in overlayfs. We were performing fchownat() indirectly via various `rsync -a` and `chown -R` invocations. In almost all the cases these commands were being used, we either don't care about file owner or were running as root, so files were already owned by the proper user/group. So, this commit replaces a bunch of `rsync -a` and `chown -R` with `rsync -rlpt`, which is the subset of rsync we care about. We could potentially drop mtime sync. But unless it causes performance problems, preserving it is best in case anything is looking at mtimes.
ansible/roles/hg-ssh/tasks/main.yml
ansible/roles/hg-web/tasks/main.yml
testing/clobber.hgmaster
testing/clobber.hgweb
testing/docker/builder-ansible-centos6/sync-and-build
testing/docker/builder-ansible-centos7/sync-and-build
testing/docker/builder-ansible-ubuntu/sync-and-build
testing/docker/builder-hgweb-chroot/bootstrap
testing/vcttesting/docker.py
--- a/ansible/roles/hg-ssh/tasks/main.yml
+++ b/ansible/roles/hg-ssh/tasks/main.yml
@@ -139,17 +139,17 @@
   command: /var/hg/venv_tools/bin/hg -R /var/hg/version-control-tools up -r {{ lookup('file', '../../../../.vctnode') }}
   when: vct_dir.stat.exists == False
 
 - name: delete ignored files from version-control-tools repo
   command: /var/hg/venv_tools/bin/hg --config extensions.purge= -R /var/hg/version-control-tools purge --all
   when: vct_dir.stat.exists == False
 
 - name: rsync version-control-tools repo (Docker only)
-  command: /usr/bin/rsync -a --delete-after /vct/ /var/hg/version-control-tools/
+  command: /usr/bin/rsync -rlpt --delete-after /vct/ /var/hg/version-control-tools/
   when: vct_dir.stat.exists == True
 
 - name: ensure version-control-tools permissions are sane (Docker only)
   command: /bin/chmod -R a+r /var/hg/version-control-tools
   when: vct_dir.stat.exists == True
 
 - name: install vcsreplicator into virtualenv
   command: /var/hg/venv_tools/bin/pip install --upgrade --no-deps --force-reinstall /var/hg/version-control-tools/pylib/vcsreplicator
--- a/ansible/roles/hg-web/tasks/main.yml
+++ b/ansible/roles/hg-web/tasks/main.yml
@@ -156,17 +156,17 @@
 
 # The hg module's purge doesn't delete ignored files. Boo. Force that
 # because old .pyc files may cause chaos.
 - name: delete ignored files from version-control-tools repo
   command: /var/hg/venv_hgweb/bin/hg --config extensions.purge= -R /var/hg/version-control-tools purge --all
   when: vct_dir.stat.exists == False
 
 - name: rsync version-control-tools repo (Docker only)
-  command: /usr/bin/rsync -a --delete-after /vct/ /var/hg/version-control-tools/
+  command: /usr/bin/rsync -rlpt --delete-after /vct/ /var/hg/version-control-tools/
   when: vct_dir.stat.exists == True
 
 - name: set up hg-experimental repo, with 3rd party extensions
   hg: repo=https://bitbucket.org/facebook/hg-experimental
       dest=/var/hg/hg-experimental
       revision=8af0e065c0099abfd17321c41351b2aacad7668d
       force=yes
       purge=yes
--- a/testing/clobber.hgmaster
+++ b/testing/clobber.hgmaster
@@ -1,1 +1,1 @@
-Switching image from CentOS 6 to CentOS 7
+Refreshed base image
--- a/testing/clobber.hgweb
+++ b/testing/clobber.hgweb
@@ -1,1 +1,1 @@
-upgrade to CentOS 7
+Refreshed base image
--- a/testing/docker/builder-ansible-centos6/sync-and-build
+++ b/testing/docker/builder-ansible-centos6/sync-and-build
@@ -1,11 +1,10 @@
 #!/bin/bash
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 set -e
 
-/usr/bin/rsync -a --delete-after /vct-mount/ /vct/
-chown -R root:root /vct
+/usr/bin/rsync -rlpt --delete-after /vct-mount/ /vct/
 cd /vct/ansible
 /usr/bin/python -u /usr/bin/ansible-playbook -c local $1
--- a/testing/docker/builder-ansible-centos7/sync-and-build
+++ b/testing/docker/builder-ansible-centos7/sync-and-build
@@ -1,11 +1,12 @@
 #!/bin/bash
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 set -e
 
-/usr/bin/rsync -a --delete-after /vct-mount/ /vct/
-chown -R root:root /vct
+# We don't preserve owner/group here because chown is slow on certain
+# Docker filesystems, like overlay2.
+/usr/bin/rsync -rlpt --delete-after /vct-mount/ /vct/
 cd /vct/ansible
 /usr/bin/python -u /usr/bin/ansible-playbook -c local $1
--- a/testing/docker/builder-ansible-ubuntu/sync-and-build
+++ b/testing/docker/builder-ansible-ubuntu/sync-and-build
@@ -1,11 +1,11 @@
 #!/bin/bash
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 set -e
 
-/usr/bin/rsync -a --delete-after /vct-mount/ /vct/
+/usr/bin/rsync -rlpt --delete-after /vct-mount/ /vct/
 chown -R root:root /vct
 cd /vct/ansible
 /usr/bin/python2.7 -u /usr/local/bin/ansible-playbook -c local $1
--- a/testing/docker/builder-hgweb-chroot/bootstrap
+++ b/testing/docker/builder-hgweb-chroot/bootstrap
@@ -6,19 +6,17 @@
 # This script runs when the container starts to produce a chroot
 # archive. It assumes a v-c-t volume is mounted at /vct-mount.
 
 set -e
 
 VCT=/python/var/hg/version-control-tools
 
 mkdir -p /python/var/hg
-/usr/bin/rsync -a --delete-after /vct-mount/ ${VCT}/
-
-chown -R root:root ${VCT}
+/usr/bin/rsync -rlpt --delete-after /vct-mount/ ${VCT}/
 
 mkdir /python/dev
 
 # setuid or setgid bits could lead to privilege escalation. Unset them.
 find /python -perm -4000 -exec chmod u-s {} \;
 find /python -perm -2000 -exec chmod g-s {} \;
 
 tar -C /python -czf /chroot.tar.gz .
--- a/testing/vcttesting/docker.py
+++ b/testing/vcttesting/docker.py
@@ -1663,17 +1663,22 @@ class Docker(object):
                 get_and_write_vct_node()
                 vct_paths = self._get_vct_files()
                 with tempfile.NamedTemporaryFile() as fh:
                     for f in sorted(vct_paths.keys()):
                         fh.write('%s\n' % f)
                     fh.write('.vctnode\n')
                     fh.flush()
 
-                    rsync('-a', '--delete-before', '--files-from', fh.name,
+                    # We don't use -a (implies -tlptgoD) because with some
+                    # filesystems used by Docker (namely overlay2), touching
+                    # files only to update attributes has a ton of overhead.
+                    # We shouldn't care about owner/group, so we don't sync
+                    # these.
+                    rsync('-rlpt', '--delete-before', '--files-from', fh.name,
                           ROOT, url)
 
                 self.state['last-vct-id'] = image
                 self.state['vct-cid'] = cid
                 self.save_state()
 
             yield state
         finally: