hgserver: convert replication hook to a Python hook (bug 1196869); r=fubar
authorGregory Szorc <gps@mozilla.com>
Thu, 20 Aug 2015 13:16:27 -0700
changeset 2952 65c6c9b387b1a954194661e8745c062968224e4a
parent 2951 38e5a18c791c4f2421c636e811d167beae137f59
child 2953 c6cb387f865b9229ef056dbffdff36817bcfe8ba
push id1053
push usergszorc@mozilla.com
push dateThu, 20 Aug 2015 20:17:08 +0000
treeherderversion-control-tools@65c6c9b387b1 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersfubar
bugs1196869
hgserver: convert replication hook to a Python hook (bug 1196869); r=fubar We want all hooks to be Python because, well, Python hooks are more powerful. As part of this, we have the hook print replication status, including time taken to replicate. This will increase understanding of what takes so long during pushes. We change the names of the changegroup hooks to enforce sorting. And we put URL printing after replication because it looks nicer.
ansible/roles/hg-ssh/files/hgrc
hghooks/mozhghooks/replicate.py
hgserver/tests/test-clone-command.t
hgserver/tests/test-obsolescence.t
hgserver/tests/test-push-basic.t
hgserver/tests/test-ssh-serve.t
--- a/ansible/roles/hg-ssh/files/hgrc
+++ b/ansible/roles/hg-ssh/files/hgrc
@@ -8,20 +8,20 @@ from = hg@hg.mozilla.org
 
 [diff]
 showfunc = True
 git = 1
 unified = 8
 
 [hooks]
 sorthooks = True
+changegroup.a_recordlogs = /repo/hg/scripts/record-pushes.sh
+changegroup.b_mirrorpush = python:mozhghooks.replicate.hook
 changegroup.push_printurls = python:mozhghooks.push_printurls.hook
-changegroup.recordlogs = /repo/hg/scripts/record-pushes.sh
-changegroup.mirrorpush = /repo/hg/scripts/push-repo.sh
-pushkey.mirrorpush = /repo/hg/scripts/push-repo.sh
+pushkey.mirrorpush = python:mozhghooks.replicate.hook
 #pretxnchangegroup.renamecase = python:mozhghooks.prevent_case_only_renames.hook
 pretxnchangegroup.single_root = python:mozhghooks.single_root.hook
 
 [extensions]
 blackbox =
 bundleclone = /repo/hg/extensions/bundleclone
 pushlog = /repo/hg/extensions/pushlog
 serverlog = /repo/hg/extensions/serverlog
new file mode 100644
--- /dev/null
+++ b/hghooks/mozhghooks/replicate.py
@@ -0,0 +1,39 @@
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+"""Hook to perform and record replication."""
+
+import os
+import pwd
+import subprocess
+import time
+
+
+# This is installed as both a changegroup and pushkey hook. They have
+# different arguments.
+def hook(ui, repo, **kwargs):
+    if not repo.root.startswith('/repo/hg/mozilla'):
+        ui.write('repository not eligible for replication\n')
+        return 0
+
+    relpath = repo.root[len('/repo/hg/mozilla/'):]
+    args = ['/usr/local/bin/repo-push.sh', relpath]
+
+    user = pwd.getpwuid(os.getuid()).pw_name
+    if user != 'hg':
+        args = ['/usr/bin/sudo', '-u', 'hg'] + args
+
+    t0 = time.time()
+
+    with open(os.devnull, 'w') as null:
+        res = subprocess.call(args, stdout=null, stderr=subprocess.STDOUT,
+                              cwd='/')
+
+    t1 = time.time()
+    status = 'completed successfully' if res == 0 else 'errored'
+    ui.write('replication to mirrors %s in %.1fs\n' % (status, t1 - t0))
+
+    # We don't currently let replication success dictate the result of the
+    # hook. This is a post transaction hook anyway, so failure likely doesn't
+    # do anything.
+    return 0
--- a/hgserver/tests/test-clone-command.t
+++ b/hgserver/tests/test-clone-command.t
@@ -16,16 +16,17 @@
   pushing to ssh://*:$HGPORT/hgcustom/version-control-tools (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 1 changesets with 1 changes to 1 files
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your change here:
   remote:   https://hg.mozilla.org/hgcustom/version-control-tools/rev/96ee1d7354c4
   $ cd ..
 
 We should get a prompt saying we are creating a new user repo.
 This also tests the exit choice.
 
@@ -294,16 +295,17 @@ We are able to push to the new user repo
   pushing to ssh://*:$HGPORT/users/user_example.com/repo-1 (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 1 changesets with 1 changes to 1 files
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your change here:
   remote:   https://hg.mozilla.org/users/user_example.com/repo-1/rev/384b668fc3c2
 
   $ cd ..
 
 TODO verify the new user repo shows up in hgweb
 
--- a/hgserver/tests/test-obsolescence.t
+++ b/hgserver/tests/test-obsolescence.t
@@ -73,16 +73,17 @@ Create initial repo content
   pushing to ssh://*:$HGPORT/users/user_example.com/repo-1 (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 3 changesets with 3 changes to 3 files (+1 heads)
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your changes here:
   remote:   https://hg.mozilla.org/users/user_example.com/repo-1/rev/96ee1d7354c4
   remote:   https://hg.mozilla.org/users/user_example.com/repo-1/rev/c7176a0c153e
   remote:   https://hg.mozilla.org/users/user_example.com/repo-1/rev/316dccdaed3c
   $ cd ..
 
 Create another clone
@@ -105,19 +106,21 @@ Create some obsolescence markers
   pushing to ssh://*:$HGPORT/users/user_example.com/repo-1 (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 1 changesets with 0 changes to 1 files
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your change here:
   remote:   https://hg.mozilla.org/users/user_example.com/repo-1/rev/d22031ed19ec
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   $ cd ..
 
 Pulling should get the obsmarkers
 
   $ cd repo-1-clone
   $ hg pull
   pulling from ssh://*:$HGPORT/users/user_example.com/repo-1 (glob)
   searching for changes
--- a/hgserver/tests/test-push-basic.t
+++ b/hgserver/tests/test-push-basic.t
@@ -31,16 +31,17 @@ Pushing a commit to a repo works
   pushing to ssh://*:$HGPORT/mozilla-central (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 1 changesets with 1 changes to 1 files
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your change here:
   remote:   https://hg.mozilla.org/mozilla-central/rev/96ee1d7354c4
 
 It got replicated to mirrors
 
   $ http --no-headers ${HGWEB_0_URL}mozilla-central/json-rev/96ee1d7354c4
   200
--- a/hgserver/tests/test-ssh-serve.t
+++ b/hgserver/tests/test-ssh-serve.t
@@ -36,13 +36,14 @@ A push works
   pushing to ssh://*:$HGPORT/repo1 (glob)
   searching for changes
   remote: adding changesets
   remote: adding manifests
   remote: adding file changes
   remote: added 1 changesets with 1 changes to 1 files
   remote: Trying to insert into pushlog.
   remote: Inserted into the pushlog db successfully.
+  remote: replication to mirrors completed successfully in \d+.\ds (re)
   remote: 
   remote: View your change here:
   remote:   https://hg.mozilla.org/repo1/rev/96ee1d7354c4
 
   $ hgmo stop