Bug 1279563 - Clone unified Firefox repository; r=glandium
authorGregory Szorc <gps@mozilla.com>
Thu, 16 Jun 2016 09:46:24 +0100
changeset 301985 b9f4f38063951cd5a8b249911aea61869f40fd1f
parent 301984 ffd61bc024b24d5d1c4e8cf1c7671a27cd06944d
child 301997 05873df0c0c268e419a714b8a6204cb852049098
push id78533
push usercbook@mozilla.com
push dateFri, 17 Jun 2016 00:03:15 +0000
treeherdermozilla-inbound@759277e45f04 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersglandium
bugs1279563, 10000
milestone50.0a1
first release with
nightly linux32
b9f4f3806395 / 50.0a1 / 20160616030228 / files
nightly linux64
b9f4f3806395 / 50.0a1 / 20160616030228 / files
nightly mac
b9f4f3806395 / 50.0a1 / 20160616030228 / files
nightly win32
b9f4f3806395 / 50.0a1 / 20160616030228 / files
nightly win64
b9f4f3806395 / 50.0a1 / 20160616030228 / files
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
releases
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1279563 - Clone unified Firefox repository; r=glandium https://hg.mozilla.org/firefox now exists. It is a unified Firefox repository containing the heads of all the major Firefox repos (mozilla-central, inbound, aurora, beta, release, esrs, etc). Having 1 unified repository is more useful and incurs less overhead than N separate repos. We want to encourage the use of the unified repository. So, we start cloning from it. The unified repo on the server is configured in such a way that manifest delta chains can become very long - over 30,000 deltas. This can make manifest reading very slow and slow down many Mercurial operations. The server compensates for this by setting format.maxchainlen=10000 to limit the delta chains to 10,000. Unfortunately, this setting is not preserved when clients do a traditional clone: the changegroup consists of a single delta chain and the client will use its own settings (often the default) to break the chain. This will result in the client re-creating very long delta chains. So, as part of initializing the new repo we configure format.maxchainlen in its .hg/hgrc so it doesn't suffer from this performance issue. We could achieve the same result by passing the --config option to `hg clone`. However, the option won't be preserved in the repo's .hg/hgrc and subsequent `hg pull` operations could result in the creation of very long delta chains. So we need to write the config option in the .hg/hgrc. `hg clone` is equivalent to `hg init` + `hg pull` anyway, so we just separate out the steps and insert a write to .hg/hgrc in between. We also set the "default" path (like `hg clone` would do). DONTBUILD (NPOTB) MozReview-Commit-ID: Fs4cz9YvdCv
python/mozboot/mozboot/bootstrap.py
--- a/python/mozboot/mozboot/bootstrap.py
+++ b/python/mozboot/mozboot/bootstrap.py
@@ -83,17 +83,17 @@ Your choice:
 
 FINISHED = '''
 Your system should be ready to build %s!
 '''
 
 SOURCE_ADVERTISE = '''
 Source code can be obtained by running
 
-    hg clone https://hg.mozilla.org/mozilla-central
+    hg clone https://hg.mozilla.org/firefox
 
 Or, if you prefer Git, you should install git-cinnabar, and follow the
 instruction here to clone from the Mercurial repository:
 
     https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development
 
 Or, if you really prefer vanilla flavor Git:
 
@@ -343,31 +343,61 @@ def update_mercurial_repo(hg, url, dest,
     finally:
         print('=' * 80)
 
 
 def clone_firefox(hg, dest):
     """Clone the Firefox repository to a specified destination."""
     print('Cloning Firefox Mercurial repository to %s' % dest)
 
+    # We create an empty repo then modify the config before adding data.
+    # This is necessary to ensure storage settings are optimally
+    # configured.
     args = [
         hg,
-        'clone',
-        'https://hg.mozilla.org/mozilla-central',
-        dest,
+        # The unified repo is generaldelta, so ensure the client is as
+        # well.
+        '--config', 'format.generaldelta=true',
+        'init',
+        dest
     ]
+    res = subprocess.call(args)
+    if res:
+        print('unable to create destination repo; please try cloning manually')
+        return False
 
-    res = subprocess.call(args)
+    # Strictly speaking, this could overwrite a config based on a template
+    # the user has installed. Let's pretend this problem doesn't exist
+    # unless someone complains about it.
+    with open(os.path.join(dest, '.hg', 'hgrc'), 'ab') as fh:
+        fh.write('[paths]\n')
+        fh.write('default = https://hg.mozilla.org/firefox\n')
+        fh.write('\n')
+
+        # The server uses aggressivemergedeltas which can blow up delta chain
+        # length. This can cause performance to tank due to delta chains being
+        # too long. Limit the delta chain length to something reasonable
+        # to bound revlog read time.
+        fh.write('[format]\n')
+        fh.write('# This is necessary to keep performance in check\n')
+        fh.write('maxchainlen = 10000\n')
+
+    res = subprocess.call([hg, 'pull', 'https://hg.mozilla.org/firefox'], cwd=dest)
     print('')
     if res:
-        print('error cloning; please try again')
+        print('error pulling; try running `hg pull https://hg.mozilla.org/firefox` manually')
         return False
-    else:
-        print('Firefox source code available at %s' % dest)
-        return True
+
+    print('updating to "central" - the development head of Gecko and Firefox')
+    res = subprocess.call([hg, 'update', '-r', 'central'], cwd=dest)
+    if res:
+        print('error updating; you will need to `hg update` manually')
+
+    print('Firefox source code available at %s' % dest)
+    return True
 
 
 def current_firefox_checkout(check_output, hg=None):
     """Determine whether we're in a Firefox checkout.
 
     Returns one of None, ``git``, or ``hg``.
     """
     HG_ROOT_REVISIONS = set([