streamclone: preserve remote phases (issue5648) draft
authorGregory Szorc <gregory.szorc@gmail.com>
Sun, 24 Dec 2017 11:00:45 -0700
changeset 41195 1562a410bb7c135d3ff93cc862a84ff1af9d524b
parent 41194 13292620537280e768f6221fc1394c744526e31c
push id628
push usergszorc@mozilla.com
push dateSun, 24 Dec 2017 18:07:24 +0000
streamclone: preserve remote phases (issue5648) As the tests added by the previous changeset demonstrate, stream clones were never taught about the existence of phases. As a result, all changesets applied via stream clone were made public, even if they were draft (or even secret) on the remote. This commit integrates phases knowledge into stream clones. As the inline comment explains, because there are scenarios where we can't disambiguate phases support, the resulting phase data may not match the server exactly. However, I believe the remaining areas of buggy behavior are confined to secret changesets. Secret changesets aren't common. And transferring them via stream clones either requires a buggy Mercurial server or a specially-configured server. I think it will require stream clone support in bundle2 before we can properly handle secret phases for stream clones. It /might/ be acceptable to force all local changesets to secret and then promote to draft or public from remote phases data. However, I was having trouble implementing this. Existing phases code doesn't seem geared towards supporting this scenario and I didn't want to incur a lot of extra work to refactor the phases code. Perfect is the enemy of good: I'm inclined to tackle secret phase as part of supporting stream clones in bundle2. This change introduces a `roots(all())` revset, which will require a linear scan of the changelog during clone bundles. On a Firefox repo, this may slow down stream clones by ~1s. Again, support for stream clones in bundle2 should help here. .. fix:: Draft changeset phase is now preserved during a stream clone. Before, all changesets pulled via a stream clone would have a public phase, even if they were draft on the remote and the remote was non-publishing. Differential Revision: https://phab.mercurial-scm.org/D1753
mercurial/streamclone.py
tests/test-clone-uncompressed.t
tests/test-http-bundle1.t
tests/test-http-proxy.t
tests/test-http.t
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -5,16 +5,19 @@
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 import struct
 
 from .i18n import _
+from .node import (
+    hex,
+)
 from . import (
     branchmap,
     error,
     phases,
     store,
     util,
 )
 
@@ -113,16 +116,18 @@ def maybeperformlegacystreamclone(pullop
     supported, requirements = canperformstreamclone(pullop)
 
     if not supported:
         return
 
     repo = pullop.repo
     remote = pullop.remote
 
+    remotephases = pullop.remote.listkeys('phases')
+
     # Save remote branchmap. We will use it later to speed up branchcache
     # creation.
     rbranchmap = None
     if remote.capable('branchmap'):
         rbranchmap = remote.branchmap()
 
     repo.ui.status(_('streaming all changes\n'))
 
@@ -153,16 +158,55 @@ def maybeperformlegacystreamclone(pullop
         # new requirements = old non-format requirements +
         #                    new format-related remote requirements
         # requirements from the streamed-in repository
         repo.requirements = requirements | (
                 repo.requirements - repo.supportedformats)
         repo._applyopenerreqs()
         repo._writerequirements()
 
+        # Application of phases data is subtly non-trivial.
+        #
+        # The following have to be considered:
+        #
+        # * The remote could be running an old version of Mercurial that
+        #   isn't phases aware. Remote phases data will be empty. (This should
+        #   not be common since phases were supported since Mercurial 1.9.)
+        # * Some versions of Mercurial (before issue 5589 was fixed) can
+        #   serve secret changesets. Versions after can be configured to serve
+        #   secret changesets.
+        # * The remote could be publishing or non-publishing.
+        # * If local changesets are secret, this impacts discovery and other
+        #   mechanisms that occur during the "pull" that is performed after
+        #   this function returns.
+        #
+        # This combination of scenarios means that we can't 100% reliably
+        # reproduce the remote phases state. For example, if the remote is
+        # configured to serve secret changesets and all changesets are secret,
+        # there won't be any phases data and we won't be able to distinguish
+        # between that repo and a Mercurial that isn't phases aware.
+        #
+        # Our strategy is to look at the remote phases data. If there isn't any,
+        # we assume we are talking to an old server that doesn't support phases.
+        # We keep all changesets as public. Otherwise, we force all changesets
+        # to draft. Then we apply the remote phases data.
+        if remotephases:
+            roots = [ctx.node() for ctx in repo.set('roots(all())')]
+            tr = pullop.gettransaction()
+            phases.retractboundary(repo, tr, phases.draft, roots)
+
+            # Strictly speaking, we may not need to apply remote phases here,
+            # as the incremental pull after this function may take care of it.
+            # But since we already have the data, we might as well use it.
+            publicheads, draftroots = phases.analyzeremotephases(
+                repo, repo.heads(), {hex(n): phases.draft for n in roots})
+
+            if publicheads:
+                phases.advanceboundary(repo, tr, phases.public, publicheads)
+
         if rbranchmap:
             branchmap.replacecache(repo, rbranchmap)
 
         repo.invalidate()
 
 def allowservergeneration(repo):
     """Whether streaming clones are allowed from the server."""
     if not repo.ui.configbool('server', 'uncompressed', untrusted=True):
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -34,16 +34,19 @@ Basic clone
   searching for changes
   no changes found
 
 Clone with background file closing enabled
 
   $ hg --debug --config worker.backgroundclose=true --config worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT clone-background | grep -v adding
   using http://localhost:$HGPORT/
   sending capabilities command
+  preparing listkeys for "phases"
+  sending listkeys command
+  received listkey for "phases": 58 bytes
   sending branchmap command
   streaming all changes
   sending stream_out command
   1027 files to transfer, 96.3 KB of data
   starting 4 threads for background file closing
   transferred 96.3 KB in * seconds (*/sec) (glob)
   query 1; heads
   sending batch command
@@ -89,36 +92,32 @@ Phases on non-publishing server are pres
 
   $ hg clone --stream -U http://localhost:$HGPORT all-draft
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
 
-TODO this is buggy
-
   $ hg -R all-draft log -T '{rev} {phase}\n'
-  1 public
-  0 public
+  1 draft
+  0 draft
 
 Mixed phase is preserved
 
   $ hg -R server phase --public -r 0
   $ hg clone --stream -U http://localhost:$HGPORT mixed-draft
   streaming all changes
   1027 files to transfer, 96.3 KB of data
   transferred 96.3 KB in * seconds (*/sec) (glob)
   searching for changes
   no changes found
 
-TODO this is buggy
-
   $ hg -R mixed-draft log -T '{rev} {phase}\n'
-  1 public
+  1 draft
   0 public
 
 Cannot stream clone when there are secret changesets
 
   $ hg -R server phase --force --secret -r tip
   $ hg clone --stream -U http://localhost:$HGPORT secret-denied
   warning: stream clone requested but server has them disabled
   requesting all changes
--- a/tests/test-http-bundle1.t
+++ b/tests/test-http-bundle1.t
@@ -286,18 +286,19 @@ test http authentication
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
   "GET /?cmd=lookup HTTP/1.1" 200 - x-hgarg-1:key=tip x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 401 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
+  "GET /?cmd=listkeys HTTP/1.1" 401 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
+  "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=branchmap HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
-  "GET /?cmd=stream_out HTTP/1.1" 401 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=stream_out HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
   "GET /?cmd=listkeys HTTP/1.1" 401 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
--- a/tests/test-http-proxy.t
+++ b/tests/test-http-proxy.t
@@ -102,16 +102,17 @@ do not use the proxy if it is in the no 
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   new changesets 83180e7845de
   updating to branch default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ cat proxy.log
   * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
+  $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=listkeys HTTP/1.1" - - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=branchmap HTTP/1.1" - - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=stream_out HTTP/1.1" - - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D83180e7845de420a1bb46896fd5fe05294f8d629 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=0&common=83180e7845de420a1bb46896fd5fe05294f8d629&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=batch HTTP/1.1" - - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   $LOCALIP - - [$LOGDATE$] "GET http://localhost:$HGPORT/?cmd=getbundle HTTP/1.1" - - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=83180e7845de420a1bb46896fd5fe05294f8d629&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ (glob)
   * - - [*] "GET http://localhost:$HGPORT/?cmd=capabilities HTTP/1.1" - - (glob)
--- a/tests/test-http.t
+++ b/tests/test-http.t
@@ -277,18 +277,19 @@ test http authentication
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
   "GET /?cmd=lookup HTTP/1.1" 200 - x-hgarg-1:key=tip x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 401 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=namespaces x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=bookmarks x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
+  "GET /?cmd=listkeys HTTP/1.1" 401 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
+  "GET /?cmd=listkeys HTTP/1.1" 200 - x-hgarg-1:namespace=phases x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=branchmap HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
-  "GET /?cmd=stream_out HTTP/1.1" 401 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=stream_out HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D5fed3813f7f5e1824344fdc9cf8f63bb662c292d x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=0&common=5fed3813f7f5e1824344fdc9cf8f63bb662c292d&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -
   "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=getbundle HTTP/1.1" 401 - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   "GET /?cmd=capabilities HTTP/1.1" 200 -