bundle2: increase payload part chunk size to 32kb
authorGregory Szorc <gregory.szorc@gmail.com>
Sat, 20 Jan 2018 22:55:42 -0800
changeset 41702 4fb2bb61597cb34c69c4af7a2d1fb0bb43145eb1
parent 41701 1d118f9f4f57682e5ddf7d2a60eedf1133ee4986
child 41703 27b6df1b5adbdf647cf5c6675b40575e1b197c60
child 41803 34287c2e3fc4d4e1a3546386132f755cb870325c
push id656
push usergszorc@mozilla.com
push dateTue, 23 Jan 2018 02:05:10 +0000
bugs32768
bundle2: increase payload part chunk size to 32kb Bundle2 payload parts are framed chunks. Esentially, we obtain data in equal size chunks of size `preferedchunksize` and emit those to a generator. That generator is fed into a compressor (which can be the no-op compressor, which just re-emits the generator). And the output from the compressor likely goes to a file descriptor or socket. What this means is that small chunk sizes create more Python objects and Python function calls than larger chunk sizes. And as we know, Python object and function call overhead in performance sensitive code matters (at least with CPython). This commit increases the bundle2 part payload chunk size from 4k to 32k. Practically speaking, this means that the chunks we feed into a compressor (implemented in C code) or feed directly into a file handle or socket write() are larger. It's possible the chunks might be larger than what the receiver can handle in one logical operation. But at that point, we're in C code, which is much more efficient at dealing with splitting up the chunk and making multiple function calls than Python is. A downside to larger chunks is that the receiver has to wait for that much data to arrive (either raw or from a decompressor) before it can process the chunk. But 32kb still feels like a small buffer to have to wait for. And in many cases, the client will convert from 8 read(4096) to 1 read(32768). That's happening in Python land. So we cut down on the number of Python objects and function calls, making the client faster as well. I don't think there are any significant concerns to increasing the payload chunk size to 32kb. The impact of this change on performance significant. Using `curl` to obtain a stream clone bundle2 payload from a server on localhost serving the mozilla-unified repository: before: 20.78 user; 7.71 system; 80.5 MB/s after: 13.90 user; 3.51 system; 132 MB/s legacy: 9.72 user; 8.16 system; 132 MB/s bundle2 stream clone generation is still more resource intensive than legacy stream clone (that's likely because of the use of a util.chunkbuffer). But the throughput is the same. We might be in territory we're this is effectively a benchmark of the networking stack or Python's syscall throughput. From the client perspective, `hg clone -U --stream`: before: 33.50 user; 7.95 system; 53.3 MB/s after: 22.82 user; 7.33 system; 72.7 MB/s legacy: 29.96 user; 7.94 system; 58.0 MB/s And for `hg clone --stream` with a working directory update of ~230k files: after: 119.55 user; 26.47 system; 0:57.08 wall legacy: 126.98 user; 26.94 system; 1:05.56 wall So, it appears that bundle2's stream clone is now definitively faster than legacy stream clone! Differential Revision: https://phab.mercurial-scm.org/D1932
mercurial/bundle2.py
tests/test-clone-uncompressed.t
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -178,17 +178,17 @@ urlreq = util.urlreq
 
 _fstreamparamsize = '>i'
 _fpartheadersize = '>i'
 _fparttypesize = '>B'
 _fpartid = '>I'
 _fpayloadsize = '>i'
 _fpartparamcount = '>BB'
 
-preferedchunksize = 4096
+preferedchunksize = 32768
 
 _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
 
 def outdebug(ui, message):
     """debug regarding output stream (bundling)"""
     if ui.configbool('devel', 'bundle2.debug'):
         ui.debug('bundle2-output: %s\n' % message)
 
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -194,25 +194,25 @@ Basic clone
 getbundle requests with stream=1 are uncompressed
 
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto '0.1 0.2 comp=zlib,none' --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
   200 Script output follows
   content-type: application/mercurial-0.2
   
 
   $ f --size --hex --bytes 256 body
-  body: size=112318
+  body: size=112222
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 68 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |h.STREAM2.......|
   0020: 05 09 04 0c 2d 62 79 74 65 63 6f 75 6e 74 39 38 |....-bytecount98|
   0030: 37 35 38 66 69 6c 65 63 6f 75 6e 74 31 30 33 30 |758filecount1030|
   0040: 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 65 |requirementsdote|
   0050: 6e 63 6f 64 65 20 66 6e 63 61 63 68 65 20 67 65 |ncode fncache ge|
   0060: 6e 65 72 61 6c 64 65 6c 74 61 20 72 65 76 6c 6f |neraldelta revlo|
-  0070: 67 76 31 20 73 74 6f 72 65 00 00 10 00 73 08 42 |gv1 store....s.B|
+  0070: 67 76 31 20 73 74 6f 72 65 00 00 80 00 73 08 42 |gv1 store....s.B|
   0080: 64 61 74 61 2f 30 2e 69 00 03 00 01 00 00 00 00 |data/0.i........|
   0090: 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 01 |................|
   00a0: ff ff ff ff ff ff ff ff 80 29 63 a0 49 d3 23 87 |.........)c.I.#.|
   00b0: bf ce fe 56 67 92 67 2c 69 d1 ec 39 00 00 00 00 |...Vg.g,i..9....|
   00c0: 00 00 00 00 00 00 00 00 75 30 73 08 42 64 61 74 |........u0s.Bdat|
   00d0: 61 2f 31 2e 69 00 03 00 01 00 00 00 00 00 00 00 |a/1.i...........|
   00e0: 02 00 00 00 01 00 00 00 00 00 00 00 01 ff ff ff |................|
   00f0: ff ff ff ff ff f9 76 da 1d 0d f2 25 6c de 08 db |......v....%l...|