Bug 1137898 - Migrate to android:versionCode scheme v1. r=rnewman,snorp a=sylvestre
authorNick Alexander <nalexander@mozilla.com>
Wed, 26 Aug 2015 18:34:48 -0700
changeset 289124 648c31e781680ded6dec0cf46915b7e4c9023b9a
parent 289123 437fb42ca4364757dacaaf65a526aaf92293b28d
child 289125 44ed86477ea22dc8fb35ead22571cfbdfd8e8f31
push id5067
push userraliiev@mozilla.com
push dateMon, 21 Sep 2015 14:04:52 +0000
treeherdermozilla-beta@14221ffe5b2f [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersrnewman, snorp, sylvestre
bugs1137898
milestone42.0a2
Bug 1137898 - Migrate to android:versionCode scheme v1. r=rnewman,snorp a=sylvestre Android version codes serve multiple masters. They indicate newer versions, yes; but they also break ties between versions with different features and requirements. High order bits effectively partition the space of versions and are valuable. Since Android version codes are signed Java integers, we have 31 bits to work with. Mozilla's traditional build ID is YYYYMMDDhhmmss. This was chopped to ten characters (YYYYMMDDhh, i.e., hourly build IDs) and then converted to a decimal. This took many high order bits. We will lose another high order bit in the 36th month of 2015 -- i.e., as soon as the year rolls over to 2016. If we waited to lose the next higher order bit, we'd lose that one in the 46th month of 2017 -- i.e., as soon as the year rolls over to 2018. The following patch sacrifices a high order bit to change the version scheme, winning us roughly 15 years of hourly build IDs before we are forced to lose another high order bit. So it's clearly to our advantage to change the scheme sooner rather than later -- we will sacrifice 1 bit for 15 years of build IDs, rather than keeping the current scheme and sacrificing (say) 2 bits for 3 years of build IDs. The resulting scheme produces build IDs that look like (in binary): 0111 1000 0010 tttt tttt tttt tttt txpg The meaning of these build IDs is documented in the Python source code that generates them.
python/mozbuild/mozbuild/android_version_code.py
python/mozbuild/mozbuild/test/test_android_version_code.py
--- a/python/mozbuild/mozbuild/android_version_code.py
+++ b/python/mozbuild/mozbuild/android_version_code.py
@@ -1,34 +1,136 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from __future__ import absolute_import, print_function
 
 import argparse
+import math
 import sys
+import time
 
+# Builds before this build ID use the v0 version scheme.  Builds after this
+# build ID use the v1 version scheme.
+V1_CUTOFF = 20150801000000 # YYYYmmddHHMMSS
 
-def android_version_code(buildid, cpu_arch=None, min_sdk=0, max_sdk=0):
+def android_version_code_v0(buildid, cpu_arch=None, min_sdk=0, max_sdk=0):
     base = int(str(buildid)[:10])
     # None is interpreted as arm.
     if not cpu_arch or cpu_arch in ['armeabi', 'armeabi-v7a']:
         # Increment by MIN_SDK_VERSION -- this adds 9 to every build ID as a
         # minimum.  Our split APK starts at 11.
         return base + min_sdk + 0
     elif cpu_arch in ['x86']:
         # Increment the version code by 3 for x86 builds so they are offered to
         # x86 phones that have ARM emulators, beating the 2-point advantage that
         # the v11+ ARMv7 APK has.  If we change our splits in the future, we'll
         # need to do this further still.
         return base + min_sdk + 3
     else:
         raise ValueError("Don't know how to compute android:versionCode "
-                         "for CPU arch " + cpu_arch)
+                         "for CPU arch %s" % cpu_arch)
+
+def android_version_code_v1(buildid, cpu_arch=None, min_sdk=0, max_sdk=0):
+    '''Generate a v1 android:versionCode.
+
+    The important consideration is that version codes be monotonically
+    increasing (per Android package name) for all published builds.  The input
+    build IDs are based on timestamps and hence are always monotonically
+    increasing.
+
+    The generated v1 version codes look like (in binary):
+
+    0111 1000 0010 tttt tttt tttt tttt txpg
+
+    The 17 bits labelled 't' represent the number of hours since midnight on
+    September 1, 2015.  (2015090100 in YYYYMMMDDHH format.)  This yields a
+    little under 15 years worth of hourly build identifiers, since 2**17 / (366
+    * 24) =~ 14.92.
+
+    The bits labelled 'x', 'p', and 'g' are feature flags.
+
+    The bit labelled 'x' is 1 if the build is for an x86 architecture and 0
+    otherwise, which means the build is for an ARM architecture.  (Fennec no
+    longer supports ARMv6, so ARM is equivalent to ARMv7 and above.)
+
+    The bit labelled 'p' is a placeholder that is always 0 (for now).
+
+    The bit labelled 'g' is 1 if the build is targetting Android API 11+ and 0
+    otherwise, which means the build targets Android API 9-10 (Gingerbread).
+    (Fennec no longer supports below Android API 9.)
+
+    We throw an explanatory exception when we are within one calendar year of
+    running out of build events.  This gives lots of time to update the version
+    scheme.  The responsible individual should then bump the range (to allow
+    builds to continue) and use the time remaining to update the version scheme
+    via the reserved high order bits.
+
+    N.B.: the reserved 0 bit to the left of the highest order 't' bit can,
+    sometimes, be used to bump the version scheme.  In addition, by reducing the
+    granularity of the build identifiers (for example, moving to identifying
+    builds every 2 or 4 hours), the version scheme may be adjusted further still
+    without losing a (valuable) high order bit.
+    '''
+    def hours_since_cutoff(buildid):
+        # The ID is formatted like YYYYMMDDHHMMSS (using
+        # datetime.now().strftime('%Y%m%d%H%M%S'); see
+        # toolkit/xre/make-platformini.py).  The inverse function is
+        # time.strptime.  N.B.: the time module expresses time as decimal
+        # seconds since the epoch.
+        fmt = '%Y%m%d%H%M%S'
+        build = time.strptime(str(buildid), fmt)
+        cutoff = time.strptime(str(V1_CUTOFF), fmt)
+        return int(math.floor((time.mktime(build) - time.mktime(cutoff)) / (60.0 * 60.0)))
+
+    # Of the 21 low order bits, we take 17 bits for builds.
+    base = hours_since_cutoff(buildid)
+    if base < 0:
+        raise ValueError("Something has gone horribly wrong: cannot calculate "
+                         "android:versionCode from build ID %s: hours underflow "
+                         "bits allotted!" % buildid)
+    if base > 2**17:
+        raise ValueError("Something has gone horribly wrong: cannot calculate "
+                         "android:versionCode from build ID %s: hours overflow "
+                         "bits allotted!" % buildid)
+    if base > 2**17 - 366 * 24:
+        raise ValueError("Running out of low order bits calculating "
+                         "android:versionCode from build ID %s: "
+                         "; YOU HAVE ONE YEAR TO UPDATE THE VERSION SCHEME." % buildid)
+
+    version = 0b1111000001000000000000000000000
+    # We reserve 1 "middle" high order bit for the future, and 3 low order bits
+    # for architecture and APK splits.
+    version |= base << 3
+
+    # None is interpreted as arm.
+    if not cpu_arch or cpu_arch in ['armeabi', 'armeabi-v7a']:
+        # 0 is interpreted as SDK 9.
+        if not min_sdk or min_sdk == 9:
+            pass
+        elif min_sdk == 11:
+            version |= 1 << 0
+        else:
+            raise ValueError("Don't know how to compute android:versionCode "
+                             "for CPU arch %s and min SDK %s" % (cpu_arch, min_sdk))
+    elif cpu_arch in ['x86']:
+        version |= 1 << 2
+    else:
+        raise ValueError("Don't know how to compute android:versionCode "
+                         "for CPU arch %s" % cpu_arch)
+
+    return version
+
+def android_version_code(buildid, *args, **kwargs):
+    base = int(str(buildid))
+    if base < V1_CUTOFF:
+        return android_version_code_v0(buildid, *args, **kwargs)
+    else:
+        return android_version_code_v1(buildid, *args, **kwargs)
 
 
 def main(argv):
     parser = argparse.ArgumentParser('Generate an android:versionCode',
                                      add_help=False)
     parser.add_argument('--verbose', action='store_true',
                         default=False,
                         help='Be verbose')
--- a/python/mozbuild/mozbuild/test/test_android_version_code.py
+++ b/python/mozbuild/mozbuild/test/test_android_version_code.py
@@ -1,23 +1,65 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from mozunit import main
 import unittest
 
-from mozbuild.android_version_code import android_version_code
+from mozbuild.android_version_code import (
+    android_version_code_v0,
+    android_version_code_v1,
+)
 
 class TestAndroidVersionCode(unittest.TestCase):
-    def test_basic(self):
+    def test_android_version_code_v0(self):
         # From https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=e25de9972a77.
         buildid = '20150708104620'
         arm_api9  = 2015070819
         arm_api11 = 2015070821
         x86_api9  = 2015070822
-        self.assertEqual(android_version_code(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None), arm_api9)
-        self.assertEqual(android_version_code(buildid, cpu_arch='armeabi-v7a', min_sdk=11, max_sdk=None), arm_api11)
-        self.assertEqual(android_version_code(buildid, cpu_arch='x86', min_sdk=9, max_sdk=None), x86_api9)
+        self.assertEqual(android_version_code_v0(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None), arm_api9)
+        self.assertEqual(android_version_code_v0(buildid, cpu_arch='armeabi-v7a', min_sdk=11, max_sdk=None), arm_api11)
+        self.assertEqual(android_version_code_v0(buildid, cpu_arch='x86', min_sdk=9, max_sdk=None), x86_api9)
+
+    def test_android_version_code_v1(self):
+        buildid = '20150825141628'
+        arm_api9  = 0b01111000001000000001001001110000
+        arm_api11 = 0b01111000001000000001001001110001
+        x86_api9  = 0b01111000001000000001001001110100
+        self.assertEqual(android_version_code_v1(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None), arm_api9)
+        self.assertEqual(android_version_code_v1(buildid, cpu_arch='armeabi-v7a', min_sdk=11, max_sdk=None), arm_api11)
+        self.assertEqual(android_version_code_v1(buildid, cpu_arch='x86', min_sdk=9, max_sdk=None), x86_api9)
+
+    def test_android_version_code_v1_underflow(self):
+        '''Verify that it is an error to ask for v1 codes predating the cutoff.'''
+        buildid = '201508010000' # Earliest possible.
+        arm_api9 = 0b01111000001000000000000000000000
+        self.assertEqual(android_version_code_v1(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None), arm_api9)
+        with self.assertRaises(ValueError) as cm:
+            underflow = '201507310000' # Latest possible (valid) underflowing date.
+            android_version_code_v1(underflow, cpu_arch='armeabi', min_sdk=9, max_sdk=None)
+        self.assertTrue('underflow' in cm.exception.message)
+
+    def test_android_version_code_v1_running_low(self):
+        '''Verify there is an informative message if one asks for v1 codes that are close to overflow.'''
+        with self.assertRaises(ValueError) as cm:
+            overflow = '20290801000000'
+            android_version_code_v1(overflow, cpu_arch='armeabi', min_sdk=9, max_sdk=None)
+        self.assertTrue('Running out of low order bits' in cm.exception.message)
+
+    def test_android_version_code_v1_overflow(self):
+        '''Verify that it is an error to ask for v1 codes that actually does overflow.'''
+        with self.assertRaises(ValueError) as cm:
+            overflow = '20310801000000'
+            android_version_code_v1(overflow, cpu_arch='armeabi', min_sdk=9, max_sdk=None)
+        self.assertTrue('overflow' in cm.exception.message)
+
+    def test_android_version_code_v0_relative_v1(self):
+        '''Verify that the first v1 code is greater than the equivalent v0 code.'''
+        buildid = '20150801000000'
+        self.assertGreater(android_version_code_v1(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None),
+                           android_version_code_v0(buildid, cpu_arch='armeabi', min_sdk=9, max_sdk=None))
 
 
 if __name__ == '__main__':
     main()