Bug 1355482 - documentation for partner repacks, r=aki
authorNick Thomas <nthomas@mozilla.com>
Sat, 16 Jun 2018 17:09:45 -0700
changeset 423999 b8ebfd75f50124e4f957e1171a5ec79bfee210fc
parent 423998 afd967c185fe7801bd6786e149ae77d2a28ce8f2
child 424000 5b2c8eebbb03e63de0788988275f2f4a31825980
push id34197
push usercsabou@mozilla.com
push dateThu, 28 Jun 2018 09:44:02 +0000
treeherdermozilla-central@db455160668d [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersaki
bugs1355482
milestone63.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1355482 - documentation for partner repacks, r=aki MozReview-Commit-ID: 5uRWpw2WJyE
taskcluster/docs/partner-repacks.rst
taskcluster/docs/release-promotion.rst
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/partner-repacks.rst
@@ -0,0 +1,252 @@
+Partner repacks
+===============
+
+We create slightly-modified Firefox releases for some extra audiences
+
+* EME-free builds, which disable DRM plugins by default
+* Funnelcake builds, which are used for Mozilla experiments
+* partner builds, which customize Firefox for external partners
+
+We use the phrase "partner repacks" to refer to all these builds because they
+use the same process of repacking regular Firefox releases with additional files.
+The specific differences depend on the type of build.
+
+We produce partner repacks for some beta builds, and for release builds, as part of the release
+automation. We don't produce any files to update these builds as they are handled automatically
+(see updates_).
+
+
+Parameters & Scheduling
+-----------------------
+
+Partner repacks have a number of parameters which control how they work:
+
+* ``release_enable_emefree``
+* ``release_enable_partners``
+* ``release_partner_config``
+* ``release_partner_build_number``
+* ``release_partners``
+
+We split the repacks into two 'paths', EME-free and everything else, to retain some
+flexibility over enabling/disabling them separately. This costs us some duplication of the kinds
+in the repacking stack. The two enable parameters are booleans to turn these two paths
+on/off. We set them in release-runner3's `is_partner_enabled() <https://dxr.mozilla
+.org/build-central/search?q=function%3Ais_partner_enabled&redirect=true>`_ when starting a
+release. They're both true for Firefox betas >= b8 and releases, but otherwise disabled.
+
+``release_partner_config`` is a dictionary of configuration data which drives the task generation
+logic. It's usually looked up during the release promotion action task, using the Github
+GraphQL API in the `get_partner_config_by_url()
+<python/taskgraph.util.html#taskgraph.util.partners.get_partner_config_by_url>`_ function, with the
+url defined in `taskcluster/ci/config.yml <https://dxr.mozilla
+.org/mozilla-release/search?q=regexp%3A^partner+path%3Aconfig.yml&redirect=true>`_.
+
+``release_partner_build_number`` is an integer used to create unique upload paths in the firefox
+candidates directory, while ``release_partners`` is a list of partners that should be
+repacked (i.e. a subset of the whole config). Both are intended for use when respinning a few partners after
+the regular Firefox has shipped. More information on that can be found in the `release-warrior docs
+<https://github.com/mozilla-releng/releasewarrior-2
+.0/blob/master/docs/misc-operations/off-cycle-partner-repacks -and-funnelcake.md>`_.
+
+Most of the machine time for generating partner repacks takes place in the `promote` phase of the
+automation, or `promote_rc` in the case of X.0 release candidates. The EME-free builds are copied into the
+Firefox releases directory in the `push` phase, along with the regular bits.
+
+
+Configuration
+-------------
+
+We need some configuration to know *what* to repack, and *how* to do that. The *what* is defined by
+default.xml manifests, as used with the `repo <https://gerrit.googlesource.com/git-repo>`_ tool
+for git. The `default.xml for EME-free <https://github
+.com/mozilla-partners/mozilla-EME-free-manifest/blob/master/default.xml>`_ illustrates this::
+
+    <?xml version="1.0" ?>
+    <manifest>
+      <remote fetch="git@github.com:mozilla-partners/" name="mozilla-partners"/>
+      <remote fetch="git@github.com:mozilla/" name="mozilla"/>
+
+      <project name="repack-scripts" path="scripts" remote="mozilla-partners" revision="master"/>
+      <project name="build-tools" path="scripts/tools" remote="mozilla" revision="master"/>
+      <project name="mozilla-EME-free" path="partners/mozilla-EME-free" remote="mozilla-partners" revision="master"/>
+    </manifest>
+
+The repack-scripts and build-tools repos are found in all manifests, and then there is a list of
+partner repositories which contain the *how* configuration. Some of these repos are not publicly
+visible.
+
+A partner repository may contain multiple configurations inside the ``desktop`` directory. Each
+subdirectory must contain a ``repack.cfg`` and a ``distribution`` directory, the latter
+containing the customizations needed. Here's `EME-free's repack.cfg <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/repack.cfg>`_::
+
+    aus="mozilla-EMEfree"
+    dist_id="mozilla-EMEfree"
+    dist_version="1.0"
+    linux-i686=false
+    linux-x86_64=false
+    locales="ach af an ar"  # truncated for display here
+    mac=true
+    win32=true
+    win64=true
+    output_dir="%(platform)s-EME-free/%(locale)s"
+
+    # Upload params
+    upload_to_candidates=true
+
+Note the list of locales and boolean toggles for enabling platforms. The ``output_dir`` and
+``upload_to_candidates`` parameters are only present for repacks which are uploaded into the
+`candidates directory <https://archive.mozilla.org/pub/firefox/candidates/>`_.
+
+All customizations will be placed in the ``distribution`` directory at the root of the Firefox
+install directory, or in the case of OS X in ``Firefox.app/Contents/Resources/distribution/``. A
+``distribution.ini`` file is the minimal requirement, here's an example from `EME-free
+<https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/distribution
+/distribution.ini>`_::
+
+    # Partner Distribution Configuration File
+    # Author: Mozilla
+    # Date: 2015-03-27
+
+    [Global]
+    id=mozilla-EMEfree
+    version=1.0
+    about=Mozilla Firefox EME-free
+
+    [Preferences]
+    media.eme.enabled=false
+    app.partner.mozilla-EMEfree="mozilla-EMEfree"
+
+Extensions and other customizations might also be included in repacks.
+
+
+Repacking process
+-----------------
+
+The stack of tasks to create partner repacks is broadly similar to localised nightlies and
+regular releases. The basic form is
+
+* partner repack - insert the customisations into the the regular builds
+* signing - sign the internals which will become the installer (Mac only)
+* repackage - create the "installer" (Mac and Windows)
+* chunking dummy - a linux only bridge to ...
+* repackage signing - sign the "installers" (mainly Windows)
+* beetmover - move the files to a partner-specific destination
+* beetmover checksums - possibly beetmove the checksums from previous step
+
+Some key divergences are:
+
+* all intermediate artifacts are uploaded with a ``releng/partner`` prefix
+* we don't insert any binaries on Windows so no need for internal signing
+* there's no need to create any complete mar files at the repackage step
+* we support both public and private destinations in beetmover
+* we only need beetmover checksums for EME-free builds
+
+
+Partner repack
+^^^^^^^^^^^^^^
+
+* kinds: ``release-partner-repack`` ``release-eme-free-repack``
+* platforms: Typically all (but depends on what's enabled by partner configuration)
+* upstreams: ``build-signing`` ``l10n-signing``
+
+There is one task per platform in this step, calling out to `scripts/desktop_partner_repacks.py
+<https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/scripts
+/desktop_partner_repacks.py>`_ in mozharness to prepare an environment and then perform the repacks.
+
+It takes as input the build-signing and l10n-signing artifacts, which are all zip/tar.gz/tar.bz2
+archives, simplifying the repack process by avoiding dmg and exe. Windows produces ``target.zip``
+& ``setup.exe``, Mac is ``target.tar.gz``, Linux is the final product ``target.tar.bz2``
+(beetmover handles pretty naming as usual).
+
+Signing
+^^^^^^^
+
+* kinds: ``release-partner-repack-signing`` ``release-eme-free-repack-signing``
+* platforms: Mac
+* upstreams: ``release-partner-repack`` ``release-eme-free-repack``
+
+We chunk the single partner repack task out to a signing task per artifact at this point. For
+example, EME-free will become ~95 tasks, one for each locale. We collect the target.tar.gz from the
+upstream, and return a signed target.tar.gz. We use a ``target.dmg`` artifact for
+nightlies/regular releases, but this is converted to ``target.tar.gz`` by the signing
+scriptworker before sending it to the signing server, so partners are equivalent.
+
+Repackage
+^^^^^^^^^
+
+* kinds: ``release-partner-repack-repackage`` ``release-eme-free-repack-repackage``
+* platforms: Mac & Windows
+* upstreams:
+
+    * Mac: ``release-partner-signing`` ``release-eme-free-signing``
+    * Windows: ``release-partner-repack`` ``release-eme-free-repack``
+
+Mac has a repackage job for each of the signing tasks. Windows repackages are chunked here to
+the same granularity as mac. Takes ``target.zip`` & ``setup.exe`` to produce ``target.exe`` on
+Windows, and ``target.tar.gz`` to produce ``target.dmg`` on Mac. There's no need to produce any
+complete.mar files here like regular release bits do because we can reuse those.
+
+Chunking dummy
+^^^^^^^^^^^^^^
+
+* kinds: ``release-partner-repack-chunking-dummy``
+* platforms: Linux
+* upstreams: ``release-partner-repack``
+
+We're need Linux chunked at the next step so this dummy takes care of that for the relatively simple path
+Linux follows. One task per sub config+locale combination, the same as Windows and Mac. This doesn't need to
+exist for EME-free because we don't need to create Linux builds there.
+
+Repackage Signing
+^^^^^^^^^^^^^^^^^
+
+* kinds: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing``
+* platforms: All
+* upstreams:
+
+   * Mac & Windows: ``release-partner-repackage`` ``release-eme-free-repackage``
+   * Linux: ``release-partner-repack-chunking-dummy``
+
+This step GPG signs all platforms, and sha2signcode signs the Windows installer.
+
+Beetmover
+^^^^^^^^^
+
+* kinds: ``release-partner-repack-beetmover`` ``release-eme-free-repack-beetmover``
+* platforms: All
+* upstreams: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing``
+
+Moves and renames the artifacts to their public location in the `candidates directory
+<https://archive.mozilla.org/pub/firefox/candidates/>`_, or a private S3 bucket. Each task will
+have the ``project:releng:beetmover:action:push-to-partner`` scope, with public uploads having
+``project:releng:beetmover:bucket:release`` and private uploads using
+``project:releng:beetmover:bucket:partner``. The ``upload_to_candidates`` key in the partner config
+controls the second scope. There's a separate partner code path in `beetmoverscript <https://github
+.com/mozilla-releng/beetmoverscript>`_.
+
+Beetmover checksums
+^^^^^^^^^^^^^^^^^^^
+
+* kinds: ``release-eme-free-repack-beetmover-checksums``
+* platforms: Mac & Windows
+* upstreams: ``release-eme-free-repack-repackage-beetmover``
+
+The EME-free builds should be present in our SHA256SUMS file and friends (`e.g. <https://archive
+.mozilla.org/pub/firefox/releases/61.0/SHA256SUMS>`_) so we beetmove the target.checksums from
+the beetmover tasks into the candidates directory. They get picked up by the
+``release-generate-checksums`` kind.
+
+.. _updates:
+
+Updates
+-------
+
+It's very rare to need to update a partner repack differently from the original
+release build but we retain that capability. A partner build with distribution name ``foo``,
+based on a release Firefox build, will query for an update on the ``release-cck-foo`` channel. If
+the update server `Balrog <http://mozilla-balrog.readthedocs.io/en/latest/>`_ finds no rule for
+that channel it will fallback to the ``release`` channel. The update files for the regular releases do not
+modify the ``distribution/`` directory, so the customizations are not modified.
+
+`Bug 1430254 <https://bugzilla.mozilla.org/show_bug.cgi?id=1430254>`_ is an example of an exception to this
+logic.
\ No newline at end of file
--- a/taskcluster/docs/release-promotion.rst
+++ b/taskcluster/docs/release-promotion.rst
@@ -45,8 +45,9 @@ In-depth relpro guide
 
 .. toctree::
 
     release-promotion-action
     balrog
     partials
     pushapk
     signing
+    partner-repacks