author | Nick Thomas <nthomas@mozilla.com> |
Wed, 27 Jun 2018 17:42:08 +1200 | |
changeset 424044 | 347196a3c2c004364d07bd1d6a8e1a94dbcb3fb2 |
parent 424043 | b8a8cac376ca7d7447f59f9e6c3df3b45f8a10c2 |
child 424045 | 4adf72720fbba7b01ce9bf70753b4489ccd68997 |
push id | 34198 |
push user | csabou@mozilla.com |
push date | Thu, 28 Jun 2018 10:05:18 +0000 |
treeherder | mozilla-central@b429b9fb68f1 [default view] [failures only] |
perfherder | [talos] [build metrics] [platform microbench] (compared to previous push) |
reviewers | aki |
bugs | 1355482 |
milestone | 63.0a1 |
first release with | nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
|
last release without | nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
|
--- a/taskcluster/docs/balrog.rst +++ b/taskcluster/docs/balrog.rst @@ -1,13 +1,11 @@ Balrog in Release Promotion =========================== -.. _overview - Overview -------- Balrog is Mozilla's update server. It is responsible for delivering newer versions of Firefox to existing Firefox installations. If you are not already, it would be useful to be familiar with Balrog's core concepts before continuing with this doc. You can find that information on `Balrog's official documentation`_. The basic interactions that Release Promotion has with Balrog are as follows: #. Submit new release metadata to Balrog with a number of ``balrog`` tasks and the ``release-balrog-submit-toplevel`` task. #. Update test channels to point at the new release in the ``release-balrog-submit-toplevel`` task. @@ -34,14 +32,14 @@ After a release has been pushed to cdns, Schedule Shipping ----------------- When we're ready to ship a release we need to let Balrog know about it by scheduling a change to the appropriate Balrog rule. If ``release_eta`` is set it will be used as the ship date and time. If not, the release will be scheduled for shipping 5 minutes in the future. In either case, signoff will need to be done in Balrog by multiple parties before the release is actually made live. This step is done by the ``release-balrog-scheduling`` task in the ``ship`` phase. ``secondary`` tasks ------------------ +------------------- You may have noticed ``secondary`` variants of the ``release-balrog-submit-toplevel``, ``release-update-verify``, ``release-final-verify``, and ``release-balrog-scheduling`` tasks. These fulfill the same function as their primary counterparts, but for the "beta" update channel. They are only used when we build Release Candidates. .. _Balrog's official documentation: http://mozilla-balrog.readthedocs.io/en/latest/ -S _balrogscript workers: https://github.com/mozilla-releng/balrogscript +.. _balrogscript workers: https://github.com/mozilla-releng/balrogscript
--- a/taskcluster/docs/kinds.rst +++ b/taskcluster/docs/kinds.rst @@ -86,17 +86,17 @@ static-analysis --------------- Static analysis builds use the compiler to perform some detailed analysis of the source code while building. The useful output from these tasks are their build logs, and while they produce a binary, they do not upload it as an artifact. static-analysis-autotest ---------------- +------------------------ Static analysis autotest utility in order to be sure that there is no regression when upgrading utilities that impact static-analysis. toolchain --------- Toolchain builds create the compiler toolchains used to build Firefox. These @@ -198,40 +198,40 @@ of the Beetmover kind above and the repa release-beetmover-push-to-release --------------------------------- release-beetmover-push-to-release publishes promoted releases from the candidates directory to the release directory. This is part of release promotion. beetmover-source -------------------- +---------------- Beetmover-source publishes release source. This is part of release promotion. checksums-signing ----------------- Checksums-signing take as input the checksums file generated by beetmover tasks and sign it via the signing scriptworkers. Returns the same file signed and additional detached signature. release-source-checksums-signing -------------------------------- +-------------------------------- release-source-checksums-signing take as input the checksums file generated by source-related beetmover task and sign it via the signing scriptworkers. Returns the same file signed and additional detached signature. beetmover-checksums ------------------- Beetmover, takes specific artifact checksums and pushes it to a location outside of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the process determines the final location and "pretty" names it (version product name) release-beetmover-source-checksums ---------------------------------- +---------------------------------- Beetmover, takes source specific artifact checksums and pushes it to a location outside of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the process determines the final location and "pretty" names it (version product name) google-play-strings ------------------- Download strings to display on Google Play from https://l10n.mozilla-community.org/stores_l10n/. Artifact is then used by push-apk. @@ -270,29 +270,29 @@ release-binary-transparency Binary transparency creates a publicly verifiable log of binary shas for downstream release auditing. https://wiki.mozilla.org/Security/Binary_Transparency release-snap-repackage ---------------------- Generate an installer using Ubuntu's Snap format. release-snap-push ----------------------- +----------------- Pushes Snap repackage on Snap store. release-notify-push ----------------------- +------------------- Notify when a release has been pushed to CDNs. release-notify-ship ----------------------- +------------------- Notify when a release has been shipped. release-secondary-notify-ship ----------------------- +----------------------------- Notify when an RC release has been shipped to the beta channel. release-notify-promote ---------------------- Notify when a release has been promoted. release-bouncer-sub ------------------- @@ -338,17 +338,17 @@ release-beetmover-signed-langpacks ---------------------------------- Publishes signed langpacks to archive.mozilla.org release-update-verify --------------------- Verifies the contents and package of release update MARs. release-secondary-update-verify ---------------------- +------------------------------- Verifies the contents and package of release update MARs. release-update-verify-config ---------------------------- Creates configs for release-update-verify tasks release-secondary-update-verify-config -------------------------------------- @@ -358,69 +358,69 @@ release-updates-builder ----------------------- Top level Balrog blob submission & patcher/update verify config updates. release-version-bump -------------------- Bumps to the next version. release-source --------------------- +-------------- Generates source for the release release-source-signing --------------------- +---------------------- Signs source for the release release-partner-repack ---------------------- Generates customized versions of releases for partners. release-partner-repack-chunking-dummy ----------------------- +------------------------------------- Chunks the partner repacks by locale. release-partner-repack-signing ------------------------------ Internal signing of partner repacks. release-partner-repack-repackage ------------------------------- +-------------------------------- Repackaging of partner repacks. release-partner-repack-repackage-signing ------------------------------- +---------------------------------------- External signing of partner repacks. release-partner-repack-beetmover ------------------------------- +-------------------------------- Moves the partner repacks to S3 buckets. release-early-tagging --------------------- Utilises treescript to perform tagging that should happen near the start of a release. release-eme-free-repack ----------------------- +----------------------- Generates customized versions of releases for eme-free repacks. release-eme-free-repack-signing ------------------------------- +------------------------------- Internal signing of eme-free repacks release-eme-free-repack-repackage ------------------------------- +--------------------------------- Repackaging of eme-free repacks. release-eme-free-repack-repackage-signing ------------------------------- +----------------------------------------- External signing of eme-free repacks. release-eme-free-repack-beetmover ------------------------------- +--------------------------------- Moves the eme-free repacks to S3 buckets. release-eme-free-repack-beetmover-checksums ------------------------------------------- Moves the beetmover checksum for eme-free repacks to S3 buckets. repackage --------- @@ -449,34 +449,33 @@ Partials takes the complete.mar files pr updates between previous nightly releases and the new one. Requires a release_history in the parameters. See ``mach release-history`` if doing this manually. partials-signing ---------------- Partials-signing takes the partial updates produced in Partials and signs them. post-balrog-dummy --------------------- +----------------- Dummy tasks to consolidate balrog dependencies to avoid taskcluster limits on number of dependencies per task. post-beetmover-dummy -------------------- Dummy tasks to consolidate beetmover dependencies to avoid taskcluster limits on number of dependencies per task. post-beetmover-checksums-dummy ------------------------------ Dummy tasks to consolidate beetmover-checksums dependencies to avoid taskcluster limits on number of dependencies per task. post-langpack-dummy ------------------------------- +------------------- Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task. fetch ----- - Tasks that obtain something from a remote service and re-expose it as a task artifact. These tasks are used to effectively cache and re-host remote content so it is reliably and deterministically available. packages -------- Tasks used to build packages for use in docker images.
--- a/taskcluster/docs/partials.rst +++ b/taskcluster/docs/partials.rst @@ -1,27 +1,23 @@ Partial Update Generation ========================= -.. _overview - Overview -------- Windows, Mac and Linux releases have partial updates, to reduce the file size end-users have to download in order to receive new versions. These are created using a docker image, some Python, ``mbsdiff``, and the tools in ``tools/update-packaging`` The task has been called 'Funsize' for quite some time. This might make sense depending on what brands of chocolate bar are available near you. -.. _how the task works - How the Task Works ------------------ Funsize uses a docker image that's built in-tree, named funsize-update-generator. The image contains some Python to examine the task definition and determine what needs to be done, but it downloads tools like ``mar`` and ``mbsdiff`` from either locations specified in the task definition, or default mozilla-central locations. @@ -40,17 +36,17 @@ artifact. "dest_mar": "target-60.0b8.partial.mar", "locale": "ach", "from_mar": "http://archive.mozilla.org/pub/firefox/candidates/60.0b8-candidates/build1/update/linux-i686/ach/firefox-60.0b8.complete.mar", "update_number": 2, "platform": "linux32", "previousVersion": "60.0b8", "previousBuildNumber": "1", "branch": "mozilla-beta" - }, + } The 'update number' indicates how many released versions there are between 'to' and the current 'from'. For example, if we are building a partial update for the current nightly from the previous one, the update number will be 1. For the release before that, it will be 2. This lets us use generic output artifact names that we can rename in the later ``beetmover`` tasks. Inside the task, for each partial it has been told to generate, it will download, unpack and virus scan the 'from_mar' and 'to_mar', download the tools, and run ``make_incremental_update.sh`` from @@ -62,36 +58,32 @@ and this allows us to save a lot of comp For Releases ------------ Partials are made as part of the ``promote`` task group. The previous versions used to create the update are specified in ship-it by Release Management. -.. _data and metrics - Data About Partials ------------------- Some metrics are collected in Datadog_ about partial update tasks. The prefix used is ``releng.releases.partials``, so the relevant metric names will all start with that. Some dashboards in Datadog are public, some require a login. If you need access, file a bug under 'Cloud Services :: Operations: Metrics/Monitoring' Some examples of potentially useful metrics: * ``releng.releases.partials.partial_mar_size`` (tagged with branch, platform and update-number) * ``releng.releases.partials.task_duration`` - the time the task took, running partial generation concurrently. * ``releng.releases.partials.generate_partial.time`` - the time taken to make one partial update -.. _nightly partials - Nightly Partials ---------------- Since nightly releases don't appear in ship-it, the partials to create are determined in the decision task. This was controversial, and so here are the assumptions and reasons, so that when an alternative solution is discovered, we can assess it in context:
--- a/taskcluster/docs/release-promotion-action.rst +++ b/taskcluster/docs/release-promotion-action.rst @@ -2,25 +2,25 @@ Release Promotion Action ======================== The release promotion action is how Releng triggers `release promotion`_ taskgraphs. The one action covers all release promotion needs: different *flavors* allow for us to trigger the different :ref:`release promotion phases` for each product. The input schema and release promotion flavors are defined in the `release promotion action`_. +.. _snowman model: + The snowman model ----------------- The `release promotion action`_ allows us to chain multiple taskgraphs (aka graphs, aka task groups) together. Essentially, we're using `optimization`_ logic to replace task labels in the current taskgraph with task IDs from the previous taskgraph(s). -.. _snowman model: - This is the ``snowman`` model. If you request the body of the snowman and point at the base, we only create the middle section of the snowman. If you request the body of the snowman and don't point it at the base, we build the first base and body of the snowman from scratch. For example, let's generate a task ``t2`` that depends on ``t1``. Let's call our new taskgraph ``G``:: G
--- a/taskcluster/docs/release-promotion.rst +++ b/taskcluster/docs/release-promotion.rst @@ -11,17 +11,17 @@ increased the end-to-end time for a give promotion removes these anti-patterns. By running our continuous integration tests against our shippable builds, we have a higher degree of confidence at release time. By separating the build phase tasks (compilation, packaging, and related tests) from the promotion phase tasks, we can schedule each phase at their own independent cadence, as needed, and the end-to-end time for promotion is reduced significantly. -.. _release promotion phases +.. _release promotion phases: Release Promotion Phases ------------------------ Currently, we have the ``build``, ``promote``, ``push``, and ``ship`` phases. The ``build`` phase creates ``shippable builds``. These optimize for correctness over speed, and are designed to be of shipping quality, should we decide to
--- a/taskcluster/docs/signing.rst +++ b/taskcluster/docs/signing.rst @@ -1,26 +1,22 @@ Signing ======= -.. _overview - Overview -------- Our `code signing`_ happens in discrete tasks, for both performance reasons and to limit which machines have access to the signing servers and keys. In general, the binary-to-be-signed is generated in one task, and the request to sign it is in a second task. We verify the request via the `chain of trust`_, sign the binary, then upload the signed binary or original binary + detached signature as artifacts. -.. _how the task works - How the Task Works ------------------ Scriptworker_ verifies the task definition and the upstream tasks until it determines the graph comes from a trusted tree; this is `chain of trust`_ verification. Part of this verification is downloading and verifying the shas of the ``upstreamArtifacts`` in the task payload. @@ -81,17 +77,17 @@ file will also need to be signed. These `release-source-signing` and `partials-signing` sign the release source tarball and partial update MARs. We generate signed checksums at the top of the releases directories, like in `60.0`_. To generate these, we have the checksums signing kinds, including `release-generate-checksums-signing`, `checksums-signing`, and `release-source-checksums-signing` -.. _signing formats +.. _signing formats: Signing formats --------------- The known signingscript formats are listed in the fourth column of the `signing password files`_. The formats are specified in the ``upstreamArtifacts`` list-of-dicts. The task @@ -109,17 +105,17 @@ This includes the ``focus-jar`` format, set of keys for the Focus app. ``macapp`` signing accepts either a ``dmg`` or ``tar.gz``; it converts ``dmg`` files to ``tar.gz`` before submitting to the signing server. The signed binary is a ``tar.gz``. ``signcode`` signing takes individual binaries or a zipfile. We sign the individual file or internals of the zipfile, skipping any already-signed files -and a select few blocklisted files (using the `_should_sign_windows`_ function). +and a select few blocklisted files (using the `should_sign_windows`_ function). It returns a signed individual binary or zipfile with signed internals, depending on the input. This format includes ``signcode``, ``osslsigncode``, ``sha2signcode``, and ``sha2signcodestub``. ``mar`` signing signs our update files (Mozilla ARchive). ``mar_sha384`` is the same, but with a different hashing algorithm. ``widevine`` and ``widevine_blessed`` are also video-related; see the @@ -174,17 +170,17 @@ development use. Because it isn't used o Mozilla Releng is able to make breaking changes on this pool without affecting any other team. .. _60.0: https://archive.mozilla.org/pub/firefox/releases/60.0/ .. _addonscript: https://github.com/mozilla-releng/addonscript/ .. _code signing: https://en.wikipedia.org/wiki/Code_signing .. _chain of trust: https://scriptworker.readthedocs.io/en/latest/chain_of_trust.html .. _depsigning: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/depsigning -.. __should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369 +.. _should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369 .. _Encrypted Media Extensions: https://hacks.mozilla.org/2014/05/reconciling-mozillas-mission-and-w3c-eme/ .. _signing password files: https://github.com/mozilla/build-puppet/tree/feff5e12ab70f2c060b29940464e77208c7f0ef2/modules/signing_scriptworker/templates .. _signingscript: https://github.com/mozilla-releng/signingscript/ .. _signing-linux-dev: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-dev .. _signing-linux-v1: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-v1 .. _signtool: https://github.com/mozilla-releng/signtool .. _Scriptworker: https://github.com/mozilla-releng/scriptworker/ .. _widevine site: https://www.widevine.com/wv_drm.html
--- a/taskcluster/taskgraph/util/partners.py +++ b/taskcluster/taskgraph/util/partners.py @@ -49,36 +49,33 @@ MANIFEST_QUERY = """query { object(expression: "master:default.xml") { ... on Blob { text } } } } """ - -r""" -Example response: -{ - "data": { - "repository": { - "object": { - "text": "<?xml version=\"1.0\" ?>\n<manifest>\n " + - "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n " + - "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n " + - "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " + - "revision=\"master\"/>\n <project name=\"build-tools\" path=\"scripts/tools\" " + - "remote=\"mozilla\" revision=\"master\"/>\n <project name=\"mozilla-EME-free\" " + - "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " + - "revision=\"master\"/>\n</manifest>\n" - } - } - } -} -""" +# Example response: +# { +# "data": { +# "repository": { +# "object": { +# "text": "<?xml version=\"1.0\" ?>\n<manifest>\n " + +# "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n " + +# "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n " + +# "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " + +# "revision=\"master\"/>\n <project name=\"build-tools\" path=\"scripts/tools\" " + +# "remote=\"mozilla\" revision=\"master\"/>\n <project name=\"mozilla-EME-free\" " + +# "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " + +# "revision=\"master\"/>\n</manifest>\n" +# } +# } +# } +# } # Returns the contents of desktop/*/repack.cfg for a partner repository REPACK_CFG_QUERY = """query{ repository(owner:"%(owner)s", name:"%(repo)s") { object(expression: "master:desktop/"){ ... on Tree { entries { name @@ -95,51 +92,49 @@ REPACK_CFG_QUERY = """query{ } } } } } } } """ -r""" -Example response: -{ - "data": { - "repository": { - "object": { - "entries": [ - { - "name": "mozilla-EME-free", - "object": { - "entries": [ - { - "name": "distribution", - "object": {} - }, - { - "name": "repack.cfg", - "object": { - "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" + - "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" + - " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" + - "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" + - "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" + - "upload_to_candidates=true\n" - } - } - ] - } - } - ] - } - } - } -} -""" +# Example response: +# { +# "data": { +# "repository": { +# "object": { +# "entries": [ +# { +# "name": "mozilla-EME-free", +# "object": { +# "entries": [ +# { +# "name": "distribution", +# "object": {} +# }, +# { +# "name": "repack.cfg", +# "object": { +# "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" + +# "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" + +# " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" + +# "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" + +# "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" + +# "upload_to_candidates=true\n" +# } +# } +# ] +# } +# } +# ] +# } +# } +# } +# } # Map platforms in repack.cfg into their equivalents in taskcluster TC_PLATFORM_PER_FTP = { 'linux-i686': 'linux-nightly', 'linux-x86_64': 'linux64-nightly', 'mac': 'macosx64-nightly', 'win32': 'win32-nightly', 'win64': 'win64-nightly',