Bug 1539649 [wpt PR 16083] - Upload MANIFEST.json.bz2 and MANIFEST.json.zst, a=testonly
☠☠ backed out by 41b4f50eae55 ☠ ☠
authorSam Sneddon <me@gsnedders.com>
Thu, 18 Apr 2019 11:56:01 +0000
changeset 470607 7d835a79647ce43f31f612eaea0ca2865be5fc1c
parent 470606 d56536bd8de95046a484a32c2ea521d18bfb0989
child 470608 4e31380b1803ce6811ce8ddcaeb2392e98e05ef8
push id112872
push userjames@hoppipolla.co.uk
push dateTue, 23 Apr 2019 14:15:53 +0000
treeherdermozilla-inbound@fc657d55eb97 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1539649, 16083, 16081
milestone68.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1539649 [wpt PR 16083] - Upload MANIFEST.json.bz2 and MANIFEST.json.zst, a=testonly Automatic update from web-platform-tests Upload MANIFEST.json.bz2 and MANIFEST.json.zst (#16083) The first half of #16081. zstandard has the best compression ratio and fast decoding. bzip2 has good compression and very slow decoding. gzip has poor compression and fast decoding. Overall zstandard is alway the best to download + decompress, but relies on a third party Python package. gzip is always the worst unless network is very fast and CPU very slow. So in the future we will move to zstandard where possible and bzip2 where not possible. -- wpt-commits: d5921ca1198a8f3ddbecd8392c1dada9c4aa8fbc wpt-pr: 16083
testing/web-platform/tests/tools/ci/action_manifest_build.sh
testing/web-platform/tests/tools/ci/tag_master.py
testing/web-platform/tests/tools/docker/github/Dockerfile
--- a/testing/web-platform/tests/tools/ci/action_manifest_build.sh
+++ b/testing/web-platform/tests/tools/ci/action_manifest_build.sh
@@ -1,9 +1,11 @@
 #!/bin/bash
 set -ex
 
 mkdir -p ~/meta
 
 WPT_MANIFEST_FILE=~/meta/MANIFEST.json
 
 ./wpt manifest -p $WPT_MANIFEST_FILE
-gzip -f --best $WPT_MANIFEST_FILE
+gzip -k -f --best $WPT_MANIFEST_FILE
+bzip2 -k -f --best $WPT_MANIFEST_FILE
+zstd -k -f --ultra -22 $WPT_MANIFEST_FILE
--- a/testing/web-platform/tests/tools/ci/tag_master.py
+++ b/testing/web-platform/tests/tools/ci/tag_master.py
@@ -104,31 +104,35 @@ def create_release(owner, repo, sha, tag
                    "body": body}
     create_data = request(create_url, "Release creation", json_data=create_data)
     if not create_data:
         return False
 
     # Upload URL contains '{?name,label}' at the end which we want to remove
     upload_url = create_data["upload_url"].split("{", 1)[0]
 
-    upload_filename = "MANIFEST-%s.json.gz" % sha
-    params = {"name": upload_filename,
-              "label": "MANIFEST.json.gz"}
+    success = True
 
-    with open(os.path.expanduser("~/meta/MANIFEST.json.gz"), "rb") as f:
-        upload_data = f.read()
+    upload_exts = [".gz", ".bz2", ".zst"]
+    for upload_ext in upload_exts:
+        upload_filename = "MANIFEST-%s.json%s" % (sha, upload_ext)
+        params = {"name": upload_filename,
+                  "label": "MANIFEST.json%s" % upload_ext}
 
-    logger.info("Uploading %s bytes" % len(upload_data))
+        with open(os.path.expanduser("~/meta/MANIFEST.json%s" % upload_ext), "rb") as f:
+            upload_data = f.read()
+
+        logger.info("Uploading %s bytes" % len(upload_data))
 
-    upload_resp = request(upload_url, "Manifest upload", data=upload_data, params=params,
-                          headers={'Content-Type': 'application/octet-stream'})
-    if not upload_resp:
-        return False
+        upload_resp = request(upload_url, "Manifest upload", data=upload_data, params=params,
+                              headers={'Content-Type': 'application/octet-stream'})
+        if not upload_resp:
+            success = False
 
-    return True
+    return success
 
 
 def should_run_action():
     with open(os.environ["GITHUB_EVENT_PATH"]) as f:
         event = json.load(f)
         logger.info(json.dumps(event, indent=2))
 
     if "pull_request" in event:
--- a/testing/web-platform/tests/tools/docker/github/Dockerfile
+++ b/testing/web-platform/tests/tools/docker/github/Dockerfile
@@ -1,21 +1,23 @@
 FROM ubuntu:18.04
 
 # No interactive frontend during docker build
 ENV DEBIAN_FRONTEND=noninteractive \
     DEBCONF_NONINTERACTIVE_SEEN=true
 
 RUN apt-get -qqy update \
   && apt-get -qqy install \
+    bzip2 \
     git \
     locales \
     python \
     python-pip \
-    tzdata
+    tzdata \
+    zstd
 
 RUN pip install --upgrade pip
 RUN pip install virtualenv
 RUN pip install requests
 
 ENV TZ "UTC"
 RUN echo "${TZ}" > /etc/timezone \
   && dpkg-reconfigure --frontend noninteractive tzdata