Bug 1533671 - Tweak some options in mozproxy r=Bebe
authorTarek Ziadé <tarek@mozilla.com>
Fri, 08 Mar 2019 13:23:28 +0000
changeset 521276 62b4aaa7ed95541f31489dc4567af4b1a1879694
parent 521275 f0eaf5676456b5bc7c8d8e642441cbb9b8d9788f
child 521277 76ba842e38f5a0c47797370e5aee393ad3c05b1c
push id10862
push userffxbld-merge
push dateMon, 11 Mar 2019 13:01:11 +0000
treeherdermozilla-beta@a2e7f5c935da [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersBebe
bugs1533671
milestone67.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1533671 - Tweak some options in mozproxy r=Bebe - adds playback_artifacts - make playback_pageset_manifest optional - adds the MOZPROXY_DIR environment variable Differential Revision: https://phabricator.services.mozilla.com/D22663
testing/mozbase/docs/mozproxy.rst
testing/mozbase/mozproxy/mozproxy/backends/mitm.py
--- a/testing/mozbase/docs/mozproxy.rst
+++ b/testing/mozbase/docs/mozproxy.rst
@@ -24,24 +24,26 @@ Mozproxy provide a function that returns
      pb.stop()
 
 **config** is a dict with the following options:
 
 - **playback_tool**: name of the backend. can be "mitmproxy", "mitmproxy-android"
 - **playback_recordings**: list of recording files
 - **playback_binary_manifest**: tooltool manifests for the proxy backend binary
 - **playback_pageset_manifest**: tooltool manifest for the pagesets archive
+- **playback_artifacts**: links to downloadable artifacts
 - **binary**: path of the browser binary
 - **obj_path**: build dir
 - **platform**: platform name (provided by mozinfo.os)
 - **run_local**: if True, the test is running locally.
 - **custom_script**: name of the mitm custom script (-s option)
 - **app**: tested app. Can be "firefox",  "geckoview", "refbrow", "fenix" or  "firefox"
 - **host**: hostname for the policies.json file
 - **local_profile_dir**: profile dir
 
 
 Supported environment variables:
 
+- **MOZPROXY_DIR**: directory used by mozproxy for all data files, set by mozproxy
 - **MOZ_UPLOAD_DIR**: upload directory path
 - **GECKO_HEAD_REPOSITORY**: used to find the certutils binary path from the CI
 - **GECKO_HEAD_REV**: used to find the certutils binary path frmo the CI
 - **HOSTUTILS_MANIFEST_PATH**: used to find the certutils binary path from the CI
--- a/testing/mozbase/mozproxy/mozproxy/backends/mitm.py
+++ b/testing/mozbase/mozproxy/mozproxy/backends/mitm.py
@@ -91,17 +91,19 @@ class Mitmproxy(Playback):
             )
 
         self.mozproxy_dir = os.path.join(self.mozproxy_dir, "testing", "mozproxy")
         self.recordings_path = self.mozproxy_dir
         LOG.info(
             "mozproxy_dir used for mitmproxy downloads and exe files: %s"
             % self.mozproxy_dir
         )
-
+        # setting up the MOZPROXY_DIR env variable so custom scripts know
+        # where to get the data
+        os.environ["MOZPROXY_DIR"] = self.mozproxy_dir
         # go ahead and download and setup mitmproxy
         self.download()
         # mitmproxy must be started before setup, so that the CA cert is available
         self.start()
 
         # In case the setup fails, we want to stop the process before raising.
         try:
             self.setup()
@@ -116,23 +118,34 @@ class Mitmproxy(Playback):
 
         LOG.info("downloading mitmproxy binary")
         _manifest = os.path.join(here, self.config["playback_binary_manifest"])
         transformed_manifest = transform_platform(_manifest, self.config["platform"])
         tooltool_download(
             transformed_manifest, self.config["run_local"], self.mozproxy_dir
         )
 
-        # we use one pageset for all platforms
-        LOG.info("downloading mitmproxy pageset")
-        _manifest = self.config["playback_pageset_manifest"]
-        transformed_manifest = transform_platform(_manifest, self.config["platform"])
-        tooltool_download(
-            transformed_manifest, self.config["run_local"], self.mozproxy_dir
-        )
+        if "playback_pageset_manifest" in self.config:
+            # we use one pageset for all platforms
+            LOG.info("downloading mitmproxy pageset")
+            _manifest = self.config["playback_pageset_manifest"]
+            transformed_manifest = transform_platform(_manifest, self.config["platform"])
+            tooltool_download(
+                transformed_manifest, self.config["run_local"], self.mozproxy_dir
+            )
+
+        if "playback_artifacts" in self.config:
+            artifacts = self.config["playback_artifacts"].split(",")
+            for artifact in artifacts:
+                artifact = artifact.strip()
+                if not artifact:
+                    continue
+                artifact_name = artifact.split("/")[-1]
+                dest = os.path.join(self.mozproxy_dir, artifact_name)
+                download_file_from_url(artifact, dest, extract=True)
 
     def start(self):
         """Start playing back the mitmproxy recording."""
 
         self.mitmdump_path = os.path.join(self.mozproxy_dir, "mitmdump")
 
         recordings_list = self.recordings.split()
         self.mitmproxy_proc = self.start_mitmproxy_playback(