Bug 1767437 - Clean up browsertime documentation of vismet installation references. r=perftest-reviewers,sparky
authorKash Shampur <kshampur@mozilla.com>
Tue, 31 May 2022 14:18:37 +0000
changeset 619309 c7418ea10c53e687c447f5c46752737322fd96b1
parent 619308 13c0b769cc038747245f504105b8a7d88ced8d93
child 619310 2dbbf356398abedf4c9aaf4841803a76a40dc751
push id39777
push usersmolnar@mozilla.com
push dateTue, 31 May 2022 21:47:21 +0000
treeherdermozilla-central@1712236ad4a0 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersperftest-reviewers, sparky
bugs1767437
milestone103.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1767437 - Clean up browsertime documentation of vismet installation references. r=perftest-reviewers,sparky Differential Revision: https://phabricator.services.mozilla.com/D147443
testing/mozharness/mozharness/mozilla/testing/raptor.py
testing/perfdocs/generated/browsertime.rst
testing/raptor/raptor/perfdocs/browsertime.rst
tools/browsertime/mach_commands.py
--- a/testing/mozharness/mozharness/mozilla/testing/raptor.py
+++ b/testing/mozharness/mozharness/mozilla/testing/raptor.py
@@ -1091,19 +1091,16 @@ class Raptor(
             requirements=[os.path.join(self.raptor_path, "requirements.txt")]
         )
 
     def setup_local_ffmpeg(self):
         """Make use of the users local ffmpeg when running browsertime visual
         metrics tests.
         """
 
-        # TODO expand this logic below for Windows and Linux (Bug 1746206 and 1746208)
-        # and refactor once all the paths are finalized across platforms
-
         if "ffmpeg" in os.environ["PATH"]:
             return
 
         platform = self.platform_name()
         btime_cache = os.path.join(self.config["mozbuild_path"], "browsertime")
         if "mac" in platform:
             path_to_ffmpeg = os.path.join(
                 btime_cache,
--- a/testing/perfdocs/generated/browsertime.rst
+++ b/testing/perfdocs/generated/browsertime.rst
@@ -152,21 +152,19 @@ For example:
 
   ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"
 
 Retriggering Browsertime Visual Metrics Tasks
 ---------------------------------------------
 
 You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).
 
-When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.
+The following metrics are collected each time: ``fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange``
 
-There is only one path that still doesn't work for retriggering Browsertime tests and that happens when you use ``--rebuild X`` in a try push submission.
-
-For details on how we previously retriggered visual metrics tasks see `VisualMetrics <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/VisualMetrics>`_ (this will stay here for a few months just in case).
+Further information regarding these metrics can be viewed at `visual-metrics <https://www.sitespeed.io/documentation/sitespeed.io/metrics/#visual-metrics>`_
 
 Gecko Profiling with Browsertime
 --------------------------------
 
 To run gecko profiling using Raptor-Browsertime you can add the ``--gecko-profile`` flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:
 
 ::
 
@@ -201,61 +199,38 @@ Comparing Before/After Browsertime Video
 
 We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/mozilla/mozperftest-tools#browsertime-side-by-side-video-comparisons
 
 Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.
 
 Mach Browsertime Setup
 ----------------------
 
-Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.
+**WARNING**
+ Raptor-Browsertime (i.e. ``./mach raptor --browsertime -t <TEST>``) is currently required to be ran first in order to acquire the Node-16 binary. In general, it is also not reccomended to use ``./mach browsertime`` for testing as it will be deprecated soon.
+
+Note that if you are running Raptor-Browsertime then it will get installed automatically and also update itself. Otherwise, you can run:
 
 - ``./mach browsertime --clobber --setup --install-vismet-reqs``
 
-This will automatically check your setup, which will output something like this:
+This will automatically check your setup and install the necessary dependencies if required. If successful, the output should read as something similar to:
 
 ::
 
-    ffmpeg:   OK
-    convert:  OK
-    compare:  OK
-    Pillow:   OK
-    SSIM:     OK
+    browsertime installed successfully!
 
-- To manually check your setup, run ``./mach browsertime --check``
+    NOTE: Your local browsertime binary is at <...>/mozilla-unified/tools/browsertime/node_modules/.bin/browsertime
+
+- To manually check your setup, you can also run ``./mach browsertime --check``
 
 Known Issues
 ^^^^^^^^^^^^
 
-**If you aren't running visual metrics, then failures in** ``Pillow`` **and** ``SSIM`` **can be ignored.**
-
-`Bug 1735410: [meta] browsertime visual metrics dependencies not installing correctly <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_
-
-Currently there are issues on all platforms installing browsertime vismet dependencies. There is a fix for Linux (`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__) but not on Windows (`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__) or OSX (`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__)
-
-Linux
-"""""
-`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__ **(resolved)**
-
-If ``ffmpeg`` is listed as FAIL, try `downloading ffmpeg manually <https://ffmpeg.org/>`_ and adding it to your PATH
-
-OSX
-"""
-
-`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__ **(resolved)**
-
-**Current Status**: ``convert`` and ``compare`` fail to install. Rebuilding Firefox and running browsertime setup has not shown to resolve this issue.
-
-Windows
-"""""""
-
-`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__ **(unresolved)**
-
-If the ImageMagick URL returns a 404 during setup, please `file a bug like this <https://bugzilla.mozilla.org/show_bug.cgi?id=1735540>_` to have the URL updated.
-
-**Current Status**: ``convert``, ``compare``, and ``ffmpeg`` fail to install. Neither adding ``ffmpeg`` to the PATH, nor rebuilding Firefox have shown to resolve this issue.
+With the replacement of ImageMagick, former cross platform installation issues have been resolved. The details of this can be viewed in the meta bug tracker
+`Bug 1735410 <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_
 
 
--  For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command.
+
+- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command or a Raptor-Browsertime test
 
 - If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the :ref: `Running on Android` section)
 
 - **If you encounter any issues not mentioned here, please** `file a bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor>`_ **in the** ``Testing::Raptor`` **component.**
--- a/testing/raptor/raptor/perfdocs/browsertime.rst
+++ b/testing/raptor/raptor/perfdocs/browsertime.rst
@@ -152,21 +152,19 @@ For example:
 
   ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"
 
 Retriggering Browsertime Visual Metrics Tasks
 ---------------------------------------------
 
 You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).
 
-When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.
+The following metrics are collected each time: ``fcp, loadtime, ContentfulSpeedIndex, PerceptualSpeedIndex, SpeedIndex, FirstVisualChange, LastVisualChange``
 
-There is only one path that still doesn't work for retriggering Browsertime tests and that happens when you use ``--rebuild X`` in a try push submission.
-
-For details on how we previously retriggered visual metrics tasks see `VisualMetrics <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/VisualMetrics>`_ (this will stay here for a few months just in case).
+Further information regarding these metrics can be viewed at `visual-metrics <https://www.sitespeed.io/documentation/sitespeed.io/metrics/#visual-metrics>`_
 
 Gecko Profiling with Browsertime
 --------------------------------
 
 To run gecko profiling using Raptor-Browsertime you can add the ``--gecko-profile`` flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:
 
 ::
 
@@ -201,61 +199,38 @@ Comparing Before/After Browsertime Video
 
 We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/mozilla/mozperftest-tools#browsertime-side-by-side-video-comparisons
 
 Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.
 
 Mach Browsertime Setup
 ----------------------
 
-Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.
+**WARNING**
+ Raptor-Browsertime (i.e. ``./mach raptor --browsertime -t <TEST>``) is currently required to be ran first in order to acquire the Node-16 binary. In general, it is also not reccomended to use ``./mach browsertime`` for testing as it will be deprecated soon.
+
+Note that if you are running Raptor-Browsertime then it will get installed automatically and also update itself. Otherwise, you can run:
 
 - ``./mach browsertime --clobber --setup --install-vismet-reqs``
 
-This will automatically check your setup, which will output something like this:
+This will automatically check your setup and install the necessary dependencies if required. If successful, the output should read as something similar to:
 
 ::
 
-    ffmpeg:   OK
-    convert:  OK
-    compare:  OK
-    Pillow:   OK
-    SSIM:     OK
+    browsertime installed successfully!
 
-- To manually check your setup, run ``./mach browsertime --check``
+    NOTE: Your local browsertime binary is at <...>/mozilla-unified/tools/browsertime/node_modules/.bin/browsertime
+
+- To manually check your setup, you can also run ``./mach browsertime --check``
 
 Known Issues
 ^^^^^^^^^^^^
 
-**If you aren't running visual metrics, then failures in** ``Pillow`` **and** ``SSIM`` **can be ignored.**
-
-`Bug 1735410: [meta] browsertime visual metrics dependencies not installing correctly <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_
-
-Currently there are issues on all platforms installing browsertime vismet dependencies. There is a fix for Linux (`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__) but not on Windows (`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__) or OSX (`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__)
-
-Linux
-"""""
-`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__ **(resolved)**
-
-If ``ffmpeg`` is listed as FAIL, try `downloading ffmpeg manually <https://ffmpeg.org/>`_ and adding it to your PATH
-
-OSX
-"""
-
-`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__ **(resolved)**
-
-**Current Status**: ``convert`` and ``compare`` fail to install. Rebuilding Firefox and running browsertime setup has not shown to resolve this issue.
-
-Windows
-"""""""
-
-`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__ **(unresolved)**
-
-If the ImageMagick URL returns a 404 during setup, please `file a bug like this <https://bugzilla.mozilla.org/show_bug.cgi?id=1735540>_` to have the URL updated.
-
-**Current Status**: ``convert``, ``compare``, and ``ffmpeg`` fail to install. Neither adding ``ffmpeg`` to the PATH, nor rebuilding Firefox have shown to resolve this issue.
+With the replacement of ImageMagick, former cross platform installation issues have been resolved. The details of this can be viewed in the meta bug tracker
+`Bug 1735410 <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_
 
 
--  For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command.
+
+- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command or a Raptor-Browsertime test
 
 - If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the :ref: `Running on Android` section)
 
 - **If you encounter any issues not mentioned here, please** `file a bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor>`_ **in the** ``Testing::Raptor`` **component.**
--- a/tools/browsertime/mach_commands.py
+++ b/tools/browsertime/mach_commands.py
@@ -377,20 +377,17 @@ def node(command_context, args):
         ensure_exit_code=False,  # Don't throw on non-zero exit code.
         cwd=mozpath.join(command_context.topsrcdir),
     )
 
 
 def append_env(command_context, append_path=True):
     fetches = host_fetches[host_platform()]
 
-    # Ensure that bare `ffmpeg` and ImageMagick commands
-    # {`convert`,`compare`,`mogrify`} are found.  The `visualmetrics.py`
-    # script doesn't take these as configuration, so we do this (for now).
-    # We should update the script itself to accept this configuration.
+    # Ensure that `ffmpeg` is found and added to the environment
     path = os.environ.get("PATH", "").split(os.pathsep) if append_path else []
     path_to_ffmpeg = mozpath.join(
         state_path(command_context), fetches["ffmpeg"]["path"]
     )
 
     path.insert(
         0,
         path_to_ffmpeg
@@ -400,30 +397,16 @@ def append_env(command_context, append_p
 
     # Ensure that bare `node` and `npm` in scripts, including post-install
     # scripts, finds the binary we're invoking with.  Without this, it's
     # easy for compiled extensions to get mismatched versions of the Node.js
     # extension API.
     node_dir = os.path.dirname(node_path(command_context))
     path = [node_dir] + path
 
-    # On macOs, we can't install our own ImageMagick because the
-    # System Integrity Protection (SIP) won't let us set DYLD_LIBRARY_PATH
-    # unless we deactivate SIP with "csrutil disable".
-    # So we're asking the user to install it.
-    #
-    # if ImageMagick was installed via brew, we want to make sure we
-    # include the PATH
-    if host_platform() == "darwin":
-        for p in os.environ["PATH"].split(os.pathsep):
-            p = p.strip()
-            if not p or p in path:
-                continue
-            path.append(p)
-
     append_env = {
         "PATH": os.pathsep.join(path),
         # Bug 1560193: The JS library browsertime uses to execute commands
         # (execa) will muck up the PATH variable and put the directory that
         # node is in first in path. If this is globally-installed node,
         # that means `/usr/bin` will be inserted first which means that we
         # will get `/usr/bin/python` for `python`.
         #
@@ -450,18 +433,17 @@ def _need_install(command_context, packa
 
 
 def activate_browsertime_virtualenv(command_context, *args, **kwargs):
     r"""Activates virtualenv.
 
     This function will also install Pillow and pyssim if needed.
     It will raise an error in case the install failed.
     """
-    # TODO: (Bug 1758990) Have packages be pulled from the Mozilla pypi mirror
-    # for consistency in user environments
+    # TODO: Remove `./mach browsertime` completely, as a follow up to Bug 1758990
     MachCommandBase.activate_virtualenv(command_context, *args, **kwargs)
 
     # installing Python deps on the fly
     for dep in (
         "Pillow==%s" % PILLOW_VERSION,
         "pyssim==%s" % PYSSIM_VERSION,
         "scipy==%s" % SCIPY_VERSION,
         "numpy==%s" % NUMPY_VERSION,
@@ -481,19 +463,17 @@ def activate_browsertime_virtualenv(comm
 
 def check(command_context):
     r"""Run `visualmetrics.py --check`."""
     command_context.activate_virtualenv()
 
     args = ["--check"]
     status = command_context.run_process(
         [command_context.virtualenv_manager.python_path, visualmetrics_path()] + args,
-        # For --check, don't allow user's path to interfere with
-        # path testing except on Linux, where ImageMagick needs to
-        # be installed manually.
+        # For --check, don't allow user's path to interfere with path testing except on Linux
         append_env=append_env(
             command_context, append_path=host_platform().startswith("linux")
         ),
         pass_thru=True,
         ensure_exit_code=False,  # Don't throw on non-zero exit code.
         cwd=mozpath.join(command_context.topsrcdir),
     )