Bug 1709281 [wpt PR 28806] - Remove documentation for wptrunner as a standalone project, a=testonly
authorPhilip Jägenstedt <philip@foolip.org>
Fri, 07 May 2021 20:00:27 +0000
changeset 579088 db044538ca897bc7b0327aed8890e2a497d4c8ad
parent 579087 666e253e9ef0c9d9e7d4796e638a387dcb684368
child 579089 c60432708400cc885b36b79317de14214be2bd12
push id38446
push usernerli@mozilla.com
push dateSat, 08 May 2021 09:56:13 +0000
treeherdermozilla-central@f9bdd1b929f2 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1709281, 28806
milestone90.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1709281 [wpt PR 28806] - Remove documentation for wptrunner as a standalone project, a=testonly Automatic update from web-platform-tests Remove documentation for wptrunner as a standalone project (#28806) wptrunner 1.14 was released in 2015, and while it can be installed in a virtualenv, these instructions should not be followed by anyone these days. Integrate some of it into from-local-system.md instead. -- wpt-commits: 823819217a9913aa686048dcc01c53913f028f6a wpt-pr: 28806
testing/web-platform/tests/docs/running-tests/from-local-system.md
testing/web-platform/tests/tools/wptrunner/README.rst
testing/web-platform/tests/tools/wptrunner/docs/usage.rst
--- a/testing/web-platform/tests/docs/running-tests/from-local-system.md
+++ b/testing/web-platform/tests/docs/running-tests/from-local-system.md
@@ -147,34 +147,63 @@ Or to run in a specified copy of Firefox
     ./wpt run --binary ~/local/firefox/firefox firefox dom/historical.html
 
 For details on the supported products and a large number of other options for
 customising the test run:
 
     ./wpt run --help
 
 [A complete listing of the command-line arguments is available
-here](command-line-arguments.md).
+here](command-line-arguments.html#run).
 
 ```eval_rst
 .. toctree::
    :hidden:
 
    command-line-arguments
 ```
 
-Additional browser-specific documentation:
+### Browser-specific instructions
 
 ```eval_rst
 .. toctree::
 
   chrome
   chrome_android
   android_webview
   safari
   webkitgtk_minibrowser
 ```
 
+### Running in parallel
+
+To speed up the testing process, use the `--processes` option to run multiple
+browser instances in parallel. For example, to run the tests in dom/ with six
+Firefox instances in parallel:
+
+    ./wpt run --processes=6 firefox dom/
+
+But note that behaviour in this mode is necessarily less deterministic than with
+a single process (the default), so there may be more noise in the test results.
+
+### Output formats
+
+By default, `./wpt run` outputs test results and a summary in a human readable
+format. For debugging, `--log-mach` can give more verbose output. For example:
+
+    ./wpt run --log-mach=- --log-mach-level=info firefox dom/
+
+A machine readable JSON report can be produced using `--log-wptreport`. This
+together with `--log-wptscreenshot` is what is used to produce results for
+[wpt.fyi](https://wpt.fyi). For example:
+
+    ./wpt run --log-wptreport=report.json --log-wptscreenshot=screenshots.txt firefox css/css-grid/
+
+(See [wpt.fyi documentation](https://github.com/web-platform-tests/wpt.fyi/blob/main/api/README.md#results-creation)
+for how results are uploaded.)
+
+### Expectation data
+
 For use in continuous integration systems, and other scenarios where regression
 tracking is required, the command-line interface supports storing and loading
 the expected result of each test in a test run. See [Expectations
 Data](../../tools/wptrunner/docs/expectation) for more information on creating
 and maintaining these files.
--- a/testing/web-platform/tests/tools/wptrunner/README.rst
+++ b/testing/web-platform/tests/tools/wptrunner/README.rst
@@ -1,14 +1,13 @@
 wptrunner: A web-platform-tests harness
 =======================================
 
 wptrunner is a harness for running the W3C `web-platform-tests testsuite`_.
 
 .. toctree::
    :maxdepth: 2
 
-   docs/usage
    docs/expectation
    docs/design
    docs/internals
 
 .. _`web-platform-tests testsuite`: https://github.com/web-platform-tests/wpt
deleted file mode 100644
--- a/testing/web-platform/tests/tools/wptrunner/docs/usage.rst
+++ /dev/null
@@ -1,242 +0,0 @@
-Getting Started
-===============
-
-Installing wptrunner
---------------------
-
-The easiest way to install wptrunner is into a virtualenv, using pip::
-
-  virtualenv wptrunner
-  cd wptrunner
-  source bin/activate
-  pip install wptrunner
-
-This will install the base dependencies for wptrunner, but not any
-extra dependencies required to test against specific browsers. In
-order to do this you must use use the extra requirements files in
-``$VIRTUAL_ENV/requirements/requirements_browser.txt``. For example,
-in order to test against Firefox you would have to run::
-
-  pip install -r requirements/requirements_firefox.txt
-
-If you intend to work on the code, the ``-e`` option to pip should be
-used in combination with a source checkout i.e. inside a virtual
-environment created as above::
-
-  git clone https://github.com/w3c/wptrunner.git
-  cd wptrunner
-  pip install -e ./
-
-In addition to the dependencies installed by pip, wptrunner requires
-a copy of the web-platform-tests repository. This can be located
-anywhere on the filesystem, but the easiest option is to put it
-under the same parent directory as the wptrunner checkout::
-
-  git clone https://github.com/web-platform-tests/wpt.git
-
-It is also necessary to generate a web-platform-tests ``MANIFEST.json``
-file. It's recommended to also put that under the same parent directory as
-the wptrunner checkout, in a directory named ``meta``::
-
-  mkdir meta
-  cd web-platform-tests
-  python manifest --path ../meta/MANIFEST.json
-
-The ``MANIFEST.json`` file needs to be regenerated each time the
-web-platform-tests checkout is updated. To aid with the update process
-there is a tool called ``wptupdate``, which is described in
-:ref:`wptupdate-label`.
-
-Running the Tests
------------------
-
-A test run is started using the ``wptrunner`` command.  The command
-takes multiple options, of which the following are most significant:
-
-``--product`` (defaults to `firefox`)
-  The product to test against: `chrome`, `firefox`, or `servo`.
-
-``--binary`` (required if product is `firefox` or `servo`)
-  The path to a binary file for the product (browser) to test against.
-
-``--webdriver-binary`` (required if product is `chrome`)
-  The path to a `*driver` binary; e.g., a `chromedriver` binary.
-
-``--certutil-binary`` (required if product is `firefox` [#]_)
-  The path to a `certutil` binary (for tests that must be run over https).
-
-``--metadata`` (required only when not `using default paths`_)
-  The path to a directory containing test metadata. [#]_
-
-``--tests`` (required only when not `using default paths`_)
-  The path to a directory containing a web-platform-tests checkout.
-
-``--prefs-root`` (required only when testing a Firefox binary)
-  The path to a directory containing Firefox test-harness preferences. [#]_
-
-``--config`` (should default to `wptrunner.default.ini`)
-  The path to the config (ini) file.
-
-.. [#] The ``--certutil-binary`` option is required when the product is
-   ``firefox`` unless ``--ssl-type=none`` is specified.
-
-.. [#] The ``--metadata`` path is to a directory that contains:
-
-  * a ``MANIFEST.json`` file (the web-platform-tests documentation has
-    instructions on generating this file)
-  * (optionally) any expectation files (see :ref:`wptupdate-label`)
-
-.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
-
-There are also a variety of other command-line options available; use
-``--help`` to list them.
-
-The following examples show how to start wptrunner with various options.
-
-------------------
-Starting wptrunner
-------------------
-
-The examples below assume the following directory layout,
-though no specific folder structure is required::
-
-  ~/testtwf/wptrunner          # wptrunner checkout
-  ~/testtwf/web-platform-tests # web-platform-tests checkout
-  ~/testtwf/meta               # metadata
-
-To test a Firefox Nightly build in an OS X environment, you might start
-wptrunner using something similar to the following example::
-
-  wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
-    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
-    --certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
-    --prefs-root=~/mozilla-central/testing/profiles
-
-
-And to test a Chromium build in an OS X environment, you might start
-wptrunner using something similar to the following example::
-
-  wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
-    --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
-    --webdriver-binary=/usr/local/bin/chromedriver --product=chrome
-
---------------------
-Running test subsets
---------------------
-
-To restrict a test run just to tests in a particular web-platform-tests
-subdirectory, specify the directory name in the positional arguments after
-the options; for example, run just the tests in the `dom` subdirectory::
-
-  wptrunner --metadata=~/testtwf/meta --tests=~/testtwf/web-platform-tests/ \
-    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
-    --prefs-root=/path/to/testing/profiles \
-    dom
-
--------------------
-Running in parallel
--------------------
-
-To speed up the testing process, use the ``--processes`` option to have
-wptrunner run multiple browser instances in parallel. For example, to
-have wptrunner attempt to run tests against with six browser instances
-in parallel, specify ``--processes=6``. But note that behaviour in this
-mode is necessarily less deterministic than with ``--processes=1`` (the
-default), so there may be more noise in the test results.
-
--------------------
-Using default paths
--------------------
-
-The (otherwise-required) ``--tests`` and ``--metadata`` command-line
-options/flags be omitted if any configuration file is found that
-contains a section specifying the ``tests`` and ``metadata`` keys.
-
-See the `Configuration File`_ section for more information about
-configuration files, including information about their expected
-locations.
-
-The content of the ``wptrunner.default.ini`` default configuration file
-makes wptrunner look for tests (that is, a web-platform-tests checkout)
-as a subdirectory of the current directory named ``tests``, and for
-metadata files in a subdirectory of the current directory named ``meta``.
-
-Output
-------
-
-wptrunner uses `the mozlog package
-<https://firefox-source-docs.mozilla.org/mozbase/mozlog.html>`_) for
-output. This structures events such as test results or log messages as
-JSON objects that can then be fed to other tools for
-interpretation. More details about the message format are given in
-`the mozlog documentation
-<https://firefox-source-docs.mozilla.org/mozbase/mozlog.html>`_.
-
-By default the raw JSON messages are dumped to stdout. This is
-convenient for piping into other tools, but not ideal for humans
-reading the output. `mozlog
-<https://firefox-source-docs.mozilla.org/mozbase/mozlog.html>`_ comes
-with several other formatters, which are accessible through command
-line options. The general format of these options is
-``--log-name=dest``, where ``name`` is the name of the format and
-``dest`` is a path to a destination file, or ``-`` for stdout. The raw
-JSON data is written by the ``raw`` formatter so, the default setup
-corresponds to ``--log-raw=-``.
-
-A reasonable output format for humans is provided as ``mach``. So in
-order to output the full raw log to a file and a human-readable
-summary to stdout, one might pass the options::
-
-  --log-raw=output.log --log-mach=-
-
-Configuration File
-------------------
-
-wptrunner uses a ``.ini`` file to control some configuration
-sections. The file has three sections; ``[products]``,
-``[manifest:default]`` and ``[web-platform-tests]``.
-
-``[products]`` is used to
-define the set of available products. By default this section is empty
-which means that all the products distributed with wptrunner are
-enabled (although their dependencies may not be installed). The set
-of enabled products can be set by using the product name as the
-key. For built in products the value is empty. It is also possible to
-provide the path to a script implementing the browser functionality
-e.g.::
-
-  [products]
-  chrome =
-  netscape4 = path/to/netscape.py
-
-``[manifest:default]`` specifies the default paths for the tests and metadata,
-relative to the config file. For example::
-
-  [manifest:default]
-  tests = ~/testtwf/web-platform-tests
-  metadata = ~/testtwf/meta
-
-
-``[web-platform-tests]`` is used to set the properties of the upstream
-repository when updating the paths. ``remote_url`` specifies the git
-url to pull from; ``branch`` the branch to sync against and
-``sync_path`` the local path, relative to the configuration file, to
-use when checking out the tests e.g.::
-
-  [web-platform-tests]
-  remote_url = https://github.com/web-platform-tests/wpt.git
-  branch = master
-  sync_path = sync
-
-A configuration file must contain all the above fields; falling back
-to the default values for unspecified fields is not yet supported.
-
-The ``wptrunner`` and ``wptupdate`` commands will use configuration
-files in the following order:
-
- * Any path supplied with a ``--config`` flag to the command.
-
- * A file called ``wptrunner.ini`` in the current directory
-
- * The default configuration file (``wptrunner.default.ini`` in the
-   source directory)