Bug 1594958 - Update the in-tree README for wpt, r=maja_zf
authorJames Graham <james@hoppipolla.co.uk>
Fri, 08 Nov 2019 17:43:19 +0000
changeset 501346 254ec8dbbb9669a9600abf1d29aa6307453b55f6
parent 501345 0ecfae07219631275f41197e3bb5e7da28702c1a
child 501347 5bc477e6c6179d34331241e3f1b9040a69d9656e
push id114168
push userdluca@mozilla.com
push dateSun, 10 Nov 2019 03:08:55 +0000
treeherdermozilla-inbound@33f64c1ef3e4 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersmaja_zf
bugs1594958
milestone72.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1594958 - Update the in-tree README for wpt, r=maja_zf The old file has not been updated for all the changes that have happened. Correct the inaccuracies and update the suggested advice to be more modern. Differential Revision: https://phabricator.services.mozilla.com/D52367
testing/web-platform/README.md
--- a/testing/web-platform/README.md
+++ b/testing/web-platform/README.md
@@ -1,82 +1,106 @@
 web-platform-tests
 ==================
 
 This directory contains the W3C
 [web-platform-tests](http://github.com/w3c/web-platform-tests). They
 can be run using `mach`:
 
-    mach web-platform-tests
+    mach wpt
+
+To run only certain tests, pass these as additional arguments to the
+command. For example to include all tests in the dom directory:
+
+    mach wpt testing/web-platform/tests/dom
 
-To limit the testrun to certain directories use the `--include` option;
-for example:
+Tests may also be passed by id; this is the path plus any query or
+fragment part of a url and is suitable for copying directly from logs
+e.g. on treeherder:
+
+    mach wpt /web-nfc/idlharness.https.window.html
+
+A single file can produce multiple tests, so passing test ids rather
+than paths is sometimes necessary to run exactly one test.
 
-    mach web-platform-tests --include=dom
+The testsuite contains a mix of various test types including
+javascript (`testharness`) tests, reftests and wdspec tests. To limit
+the type of tests that get run, use `--test-type=<type>` e.g.
+`--test-type=reftest` for reftests.
+
+Note that if only a single testharness test is run the browser will
+stay open by default (matching the behaviour of mochitest). To prevent
+this pass `--no-pause-after-test` to `mach wpt`.
+
+Tests can be run in headless mode using the `--headless` command line
+argument.
 
-The testsuite contains a mix of javascript tests and reftests. To
-limit the type of tests that get run, use `--test-type=testharness` for
-javascript tests or `--test-type=reftest` for reftests.
+Running in Android (GeckoView)
+------------------------------
+
+You can run the tests against a Gecko-based browser (GeckoView) on an
+Android emulator. As shown below, to do so you must start an emulator,
+build Firefox for Android and then run mach wpt with the
+`org.mozilla.geckoview.test` package. The package will be installed
+interactively by `mach` and tests will run against TestRunnerActivity.
+
+    ./mach android-emulator --version x86-7.0
+    ./mach build
+    ./mach wpt --package=org.mozilla.geckoview.test
 
 FAQ
 ---
 
 * I fixed a bug and some tests have started to pass. How do I fix the
   UNEXPECTED-PASS messages when web-platform-tests is run?
 
   You need to update the expectation data for those tests. See the
   section on expectations below.
 
 * I want to write some new tests for the web-platform-tests
   testsuite. How do I do that?
 
-  See the section on tests below. You can commit the tests directly to
-  the Mozilla repository under `testing/web-platform/tests` and they
-  will be upstreamed next time the test is imported. For this reason
-  please ensure that any tests you write are testing correct-per-spec
-  behaviour even if we don't yet pass, get proper review, and have a
-  commit message that makes sense outside of the Mozilla
-  context. If you are writing tests that should not be upstreamed yet
-  for some reason they must be located under
-  `testing/web-platform/mozilla/tests`.
-
-  It is important to note that in order for the tests to run the
-  manifest file must be updated; this should not be done by hand, but
-  by running `mach wpt-manifest-update` (or `mach web-platform-tests
-  --manifest-update`, if you also wish to run some tests).
-
-  `mach web-platform-tests-create <path>` is a helper script designed
-  to help create new web-platform-tests. It opens a locally configured
-  editor at `<path>` with web-platform-tests boilerplate filled in,
-  and in the background runs `mach web-platform-tests
-  --manifest-update <path>`, so the test being developed is added to
-  the manifest and opened for interactive development.
+  See the section on writing tests below. You can commit the tests
+  directly to the Mozilla repository under
+  `testing/web-platform/tests` and they will be automatically
+  upstreamed when the patch lands. For this reason please ensure that
+  any tests you write are testing correct-per-spec behaviour even if
+  we don't yet pass, get proper review, and have a commit message that
+  makes sense outside of the Mozilla context. If you are writing tests
+  that should not be upstreamed yet for some reason they must be
+  located under `testing/web-platform/mozilla/tests`.
 
 * How do I write a test that requires the use of a Mozilla-specific
   feature?
 
   Tests in the `mozilla/tests/` directory use the same harness but are
   not synced with any upstream. Be aware that these tests run on the
   server with a `/_mozilla/` prefix to their URLs.
 
-* A test is unstable; how do I disable it?
+* How do I write tests that require user interaction or other features
+  not accessible from content js?
+
+  For testharness tests this is possible using the
+  [testdriver](https://web-platform-tests.org/writing-tests/testdriver.html)
+  API.
 
-  See the section on disabling tests.
+Writing tests
+-------------
+
+Documentation for writing tests, and everything else that isn't
+specific to the gecko integration can be found at
+[https://web-platform-tests.org]().
 
 Directories
 -----------
 
 `tests/` contains the tests themselves. This is a copy of a certain
 revision of web-platform-tests. Any patches modifying this directory
 will be upstreamed next time the tests are imported.
 
-`harness/` contains the [wptrunner](http://github.com/w3c/wptrunner)
-test runner. Again the contents of this directory will be overwritten
-on update.
-
 `meta/` contains Gecko-specific expectation data. This is explained in
 the following section.
 
 `mozilla/tests` contains tests that will not be upstreamed and may
 make use of Mozilla-specific features.
 
 `mozilla/meta` contains metadata for the Mozilla-specific tests.
 
@@ -96,73 +120,95 @@ directory rather than the `tests` direct
 
 The format of these files is similar to `ini` files, but with a couple
 of important differences; sections can be nested using indentation,
 and only `:` is permitted as a key-value separator. For example the
 expectation file for a test with one failing subtest and one erroring
 subtest might look like:
 
     [filename.html]
-        type: testharness
-
         [Subtest name for failing test]
             expected: FAIL
 
         [Subtest name for erroring test]
             expected: ERROR
 
 Expectations can also be made platform-specific using a simple
 python-like conditional syntax e.g. for a test that times out on linux
 but otherwise fails:
 
     [filename.html]
-        type: reftest
         expected:
             if os == "linux": TIMEOUT
             FAIL
 
 The available variables for the conditions are those provided by
 [mozinfo](https://firefox-source-docs.mozilla.org/mozbase/mozinfo.html).
 
-For more information on manifest files, see the
-[wptrunner documentation](http://wptrunner.readthedocs.org/en/latest/expectation.html).
+Tests that are intermittent may be marked with multiple statuses using
+a list of possibilities e.g. for a test that usually passes, but
+intermittently fails:
+
+    [filename.html]
+        [Subtest name for intermittent test]
+            expected: [PASS, FAIL]
+
+
+For more information on manifest files, see the [wptrunner
+documentation](https://web-platform-tests.org/tools/wptrunner/docs/expectation.html).
 
 Autogenerating Expectation Data
 -------------------------------
 
 After changing some code it may be necessary to update the expectation
 data for the relevant tests. This can of course be done manually, but
 tools are available to automate much of the process.
 
 First one must run the tests that have changed status, and save the
 raw log output to a file:
 
-    mach web-platform-tests --include=url/of/test.html --log-raw=new_results.log
-
-Then the `web-platform-tests-update` command may be run using this log
-data to update the expectation files:
+    mach wpt /url/of/test.html --log-wptreport new_results.json
 
-    mach web-platform-tests-update --no-check-clean new_results.log
+Then the `wpt-update` command may be run using this log data to update
+the expectation files:
 
-By default this only updates the results data for the current
-platform. To forcibly overwrite all existing result data, use the
-`--ignore-existing` option to the update command.
+    mach wpt-update new_results.json
 
 Disabling Tests
 ---------------
 
 Tests are disabled using the same manifest files used to set
 expectation values. For example, if a test is unstable on Windows, it
 can be disabled using an ini file with the contents:
 
     [filename.html]
-        type: testharness
         disabled:
             if os == "win": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567
 
+For intermittents it's generally preferable to give the test multiple
+expectations rather than disable it.
+
+Fuzzy Reftests
+--------------
+
+Reftests where the test doesn't exactly match the reference can be
+marked as fuzzy. If the difference is inherent to the test, it should
+be encoded in a [meta
+element](https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching),
+but where it's a Gecko-specific difference it can be added to the
+metadata file, using the same syntax e.g.
+
+    [filename.html]
+        fuzzy: maxDifference=10-15;totalPixels=200-300
+
+In this case we specify that we expect between 200 and 300 pixels,
+inclusive, to be different, and the maximum difference in any colour
+channel to be between 10 and 15.
+
+
 Enabling Prefs
 --------------
 
 Some tests require specific prefs to be enabled before running. These
 prefs can be set in the expectation data using a `prefs` key with a
 comma-seperate list of `pref.name:value` items to set e.g.
 
     [filename.html]
@@ -174,17 +220,16 @@ Disabling Leak Checks
 ----------------------
 
 When a test is imported that leaks, it may be necessary to temporarily
 disable leak checking for that test in order to allow the import to
 proceed. This works in basically the same way as disabling a test, but
 with the key 'leaks' e.g.
 
     [filename.html]
-        type: testharness
         leaks:
             if os == "linux": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567
 
 Setting per-Directory Metadata
 ------------------------------
 
 Occasionally it is useful to set metadata for an entire directory of
 tests e.g. to disable then all, or to enable prefs for every test. In
@@ -227,39 +272,26 @@ would cause problems later:
         if nightly_build: PASS
         FAIL
 
 This is because on import the automatic metadata updates are run
 against the results of nightly builds, and we remove any existing
 conditions that match that configuration to avoid building up stale
 configuration options.
 
-Test Format
------------
-
-Javascript tests are written using
-[testharness.js](http://github.com/w3c/testharness.js/). Reftests are
-similar to standard Gecko reftests without an explicit manifest file,
-but with in-test or filename conventions for identifying the
-reference.
-
-Full documentation on test authoring and submission can be found on
-[testthewebforward.org](http://testthewebforward.org/docs).
-
 Test Manifest
 -------------
 
 web-platform-tests use a large auto-generated JSON file as their
 manifest. This stores data about the type of tests, their references,
 if any, and their timeout, gathered by inspecting the filenames and
-the contents of the test files.
+the contents of the test files. It it not necessary to manually add
+new tests to the manifest; it is automatically kept up to date when
+running `mach wpt`.
 
-In order to update the manifest it is recommended that you run `mach
-web-platform-tests --manifest-update`. This rescans the test directory
-looking for new, removed, or altered tests.
 
 Running Tests In Other Browsers
 -------------------------------
 
 web-platform-tests is cross browser, and the runner is compatible with
 multiple browsers. Therefore it's possible to check the behaviour of
 tests in other browsers. By default Chrome, Edge and Servo are
 supported. In order to run the tests in these browsers use the
@@ -267,8 +299,14 @@ supported. In order to run the tests in 
 
     mach wpt --product chrome dom/historical.html
 
 By default these browsers run without expectation metadata, but it can
 be added in the `testing/web-platform/products/<product>`
 directory. To run with the same metadata as for Firefox (so that
 differences are reported as unexpected results), pass `--meta
 testing/web-platform/meta` to the mach command.
+
+Results from the upstream CI for many browser, including Chrome and
+Safari, are available on [wpt.fyi](https://wpt.fyi). There is also a
+[gecko dashboard](https://jgraham.github.io/wptdash/) which by default
+shows tests that are failing in Gecko but not in Chrome and Safari,
+organised by component, based on the wpt.fyi data.