Bug 1493024 [wpt PR 13125] - [docs] Reorganize information on running the tests, a=testonly
authorMike Pennisi <mike@mikepennisi.com>
Wed, 06 Mar 2019 12:36:20 +0000
changeset 522567 0a9bd27b33902cb0a5a6baca2ca7b6a74ea1dd37
parent 522566 80f87e0a5ed3cd2f37f5f7d0664d61c479826b72
child 522568 51db6656704a7c823764c2caf252b9a125662222
push id10871
push usercbrindusan@mozilla.com
push dateMon, 18 Mar 2019 15:49:32 +0000
treeherdermozilla-beta@018abdd16060 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1493024, 13125
milestone67.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1493024 [wpt PR 13125] - [docs] Reorganize information on running the tests, a=testonly Automatic update from web-platform-tests [docs] Reorganize information on running the tests Previously, information was organized as follows: - Section: Introduction - Page: Introduction - Introductory material - Instructions for running the tests manually - Abbreviated instructions for `wpt run` - Section: Running the Tests - Page: Running the Tests - Instructions for `wpt run` - Instructions for the web runner - Notes on writing a custom runner Dispersing the information about "test running" across multiple sections tended to dilute the meaning of each section. It also made the introduction lengthier and possibly more intimidating (especially for readers who have no intention of running the tests manually). Restructure the information as follows: - Section: Introduction - Page: Introduction - Introductory material - Section: Running the Tests - Page: Running the Tests - Instructions for running the tests manually - Reference to "Running the Tests in Automation" - Page: Running the Tests in Automation - Instructions for `wpt run` - Instructions for the web runner - Notes on writing a custom runner -- wpt-commits: c271bdc85efc27839b01ec5aecf4f3a22084d820 wpt-pr: 13125
testing/web-platform/tests/docs/_running-tests/custom-runner.md
testing/web-platform/tests/docs/_running-tests/from-local-system.md
testing/web-platform/tests/docs/_running-tests/from-web.md
testing/web-platform/tests/docs/_running-tests/index.md
testing/web-platform/tests/docs/introduction.md
new file mode 100644
--- /dev/null
+++ b/testing/web-platform/tests/docs/_running-tests/custom-runner.md
@@ -0,0 +1,25 @@
+---
+layout: page
+title: Writing Your Own Runner
+---
+
+Most test runners have two stages: finding all tests, followed by
+executing them (or a subset thereof).
+
+To find all tests in the repository, it is **strongly** recommended to
+use the included `wpt manifest` tool: the required behaviors are more
+complex than what are documented (especially when it comes to
+precedence of the various possibilities and some undocumented legacy
+ways to define test types), and hence its behavior should be
+considered the canonical definition of how to enumerate tests and find
+their type in the repository.
+
+For test execution, please read the documentation for the various test types
+very carefully and then check your understanding on
+the [mailing list][public-test-infra] or [IRC][] ([webclient][web irc], join
+channel `#testing`). It's possible edge-case behavior isn't properly
+documented!
+
+[public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/
+[IRC]: irc://irc.w3.org:6667/testing
+[web irc]: http://irc.w3.org
new file mode 100644
--- /dev/null
+++ b/testing/web-platform/tests/docs/_running-tests/from-local-system.md
@@ -0,0 +1,114 @@
+---
+layout: page
+title: Running Tests from the Local System
+---
+
+The tests are designed to be run from your local computer.
+
+## System Setup
+
+The test environment requires [Python 2.7+](http://www.python.org/downloads)
+(but not Python 3.x).
+
+On Windows, be sure to add the Python directory (`c:\python2x`, by default) to
+your `%Path%` [Environment Variable](http://www.computerhope.com/issues/ch000549.htm),
+and read the [Windows Notes](#windows-notes) section below.
+
+To get the tests running, you need to set up the test domains in your
+[`hosts` file](http://en.wikipedia.org/wiki/Hosts_%28file%29%23Location_in_the_file_system).
+
+The necessary content can be generated with `./wpt make-hosts-file`; on
+Windows, you will need to preceed the prior command with `python` or
+the path to the Python binary (`python wpt make-hosts-file`).
+
+For example, on most UNIX-like systems, you can setup the hosts file with:
+
+```bash
+./wpt make-hosts-file | sudo tee -a /etc/hosts
+```
+
+And on Windows (this must be run in a PowerShell session with Administrator privileges):
+
+```bash
+python wpt make-hosts-file | Out-File %SystemRoot%\System32\drivers\etc\hosts -Encoding ascii -Append
+```
+
+If you are behind a proxy, you also need to make sure the domains above are
+excluded from your proxy lookups.
+
+### Windows Notes
+
+Generally Windows Subsystem for Linux will provide the smoothest user
+experience for running web-platform-tests on Windows.
+
+The standard Windows shell requires that all `wpt` commands are prefixed
+by the Python binary i.e. assuming `python` is on your path the server is
+started using:
+
+`python wpt serve`
+
+## Via the browser
+
+The test environment can then be started using
+
+    ./wpt serve
+
+This will start HTTP servers on two ports and a websockets server on
+one port. By default the web servers start on ports 8000 and 8443 and the other
+ports are randomly-chosen free ports. Tests must be loaded from the
+*first* HTTP server in the output. To change the ports,
+create a `config.json` file in the wpt root directory, and add
+port definitions of your choice e.g.:
+
+```
+{
+  "ports": {
+    "http": [1234, "auto"],
+    "https":[5678]
+  }
+}
+```
+
+After your `hosts` file is configured, the servers will be locally accessible at:
+
+http://web-platform.test:8000/<br>
+https://web-platform.test:8443/ *
+
+This server has all the capabilities of the publicly-deployed version--see
+[Running the Tests from the Web][from-web].
+
+\**See [Trusting Root CA](https://github.com/web-platform-tests/wpt/blob/master/README.md#trusting-root-ca)*
+
+## Via the command line
+
+Many tests can be automatically executed in a new browser instance using
+
+    ./wpt run [browsername] [tests]
+
+This will automatically load the tests in the chosen browser and extract the
+test results. For example to run the `dom/historical.html` tests in a local
+copy of Chrome:
+
+    ./wpt run chrome dom/historical.html
+
+Or to run in a specified copy of Firefox:
+
+    ./wpt run --binary ~/local/firefox/firefox firefox dom/historical.html
+
+For details on the supported products and a large number of other options for
+customising the test run:
+
+    ./wpt run --help
+
+Additional browser-specific documentation:
+
+  * [Chrome][chrome]
+
+  * [Chrome for Android][chrome android]
+
+  * [Safari][safari]
+
+[from-web]: {{ site.baseurl }}{% link _running-tests/from-web.md %}
+[chrome]: {{ site.baseurl }}{% link _running-tests/chrome.md %}
+[chrome android]: {{ site.baseurl }}{% link _running-tests/chrome_android.md %}
+[safari]: {{ site.baseurl }}{% link _running-tests/safari.md %}
new file mode 100644
--- /dev/null
+++ b/testing/web-platform/tests/docs/_running-tests/from-web.md
@@ -0,0 +1,26 @@
+---
+layout: page
+title: Running Tests from the Web
+---
+
+Tests that have been merged on GitHub are mirrored at [http://w3c-test.org/][w3c-test].
+[On properly-configured systems](from-local-system), local files may also be
+served from the URL [http://web-platform.test](http://web-platform.test).
+
+For running multiple tests inside a browser, there is a test runner
+located at `/tools/runner/index.html`.
+
+This allows all the tests, or those matching a specific prefix
+(e.g. all tests under `/dom/`) to be run. For testharness.js tests,
+the results will be automatically collected, while the runner
+provides a simple UI for manually comparing reftest rendering and
+running manual tests.
+
+Note, however, it does not currently handle more complex reftests with
+more than one reference involved.
+
+Because it runs entirely in-browser, this runner cannot deal with
+edge-cases like tests that cause the browser to crash or hang.
+
+[w3c-test]: http://w3c-test.org
+[from-local-system]: {{ site.baseurl }}{% link _running-tests/from-local-system.md %}
--- a/testing/web-platform/tests/docs/_running-tests/index.md
+++ b/testing/web-platform/tests/docs/_running-tests/index.md
@@ -1,80 +1,18 @@
 ---
 layout: page
 title: Running Tests
 ---
-In simple cases individual tests can be run by simply loading the page
-in a browser window. For running larger groups of tests, or running
-tests frequently, this is not a practical approach and several better
-options exist.
-
-## From the Command Line
-
-The simplest way to run tests is to use the `wpt run` command from the
-root of the repository. This will automatically load the tests in the
-chosen browser, and extract the test results. For example to run the
-`dom/historical.html` tests in a local copy of Chrome:
-
-    ./wpt run chrome dom/historical.html
-
-Or to run in a specified copy of Firefox:
-
-    ./wpt run --binary ~/local/firefox/firefox firefox dom/historical.html
 
-For details on the supported products and a large number of other options for
-customising the test run:
-
-    ./wpt run --help
-
-Additional browser-specific documentation:
-
-  * [Chrome][chrome]
+The simplest way to run the tests is via the public website. More detail on
+that approach is available in [Running tests from the Web][from-web].
 
-  * [Chrome for Android][chrome android]
-
-  * [Safari][safari]
-
-## From Inside a Browser
-Tests that have been merged on GitHub are mirrored at [http://w3c-test.org/][w3c-test].
-
-For running multiple tests inside a browser, there is a test runner
-located at `/tools/runner/index.html`.
+Contributors who are interested in modifying and creating tests should refer to
+[Running Tests from the Local System][from-local-system].
 
-This allows all the tests, or those matching a specific prefix
-(e.g. all tests under `/dom/`) to be run. For testharness.js tests,
-the results will be automatically collected, while the runner
-provides a simple UI for manually comparing reftest rendering and
-running manual tests.
-
-Note, however, it does not currently handle more complex reftests with
-more than one reference involved.
-
-Because it runs entirely in-browser, this runner cannot deal with
-edge-cases like tests that cause the browser to crash or hang.
-
-## Writing Your Own Runner
-
-Most test runners have two stages: finding all tests, followed by
-executing them (or a subset thereof).
+Advanced use cases may call for a customized method of executing the tests.
+Guidelines for writing a custom "runner" are available at [Writing Your Own
+Runner][custom-runner].
 
-To find all tests in the repository, it is **strongly** recommended to
-use the included `wpt manifest` tool: the required behaviors are more
-complex than what are documented (especially when it comes to
-precedence of the various possibilities and some undocumented legacy
-ways to define test types), and hence its behavior should be
-considered the canonical definition of how to enumerate tests and find
-their type in the repository.
-
-For test execution, please read the documentation for the various test types
-very carefully and then check your understanding on
-the [mailing list][public-test-infra] or [IRC][] ([webclient][web irc], join
-channel `#testing`). It's possible edge-case behavior isn't properly
-documented!
-
-
-[chrome]: {{ site.baseurl }}{% link _running-tests/chrome.md %}
-[chrome android]: {{ site.baseurl }}{% link _running-tests/chrome_android.md %}
-[safari]: {{ site.baseurl }}{% link _running-tests/safari.md %}
-[public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/
-[IRC]: irc://irc.w3.org:6667/testing
-[web irc]: http://irc.w3.org
-[w3c-test]: http://w3c-test.org
+[custom-runner]: {{ site.baseurl }}{% link _running-tests/custom-runner.md %}
+[from-web]: {{ site.baseurl }}{% link _running-tests/from-web.md %}
+[from-local-system]: {{ site.baseurl }}{% link _running-tests/from-local-system.md %}
--- a/testing/web-platform/tests/docs/introduction.md
+++ b/testing/web-platform/tests/docs/introduction.md
@@ -91,102 +91,16 @@ files they change are in; there are also
 to notify a number of people: this list of people comes from META.yml
 files in those same directories and their parents (i.e., they work
 recursively: `a/META.yml` will get notified for `a/foo.html` and
 `a/b/bar.html`).
 
 If you want to be notified about changes to tests in a directory, feel
 free to add yourself to the META.yml file!
 
-
-## Local Setup
-
-The tests are designed to be run from your local computer. The test
-environment requires [Python 2.7+](http://www.python.org/downloads) (but not Python 3.x).
-
-On Windows, be sure to add the Python directory (`c:\python2x`, by default) to
-your `%Path%` [Environment Variable](http://www.computerhope.com/issues/ch000549.htm),
-and read the [Windows Notes](#windows-notes) section below.
-
-To get the tests running, you need to set up the test domains in your
-[`hosts` file](http://en.wikipedia.org/wiki/Hosts_%28file%29%23Location_in_the_file_system).
-
-The necessary content can be generated with `./wpt make-hosts-file`; on
-Windows, you will need to preceed the prior command with `python` or
-the path to the Python binary (`python wpt make-hosts-file`).
-
-For example, on most UNIX-like systems, you can setup the hosts file with:
-
-```bash
-./wpt make-hosts-file | sudo tee -a /etc/hosts
-```
-
-And on Windows (this must be run in a PowerShell session with Administrator privileges):
-
-```bash
-python wpt make-hosts-file | Out-File %SystemRoot%\System32\drivers\etc\hosts -Encoding ascii -Append
-```
-
-If you are behind a proxy, you also need to make sure the domains above are
-excluded from your proxy lookups.
-
-The test environment can then be started using
-
-    ./wpt serve
-
-This will start HTTP servers on two ports and a websockets server on
-one port. By default the web servers start on ports 8000 and 8443 and the other
-ports are randomly-chosen free ports. Tests must be loaded from the
-*first* HTTP server in the output. To change the ports,
-create a `config.json` file in the wpt root directory, and add
-port definitions of your choice e.g.:
-
-```
-{
-  "ports": {
-    "http": [1234, "auto"],
-    "https":[5678]
-  }
-}
-```
-
-After your `hosts` file is configured, the servers will be locally accessible at:
-
-http://web-platform.test:8000/<br>
-https://web-platform.test:8443/ *
-
-\**See [Trusting Root CA](https://github.com/web-platform-tests/wpt/blob/master/README.md#trusting-root-ca)*
-
-## Running tests automatically
-
-The `wpt run` command provides a frontend for running tests automatically
-in various browsers. The general syntax is:
-
-```
-wpt run [options] <product> [test paths]
-```
-
-e.g. to run `dom/historical.html` in Firefox, the required command is:
-
-```
-wpt run firefox dom/historical.html
-```
-
-### Windows Notes
-
-Generally Windows Subsystem for Linux will provide the smoothest user
-experience for running web-platform-tests on Windows.
-
-The standard Windows shell requires that all `wpt` commands are prefixed
-by the Python binary i.e. assuming `python` is on your path the server is
-started using:
-
-`python wpt serve`
-
-
 [web-platform]: https://platform.html5.org
 [test262]: https://github.com/tc39/test262
 [webgl]: https://github.com/KhronosGroup/WebGL
 [public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/
 [IRC]: irc://irc.w3.org:6667/testing
 [web irc]: http://irc.w3.org
 
 [reftests]: {{ site.baseurl }}{% link _writing-tests/reftests.md %}