testing/web-platform/tests/docs/_running-tests/custom-runner.md
author Mike Pennisi <mike@mikepennisi.com>
Wed, 06 Mar 2019 12:36:20 +0000
changeset 525286 6b60020595e006e225c212c9880dd149d42daf29
permissions -rw-r--r--
Bug 1493024 [wpt PR 13125] - [docs] Reorganize information on running the tests, a=testonly Automatic update from web-platform-tests [docs] Reorganize information on running the tests Previously, information was organized as follows: - Section: Introduction - Page: Introduction - Introductory material - Instructions for running the tests manually - Abbreviated instructions for `wpt run` - Section: Running the Tests - Page: Running the Tests - Instructions for `wpt run` - Instructions for the web runner - Notes on writing a custom runner Dispersing the information about "test running" across multiple sections tended to dilute the meaning of each section. It also made the introduction lengthier and possibly more intimidating (especially for readers who have no intention of running the tests manually). Restructure the information as follows: - Section: Introduction - Page: Introduction - Introductory material - Section: Running the Tests - Page: Running the Tests - Instructions for running the tests manually - Reference to "Running the Tests in Automation" - Page: Running the Tests in Automation - Instructions for `wpt run` - Instructions for the web runner - Notes on writing a custom runner -- wpt-commits: c271bdc85efc27839b01ec5aecf4f3a22084d820 wpt-pr: 13125

---
layout: page
title: Writing Your Own Runner
---

Most test runners have two stages: finding all tests, followed by
executing them (or a subset thereof).

To find all tests in the repository, it is **strongly** recommended to
use the included `wpt manifest` tool: the required behaviors are more
complex than what are documented (especially when it comes to
precedence of the various possibilities and some undocumented legacy
ways to define test types), and hence its behavior should be
considered the canonical definition of how to enumerate tests and find
their type in the repository.

For test execution, please read the documentation for the various test types
very carefully and then check your understanding on
the [mailing list][public-test-infra] or [IRC][] ([webclient][web irc], join
channel `#testing`). It's possible edge-case behavior isn't properly
documented!

[public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/
[IRC]: irc://irc.w3.org:6667/testing
[web irc]: http://irc.w3.org