Bug 1444049 - [marionette] Enhance documentation for Testing.md. r=whimboo
authorHenrik Skupin <mail@hskupin.info>
Thu, 08 Mar 2018 14:33:03 +0100
changeset 462398 1c15488a83eb58692074844ad2080c6726a6ab44
parent 462397 354e4a21f31095c749db01a8d75a6177f89db497
child 462399 91d8b7ad008b730c5220b0422f1a75b5bcd3f767
push id1683
push usersfraser@mozilla.com
push dateThu, 26 Apr 2018 16:43:40 +0000
treeherdermozilla-release@5af6cb21869d [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerswhimboo
bugs1444049
milestone60.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1444049 - [marionette] Enhance documentation for Testing.md. r=whimboo This fixes spelling mistakes, and adds sections for the setup of tests, and running Marionette harness unit tests. MozReview-Commit-ID: DO11Si67Zkp
testing/marionette/doc/Debugging.md
testing/marionette/doc/Patches.md
testing/marionette/doc/Testing.md
--- a/testing/marionette/doc/Debugging.md
+++ b/testing/marionette/doc/Debugging.md
@@ -12,16 +12,24 @@ working directory.  With Fennec it can b
 `mach marionette test` takes a `--gecko-log` option which lets
 you redirect this output stream.  This is convenient if you want to
 “merge” the test harness output with the stdout from the browser.
 Per Unix conventions you can use `-` (dash) to have Firefox write
 its log to stdout instead of file:
 
 	% ./mach marionette test --gecko-log -
 
+It is common to use this in conjunction with an option to increase
+the Marionette log level:
+
+  % ./mach test --gecko-log - -vv TEST
+
+A single `-v` enables debug logging, and a double `-vv` enables
+trace logging.
+
 This debugging technique can be particularly effective when combined
 with using [pdb] in the Python client or the JS remote debugger
 that is described below.
 
 [pdb]: https://docs.python.org/2/library/pdb.html
 
 
 JavaScript debugger
--- a/testing/marionette/doc/Patches.md
+++ b/testing/marionette/doc/Patches.md
@@ -4,11 +4,15 @@ Submitting patches
 You can submit patches by uploading .diff files to Bugzilla or by
 sending them to [MozReview].
 
 Once you have contributed a couple of patches, we are happy to
 sponsor you in [becoming a Mozilla committer].  When you have been
 granted commit access level 1 you will have permission to use the
 [Firefox CI] to trigger your own “try runs” to test your changes.
 
+This is a good try syntax to use when testing Marionette changes:
+
+    -b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional -t none
+
 [MozReview]: http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
 [becoming a Mozilla committer]: https://www.mozilla.org/en-US/about/governance/policies/commit/
 [Firefox CI]: https://treeherder.mozilla.org/
--- a/testing/marionette/doc/Testing.md
+++ b/testing/marionette/doc/Testing.md
@@ -1,68 +1,108 @@
-Running tests
-=============
+# Running Tests
 
 We verify and test Marionette in a couple of different ways.
-Marionette has a set of [xpcshell] unit tests located in
-_testing/marionette/test_*.js_.  These can be run this way:
+While for the server component XPCShell tests are used, the
+client and harness packages both use Python.
+
+## Setting up the tests
+
+Marionette-based tests can be run in-tree with a locally built
+or downloaded Firefox through `mach`, as well as with out-of-tree
+tests with `marionette`.
+
+Running in-tree tests, `mach` will automatically manage the Python
+virtual environment in which your tests are run.  The Marionette
+client that is picked up is the one that is in-tree at
+_testing/marionette/client_.
 
-	% ./mach test testing/marionette/test_*.js
+If you want to run tests from a downloaded test archive, you will
+need to download the `target.common.tests.zip` artifact as attached to
+Treeherder [build jobs] `B` for your system.  Extract that file and set up
+the Python Marionette client and harness by executing the following
+command:
+
+    % pip install -r config/marionette_requirements.txt
 
-Because tests are run in parallell and xpcshell itself is quite
+The tests can then be found under
+*marionette/tests/testing/marionette/harness/marionette_harness/tests* and
+can be executed with the command `marionette`.  It supports the same options as
+described below for `mach`.
+
+[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build
+
+
+## XPCShell tests
+
+Marionette has a set of [xpcshell] unit tests located in
+_testing/marionette/test*.js_.  These can be run this way:
+
+    % ./mach test testing/marionette/test_*.js
+
+Because tests are run in parallel and xpcshell itself is quite
 chatty, it can sometimes be useful to run the tests sequentially:
 
-	% ./mach test --sequential testing/marionette/test_error.js
+    % ./mach test --sequential testing/marionette/test_error.js
+
+These unit tests run as part of the `X` jobs on Treeherder.
 
-These unit tests run as part of the _X_ jobs on Treeherder.
+[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
+
+
+## Python functional tests
 
 We also have a set of functional tests that make use of the Marionette
 Python client.  These start a Firefox process and tests the Marionette
 protocol input and output.  The following command will run all tests:
 
-	% ./mach test mn
+    % ./mach marionette-test
 
 But you can also run individual tests:
 
-	% ./mach test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+    % ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+
+In case you want to run the tests with eg. a Nightly build of Firefox,
+mach let you do this with the --binary option:
+
+    % ./mach marionette-test --binary=/path/to/firefox-executable TEST
 
 When working on Marionette code it is often useful to surface the
-stdout from Firefox:
-
-	% ./mach test --gecko-log - TEST
-
-It is common to use this in conjunction with an option to increase
-the Marionette log level:
-
-	% ./mach test --gecko-log - -vv TEST
-
-A single `-v` enables debug logging, and a double `-vv` enables
-trace logging.
+stdout from Firefox, which can be achived with the `--gecko-log` option.
+See [Debugging](Debugging.html) for usage instructions.
 
 As these are functional integration tests and pop up Firefox windows
 sporadically, a helpful tip is to surpress the window whilst you
 are running them by using Firefox’ [headless mode]:
 
-	% ./mach test -z TEST
+    % ./mach marionette-test -z TEST
 
 `-z` is an alias for `--headless` and equivalent to setting the
 `MOZ_HEADLESS` output variable.  In addition to `MOZ_HEADLESS`
 there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
 controlling the dimensions of the no-op virtual display.  This is
 similar to using xvfb(1) which you may know from the X windowing system,
 but has the additional benefit of also working on macOS and Windows.
 
-We have a separate page documenting how to write good Python tests in
-<doc/PythonTests.md>.  These tests will run as part of the _Mn_
-job on Treeherder.
+These functional tests will run as part of the `Mn` job on Treeherder.
 
 In addition to these two test types that specifically test the
 Marionette protocol, Marionette is used as the backend for the
 [geckodriver] WebDriver implementation.  It is served by a WPT test
 suite which effectively tests conformance to the W3C specification.
 
-This is a good try syntax to use when testing Marionette changes:
-
-	-b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional-local-e10s,firefox-ui-functional-remote-e10s -t none
-
-[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
 [headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
 [geckodriver]: ../geckodriver/README.md
+
+
+## Python harness tests
+
+The Marionette harness Python package has a set of unit tests, which
+are written by using the [pytest] framework. The following command will
+run all tests:
+
+    % ./mach python-test testing/marionette/
+
+To run a specific test specify the full path to the module:
+
+    % ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
+
+[pytest]: https://docs.pytest.org/en/latest/