Bug 1670286 - [remote] Update Puppeteer-vendoring documentation r=remote-protocol-reviewers,whimboo
authorMaja Frydrychowicz <mjzffr@gmail.com>
Fri, 04 Dec 2020 15:08:14 +0000
changeset 559506 b6931471192607726aa261f7aa954f9c914ea0ab
parent 559505 39b40e265fbd813e5346d9232ea7ff9dcbb78e0e
child 559507 80442bbcf25513662b170d0a82d27125418b7fdf
push id38005
push userbtara@mozilla.com
push dateSat, 05 Dec 2020 09:38:58 +0000
treeherdermozilla-central@7ce95b6cde26 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersremote-protocol-reviewers, whimboo
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1670286 - [remote] Update Puppeteer-vendoring documentation r=remote-protocol-reviewers,whimboo Differential Revision: https://phabricator.services.mozilla.com/D97810
--- a/remote/doc/PuppeteerVendor.md
+++ b/remote/doc/PuppeteerVendor.md
@@ -1,14 +1,77 @@
 Vendoring Puppeteer
 As mentioned in the chapter on [Testing], we run the full [Puppeteer
 test suite] on try.  These tests are vendored in central under
-_remote/test/puppeteer/_ and we have a script to pull in the most
-recent changes.
+_remote/test/puppeteer/_ and we have a script to pull in upstream changes.
+We periodically perform a manual two-way sync. Below is an outline of the
+process interspersed with some tips.
+1. Clone the Puppeteer git repository and checkout the release tag you want
+   to vendor into mozilla-central.
+	 % git checkout tags/v10.0 -b sync-v10.0
+2. Apply any recent changes in `remote/test/puppeteer` to your Puppeteer branch.
+	 You might want to [install the project] at this point and make sure unit
+	 tests pass. Check the project's `package.json` for relevant testing commands.
+   You should use this as basis for a PR to the Puppeteer project once you are
+	 satisfied that the two-way sync will be successful in mozilla-central. See
+	 their [CONTRIBUTING.md].
+	 Typically, the changes we push to Puppeteer include unskipping newly passing
+	 unit tests for Firefox along with minor fixes to the tests or
+	 to Firefox-specific browser-fetching and launch code.
+	 Be sure to [run tests against both Chromium and Firefox] in the Puppeteer
+	 repo. You can specify your local Firefox build when you do so:
+	 % BINARY=<path-to-objdir-binary> npm run funit
-To update the vendored copy of Puppeteer under _remote/test/puppeteer/_:
+3. Now back in mozilla-central, you can run the following mach command to
+	 copy over the Puppeteer branch you just prepared. The mach command has
+	 flags to specify a local or remote repository as well as a commit.
+	 % ./mach remote vendor-puppeteer
+4. Go through the changes under `remote/test/puppeteer/test` and [unskip] any
+	 newly-skipped tests (e.g. change `itFailsFirefox` to `it`).
+	 A mass-change with `awk` might be useful here.
+	 Why do we do this? The Puppeteer team runs their unit tests against Firefox
+	 in their CI with many tests skipped. In contrast, we leave these tests
+	 unskipped in Mozilla CI and track test expectation metadata
+	 in [puppeteer-expected.json] instead.
-	% ./mach remote vendor-puppeteer
+5. Use `./mach puppeteer-test` (see [Testing]) to run Puppeteer tests against
+   both Chromium and Firefox in headless mode. Again, only running a subset of
+	 tests against Firefox is fine -- at this point you just want to check that
+	 the typescript compiles and the browser binaries are launched successfully.
+6. Next you want to update the test expectation metadata: test results might
+   have changed, tests may have been renamed, removed or added. The
+	 easiest way to do this is to run the Puppeteer test job on try
+	 (see [Testing]). You will find the new test metadata as an artifact on that
+	 job and you can copy it over into your sync patch if it looks reasonable.
+	 Examine the job logs and makes sure the run didn't get interrupted early
+	 by a crash or a hang, especially if you see a lot of
+	 `TEST-UNEXPECTED-MISSING` in the Treeherder Failure Summary. You might need
+	 to add new test skips or fix some new bug in the unit tests. This is the
+	 fun part. 
+7. Once you are happy with the metadata and are ready to submit the sync patch
+   up for review, run the Puppeteer test job on try again with `--rebuild 10`
+	 to check for stability.
 [Testing]: ./Testing.html
 [Puppeteer test suite]: https://github.com/GoogleChrome/puppeteer/tree/master/test
+[re-install the project]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md#getting-code
+[run tests against both Chromium and Firefox]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md
+[puppeteer-expected.json]: https://searchfox.org/mozilla-central/source/remote/puppeteer-expected.json
+[CONTRIBUTING.md]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md
+[unskip]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md#skipping-tests-in-specific-conditions
--- a/remote/doc/Testing.md
+++ b/remote/doc/Testing.md
@@ -107,16 +107,19 @@ original `add_task()`.
 Puppeteer tests
 In addition to our own Firefox-specific tests, we run the upstream
 [Puppeteer test suite] against our implementation to [track progress]
 towards achieving full [Puppeteer support] in Firefox. The tests are written
 in the behaviour-driven testing framework [Mocha].
+Check the upstream [Puppeteer test suite] documentation for instructions on
+how to skip tests, run only one test or a subsuite of tests.
 Puppeteer tests are vendored under _remote/test/puppeteer/_ and are
 run locally like this:
 	% ./mach puppeteer-test
 You can also run them against Chrome as:
   % ./mach puppeteer-test --product=chrome --subset