Bug 1670286 - [remote] Update Puppeteer-vendoring documentation r=remote-protocol-reviewers,whimboo
authorMaja Frydrychowicz <mjzffr@gmail.com>
Fri, 04 Dec 2020 15:08:14 +0000
changeset 559506 b6931471192607726aa261f7aa954f9c914ea0ab
parent 559505 39b40e265fbd813e5346d9232ea7ff9dcbb78e0e
child 559507 80442bbcf25513662b170d0a82d27125418b7fdf
push id38005
push userbtara@mozilla.com
push dateSat, 05 Dec 2020 09:38:58 +0000
treeherdermozilla-central@7ce95b6cde26 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersremote-protocol-reviewers, whimboo
bugs1670286
milestone85.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1670286 - [remote] Update Puppeteer-vendoring documentation r=remote-protocol-reviewers,whimboo Differential Revision: https://phabricator.services.mozilla.com/D97810
remote/doc/PuppeteerVendor.md
remote/doc/Testing.md
--- a/remote/doc/PuppeteerVendor.md
+++ b/remote/doc/PuppeteerVendor.md
@@ -1,14 +1,77 @@
 Vendoring Puppeteer
 ===================
 
 As mentioned in the chapter on [Testing], we run the full [Puppeteer
 test suite] on try.  These tests are vendored in central under
-_remote/test/puppeteer/_ and we have a script to pull in the most
-recent changes.
+_remote/test/puppeteer/_ and we have a script to pull in upstream changes.
+
+We periodically perform a manual two-way sync. Below is an outline of the
+process interspersed with some tips.
+
+1. Clone the Puppeteer git repository and checkout the release tag you want
+   to vendor into mozilla-central.
+
+	 % git checkout tags/v10.0 -b sync-v10.0
+
+
+2. Apply any recent changes in `remote/test/puppeteer` to your Puppeteer branch.
+
+	 You might want to [install the project] at this point and make sure unit
+	 tests pass. Check the project's `package.json` for relevant testing commands.
+
+   You should use this as basis for a PR to the Puppeteer project once you are
+	 satisfied that the two-way sync will be successful in mozilla-central. See
+	 their [CONTRIBUTING.md].
+
+	 Typically, the changes we push to Puppeteer include unskipping newly passing
+	 unit tests for Firefox along with minor fixes to the tests or
+	 to Firefox-specific browser-fetching and launch code.
+
+	 Be sure to [run tests against both Chromium and Firefox] in the Puppeteer
+	 repo. You can specify your local Firefox build when you do so:
+
+	 % BINARY=<path-to-objdir-binary> npm run funit
 
-To update the vendored copy of Puppeteer under _remote/test/puppeteer/_:
+3. Now back in mozilla-central, you can run the following mach command to
+	 copy over the Puppeteer branch you just prepared. The mach command has
+	 flags to specify a local or remote repository as well as a commit.
+
+	 % ./mach remote vendor-puppeteer
+
+4. Go through the changes under `remote/test/puppeteer/test` and [unskip] any
+	 newly-skipped tests (e.g. change `itFailsFirefox` to `it`).
+	 A mass-change with `awk` might be useful here.
+
+	 Why do we do this? The Puppeteer team runs their unit tests against Firefox
+	 in their CI with many tests skipped. In contrast, we leave these tests
+	 unskipped in Mozilla CI and track test expectation metadata
+	 in [puppeteer-expected.json] instead.
 
-	% ./mach remote vendor-puppeteer
+5. Use `./mach puppeteer-test` (see [Testing]) to run Puppeteer tests against
+   both Chromium and Firefox in headless mode. Again, only running a subset of
+	 tests against Firefox is fine -- at this point you just want to check that
+	 the typescript compiles and the browser binaries are launched successfully.
+
+6. Next you want to update the test expectation metadata: test results might
+   have changed, tests may have been renamed, removed or added. The
+	 easiest way to do this is to run the Puppeteer test job on try
+	 (see [Testing]). You will find the new test metadata as an artifact on that
+	 job and you can copy it over into your sync patch if it looks reasonable.
+
+	 Examine the job logs and makes sure the run didn't get interrupted early
+	 by a crash or a hang, especially if you see a lot of
+	 `TEST-UNEXPECTED-MISSING` in the Treeherder Failure Summary. You might need
+	 to add new test skips or fix some new bug in the unit tests. This is the
+	 fun part. 
+
+7. Once you are happy with the metadata and are ready to submit the sync patch
+   up for review, run the Puppeteer test job on try again with `--rebuild 10`
+	 to check for stability.
 
 [Testing]: ./Testing.html
 [Puppeteer test suite]: https://github.com/GoogleChrome/puppeteer/tree/master/test
+[re-install the project]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md#getting-code
+[run tests against both Chromium and Firefox]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md
+[puppeteer-expected.json]: https://searchfox.org/mozilla-central/source/remote/puppeteer-expected.json
+[CONTRIBUTING.md]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md
+[unskip]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md#skipping-tests-in-specific-conditions
--- a/remote/doc/Testing.md
+++ b/remote/doc/Testing.md
@@ -107,16 +107,19 @@ original `add_task()`.
 Puppeteer tests
 ---------------
 
 In addition to our own Firefox-specific tests, we run the upstream
 [Puppeteer test suite] against our implementation to [track progress]
 towards achieving full [Puppeteer support] in Firefox. The tests are written
 in the behaviour-driven testing framework [Mocha].
 
+Check the upstream [Puppeteer test suite] documentation for instructions on
+how to skip tests, run only one test or a subsuite of tests.
+
 Puppeteer tests are vendored under _remote/test/puppeteer/_ and are
 run locally like this:
 
 	% ./mach puppeteer-test
 
 You can also run them against Chrome as:
 
   % ./mach puppeteer-test --product=chrome --subset