author | Maja Frydrychowicz <mjzffr@gmail.com> |
Fri, 04 Dec 2020 15:08:14 +0000 | |
changeset 559506 | b6931471192607726aa261f7aa954f9c914ea0ab |
parent 559505 | 39b40e265fbd813e5346d9232ea7ff9dcbb78e0e |
child 559507 | 80442bbcf25513662b170d0a82d27125418b7fdf |
push id | 38005 |
push user | btara@mozilla.com |
push date | Sat, 05 Dec 2020 09:38:58 +0000 |
treeherder | mozilla-central@7ce95b6cde26 [default view] [failures only] |
perfherder | [talos] [build metrics] [platform microbench] (compared to previous push) |
reviewers | remote-protocol-reviewers, whimboo |
bugs | 1670286 |
milestone | 85.0a1 |
first release with | nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
|
last release without | nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
|
remote/doc/PuppeteerVendor.md | file | annotate | diff | comparison | revisions | |
remote/doc/Testing.md | file | annotate | diff | comparison | revisions |
--- a/remote/doc/PuppeteerVendor.md +++ b/remote/doc/PuppeteerVendor.md @@ -1,14 +1,77 @@ Vendoring Puppeteer =================== As mentioned in the chapter on [Testing], we run the full [Puppeteer test suite] on try. These tests are vendored in central under -_remote/test/puppeteer/_ and we have a script to pull in the most -recent changes. +_remote/test/puppeteer/_ and we have a script to pull in upstream changes. + +We periodically perform a manual two-way sync. Below is an outline of the +process interspersed with some tips. + +1. Clone the Puppeteer git repository and checkout the release tag you want + to vendor into mozilla-central. + + % git checkout tags/v10.0 -b sync-v10.0 + + +2. Apply any recent changes in `remote/test/puppeteer` to your Puppeteer branch. + + You might want to [install the project] at this point and make sure unit + tests pass. Check the project's `package.json` for relevant testing commands. + + You should use this as basis for a PR to the Puppeteer project once you are + satisfied that the two-way sync will be successful in mozilla-central. See + their [CONTRIBUTING.md]. + + Typically, the changes we push to Puppeteer include unskipping newly passing + unit tests for Firefox along with minor fixes to the tests or + to Firefox-specific browser-fetching and launch code. + + Be sure to [run tests against both Chromium and Firefox] in the Puppeteer + repo. You can specify your local Firefox build when you do so: + + % BINARY=<path-to-objdir-binary> npm run funit -To update the vendored copy of Puppeteer under _remote/test/puppeteer/_: +3. Now back in mozilla-central, you can run the following mach command to + copy over the Puppeteer branch you just prepared. The mach command has + flags to specify a local or remote repository as well as a commit. + + % ./mach remote vendor-puppeteer + +4. Go through the changes under `remote/test/puppeteer/test` and [unskip] any + newly-skipped tests (e.g. change `itFailsFirefox` to `it`). + A mass-change with `awk` might be useful here. + + Why do we do this? The Puppeteer team runs their unit tests against Firefox + in their CI with many tests skipped. In contrast, we leave these tests + unskipped in Mozilla CI and track test expectation metadata + in [puppeteer-expected.json] instead. - % ./mach remote vendor-puppeteer +5. Use `./mach puppeteer-test` (see [Testing]) to run Puppeteer tests against + both Chromium and Firefox in headless mode. Again, only running a subset of + tests against Firefox is fine -- at this point you just want to check that + the typescript compiles and the browser binaries are launched successfully. + +6. Next you want to update the test expectation metadata: test results might + have changed, tests may have been renamed, removed or added. The + easiest way to do this is to run the Puppeteer test job on try + (see [Testing]). You will find the new test metadata as an artifact on that + job and you can copy it over into your sync patch if it looks reasonable. + + Examine the job logs and makes sure the run didn't get interrupted early + by a crash or a hang, especially if you see a lot of + `TEST-UNEXPECTED-MISSING` in the Treeherder Failure Summary. You might need + to add new test skips or fix some new bug in the unit tests. This is the + fun part. + +7. Once you are happy with the metadata and are ready to submit the sync patch + up for review, run the Puppeteer test job on try again with `--rebuild 10` + to check for stability. [Testing]: ./Testing.html [Puppeteer test suite]: https://github.com/GoogleChrome/puppeteer/tree/master/test +[re-install the project]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md#getting-code +[run tests against both Chromium and Firefox]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md +[puppeteer-expected.json]: https://searchfox.org/mozilla-central/source/remote/puppeteer-expected.json +[CONTRIBUTING.md]: https://github.com/puppeteer/puppeteer/blob/main/CONTRIBUTING.md +[unskip]: https://github.com/puppeteer/puppeteer/blob/main/test/README.md#skipping-tests-in-specific-conditions
--- a/remote/doc/Testing.md +++ b/remote/doc/Testing.md @@ -107,16 +107,19 @@ original `add_task()`. Puppeteer tests --------------- In addition to our own Firefox-specific tests, we run the upstream [Puppeteer test suite] against our implementation to [track progress] towards achieving full [Puppeteer support] in Firefox. The tests are written in the behaviour-driven testing framework [Mocha]. +Check the upstream [Puppeteer test suite] documentation for instructions on +how to skip tests, run only one test or a subsuite of tests. + Puppeteer tests are vendored under _remote/test/puppeteer/_ and are run locally like this: % ./mach puppeteer-test You can also run them against Chrome as: % ./mach puppeteer-test --product=chrome --subset