Backed out 3 changesets (bug 1510695, bug 1510696, bug 1510693) for causing marionette reftest failures.
authorCosmin Sabou <csabou@mozilla.com>
Thu, 10 Jan 2019 22:04:11 +0200
changeset 453364 3994da3447c34fbf333b51776fd91afeb7fa1dbf
parent 453363 16ce9a71ff04850dd34a71759852bca9a62a413d
child 453365 f747aad08cbcc77d2b93dbe2b30a04b2096e79b4
push id111085
push userdvarga@mozilla.com
push dateFri, 11 Jan 2019 04:23:32 +0000
treeherdermozilla-inbound@9cc1cf173fce [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
bugs1510695, 1510696, 1510693
milestone66.0a1
backs outdaf05866f9ddcd5a362b4d436477f3e01abd4234
aa26fbe413c4039675a4d45bb83c73e8d4e9af5d
bd2cb6b0a391056bd88a2e0d00577a80a3fef4fa
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Backed out 3 changesets (bug 1510695, bug 1510696, bug 1510693) for causing marionette reftest failures. Backed out changeset daf05866f9dd (bug 1510696) Backed out changeset aa26fbe413c4 (bug 1510695) Backed out changeset bd2cb6b0a391 (bug 1510693)
testing/marionette/driver.js
testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py
testing/marionette/harness/marionette_harness/tests/unit/unit-tests.ini
testing/marionette/harness/marionette_harness/www/reftest/mostly-teal-700x700.html
testing/marionette/harness/marionette_harness/www/reftest/teal-700x700.html
testing/marionette/listener.js
testing/marionette/reftest.js
testing/web-platform/meta/css/CSS2/generated-content/content-175.xht.ini
testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-collapse-001.xht.ini
testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-103.xht.ini
testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-104.xht.ini
testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-113.xht.ini
testing/web-platform/meta/css/CSS2/normal-flow/blocks-020.xht.ini
testing/web-platform/meta/css/CSS2/normal-flow/blocks-021.xht.ini
testing/web-platform/meta/css/CSS2/normal-flow/blocks-022.xht.ini
testing/web-platform/meta/css/CSS2/normal-flow/inline-replaced-width-014.xht.ini
testing/web-platform/meta/css/CSS2/positioning/absolute-replaced-width-003c.xht.ini
testing/web-platform/meta/css/CSS2/positioning/left-offset-percentage-002.xht.ini
testing/web-platform/meta/css/CSS2/positioning/top-offset-percentage-002.xht.ini
testing/web-platform/meta/css/CSS2/tables/separated-border-model-007.xht.ini
testing/web-platform/meta/css/CSS2/tables/separated-border-model-008.xht.ini
testing/web-platform/meta/css/CSS2/tables/separated-border-model-009.xht.ini
testing/web-platform/meta/css/CSS2/visuren/top-114.xht.ini
testing/web-platform/meta/css/css-backgrounds/border-image-outset-003.html.ini
testing/web-platform/meta/css/css-flexbox/flexbox_visibility-collapse-line-wrapping.html.ini
testing/web-platform/meta/css/css-images/tiled-radial-gradients.html.ini
testing/web-platform/meta/css/css-multicol/multicol-fill-auto-001.xht.ini
testing/web-platform/meta/css/css-multicol/multicol-inherit-001.xht.ini
testing/web-platform/meta/css/css-multicol/multicol-overflow-000.xht.ini
testing/web-platform/meta/css/css-transforms/matrix/svg-matrix-005.html.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-007.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-013.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-019.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-025.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-031.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-037.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-043.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-049.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-055.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-061.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-067.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-073.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-079.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-085.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-091.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-097.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-107.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-115.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-123.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-131.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-139.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-147.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-155.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-163.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-171.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-179.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-187.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-195.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-203.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-211.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-219.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-227.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-006.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-012.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-018.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-024.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-030.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-036.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-042.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-048.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-054.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-060.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-066.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-072.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-078.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-084.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-090.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-096.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-106.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-114.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-122.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-130.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-138.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-146.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-154.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-162.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-170.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-178.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-186.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-194.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-202.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-210.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-218.xht.ini
testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-226.xht.ini
testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-007.xht.ini
testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-008.xht.ini
testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vrl-008.xht.ini
testing/web-platform/meta/css/css-writing-modes/sizing-orthog-prct-htb-in-vrl-001.xht.ini
testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-011.xht.ini
testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-013.xht.ini
testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-010.xht.ini
testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-012.xht.ini
testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py
--- a/testing/marionette/driver.js
+++ b/testing/marionette/driver.js
@@ -3501,35 +3501,35 @@ GeckoDriver.prototype.setupReftest = asy
 
   let {urlCount = {}, screenshot = "unexpected"} = cmd.parameters;
   if (!["always", "fail", "unexpected"].includes(screenshot)) {
     throw new InvalidArgumentError(
         "Value of `screenshot` should be 'always', 'fail' or 'unexpected'");
   }
 
   this._reftest = new reftest.Runner(this);
-  this._reftest.setup(urlCount, screenshot);
+  await this._reftest.setup(urlCount, screenshot);
 };
 
 
 /** Run a reftest. */
 GeckoDriver.prototype.runReftest = async function(cmd) {
-  let {test, references, expected, timeout, width, height} = cmd.parameters;
+  let {test, references, expected, timeout} = cmd.parameters;
 
   if (!this._reftest) {
     throw new UnsupportedOperationError(
         "Called reftest:run before reftest:start");
   }
 
   assert.string(test);
   assert.string(expected);
   assert.array(references);
 
   return {value: await this._reftest.run(
-      test, references, expected, timeout, width, height)};
+      test, references, expected, timeout)};
 };
 
 /**
  * End a reftest run.
  *
  * Closes the reftest window (without changing the current window handle),
  * and removes cached canvases.
  */
deleted file mode 100644
--- a/testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py
+++ /dev/null
@@ -1,84 +0,0 @@
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-from __future__ import absolute_import, print_function
-
-from marionette_driver.errors import InvalidArgumentException, UnsupportedOperationException
-from marionette_harness import MarionetteTestCase
-
-
-class TestReftest(MarionetteTestCase):
-    def setUp(self):
-        super(TestReftest, self).setUp()
-
-        self.original_window = self.marionette.current_window_handle
-
-        self.marionette.set_context(self.marionette.CONTEXT_CHROME)
-        self.marionette.set_pref("dom.send_after_paint_to_content", True)
-
-    def tearDown(self):
-        try:
-            # make sure we've teared down any reftest context
-            self.marionette._send_message("reftest:teardown", {})
-        except UnsupportedOperationException:
-            # this will throw if we aren't currently in a reftest context
-            pass
-
-        self.marionette.switch_to_window(self.original_window)
-
-        self.marionette.set_context(self.marionette.CONTEXT_CONTENT)
-        self.marionette.clear_pref("dom.send_after_paint_to_content")
-
-        super(TestReftest, self).tearDown()
-
-    def test_basic(self):
-        self.marionette._send_message("reftest:setup", {"screenshot": "unexpected"})
-        rv = self.marionette._send_message("reftest:run",
-                                           {"test": "about:blank",
-                                            "references": [["about:blank", [], "=="]],
-                                            "expected": "PASS",
-                                            "timeout": 10 * 1000})
-        self.marionette._send_message("reftest:teardown", {})
-        expected = {u'value': {u'extra': {},
-                               u'message': u'Testing about:blank == about:blank\n',
-                               u'stack': None,
-                               u'status': u'PASS'}}
-        self.assertEqual(expected, rv)
-
-    def test_url_comparison(self):
-        test_page = self.fixtures.where_is("test.html")
-        test_page_2 = self.fixtures.where_is("foo/../test.html")
-
-        self.marionette._send_message("reftest:setup", {"screenshot": "unexpected"})
-        rv = self.marionette._send_message("reftest:run",
-                                           {"test": test_page,
-                                            "references": [[test_page_2, [], "=="]],
-                                            "expected": "PASS",
-                                            "timeout": 10 * 1000})
-        self.marionette._send_message("reftest:teardown", {})
-        self.assertEqual(u"PASS", rv[u"value"][u"status"])
-
-    def test_cache_multiple_sizes(self):
-        teal = self.fixtures.where_is("reftest/teal-700x700.html")
-        mostly_teal = self.fixtures.where_is("reftest/mostly-teal-700x700.html")
-
-        self.marionette._send_message("reftest:setup", {"screenshot": "unexpected"})
-        rv = self.marionette._send_message("reftest:run",
-                                           {"test": teal,
-                                            "references": [[mostly_teal, [], "=="]],
-                                            "expected": "PASS",
-                                            "timeout": 1 * 1000,
-                                            "width": 600,
-                                            "height": 600})
-        self.assertEqual(u"PASS", rv[u"value"][u"status"])
-
-        rv = self.marionette._send_message("reftest:run",
-                                           {"test": teal,
-                                            "references": [[mostly_teal, [], "=="]],
-                                            "expected": "PASS",
-                                            "timeout": 1 * 1000,
-                                            "width": 700,
-                                            "height": 700})
-        self.assertEqual(u"FAIL", rv[u"value"][u"status"])
-        self.marionette._send_message("reftest:teardown", {})
--- a/testing/marionette/harness/marionette_harness/tests/unit/unit-tests.ini
+++ b/testing/marionette/harness/marionette_harness/tests/unit/unit-tests.ini
@@ -121,10 +121,8 @@ skip-if = appname == 'fennec'
 
 [test_addons.py]
 skip-if = appname == 'fennec' # Bug 1330598
 
 [test_select.py]
 [test_crash.py]
 skip-if = asan || manage_instance == false || appname == 'fennec' # Bug 1298921
 [test_localization.py]
-
-[test_reftest.py]
deleted file mode 100644
--- a/testing/marionette/harness/marionette_harness/www/reftest/mostly-teal-700x700.html
+++ /dev/null
@@ -1,21 +0,0 @@
-<!doctype html>
-<style>
-  * {
-      margin: 0;
-      padding: 0;
-  }
-
-  html {
-      background: teal;
-  }
-
-  div {
-      position: absolute;
-      top: 600px;
-      left: 600px;
-      height: 100px;
-      width: 100px;
-      background: orange;
-  }
-</style>
-<div></div>
deleted file mode 100644
--- a/testing/marionette/harness/marionette_harness/www/reftest/teal-700x700.html
+++ /dev/null
@@ -1,21 +0,0 @@
-<!doctype html>
-<style>
-  * {
-      margin: 0;
-      padding: 0;
-  }
-
-  html {
-      background: teal;
-  }
-
-  div {
-      position: absolute;
-      top: 600px;
-      left: 600px;
-      height: 100px;
-      width: 100px;
-      background: transparent;
-  }
-</style>
-<div></div>
--- a/testing/marionette/listener.js
+++ b/testing/marionette/listener.js
@@ -1615,18 +1615,16 @@ function flushRendering() {
 async function reftestWait(url, remote) {
   let win = curContainer.frame;
   let document = curContainer.frame.document;
 
   let windowUtils = content.windowUtils;
 
   let reftestWait = false;
 
-  url = new URL(url).href;
-
   if (document.location.href !== url || document.readyState != "complete") {
     logger.debug(truncate`Waiting for page load of ${url}`);
     await new Promise(resolve => {
       let maybeResolve = event => {
         if (event.target === curContainer.frame.document &&
             event.target.location.href === url) {
           win = curContainer.frame;
           document = curContainer.frame.document;
--- a/testing/marionette/reftest.js
+++ b/testing/marionette/reftest.js
@@ -28,35 +28,35 @@ const SCREENSHOT_MODE = {
 
 const STATUS = {
   PASS: "PASS",
   FAIL: "FAIL",
   ERROR: "ERROR",
   TIMEOUT: "TIMEOUT",
 };
 
-const DEFAULT_REFTEST_WIDTH = 600;
-const DEFAULT_REFTEST_HEIGHT = 600;
+const REFTEST_WIDTH = 600;
+const REFTEST_HEIGHT = 600;
 
 /**
  * Implements an fast runner for web-platform-tests format reftests
  * c.f. http://web-platform-tests.org/writing-tests/reftests.html.
  *
  * @namespace
  */
 this.reftest = {};
 
 /**
  * @memberof reftest
  * @class Runner
  */
 reftest.Runner = class {
   constructor(driver) {
     this.driver = driver;
-    this.canvasCache = new DefaultMap(undefined, () => new Map([[null, []]]));
+    this.canvasCache = new Map([[null, []]]);
     this.windowUtils = null;
     this.lastURL = null;
     this.remote = Preferences.get(PREF_E10S);
   }
 
   /**
    * Setup the required environment for running reftests.
    *
@@ -65,83 +65,71 @@ reftest.Runner = class {
    *
    * @param {Object.<Number>} urlCount
    *     Object holding a map of URL: number of times the URL
    *     will be opened during the reftest run, where that's
    *     greater than 1.
    * @param {string} screenshotMode
    *     String enum representing when screenshots should be taken
    */
-  setup(urlCount, screenshotMode) {
+  async setup(urlCount, screenshotMode) {
     this.parentWindow =  assert.open(this.driver.getCurrentWindow());
 
     this.screenshotMode = SCREENSHOT_MODE[screenshotMode] ||
         SCREENSHOT_MODE.unexpected;
 
     this.urlCount = Object.keys(urlCount || {})
         .reduce((map, key) => map.set(key, urlCount[key]), new Map());
+
+    await this.ensureWindow();
   }
 
-  async ensureWindow(timeout, width, height) {
-    logger.debug(`ensuring we have a window ${width}x${height}`);
-
+  async ensureWindow(timeout) {
     if (this.reftestWin && !this.reftestWin.closed) {
-      let browserRect = this.reftestWin.gBrowser.getBoundingClientRect();
-      if (browserRect.width === width && browserRect.height === height) {
-        return this.reftestWin;
-      }
-      logger.debug(`current: ${browserRect.width}x${browserRect.height}`);
+      return this.reftestWin;
     }
 
     let reftestWin;
     if (Services.appinfo.OS == "Android") {
       logger.debug("Using current window");
       reftestWin = this.parentWindow;
       await this.driver.listener.get({
         commandID: this.driver.listener.activeMessageId,
         pageTimeout: timeout,
         url: "about:blank",
         loadEventExpected: false});
+
     } else {
       logger.debug("Using separate window");
-      if (this.reftestWin && !this.reftestWin.closed) {
-        this.reftestWin.close();
-      }
-      reftestWin = await this.openWindow(width, height);
+      reftestWin = await this.openWindow();
     }
 
-    this.setupWindow(reftestWin, width, height);
+    this.setupWindow(reftestWin);
     this.windowUtils = reftestWin.windowUtils;
     this.reftestWin = reftestWin;
 
     let found = this.driver.findWindow([reftestWin], () => true);
     await this.driver.setWindowHandle(found, true);
 
-    let browserRect = reftestWin.gBrowser.getBoundingClientRect();
-    logger.debug(`new: ${browserRect.width}x${browserRect.height}`);
-
     return reftestWin;
   }
 
-  async openWindow(width, height) {
-    assert.positiveInteger(width);
-    assert.positiveInteger(height);
-
+  async openWindow() {
     let reftestWin = this.parentWindow.open(
         "chrome://marionette/content/reftest.xul",
         "reftest",
-        `chrome,height=${width},width=${height}`);
+        `chrome,height=${REFTEST_WIDTH},width=${REFTEST_HEIGHT}`);
 
     await new Promise(resolve => {
       reftestWin.addEventListener("load", resolve, {once: true});
     });
     return reftestWin;
   }
 
-  setupWindow(reftestWin, width, height) {
+  setupWindow(reftestWin) {
     let browser;
     if (Services.appinfo.OS === "Android") {
       browser = reftestWin.document.getElementsByTagName("browser")[0];
       browser.setAttribute("remote", "false");
     } else {
       browser = reftestWin.document.createElementNS(XUL_NS, "xul:browser");
       browser.permanentKey = {};
       browser.setAttribute("id", "browser");
@@ -150,21 +138,21 @@ reftest.Runner = class {
       browser.setAttribute("primary", "true");
       if (this.remote) {
         browser.setAttribute("remote", "true");
         browser.setAttribute("remoteType", "web");
       } else {
         browser.setAttribute("remote", "false");
       }
     }
-    // Make sure the browser element is exactly the right size, no matter
+    // Make sure the browser element is exactly 600x600, no matter
     // what size our window is
     const windowStyle = `padding: 0px; margin: 0px; border:none;
-min-width: ${width}px; min-height: ${height}px;
-max-width: ${width}px; max-height: ${height}px`;
+min-width: ${REFTEST_WIDTH}px; min-height: ${REFTEST_HEIGHT}px;
+max-width: ${REFTEST_WIDTH}px; max-height: ${REFTEST_HEIGHT}px`;
     browser.setAttribute("style", windowStyle);
 
     if (Services.appinfo.OS !== "Android") {
       let doc = reftestWin.document.documentElement;
       while (doc.firstChild) {
         doc.firstChild.remove();
       }
       doc.appendChild(browser);
@@ -220,33 +208,30 @@ max-width: ${width}px; max-height: ${hei
    * @param {string} expected
    *     Expected test outcome (e.g. <tt>PASS</tt>, <tt>FAIL</tt>).
    * @param {number} timeout
    *     Test timeout in milliseconds.
    *
    * @return {Object}
    *     Result object with fields status, message and extra.
    */
-  async run(testUrl, references, expected, timeout,
-      width = DEFAULT_REFTEST_WIDTH,
-      height = DEFAULT_REFTEST_HEIGHT) {
+  async run(testUrl, references, expected, timeout) {
 
     let timeoutHandle;
 
     let timeoutPromise = new Promise(resolve => {
       timeoutHandle = this.parentWindow.setTimeout(() => {
         resolve({status: STATUS.TIMEOUT, message: null, extra: {}});
       }, timeout);
     });
 
     let testRunner = (async () => {
       let result;
       try {
-        result = await this.runTest(testUrl, references, expected, timeout,
-            width, height);
+        result = await this.runTest(testUrl, references, expected, timeout);
       } catch (e) {
         result = {
           status: STATUS.ERROR,
           message: String(e),
           stack: e.stack,
           extra: {},
         };
       }
@@ -257,18 +242,18 @@ max-width: ${width}px; max-height: ${hei
     this.parentWindow.clearTimeout(timeoutHandle);
     if (result.status === STATUS.TIMEOUT) {
       this.abort();
     }
 
     return result;
   }
 
-  async runTest(testUrl, references, expected, timeout, width, height) {
-    let win = await this.ensureWindow(timeout, width, height);
+  async runTest(testUrl, references, expected, timeout) {
+    let win = await this.ensureWindow(timeout);
 
     function toBase64(screenshot) {
       let dataURL = screenshot.canvas.toDataURL();
       return dataURL.split(",")[1];
     }
 
     let result = {
       status: STATUS.FAIL,
@@ -337,24 +322,23 @@ max-width: ${width}px; max-height: ${hei
         let isFail = this.screenshotMode === SCREENSHOT_MODE.fail;
         let isUnexpected = this.screenshotMode === SCREENSHOT_MODE.unexpected;
         if (isFail || (isUnexpected && expected != result.status)) {
           recordScreenshot();
         }
       }
 
       // Return any reusable canvases to the pool
-      let cacheKey = width + "x" + height;
-      let canvasPool = this.canvasCache.get(cacheKey).get(null);
+      let canvasPool = this.canvasCache.get(null);
       [comparison.lhs, comparison.rhs].map(screenshot => {
         if (screenshot !== null && screenshot.reuseCanvas) {
           canvasPool.push(screenshot.canvas);
         }
       });
-      logger.debug(`Canvas pool (${cacheKey}) is of length ${canvasPool.length}`);
+      logger.debug(`Canvas pool is of length ${canvasPool.length}`);
 
     }
 
     if (screenshotData.length) {
       // For now the tbpl formatter only accepts one screenshot, so just
       // return the last one we took.
       let lastScreenshot = screenshotData[screenshotData.length - 1];
       // eslint-disable-next-line camelcase
@@ -428,41 +412,40 @@ max-width: ${width}px; max-height: ${hei
     const focusManager = Services.focus;
     if (focusManager.activeWindow != win) {
       focusManager.activeWindow = win;
     }
     this.driver.curBrowser.contentBrowser.focus();
   }
 
   async screenshot(win, url, timeout) {
+    win.innerWidth = REFTEST_WIDTH;
+    win.innerHeight = REFTEST_HEIGHT;
+
     // On windows the above doesn't *actually* set the window to be the
     // reftest size; but *does* set the content area to be the right size;
     // the window is given some extra borders that aren't explicable from CSS
     let browserRect = win.gBrowser.getBoundingClientRect();
     let canvas = null;
     let remainingCount = this.urlCount.get(url) || 1;
     let cache = remainingCount > 1;
-    let cacheKey = browserRect.width + "x" + browserRect.height;
     logger.debug(`screenshot ${url} remainingCount: ` +
-        `${remainingCount} cache: ${cache} cacheKey: ${cacheKey}`);
+        `${remainingCount} cache: ${cache}`);
     let reuseCanvas = false;
-    let sizedCache = this.canvasCache.get(cacheKey);
-    if (sizedCache.has(url)) {
+    if (this.canvasCache.has(url)) {
       logger.debug(`screenshot ${url} taken from cache`);
-      canvas = sizedCache.get(url);
+      canvas = this.canvasCache.get(url);
       if (!cache) {
-        sizedCache.delete(url);
+        this.canvasCache.delete(url);
       }
     } else {
-      let canvasPool = sizedCache.get(null);
-      if (canvasPool.length) {
-        logger.debug("reusing canvas from canvas pool");
-        canvas = canvasPool.pop();
+      let canvases = this.canvasCache.get(null);
+      if (canvases.length) {
+        canvas = canvases.pop();
       } else {
-        logger.debug("using new canvas");
         canvas = null;
       }
       reuseCanvas = !cache;
 
       let ctxInterface = win.CanvasRenderingContext2D;
       let flags = ctxInterface.DRAWWINDOW_DRAW_CARET |
           ctxInterface.DRAWWINDOW_DRAW_VIEW;
 
@@ -497,38 +480,20 @@ max-width: ${width}px; max-height: ${hei
       canvas = capture.canvas(
           win,
           0, // left
           0, // top
           browserRect.width,
           browserRect.height,
           {canvas, flags});
     }
-    if (canvas.width !== browserRect.width ||
-        canvas.height !== browserRect.height) {
+    if (canvas.width !== REFTEST_WIDTH || canvas.height !== REFTEST_HEIGHT) {
       logger.warn(`Canvas dimensions changed to ${canvas.width}x${canvas.height}`);
       reuseCanvas = false;
       cache = false;
     }
     if (cache) {
-      sizedCache.set(url, canvas);
+      this.canvasCache.set(url, canvas);
     }
     this.urlCount.set(url, remainingCount - 1);
     return {canvas, reuseCanvas};
   }
 };
-
-class DefaultMap extends Map {
-  constructor(iterable, defaultFactory) {
-    super(iterable);
-    this.defaultFactory = defaultFactory;
-  }
-
-  get(key) {
-    if (this.has(key)) {
-      return super.get(key);
-    }
-
-    let v = this.defaultFactory();
-    this.set(key, v);
-    return v;
-  }
-}
--- a/testing/web-platform/meta/css/CSS2/generated-content/content-175.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/generated-content/content-175.xht.ini
@@ -1,2 +1,4 @@
 [content-175.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-collapse-001.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-collapse-001.xht.ini
@@ -1,2 +1,4 @@
 [margin-collapse-001.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-103.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-103.xht.ini
@@ -1,2 +1,4 @@
 [margin-right-103.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-104.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-104.xht.ini
@@ -1,2 +1,4 @@
 [margin-right-104.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-113.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/margin-padding-clear/margin-right-113.xht.ini
@@ -1,2 +1,4 @@
 [margin-right-113.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/normal-flow/blocks-020.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/normal-flow/blocks-020.xht.ini
@@ -1,2 +1,4 @@
 [blocks-020.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/normal-flow/blocks-021.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/normal-flow/blocks-021.xht.ini
@@ -1,2 +1,4 @@
 [blocks-021.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/normal-flow/blocks-022.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/normal-flow/blocks-022.xht.ini
@@ -1,2 +1,4 @@
 [blocks-022.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/normal-flow/inline-replaced-width-014.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/normal-flow/inline-replaced-width-014.xht.ini
@@ -1,2 +1,4 @@
 [inline-replaced-width-014.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/positioning/absolute-replaced-width-003c.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/positioning/absolute-replaced-width-003c.xht.ini
@@ -1,2 +1,4 @@
 [absolute-replaced-width-003c.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/positioning/left-offset-percentage-002.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/positioning/left-offset-percentage-002.xht.ini
@@ -1,2 +1,4 @@
 [left-offset-percentage-002.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/positioning/top-offset-percentage-002.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/positioning/top-offset-percentage-002.xht.ini
@@ -1,2 +1,4 @@
 [top-offset-percentage-002.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/tables/separated-border-model-007.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/tables/separated-border-model-007.xht.ini
@@ -1,2 +1,4 @@
 [separated-border-model-007.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/tables/separated-border-model-008.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/tables/separated-border-model-008.xht.ini
@@ -1,2 +1,4 @@
 [separated-border-model-008.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/tables/separated-border-model-009.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/tables/separated-border-model-009.xht.ini
@@ -1,2 +1,4 @@
 [separated-border-model-009.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/CSS2/visuren/top-114.xht.ini
+++ b/testing/web-platform/meta/css/CSS2/visuren/top-114.xht.ini
@@ -1,7 +1,6 @@
 [top-114.xht]
   expected:
     if os == "win": FAIL
     if os == "linux": FAIL
     if not debug and not webrender and e10s and (os == "mac") and (version == "OS X 10.10.5") and (processor == "x86_64") and (bits == 64): FAIL
     if debug and not webrender and e10s and (os == "mac") and (version == "OS X 10.10.5") and (processor == "x86_64") and (bits == 64): FAIL
-    if not debug and not webrender and not e10s and (os == "android") and (version == "Ubuntu 16.04") and (processor == "x86") and (bits == 32): FAIL
--- a/testing/web-platform/meta/css/css-backgrounds/border-image-outset-003.html.ini
+++ b/testing/web-platform/meta/css/css-backgrounds/border-image-outset-003.html.ini
@@ -1,2 +1,4 @@
 [border-image-outset-003.html]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-flexbox/flexbox_visibility-collapse-line-wrapping.html.ini
+++ b/testing/web-platform/meta/css/css-flexbox/flexbox_visibility-collapse-line-wrapping.html.ini
@@ -1,2 +1,4 @@
 [flexbox_visibility-collapse-line-wrapping.html]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-images/tiled-radial-gradients.html.ini
+++ b/testing/web-platform/meta/css/css-images/tiled-radial-gradients.html.ini
@@ -1,2 +1,4 @@
 [tiled-radial-gradients.html]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-multicol/multicol-fill-auto-001.xht.ini
+++ b/testing/web-platform/meta/css/css-multicol/multicol-fill-auto-001.xht.ini
@@ -1,2 +1,4 @@
 [multicol-fill-auto-001.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-multicol/multicol-inherit-001.xht.ini
+++ b/testing/web-platform/meta/css/css-multicol/multicol-inherit-001.xht.ini
@@ -1,2 +1,4 @@
 [multicol-inherit-001.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-multicol/multicol-overflow-000.xht.ini
+++ b/testing/web-platform/meta/css/css-multicol/multicol-overflow-000.xht.ini
@@ -1,2 +1,4 @@
 [multicol-overflow-000.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-transforms/matrix/svg-matrix-005.html.ini
+++ b/testing/web-platform/meta/css/css-transforms/matrix/svg-matrix-005.html.ini
@@ -1,2 +1,4 @@
 [svg-matrix-005.html]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-007.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-007.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-007.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-013.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-013.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-013.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-019.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-019.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-019.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-025.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-025.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-025.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-031.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-031.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-031.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-037.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-037.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-037.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-043.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-043.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-043.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-049.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-049.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-049.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-055.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-055.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-055.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-061.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-061.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-061.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-067.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-067.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-067.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-073.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-073.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-073.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-079.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-079.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-079.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-085.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-085.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-085.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-091.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-091.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-091.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-097.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-097.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-097.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-107.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-107.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-107.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-115.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-115.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-115.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-123.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-123.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-123.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-131.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-131.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-131.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-139.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-139.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-139.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-147.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-147.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-147.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-155.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-155.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-155.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-163.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-163.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-163.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-171.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-171.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-171.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-179.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-179.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-179.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-187.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-187.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-187.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-195.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-195.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-195.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-203.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-203.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-203.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-211.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-211.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-211.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-219.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-219.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-219.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-227.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vlr-227.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vlr-227.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-006.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-006.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-006.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-012.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-012.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-012.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-018.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-018.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-018.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-024.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-024.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-024.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-030.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-030.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-030.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-036.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-036.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-036.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-042.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-042.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-042.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-048.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-048.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-048.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-054.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-054.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-054.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-060.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-060.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-060.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-066.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-066.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-066.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-072.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-072.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-072.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-078.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-078.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-078.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-084.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-084.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-084.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-090.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-090.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-090.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-096.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-096.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-096.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-106.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-106.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-106.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-114.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-114.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-114.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-122.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-122.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-122.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-130.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-130.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-130.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-138.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-138.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-138.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-146.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-146.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-146.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-154.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-154.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-154.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-162.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-162.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-162.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-170.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-170.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-170.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-178.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-178.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-178.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-186.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-186.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-186.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-194.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-194.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-194.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-202.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-202.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-202.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-210.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-210.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-210.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-218.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-218.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-218.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-226.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/abs-pos-non-replaced-vrl-226.xht.ini
@@ -1,2 +1,4 @@
 [abs-pos-non-replaced-vrl-226.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-007.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-007.xht.ini
@@ -1,2 +1,4 @@
 [sizing-orthog-htb-in-vlr-007.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-008.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vlr-008.xht.ini
@@ -1,2 +1,4 @@
 [sizing-orthog-htb-in-vlr-008.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vrl-008.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-htb-in-vrl-008.xht.ini
@@ -1,2 +1,4 @@
 [sizing-orthog-htb-in-vrl-008.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-prct-htb-in-vrl-001.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/sizing-orthog-prct-htb-in-vrl-001.xht.ini
@@ -1,2 +1,4 @@
 [sizing-orthog-prct-htb-in-vrl-001.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-011.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-011.xht.ini
@@ -1,2 +1,4 @@
 [text-indent-vlr-011.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-013.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/text-indent-vlr-013.xht.ini
@@ -1,2 +1,4 @@
 [text-indent-vlr-013.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-010.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-010.xht.ini
@@ -1,2 +1,4 @@
 [text-indent-vrl-010.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-012.xht.ini
+++ b/testing/web-platform/meta/css/css-writing-modes/text-indent-vrl-012.xht.ini
@@ -1,2 +1,4 @@
 [text-indent-vrl-012.xht]
-  expected: FAIL
+  expected:
+    if os == "android": PASS
+    FAIL
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py
@@ -144,17 +144,17 @@ class FennecBrowser(FirefoxBrowser):
             self.profile.set_preferences({"browser.viewport.desktopWidth": 600,
                                           # Disable high DPI
                                           "layout.css.devPixelsPerPx": "1.0",
                                           # Ensure that the full browser element
                                           # appears in the screenshot
                                           "apz.allow_zooming": False,
                                           "android.widget_paints_background": False,
                                           # Ensure that scrollbars are always painted
-                                          "layout.testing.overlay-scrollbars.always-visible": True})
+                                          "ui.scrollbarFadeBeginDelay": 100000})
 
         if self.install_fonts:
             self.logger.debug("Copying Ahem font to profile")
             font_dir = os.path.join(self.profile.profile, "fonts")
             if not os.path.exists(font_dir):
                 os.makedirs(font_dir)
             with open(os.path.join(self.tests_root, "fonts", "Ahem.ttf"), "rb") as src:
                 with open(os.path.join(font_dir, "Ahem.ttf"), "wb") as dest: