Bug 1097230 - Update to latest wptrunner, a=testonly
authorJames Graham <james@hoppipolla.co.uk>
Thu, 20 Nov 2014 16:29:59 +0000
changeset 240971 133cef79f41bfcd43047121ecc9906f53f4668af
parent 240970 ee52bc0981762ed397beb3c68d3ece02e3d4c305
child 240972 a6edf0bb1e972735f1681f9baa47812f3fab9fb5
push id4311
push userraliiev@mozilla.com
push dateMon, 12 Jan 2015 19:37:41 +0000
treeherdermozilla-beta@150c9fed433b [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1097230
milestone36.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1097230 - Update to latest wptrunner, a=testonly
testing/web-platform/harness/README.rst
testing/web-platform/harness/docs/usage.rst
testing/web-platform/harness/setup.py
testing/web-platform/harness/wptrunner/config.py
testing/web-platform/harness/wptrunner/executors/executorservo.py
testing/web-platform/harness/wptrunner/manifestexpected.py
testing/web-platform/harness/wptrunner/manifestupdate.py
testing/web-platform/harness/wptrunner/metadata.py
testing/web-platform/harness/wptrunner/testloader.py
testing/web-platform/harness/wptrunner/testrunner.py
testing/web-platform/harness/wptrunner/update.py
testing/web-platform/harness/wptrunner/wptcommandline.py
testing/web-platform/harness/wptrunner/wptrunner.py
--- a/testing/web-platform/harness/README.rst
+++ b/testing/web-platform/harness/README.rst
@@ -1,39 +1,107 @@
-web-platform-tests Harness
-==========================
+wptrunner: A web-platform-tests harness
+=======================================
 
-This harness is designed for running the W3C web-platform-tests
-`testsuite_`.
+wptrunner is a harness for running the W3C `web-platform-tests testsuite`_.
 
-The code hasn't been merged to master yet, but when it is the
-documentation below might be quite relevant.
+.. contents::
 
 Installation
 ~~~~~~~~~~~~
 
 wptrunner is expected to be installed into a virtualenv using pip. For
 development, it can be installed using the `-e` option::
 
   pip install -e ./
 
 Running the Tests
 ~~~~~~~~~~~~~~~~~
 
-After installation the command `wptrunner` should be avaliable to run
-the tests. This takes two arguments; the path to the metadata
-directory containing expectation files (see below) and a MANIFEST.json
-file (see the web-platform-tests documentation for isntructions on
-generating this file), and the path to the web-platform-tests
-checkout::
+After installation, the command ``wptrunner`` should be available to run
+the tests.
+
+The ``wptrunner`` command  takes multiple options, of which the
+following are most significant:
+
+``--product`` (defaults to `firefox`)
+  The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
+
+``--binary`` (required)
+  The path to a binary file for the product (browser) to test against.
+
+``--metadata`` (required)
+  The path to a directory containing test metadata. [#]_
+
+``--tests`` (required)
+  The path to a directory containing a web-platform-tests checkout.
+
+``--prefs-root`` (required only when testing a Firefox binary)
+  The path to a directory containing Firefox test-harness preferences. [#]_
+
+.. [#] The ``--metadata`` path is to a directory that contains:
+
+  * a ``MANIFEST.json`` file (the web-platform-tests documentation has
+    instructions on generating this file); and
+  * (optionally) any expectation files (see below)
+
+.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
+
+There are also a variety of other options available; use ``--help`` to
+list them.
+
+-------------------------------
+Example: How to start wptrunner
+-------------------------------
+
+To test a Firefox Nightly build in an OS X environment, you might start
+wptrunner using something similar to the following example::
 
-  wptrunner /path/to/metadata /path/to/tests
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  --binary=~/mozilla-central/obj-x86_64-apple-darwin14.0.0/dist/Nightly.app/Contents/MacOS/firefox \
+  --prefs-root=~/mozilla-central/testing/profiles
+
+And to test a Chromium build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
+  --product=chrome
+
+-------------------------------------
+Example: How to run a subset of tests
+-------------------------------------
+
+To restrict a test run just to tests in a particular web-platform-tests
+subdirectory, use ``--include`` with the directory name; for example::
 
-There are also a variety of other options available; use `--help` to
-list them.
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles \
+  --include=dom
+
+Output
+~~~~~~
+
+By default wptrunner just dumps its entire output as raw JSON messages
+to stdout. This is convenient for piping into other tools, but not ideal
+for humans reading the output.
+
+As an alternative, you can use the ``--log-mach`` option, which provides
+output in a reasonable format for humans. The option requires a value:
+either the path for a file to write the `mach`-formatted output to, or
+"`-`" (a hyphen) to write the `mach`-formatted output to stdout.
+
+When using ``--log-mach``, output of the full raw JSON log is still
+available, from the ``--log-raw`` option. So to output the full raw JSON
+log to a file and a human-readable summary to stdout, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles
+  --log-raw=output.log --log-mach=-
 
 Expectation Data
 ~~~~~~~~~~~~~~~~
 
 wptrunner is designed to be used in an environment where it is not
 just necessary to know which tests passed, but to compare the results
 between runs. For this reason it is possible to store the results of a
 previous run in a set of ini-like "expectation files". This format is
@@ -148,9 +216,9 @@ The web-platform-test harness knows abou
   The test type e.g. `testharness` or `reftest`.
 
 `reftype`
   The type of comparison for reftests; either `==` or `!=`.
 
 `refurl`
   The reference url for reftests.
 
-_testsuite: https://github.com/w3c/web-platform-tests
+.. _`web-platform-tests testsuite`: https://github.com/w3c/web-platform-tests
--- a/testing/web-platform/harness/docs/usage.rst
+++ b/testing/web-platform/harness/docs/usage.rst
@@ -23,67 +23,125 @@ If you intend to work on the code, the `
 used in combination with a source checkout i.e. inside a virtual
 environment created as above::
 
   git clone https://github.com/w3c/wptrunner.git
   cd wptrunner
   pip install -e ./
 
 In addition to the dependencies installed by pip, wptrunner requires
-a copy of the web-platform-tests. This can be located anywhere on
-the filesystem, but the easiest option is to put it in a sibling
-directory of the wptrunner checkout called `tests`::
+a copy of the web-platform-tests repository. That can be located
+anywhere on the filesystem, but the easiest option is to put it within
+the wptrunner checkout directory, as a subdirectory named ``tests``::
 
   git clone https://github.com/w3c/web-platform-tests.git tests
 
-It is also necessary to generate the ``MANIFEST.json`` file for the
-web-platform-tests. It is recommended to put this file in a separate
-directory called ``meta``::
+It is also necessary to generate a web-platform-tests ``MANIFEST.json``
+file. It's recommended to put that within the wptrunner
+checkout directory, in a subdirectory named ``meta``::
 
   mkdir meta
-  cd web-platform-tests
+  cd tests
   python tools/scripts/manifest.py ../meta/MANIFEST.json
 
-This file needs to be regenerated every time that the
+The ``MANIFEST.json`` file needs to be regenerated each time the
 web-platform-tests checkout is updated. To aid with the update process
 there is a tool called ``wptupdate``, which is described in
 :ref:`wptupdate-label`.
 
 Running the Tests
 -----------------
 
-A test run is started using the ``wptrunner`` command. By default this
-assumes that tests are in a subdirectory of the current directory
-called ``tests`` and the metadata is in a subdirectory called
-``meta``. These defaults can be changed using either a command line
-flag or a configuration file.
+A test run is started using the ``wptrunner`` command.  The command
+takes multiple options, of which the following are most significant:
+
+``--product`` (defaults to `firefox`)
+  The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
+
+``--binary`` (required)
+  The path to a binary file for the product (browser) to test against.
+
+``--metadata`` (required only when not `using default paths`_)
+  The path to a directory containing test metadata. [#]_
+
+``--tests`` (required only when not `using default paths`_)
+  The path to a directory containing a web-platform-tests checkout.
+
+``--prefs-root`` (required only when testing a Firefox binary)
+  The path to a directory containing Firefox test-harness preferences. [#]_
+
+.. [#] The ``--metadata`` path is to a directory that contains:
 
-To specify the browser product to test against, use the ``--product``
-flag. If no product is specified, the default is ``firefox`` which
-tests Firefox desktop. ``wptrunner --help`` can be used to see a list
-of supported products. Note that this does not take account of the
-products for which the correct dependencies have been installed.
+  * a ``MANIFEST.json`` file (the web-platform-tests documentation has
+    instructions on generating this file)
+  * (optionally) any expectation files (see :ref:`wptupdate-label`)
+
+.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
+
+There are also a variety of other command-line options available; use
+``--help`` to list them.
+
+The following examples show how to start wptrunner with various options.
+
+------------------
+Starting wptrunner
+------------------
+
+To test a Firefox Nightly build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.0.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --prefs-root=~/mozilla-central/testing/profiles
 
-Depending on the product, further arguments may be required. For
-example when testing desktop browsers ``--binary`` is commonly needed
-to specify the path to the browser executable. So a complete command
-line for running tests on firefox desktop might be::
+And to test a Chromium build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
+    --product=chrome
+
+--------------------
+Running test subsets
+--------------------
 
-  wptrunner --product=firefox --binary=/usr/bin/firefox
+To restrict a test run just to tests in a particular web-platform-tests
+subdirectory, use ``--include`` with the directory name; for example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles \
+    --include=dom
+
+-------------------
+Running in parallel
+-------------------
 
-It is also possible to run multiple browser instances in parallel to
-speed up the testing process. This is achieved through the
-``--processes=N`` argument e.g. ``--processes=6`` would attempt to run
-6 browser instances in parallel. Note that behaviour in this mode is
-necessarily less deterministic than with ``--processes=1`` (the
-default) so there may be more noise in the test results.
+To speed up the testing process, use the ``--processes`` option to have
+wptrunner run multiple browser instances in parallel. For example, to
+have wptrunner attempt to run tests against with six browser instances
+in parallel, specify ``--processes=6``. But note that behaviour in this
+mode is necessarily less deterministic than with ``--processes=1`` (the
+default), so there may be more noise in the test results.
+
+-------------------
+Using default paths
+-------------------
 
-Further help can be obtained from::
+The (otherwise-required) ``--tests`` and ``--metadata`` command-line
+options/flags be omitted if any configuration file is found that
+contains a section specifying the ``tests`` and ``metadata`` keys.
 
-  wptrunner --help
+See the `Configuration File`_ section for more information about
+configuration files, including information about their expected
+locations.
+
+The content of the ``wptrunner.default.ini`` default configuration file
+makes wptrunner look for tests (that is, a web-platform-tests checkout)
+as a subdirectory of the current directory named ``tests``, and for
+metadata files in a subdirectory of the current directory named ``meta``.
 
 Output
 ------
 
 wptrunner uses the :py:mod:`mozlog.structured` package for output. This
 structures events such as test results or log messages as JSON objects
 that can then be fed to other tools for interpretation. More details
 about the message format are given in the
--- a/testing/web-platform/harness/setup.py
+++ b/testing/web-platform/harness/setup.py
@@ -7,17 +7,17 @@ import os
 import sys
 import textwrap
 
 from setuptools import setup, find_packages
 
 here = os.path.split(__file__)[0]
 
 PACKAGE_NAME = 'wptrunner'
-PACKAGE_VERSION = '1.4'
+PACKAGE_VERSION = '1.5'
 
 # Dependencies
 with open(os.path.join(here, "requirements.txt")) as f:
     deps = f.read().splitlines()
 
 # Browser-specific requirements
 requirements_files = glob.glob(os.path.join(here, "requirements_*.txt"))
 
--- a/testing/web-platform/harness/wptrunner/config.py
+++ b/testing/web-platform/harness/wptrunner/config.py
@@ -1,39 +1,41 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 import ConfigParser
 import os
 import sys
+from collections import OrderedDict
 
 here = os.path.split(__file__)[0]
 
 class ConfigDict(dict):
     def __init__(self, base_path, *args, **kwargs):
         self.base_path = base_path
         dict.__init__(self, *args, **kwargs)
 
-    def get_path(self, key):
-        pwd = os.path.abspath(os.path.curdir)
+    def get_path(self, key, default=None):
+        if key not in self:
+            return default
         path = self[key]
         os.path.expanduser(path)
-        return os.path.join(self.base_path, path)
+        return os.path.abspath(os.path.join(self.base_path, path))
 
 def read(config_path):
     config_path = os.path.abspath(config_path)
     config_root = os.path.split(config_path)[0]
     parser = ConfigParser.SafeConfigParser()
     success = parser.read(config_path)
     assert config_path in success, success
 
     subns = {"pwd": os.path.abspath(os.path.curdir)}
 
-    rv = {}
+    rv = OrderedDict()
     for section in parser.sections():
         rv[section] = ConfigDict(config_root)
         for key in parser.options(section):
             rv[section][key] = parser.get(section, key, False, subns)
 
     return rv
 
 def path(argv=None):
--- a/testing/web-platform/harness/wptrunner/executors/executorservo.py
+++ b/testing/web-platform/harness/wptrunner/executors/executorservo.py
@@ -28,31 +28,32 @@ class ServoTestharnessExecutor(ProcessTe
         self.command = [self.binary, "--cpu", "--hard-fail",
                         urlparse.urljoin(self.http_server_url, test.url)]
 
         if self.debug_args:
             self.command = list(self.debug_args) + self.command
 
 
         self.proc = ProcessHandler(self.command,
-                                   processOutputLine=[self.on_output])
+                                   processOutputLine=[self.on_output],
+                                   onFinish=self.on_finish)
         self.proc.run()
 
         timeout = test.timeout * self.timeout_multiplier
 
         # Now wait to get the output we expect, or until we reach the timeout
         self.result_flag.wait(timeout + 5)
 
         if self.result_flag.is_set():
             assert self.result_data is not None
             self.result_data["test"] = test.url
             result = self.convert_result(test, self.result_data)
             self.proc.kill()
         else:
-            if self.proc.pid is None:
+            if self.proc.proc.poll() is not None:
                 result = (test.result_cls("CRASH", None), [])
             else:
                 self.proc.kill()
                 result = (test.result_cls("TIMEOUT", None), [])
         self.runner.send_message("test_ended", test, result)
 
     def on_output(self, line):
         prefix = "ALERT: RESULT: "
@@ -62,8 +63,11 @@ class ServoTestharnessExecutor(ProcessTe
             self.result_flag.set()
         else:
             if self.interactive:
                 print line
             else:
                 self.logger.process_output(self.proc.pid,
                                            line,
                                            " ".join(self.command))
+
+    def on_finish(self):
+        self.result_flag.set()
--- a/testing/web-platform/harness/wptrunner/manifestexpected.py
+++ b/testing/web-platform/harness/wptrunner/manifestexpected.py
@@ -1,12 +1,15 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
+import os
+import urlparse
+
 from wptmanifest.backends import static
 from wptmanifest.backends.static import ManifestItem
 
 import expected
 
 """Manifest structure used to store expected results of a test.
 
 Each manifest file is represented by an ExpectedManifest that
@@ -22,31 +25,35 @@ def data_cls_getter(output_node, visited
     if isinstance(output_node, ExpectedManifest):
         return TestNode
     if isinstance(output_node, TestNode):
         return SubtestNode
     raise ValueError
 
 
 class ExpectedManifest(ManifestItem):
-    def __init__(self, name, test_path):
+    def __init__(self, name, test_path, url_base):
         """Object representing all the tests in a particular manifest
 
         :param name: Name of the AST Node associated with this object.
                      Should always be None since this should always be associated with
                      the root node of the AST.
         :param test_path: Path of the test file associated with this manifest.
+        :param url_base: Base url for serving the tests in this manifest
         """
         if name is not None:
             raise ValueError("ExpectedManifest should represent the root node")
         if test_path is None:
             raise ValueError("ExpectedManifest requires a test path")
+        if url_base is None:
+            raise ValueError("ExpectedManifest requires a base url")
         ManifestItem.__init__(self, name)
         self.child_map = {}
         self.test_path = test_path
+        self.url_base = url_base
 
     def append(self, child):
         """Add a test to the manifest"""
         ManifestItem.append(self, child)
         self.child_map[child.id] = child
         assert len(self.child_map) == len(self.children)
 
     def _remove_child(self, child):
@@ -55,16 +62,21 @@ class ExpectedManifest(ManifestItem):
         assert len(self.child_map) == len(self.children)
 
     def get_test(self, test_id):
         """Get a test from the manifest by ID
 
         :param test_id: ID of the test to return."""
         return self.child_map.get(test_id)
 
+    @property
+    def url(self):
+        return urlparse.urljoin(self.url_base,
+                                "/".join(self.test_path.split(os.path.sep)))
+
 
 class TestNode(ManifestItem):
     def __init__(self, name):
         """Tree node associated with a particular test in a manifest
 
         :param name: name of the test"""
         assert name is not None
         ManifestItem.__init__(self, name)
@@ -84,19 +96,17 @@ class TestNode(ManifestItem):
         return all(child.is_empty for child in self.children)
 
     @property
     def test_type(self):
         return self.get("type")
 
     @property
     def id(self):
-        components = self.parent.test_path.split("/")[:-1]
-        components.append(self.name)
-        url = "/" + "/".join(components)
+        url = urlparse.urljoin(self.parent.url, self.name)
         if self.test_type == "reftest":
             return (url, self.get("reftype"), self.get("refurl"))
         else:
             return url
 
     def disabled(self):
         """Boolean indicating whether the test is disabled"""
         try:
@@ -129,25 +139,27 @@ class SubtestNode(TestNode):
 
     @property
     def is_empty(self):
         if self._data:
             return False
         return True
 
 
-def get_manifest(metadata_root, test_path, run_info):
+def get_manifest(metadata_root, test_path, url_base, run_info):
     """Get the ExpectedManifest for a particular test path, or None if there is no
     metadata stored for that test path.
 
     :param metadata_root: Absolute path to the root of the metadata directory
     :param test_path: Path to the test(s) relative to the test root
+    :param url_base: Base url for serving the tests in this manifest
     :param run_info: Dictionary of properties of the test run for which the expectation
                      values should be computed.
     """
     manifest_path = expected.expected_path(metadata_root, test_path)
     try:
         with open(manifest_path) as f:
             return static.compile(f, run_info,
                                   data_cls_getter=data_cls_getter,
-                                  test_path=test_path)
+                                  test_path=test_path,
+                                  url_base=url_base)
     except IOError:
         return None
--- a/testing/web-platform/harness/wptrunner/manifestupdate.py
+++ b/testing/web-platform/harness/wptrunner/manifestupdate.py
@@ -1,13 +1,14 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 import os
+import urlparse
 from collections import namedtuple, defaultdict
 
 from wptmanifest.node import (DataNode, ConditionalNode, BinaryExpressionNode,
                               BinaryOperatorNode, VariableNode, StringNode, NumberNode,
                               UnaryExpressionNode, UnaryOperatorNode, KeyValueNode)
 from wptmanifest.backends import conditional
 from wptmanifest.backends.conditional import ManifestItem
 
@@ -43,28 +44,31 @@ def data_cls_getter(output_node, visited
         return TestNode
     elif isinstance(output_node, TestNode):
         return SubtestNode
     else:
         raise ValueError
 
 
 class ExpectedManifest(ManifestItem):
-    def __init__(self, node, test_path=None):
+    def __init__(self, node, test_path=None, url_base=None):
         """Object representing all the tests in a particular manifest
 
         :param node: AST Node associated with this object. If this is None,
                      a new AST is created to associate with this manifest.
         :param test_path: Path of the test file associated with this manifest.
+        :param url_base: Base url for serving the tests in this manifest
         """
         if node is None:
             node = DataNode(None)
         ManifestItem.__init__(self, node)
         self.child_map = {}
         self.test_path = test_path
+        self.url_base = url_base
+        assert self.url_base is not None
         self.modified = False
 
     def append(self, child):
         ManifestItem.append(self, child)
         self.child_map[child.id] = child
         assert len(self.child_map) == len(self.children)
 
     def _remove_child(self, child):
@@ -82,16 +86,20 @@ class ExpectedManifest(ManifestItem):
     def has_test(self, test_id):
         """Boolean indicating whether the current test has a known child test
         with id test id
 
         :param test_id: The id of the test to look up"""
 
         return test_id in self.child_map
 
+    @property
+    def url(self):
+        return urlparse.urljoin(self.url_base,
+                                "/".join(self.test_path.split(os.path.sep)))
 
 class TestNode(ManifestItem):
     def __init__(self, node):
         """Tree node associated with a particular test in a manifest
 
         :param node: AST node associated with the test"""
 
         ManifestItem.__init__(self, node)
@@ -136,20 +144,17 @@ class TestNode(ManifestItem):
     def test_type(self):
         """The type of the test represented by this TestNode"""
 
         return self.get("type", None)
 
     @property
     def id(self):
         """The id of the test represented by this TestNode"""
-
-        components = self.parent.test_path.split(os.path.sep)[:-1]
-        components.append(self.name)
-        url = "/" + "/".join(components)
+        url = urlparse.urljoin(self.parent.url, self.name)
         if self.test_type == "reftest":
             return (url, self.get("reftype", None), self.get("refurl", None))
         else:
             return url
 
     def disabled(self, run_info):
         """Boolean indicating whether this test is disabled when run in an
         environment with the given run_info
@@ -399,27 +404,29 @@ def make_expr(prop_set, status):
         node = expressions[0]
 
     root.append(node)
     root.append(StringNode(status))
 
     return root
 
 
-def get_manifest(metadata_root, test_path):
+def get_manifest(metadata_root, test_path, url_base):
     """Get the ExpectedManifest for a particular test path, or None if there is no
     metadata stored for that test path.
 
     :param metadata_root: Absolute path to the root of the metadata directory
     :param test_path: Path to the test(s) relative to the test root
+    :param url_base: Base url for serving the tests in this manifest
     """
     manifest_path = expected.expected_path(metadata_root, test_path)
     try:
         with open(manifest_path) as f:
-            return compile(f, test_path)
+            return compile(f, test_path, url_base)
     except IOError:
         return None
 
 
-def compile(manifest_file, test_path):
+def compile(manifest_file, test_path, url_base):
     return conditional.compile(manifest_file,
                                data_cls_getter=data_cls_getter,
-                               test_path=test_path)
+                               test_path=test_path,
+                               url_base=url_base)
--- a/testing/web-platform/harness/wptrunner/metadata.py
+++ b/testing/web-platform/harness/wptrunner/metadata.py
@@ -10,74 +10,69 @@ import types
 import uuid
 from collections import defaultdict
 
 from mozlog.structured import reader
 from mozlog.structured import structuredlog
 
 import expected
 import manifestupdate
+import testloader
 import wptmanifest
 import wpttest
 from vcs import git
 manifest = None  # Module that will be imported relative to test_root
 
 logger = structuredlog.StructuredLogger("web-platform-tests")
 
 
-def manifest_path(metadata_root):
-    return os.path.join(metadata_root, "MANIFEST.json")
-
-
-def load_test_manifest(test_root, metadata_root):
-    do_test_relative_imports(test_root)
-    return manifest.load(manifest_path(metadata_root))
+def load_test_manifests(serve_root, test_paths):
+    do_delayed_imports(serve_root)
+    manifest_loader = testloader.ManifestLoader(test_paths, False)
+    return manifest_loader.load()
 
 
-def update_manifest(git_root, metadata_root):
-    manifest.setup_git(git_root)
-    # Create an entirely new manifest
-    new_manifest = manifest.Manifest(None)
-    manifest.update(new_manifest)
-    manifest.write(new_manifest, manifest_path(metadata_root))
-    return new_manifest
-
-
-def update_expected(test_root, metadata_root, log_file_names, rev_old=None, rev_new="HEAD",
-                    ignore_existing=False, sync_root=None):
+def update_expected(test_paths, serve_root, log_file_names,
+                    rev_old=None, rev_new="HEAD", ignore_existing=False,
+                    sync_root=None):
     """Update the metadata files for web-platform-tests based on
     the results obtained in a previous run"""
 
-    manifest = load_test_manifest(test_root, metadata_root)
+    manifests = load_test_manifests(serve_root, test_paths)
+
+    change_data = {}
 
     if sync_root is not None:
         if rev_old is not None:
-            rev_old = git("rev-parse", rev_old, repo=test_root).strip()
-        rev_new = git("rev-parse", rev_new, repo=test_root).strip()
+            rev_old = git("rev-parse", rev_old, repo=sync_root).strip()
+        rev_new = git("rev-parse", rev_new, repo=sync_root).strip()
+
+        if rev_old is not None:
+            change_data = load_change_data(rev_old, rev_new, repo=sync_root)
+
 
-    if rev_old is not None:
-        change_data = load_change_data(rev_old, rev_new, repo=test_root)
-    else:
-        change_data = {}
+    expected_map_by_manifest = update_from_logs(manifests,
+                                                *log_file_names,
+                                                ignore_existing=ignore_existing)
 
-    expected_map = update_from_logs(metadata_root, manifest, *log_file_names,
-                                    ignore_existing=ignore_existing)
-
-    write_changes(metadata_root, expected_map)
+    for test_manifest, expected_map in expected_map_by_manifest.iteritems():
+        url_base = manifests[test_manifest]["url_base"]
+        metadata_path = test_paths[url_base]["metadata_path"]
+        write_changes(metadata_path, expected_map)
 
     results_changed = [item.test_path for item in expected_map.itervalues() if item.modified]
 
-    return unexpected_changes(change_data, results_changed)
+    return unexpected_changes(manifests, change_data, results_changed)
 
 
-def do_test_relative_imports(test_root):
+def do_delayed_imports(serve_root):
     global manifest
 
-    sys.path.insert(0, os.path.join(test_root))
-    sys.path.insert(0, os.path.join(test_root, "tools", "scripts"))
+    sys.path.insert(0, os.path.join(serve_root))
+    sys.path.insert(0, os.path.join(serve_root, "tools", "scripts"))
     import manifest
 
 
 def files_in_repo(repo_root):
     return git("ls-tree", "-r", "--name-only", "HEAD").split("\n")
 
 
 def rev_range(rev_old, rev_new, symmetric=False):
@@ -100,47 +95,69 @@ def load_change_data(rev_old, rev_new, r
                    "A": "new",
                    "D": "deleted"}
     # TODO: deal with renames
     for item in changes:
         rv[item[1]] = status_keys[item[0]]
     return rv
 
 
-def unexpected_changes(change_data, files_changed):
-    return [fn for fn in files_changed if change_data.get(fn) != "M"]
+def unexpected_changes(manifests, change_data, files_changed):
+    files_changed = set(files_changed)
+
+    root_manifest = None
+    for manifest, paths in manifests.iteritems():
+        if paths["url_base"] == "/":
+            root_manifest = manifest
+            break
+    else:
+        return []
+
+    rv = []
+
+    return [fn for fn, tests in root_manifest if fn in files_changed and change_data.get(fn) != "M"]
 
 # For each testrun
 # Load all files and scan for the suite_start entry
 # Build a hash of filename: properties
 # For each different set of properties, gather all chunks
 # For each chunk in the set of chunks, go through all tests
 # for each test, make a map of {conditionals: [(platform, new_value)]}
 # Repeat for each platform
 # For each test in the list of tests:
 #   for each conditional:
 #      If all the new values match (or there aren't any) retain that conditional
 #      If any new values mismatch mark the test as needing human attention
 #   Check if all the RHS values are the same; if so collapse the conditionals
 
 
-def update_from_logs(metadata_path, manifest, *log_filenames, **kwargs):
+def update_from_logs(manifests, *log_filenames, **kwargs):
     ignore_existing = kwargs.pop("ignore_existing", False)
 
-    expected_map, id_path_map = create_test_tree(metadata_path, manifest)
-    updater = ExpectedUpdater(expected_map, id_path_map, ignore_existing=ignore_existing)
+    expected_map = {}
+    id_test_map = {}
+
+    for test_manifest, paths in manifests.iteritems():
+        expected_map_manifest, id_path_map_manifest = create_test_tree(paths["metadata_path"],
+                                                                       test_manifest)
+        expected_map[test_manifest] = expected_map_manifest
+        id_test_map.update(id_path_map_manifest)
+
+    updater = ExpectedUpdater(manifests, expected_map, id_test_map,
+                              ignore_existing=ignore_existing)
     for log_filename in log_filenames:
         with open(log_filename) as f:
             updater.update_from_log(f)
 
-    for tree in expected_map.itervalues():
-        for test in tree.iterchildren():
-            for subtest in test.iterchildren():
-                subtest.coalesce_expected()
-            test.coalesce_expected()
+    for manifest_expected in expected_map.itervalues():
+        for tree in manifest_expected.itervalues():
+            for test in tree.iterchildren():
+                for subtest in test.iterchildren():
+                    subtest.coalesce_expected()
+                test.coalesce_expected()
 
     return expected_map
 
 
 def write_changes(metadata_path, expected_map):
     # First write the new manifest files to a temporary directory
     temp_path = tempfile.mkdtemp(dir=os.path.split(metadata_path)[0])
     write_new_expected(temp_path, expected_map)
@@ -165,17 +182,18 @@ def write_new_expected(metadata_path, ex
             dir = os.path.split(path)[0]
             if not os.path.exists(dir):
                 os.makedirs(dir)
             with open(path, "w") as f:
                 f.write(manifest_str.encode("utf8"))
 
 
 class ExpectedUpdater(object):
-    def __init__(self, expected_tree, id_path_map, ignore_existing=False):
+    def __init__(self, test_manifests, expected_tree, id_path_map, ignore_existing=False):
+        self.test_manifests = test_manifests
         self.expected_tree = expected_tree
         self.id_path_map = id_path_map
         self.ignore_existing = ignore_existing
         self.run_info = None
         self.action_map = {"suite_start": self.suite_start,
                            "test_start": self.test_start,
                            "test_status": self.test_status,
                            "test_end": self.test_end}
@@ -195,25 +213,26 @@ class ExpectedUpdater(object):
         if type(id) in types.StringTypes:
             return id
         else:
             return tuple(id)
 
     def test_start(self, data):
         test_id = self.test_id(data["test"])
         try:
-            test = self.expected_tree[self.id_path_map[test_id]].get_test(test_id)
+            test_manifest, test = self.id_path_map[test_id]
+            expected_node = self.expected_tree[test_manifest][test].get_test(test_id)
         except KeyError:
             print "Test not found %s, skipping" % test_id
             return
-        self.test_cache[test_id] = test
+        self.test_cache[test_id] = expected_node
 
         if test_id not in self.tests_visited:
             if self.ignore_existing:
-                test.clear_expected()
+                expected_node.clear_expected()
             self.tests_visited[test_id] = set()
 
     def test_status(self, data):
         test_id = self.test_id(data["test"])
         test = self.test_cache.get(test_id)
         if test is None:
             return
         test_cls = wpttest.manifest_test_cls[test.test_type]
@@ -242,44 +261,44 @@ class ExpectedUpdater(object):
         result = test_cls.result_cls(
             data["status"],
             data.get("message"))
 
         test.set_result(self.run_info, result)
         del self.test_cache[test_id]
 
 
-def create_test_tree(metadata_path, manifest):
+def create_test_tree(metadata_path, test_manifest):
     expected_map = {}
-    test_id_path_map = {}
+    id_test_map = {}
     exclude_types = frozenset(["stub", "helper", "manual"])
     include_types = set(manifest.item_types) ^ exclude_types
-    for test_path, tests in manifest.itertypes(*include_types):
-
-        expected_data = load_expected(metadata_path, test_path, tests)
+    for test_path, tests in test_manifest.itertypes(*include_types):
+        expected_data = load_expected(test_manifest, metadata_path, test_path, tests)
         if expected_data is None:
-            expected_data = create_expected(test_path, tests)
-
-        expected_map[test_path] = expected_data
+            expected_data = create_expected(test_manifest, test_path, tests)
 
         for test in tests:
-            test_id_path_map[test.id] = test_path
+            id_test_map[test.id] = (test_manifest, test)
+            expected_map[test] = expected_data
 
-    return expected_map, test_id_path_map
+    return expected_map, id_test_map
 
 
-def create_expected(test_path, tests):
-    expected = manifestupdate.ExpectedManifest(None, test_path)
+def create_expected(test_manifest, test_path, tests):
+    expected = manifestupdate.ExpectedManifest(None, test_path, test_manifest.url_base)
     for test in tests:
         expected.append(manifestupdate.TestNode.create(test.item_type, test.id))
     return expected
 
 
-def load_expected(metadata_path, test_path, tests):
-    expected_manifest = manifestupdate.get_manifest(metadata_path, test_path)
+def load_expected(test_manifest, metadata_path, test_path, tests):
+    expected_manifest = manifestupdate.get_manifest(metadata_path,
+                                                    test_path,
+                                                    test_manifest.url_base)
     if expected_manifest is None:
         return
 
     tests_by_id = {item.id: item for item in tests}
 
     # Remove expected data for tests that no longer exist
     for test in expected_manifest.iterchildren():
         if not test.id in tests_by_id:
new file mode 100644
--- /dev/null
+++ b/testing/web-platform/harness/wptrunner/testloader.py
@@ -0,0 +1,421 @@
+import os
+import urlparse
+from abc import ABCMeta, abstractmethod
+from Queue import Empty
+from collections import defaultdict, OrderedDict
+from multiprocessing import Queue
+
+import manifestinclude
+import manifestexpected
+import wpttest
+from mozlog import structured
+
+manifest = None
+
+def do_delayed_imports():
+    # This relies on an already loaded module having set the sys.path correctly :(
+    global manifest
+    import manifest
+
+class TestChunker(object):
+    def __init__(self, total_chunks, chunk_number):
+        self.total_chunks = total_chunks
+        self.chunk_number = chunk_number
+        assert self.chunk_number <= self.total_chunks
+
+    def __call__(self, manifest):
+        raise NotImplementedError
+
+
+class Unchunked(TestChunker):
+    def __init__(self, *args, **kwargs):
+        TestChunker.__init__(self, *args, **kwargs)
+        assert self.total_chunks == 1
+
+    def __call__(self, manifest):
+        for item in manifest:
+            yield item
+
+
+class HashChunker(TestChunker):
+    def __call__(self):
+        chunk_index = self.chunk_number - 1
+        for test_path, tests in manifest:
+            if hash(test_path) % self.total_chunks == chunk_index:
+                yield test_path, tests
+
+
+class EqualTimeChunker(TestChunker):
+    """Chunker that uses the test timeout as a proxy for the running time of the test"""
+
+    def _get_chunk(self, manifest_items):
+        # For each directory containing tests, calculate the maximum execution time after running all
+        # the tests in that directory. Then work out the index into the manifest corresponding to the
+        # directories at fractions of m/N of the running time where m=1..N-1 and N is the total number
+        # of chunks. Return an array of these indicies
+
+        total_time = 0
+        by_dir = OrderedDict()
+
+        class PathData(object):
+            def __init__(self, path):
+                self.path = path
+                self.time = 0
+                self.tests = []
+
+        class Chunk(object):
+            def __init__(self):
+                self.paths = []
+                self.tests = []
+                self.time = 0
+
+            def append(self, path_data):
+                self.paths.append(path_data.path)
+                self.tests.extend(path_data.tests)
+                self.time += path_data.time
+
+        class ChunkList(object):
+            def __init__(self, total_time, n_chunks):
+                self.total_time = total_time
+                self.n_chunks = n_chunks
+
+                self.remaining_chunks = n_chunks
+
+                self.chunks = []
+
+                self.update_time_per_chunk()
+
+            def __iter__(self):
+                for item in self.chunks:
+                    yield item
+
+            def __getitem__(self, i):
+                return self.chunks[i]
+
+            def sort_chunks(self):
+                self.chunks = sorted(self.chunks, key=lambda x:x.paths[0])
+
+            def get_tests(self, chunk_number):
+                return self[chunk_number - 1].tests
+
+            def append(self, chunk):
+                if len(self.chunks) == self.n_chunks:
+                    raise ValueError("Tried to create more than %n chunks" % self.n_chunks)
+                self.chunks.append(chunk)
+                self.remaining_chunks -= 1
+
+            @property
+            def current_chunk(self):
+                if self.chunks:
+                    return self.chunks[-1]
+
+            def update_time_per_chunk(self):
+                self.time_per_chunk = (self.total_time - sum(item.time for item in self)) / self.remaining_chunks
+
+            def create(self):
+                rv = Chunk()
+                self.append(rv)
+                return rv
+
+            def add_path(self, path_data):
+                sum_time = self.current_chunk.time + path_data.time
+                if sum_time > self.time_per_chunk and self.remaining_chunks > 0:
+                    overshoot = sum_time - self.time_per_chunk
+                    undershoot = self.time_per_chunk - self.current_chunk.time
+                    if overshoot < undershoot:
+                        self.create()
+                        self.current_chunk.append(path_data)
+                    else:
+                        self.current_chunk.append(path_data)
+                        self.create()
+                else:
+                    self.current_chunk.append(path_data)
+
+        for i, (test_path, tests) in enumerate(manifest_items):
+            test_dir = tuple(os.path.split(test_path)[0].split(os.path.sep)[:3])
+
+            if not test_dir in by_dir:
+                by_dir[test_dir] = PathData(test_dir)
+
+            data = by_dir[test_dir]
+            time = sum(wpttest.DEFAULT_TIMEOUT if test.timeout !=
+                       "long" else wpttest.LONG_TIMEOUT for test in tests)
+            data.time += time
+            data.tests.append((test_path, tests))
+
+            total_time += time
+
+        chunk_list = ChunkList(total_time, self.total_chunks)
+
+        if len(by_dir) < self.total_chunks:
+            raise ValueError("Tried to split into %i chunks, but only %i subdirectories included" % (
+                self.total_chunks, len(by_dir)))
+
+        # Put any individual dirs with a time greater than the time per chunk into their own
+        # chunk
+        while True:
+            to_remove = []
+            for path_data in by_dir.itervalues():
+                if path_data.time > chunk_list.time_per_chunk:
+                    to_remove.append(path_data)
+            if to_remove:
+                for path_data in to_remove:
+                    chunk = chunk_list.create()
+                    chunk.append(path_data)
+                    del by_dir[path_data.path]
+                chunk_list.update_time_per_chunk()
+            else:
+                break
+
+        chunk = chunk_list.create()
+        for path_data in by_dir.itervalues():
+            chunk_list.add_path(path_data)
+
+        assert len(chunk_list.chunks) == self.total_chunks, len(chunk_list.chunks)
+        assert sum(item.time for item in chunk_list) == chunk_list.total_time
+
+        chunk_list.sort_chunks()
+
+        return chunk_list.get_tests(self.chunk_number)
+
+    def __call__(self, manifest_iter):
+        manifest = list(manifest_iter)
+        tests = self._get_chunk(manifest)
+        for item in tests:
+            yield item
+
+
+class TestFilter(object):
+    def __init__(self, include=None, exclude=None, manifest_path=None):
+        if manifest_path is not None and include is None:
+            self.manifest = manifestinclude.get_manifest(manifest_path)
+        else:
+            self.manifest = manifestinclude.IncludeManifest.create()
+
+        if include is not None:
+            self.manifest.set("skip", "true")
+            for item in include:
+                self.manifest.add_include(item)
+
+        if exclude is not None:
+            for item in exclude:
+                self.manifest.add_exclude(item)
+
+    def __call__(self, manifest_iter):
+        for test_path, tests in manifest_iter:
+            include_tests = set()
+            for test in tests:
+                if self.manifest.include(test):
+                    include_tests.add(test)
+
+            if include_tests:
+                yield test_path, include_tests
+
+
+class ManifestLoader(object):
+
+    def __init__(self, test_paths, force_manifest_update=False):
+        do_delayed_imports()
+        self.test_paths = test_paths
+        self.force_manifest_update = force_manifest_update
+        self.logger = structured.get_default_logger()
+        if self.logger is None:
+            self.logger = structured.structuredlog.StructuredLogger("ManifestLoader")
+
+    def load(self):
+        rv = {}
+        for url_base, paths in self.test_paths.iteritems():
+            manifest_file = self.load_manifest(url_base=url_base,
+                                               **paths)
+            path_data = {"url_base": url_base}
+            path_data.update(paths)
+            rv[manifest_file] = path_data
+        return rv
+
+    def create_manifest(self, manifest_path, tests_path, url_base="/"):
+        self.logger.info("Creating test manifest %s" % manifest_path)
+        manifest_file = manifest.Manifest(None, url_base)
+        manifest.update(tests_path, url_base, manifest_file)
+        manifest.write(manifest_file, manifest_path)
+
+    def load_manifest(self, tests_path, metadata_path, url_base="/"):
+        manifest_path = os.path.join(metadata_path, "MANIFEST.json")
+        if (not os.path.exists(manifest_path) or
+            self.force_manifest_update):
+            self.create_manifest(manifest_path, tests_path, url_base)
+        manifest_file = manifest.load(manifest_path)
+        if manifest_file.url_base != url_base:
+            self.logger.info("Updating url_base in manifest from %s to %s" % (manifest_file.url_base,
+                                                                              url_base))
+            manifest_file.url_base = url_base
+            manifest.write(manifest_file, manifest_path)
+
+        return manifest_file
+
+class TestLoader(object):
+    def __init__(self,
+                 test_paths,
+                 test_types,
+                 test_filter,
+                 run_info,
+                 chunk_type="none",
+                 total_chunks=1,
+                 chunk_number=1,
+                 force_manifest_update=False):
+
+        self.test_paths = test_paths
+        self.test_types = test_types
+        self.test_filter = test_filter
+        self.run_info = run_info
+        self.manifests = ManifestLoader(test_paths, force_manifest_update).load()
+        self.tests = None
+        self.disabled_tests = None
+
+        self.chunk_type = chunk_type
+        self.total_chunks = total_chunks
+        self.chunk_number = chunk_number
+
+        self.chunker = {"none": Unchunked,
+                        "hash": HashChunker,
+                        "equal_time": EqualTimeChunker}[chunk_type](total_chunks,
+                                                                    chunk_number)
+
+        self._test_ids = None
+        self._load_tests()
+
+    @property
+    def test_ids(self):
+        if self._test_ids is None:
+            self._test_ids = []
+            for test_dict in [self.disabled_tests, self.tests]:
+                for test_type in self.test_types:
+                    self._test_ids += [item.id for item in test_dict[test_type]]
+        return self._test_ids
+
+    def get_test(self, manifest_test, expected_file):
+        if expected_file is not None:
+            expected = expected_file.get_test(manifest_test.id)
+        else:
+            expected = None
+        return wpttest.from_manifest(manifest_test, expected)
+
+    def load_expected_manifest(self, test_manifest, metadata_path, test_path):
+        return manifestexpected.get_manifest(metadata_path, test_path, test_manifest.url_base, self.run_info)
+
+    def iter_tests(self):
+        manifest_items = []
+
+        for manifest in self.manifests.keys():
+            manifest_items.extend(self.test_filter(manifest.itertypes(*self.test_types)))
+
+        if self.chunker is not None:
+            manifest_items = self.chunker(manifest_items)
+
+        for test_path, tests in manifest_items:
+            manifest_file = iter(tests).next().manifest
+            metadata_path = self.manifests[manifest_file]["metadata_path"]
+            expected_file = self.load_expected_manifest(manifest_file, metadata_path, test_path)
+
+            for manifest_test in tests:
+                test = self.get_test(manifest_test, expected_file)
+                test_type = manifest_test.item_type
+                yield test_path, test_type, test
+
+    def _load_tests(self):
+        """Read in the tests from the manifest file and add them to a queue"""
+        tests = {"enabled":defaultdict(list),
+                 "disabled":defaultdict(list)}
+
+        for test_path, test_type, test in self.iter_tests():
+            key = "enabled" if not test.disabled() else "disabled"
+            tests[key][test_type].append(test)
+
+        self.tests = tests["enabled"]
+        self.disabled_tests = tests["disabled"]
+
+    def groups(self, test_types, chunk_type="none", total_chunks=1, chunk_number=1):
+        groups = set()
+
+        for test_type in test_types:
+            for test in self.tests[test_type]:
+                group = test.url.split("/")[1]
+                groups.add(group)
+
+        return groups
+
+
+class TestSource(object):
+    __metaclass__ = ABCMeta
+
+    @abstractmethod
+    def queue_tests(self, test_queue):
+        pass
+
+    @abstractmethod
+    def requeue_test(self, test):
+        pass
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args, **kwargs):
+        pass
+
+
+class SingleTestSource(TestSource):
+    def __init__(self, test_queue):
+        self.test_queue = test_queue
+
+    @classmethod
+    def queue_tests(cls, test_queue, test_type, tests):
+        for test in tests[test_type]:
+            test_queue.put(test)
+
+    def get_queue(self):
+        if self.test_queue.empty():
+            return None
+        return self.test_queue
+
+    def requeue_test(self, test):
+        self.test_queue.put(test)
+
+class PathGroupedSource(TestSource):
+    def __init__(self, test_queue):
+        self.test_queue = test_queue
+        self.current_queue = None
+
+    @classmethod
+    def queue_tests(cls, test_queue, test_type, tests, depth=None):
+        if depth is True:
+            depth = None
+
+        prev_path = None
+        group = None
+
+        for test in tests[test_type]:
+            path = urlparse.urlsplit(test.url).path.split("/")[1:-1][:depth]
+            if path != prev_path:
+                group = []
+                test_queue.put(group)
+                prev_path = path
+
+            group.append(test)
+
+    def get_queue(self):
+        if not self.current_queue or self.current_queue.empty():
+            try:
+                data = self.test_queue.get(block=True, timeout=1)
+                self.current_queue = Queue()
+                for item in data:
+                    self.current_queue.put(item)
+            except Empty:
+                return None
+
+        return self.current_queue
+
+    def requeue_test(self, test):
+        self.current_queue.put(test)
+
+    def __exit__(self, *args, **kwargs):
+        if self.current_queue:
+            self.current_queue.close()
--- a/testing/web-platform/harness/wptrunner/testrunner.py
+++ b/testing/web-platform/harness/wptrunner/testrunner.py
@@ -532,27 +532,38 @@ class TestRunnerManager(threading.Thread
 
 class TestQueue(object):
     def __init__(self, test_source_cls, test_type, tests, **kwargs):
         self.queue = None
         self.test_source_cls = test_source_cls
         self.test_type = test_type
         self.tests = tests
         self.kwargs = kwargs
+        self.queue = None
 
     def __enter__(self):
+        if not self.tests[self.test_type]:
+            return None
+
         self.queue = Queue()
-        self.test_source_cls.queue_tests(self.queue,
-                                         self.test_type,
-                                         self.tests,
-                                         **self.kwargs)
+        has_tests = self.test_source_cls.queue_tests(self.queue,
+                                                     self.test_type,
+                                                     self.tests,
+                                                     **self.kwargs)
+        # There is a race condition that means sometimes we continue
+        # before the tests have been written to the underlying pipe.
+        # Polling the pipe for data here avoids that
+        self.queue._reader.poll(10)
+        assert not self.queue.empty()
         return self.queue
 
     def __exit__(self, *args, **kwargs):
-        self.queue.close()
+        if self.queue is not None:
+            self.queue.close()
+            self.queue = None
 
 
 class ManagerGroup(object):
     def __init__(self, suite_name, size, test_source_cls, test_source_kwargs,
                  browser_cls, browser_kwargs,
                  executor_cls, executor_kwargs, pause_on_unexpected=False):
         """Main thread object that owns all the TestManager threads."""
         self.suite_name = suite_name
@@ -581,16 +592,19 @@ class ManagerGroup(object):
         """Start all managers in the group"""
         self.logger.debug("Using %i processes" % self.size)
 
         self.test_queue = TestQueue(self.test_source_cls,
                                     test_type,
                                     tests,
                                     **self.test_source_kwargs)
         with self.test_queue as test_queue:
+            if test_queue is None:
+                self.logger.info("No %s tests to run" % test_type)
+                return
             for _ in range(self.size):
                 manager = TestRunnerManager(self.suite_name,
                                             test_queue,
                                             self.test_source_cls,
                                             self.browser_cls,
                                             self.browser_kwargs,
                                             self.executor_cls,
                                             self.executor_kwargs,
--- a/testing/web-platform/harness/wptrunner/update.py
+++ b/testing/web-platform/harness/wptrunner/update.py
@@ -8,16 +8,17 @@ import subprocess
 import sys
 import traceback
 import uuid
 
 import vcs
 from vcs import git, hg
 manifest = None
 import metadata
+import testloader
 import wptcommandline
 
 base_path = os.path.abspath(os.path.split(__file__)[0])
 
 bsd_license = """W3C 3-clause BSD License
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are
@@ -43,21 +44,19 @@ LIABLE FOR ANY DIRECT, INDIRECT, INCIDEN
 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGE.
 """
 
-def do_test_relative_imports(test_root):
+def do_delayed_imports(serve_root):
     global manifest
-
-    sys.path.insert(0, os.path.join(test_root))
-    sys.path.insert(0, os.path.join(test_root, "tools", "scripts"))
+    sys.path.insert(0, os.path.join(serve_root, "tools", "scripts"))
     import manifest
 
 
 class RepositoryError(Exception):
     pass
 
 
 class WebPlatformTests(object):
@@ -306,34 +305,41 @@ class Runner(object):
 def ensure_exists(path):
     if not os.path.exists(path):
         os.makedirs(path)
 
 
 def sync_tests(paths, local_tree, wpt, bug):
     wpt.update()
 
+    do_delayed_imports(paths["sync"])
+
     try:
         #bug.comment("Updating to %s" % wpt.rev)
-        initial_manifest = metadata.load_test_manifest(paths["sync"], paths["metadata"])
-        wpt.copy_work_tree(paths["test"])
-        new_manifest = metadata.update_manifest(paths["sync"], paths["metadata"])
+        sync_paths = {"/": {"tests_path": paths["sync"],
+                            "metadata_path": paths["sync_dest"]["metadata_path"]}}
+
+        manifest_loader = testloader.ManifestLoader(sync_paths)
+        initial_manifests = manifest_loader.load()
+        wpt.copy_work_tree(paths["sync_dest"]["tests_path"])
 
         local_tree.create_patch("web-platform-tests_update_%s" % wpt.rev,
                                 "Update web-platform-tests to revision %s" % wpt.rev)
-        local_tree.add_new(os.path.relpath(paths["test"], local_tree.root))
-        local_tree.update_patch(include=[paths["test"], paths["metadata"]])
+        local_tree.add_new(os.path.relpath(paths["sync_dest"]["tests_path"],
+                                           local_tree.root))
+        local_tree.update_patch(include=[paths["sync_dest"]["tests_path"],
+                                         paths["sync_dest"]["metadata_path"]])
     except Exception as e:
         #bug.comment("Update failed with error:\n %s" % traceback.format_exc())
         sys.stderr.write(traceback.format_exc())
         raise
     finally:
         pass  # wpt.clean()
 
-    return initial_manifest, new_manifest
+    return initial_manifests
 
 
 def update_metadata(paths, local_tree, initial_rev, bug, log_files, ignore_existing,
                     wpt_repo=None):
     try:
         try:
             if wpt_repo is not None:
                 name = "web-platform-tests_update_%s_metadata" % wpt_repo.rev
@@ -341,48 +347,55 @@ def update_metadata(paths, local_tree, i
             else:
                 name = "web-platform-tests_update_metadata"
                 message = "Update web-platform-tests expected data"
 
             local_tree.create_patch(name, message)
         except subprocess.CalledProcessError:
             # Patch with that name already exists, probably
             pass
-        needs_human = metadata.update_expected(paths["test"],
-                                               paths["metadata"],
+        needs_human = metadata.update_expected(paths["test_paths"],
+                                               paths["serve"],
                                                log_files,
                                                rev_old=initial_rev,
                                                ignore_existing=ignore_existing,
                                                sync_root=paths.get("sync", None))
 
         if needs_human:
             #TODO: List all the files that should be checked carefully for changes.
             pass
 
         if not local_tree.is_clean():
-            local_tree.add_new(os.path.relpath(paths["metadata"], local_tree.root))
-            local_tree.update_patch(include=[paths["metadata"]])
+            metadata_paths = [manifest_path["metadata_path"]
+                              for manifest_path in paths["test_paths"].itervalues()]
+            for path in metadata_paths:
+                local_tree.add_new(os.path.relpath(path, local_tree.root))
+            local_tree.update_patch(include=metadata_paths)
 
     except Exception as e:
         #bug.comment("Update failed with error:\n %s" % traceback.format_exc())
         sys.stderr.write(traceback.format_exc())
         raise
 
 
 def run_update(**kwargs):
     config = kwargs["config"]
 
-    paths = {"test": kwargs["tests_root"],
-             "metadata": kwargs["metadata_root"]}
+    paths = {}
+    paths["test_paths"] = kwargs["test_paths"]
 
     if kwargs["sync"]:
+        paths["sync_dest"] = kwargs["test_paths"]["/"]
         paths["sync"] = kwargs["sync_path"]
 
-    for path in paths.itervalues():
-        ensure_exists(path)
+
+    paths["serve"] = kwargs["serve_root"] if kwargs["serve_root"] else paths["test_paths"]["/"]["tests_path"]
+
+#    for path in paths.itervalues():
+#        ensure_exists(path)
 
     if not kwargs["sync"] and not kwargs["run_log"]:
         print """Nothing to do.
 
 Specify --sync to checkout latest upstream or one or more log files to update
 expected data."""
 
     if kwargs["patch"]:
@@ -407,18 +420,22 @@ expected data."""
 
     initial_rev = None
     wpt_repo = None
 
     if kwargs["sync"]:
         wpt_repo = WebPlatformTests(config["web-platform-tests"]["remote_url"],
                                     paths["sync"],
                                     rev=rev)
-        initial_manifest, new_manifest = sync_tests(paths, local_tree, wpt_repo, bug)
-        initial_rev = initial_manifest.rev
+        initial_manifests = sync_tests(paths, local_tree, wpt_repo, bug)
+        initial_rev = None
+        for manifest, path_data in initial_manifests.iteritems():
+            if path_data["url_base"] == "/":
+                initial_rev = manifest.rev
+                break
 
     if kwargs["run_log"]:
         update_metadata(paths,
                         local_tree,
                         initial_rev,
                         bug,
                         kwargs["run_log"],
                         kwargs["ignore_existing"],
--- a/testing/web-platform/harness/wptrunner/wptcommandline.py
+++ b/testing/web-platform/harness/wptrunner/wptcommandline.py
@@ -1,15 +1,17 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 import argparse
+import ast
 import os
 import sys
+from collections import OrderedDict
 
 import config
 
 def abs_path(path):
     return os.path.abspath(os.path.expanduser(path))
 
 def url_or_path(path):
     import urlparse
@@ -43,21 +45,30 @@ def create_parser(product_choices=None):
     if product_choices is None:
         config_data = config.load()
         product_choices = products.products_enabled(config_data)
 
     parser = argparse.ArgumentParser(description="Runner for web-platform-tests tests.")
     parser.add_argument("--metadata", action="store", type=abs_path, dest="metadata_root",
                         help="Path to the folder containing test metadata"),
     parser.add_argument("--tests", action="store", type=abs_path, dest="tests_root",
-                        help="Path to web-platform-tests"),
+                        help="Path to test files"),
     parser.add_argument("--prefs-root", dest="prefs_root", action="store", type=abs_path,
                         help="Path to the folder containing browser prefs"),
-    parser.add_argument("--config", action="store", type=abs_path,
+    parser.add_argument("--serve-root", action="store", type=abs_path, dest="serve_root",
+                        help="Path to web-platform-tests checkout containing serve.py and manifest.py"
+                        " (defaults to test_root)")
+    parser.add_argument("--run-info", action="store", type=abs_path,
+                        help="Path to directory containing extra json files to add to run info")
+    parser.add_argument("--config", action="store", type=abs_path, dest="config",
                         help="Path to config file")
+
+    parser.add_argument("--manifest-update", action="store_true", default=False,
+                        help="Force regeneration of the test manifest")
+
     parser.add_argument("--binary", action="store",
                         type=abs_path, help="Binary to run tests against")
     parser.add_argument("--test-types", action="store",
                         nargs="*", default=["testharness", "reftest"],
                         choices=["testharness", "reftest"],
                         help="Test types to run")
     parser.add_argument("--processes", action="store", type=int, default=1,
                         help="Number of simultaneous processes to use")
@@ -113,52 +124,91 @@ def create_parser(product_choices=None):
                         help="Don't backup device before testrun with --product=b2g")
 
     commandline.add_logging_group(parser)
     return parser
 
 
 def set_from_config(kwargs):
     if kwargs["config"] is None:
-        kwargs["config"] = config.path()
+        config_path = config.path()
+    else:
+        config_path = kwargs["config"]
 
-    kwargs["config"] = config.read(kwargs["config"])
+    kwargs["config_path"] = config_path
+    kwargs["config"] = config.read(kwargs["config_path"])
+    kwargs["test_paths"] = OrderedDict()
 
-    keys = {"paths": [("tests", "tests_root", True),
-                      ("metadata", "metadata_root", True)],
+    keys = {"paths": [("serve", "serve_root", True),
+                      ("prefs", "prefs_root", True),
+                      ("run_info", "run_info", True)],
             "web-platform-tests": [("remote_url", "remote_url", False),
                                    ("branch", "branch", False),
                                    ("sync_path", "sync_path", True)]}
 
     for section, values in keys.iteritems():
         for config_value, kw_value, is_path in values:
             if kw_value in kwargs and kwargs[kw_value] is None:
                 if not is_path:
-                    new_value = kwargs["config"].get(section, {}).get(config_value, None)
+                    new_value = kwargs["config"].get(section, {}).get(config_value)
                 else:
                     new_value = kwargs["config"].get(section, {}).get_path(config_value)
                 kwargs[kw_value] = new_value
 
+    # Set up test_paths
+
+    for section in kwargs["config"].iterkeys():
+        if section.startswith("manifest:"):
+            manifest_opts = kwargs["config"].get(section)
+            url_base = manifest_opts.get("url_base", "/")
+            kwargs["test_paths"][url_base] = {
+                "tests_path": manifest_opts.get_path("tests"),
+                "metadata_path": manifest_opts.get_path("metadata")}
+
+    if kwargs["tests_root"]:
+        if "/" not in kwargs["test_paths"]:
+            kwargs["test_paths"]["/"] = {}
+        kwargs["test_paths"]["/"]["tests_path"] = kwargs["tests_root"]
+
+    if kwargs["metadata_root"]:
+        if "/" not in kwargs["test_paths"]:
+            kwargs["test_paths"]["/"] = {}
+        kwargs["test_paths"]["/"]["metadata_path"] = kwargs["metadata_root"]
+
 
 def check_args(kwargs):
     from mozrunner import cli
 
     set_from_config(kwargs)
 
-    for key in ["tests_root", "metadata_root"]:
-        name = key.split("_", 1)[0]
-        path = kwargs[key]
+    for test_paths in kwargs["test_paths"].itervalues():
+        if not ("tests_path" in test_paths and
+                "metadata_path" in test_paths):
+            print "Fatal: must specify both a test path and metadata path"
+            sys.exit(1)
+        for key, path in test_paths.iteritems():
+            name = key.split("_", 1)[0]
+
+            if not os.path.exists(path):
+                print "Fatal: %s path %s does not exist" % (name, path)
+                sys.exit(1)
 
-        if not os.path.exists(path):
-            print "Fatal: %s path %s does not exist" % (name, path)
-            sys.exit(1)
+            if not os.path.isdir(path):
+                print "Fatal: %s path %s is not a directory" % (name, path)
+                sys.exit(1)
 
-        if not os.path.isdir(path):
-            print "Fatal: %s path %s is not a directory" % (name, path)
-            sys.exit(1)
+    if kwargs["serve_root"] is None:
+        if "/" in kwargs["test_paths"]:
+            kwargs["serve_root"] = kwargs["test_paths"]["/"]["tests_path"]
+    else:
+        print >> sys.stderr, "Unable to determine server root path"
+        sys.exit(1)
+
+    if kwargs["run_info"] is None:
+        kwargs["run_info"] = kwargs["config_path"]
 
     if kwargs["this_chunk"] > 1:
         require_arg(kwargs, "total_chunks", lambda x: x >= kwargs["this_chunk"])
 
     if kwargs["chunk_type"] is None:
         if kwargs["total_chunks"] > 1:
             kwargs["chunk_type"] = "equal_time"
         else:
@@ -182,27 +232,30 @@ def check_args(kwargs):
             sys.exit(1)
 
     return kwargs
 
 
 def create_parser_update():
     parser = argparse.ArgumentParser("web-platform-tests-update",
                                      description="Update script for web-platform-tests tests.")
+    parser.add_argument("--config", action="store", type=abs_path, help="Path to config file")
     parser.add_argument("--metadata", action="store", type=abs_path, dest="metadata_root",
                         help="Path to the folder containing test metadata"),
     parser.add_argument("--tests", action="store", type=abs_path, dest="tests_root",
                         help="Path to web-platform-tests"),
     parser.add_argument("--sync-path", action="store", type=abs_path,
                         help="Path to store git checkout of web-platform-tests during update"),
+    parser.add_argument("--serve-root", action="store", type=abs_path, dest="serve_root",
+                        help="Path to web-platform-tests checkout containing serve.py and manifest.py"
+                        " (defaults to test_root)")
     parser.add_argument("--remote_url", action="store",
                         help="URL of web-platfrom-tests repository to sync against"),
     parser.add_argument("--branch", action="store", type=abs_path,
                         help="Remote branch to sync against")
-    parser.add_argument("--config", action="store", type=abs_path, help="Path to config file")
     parser.add_argument("--rev", action="store", help="Revision to sync to")
     parser.add_argument("--no-check-clean", action="store_true", default=False,
                         help="Don't check the working directory is clean before updating")
     parser.add_argument("--patch", action="store_true",
                         help="Create an mq patch or git commit containing the changes.")
     parser.add_argument("--sync", dest="sync", action="store_true", default=False,
                         help="Sync the tests with the latest from upstream")
     parser.add_argument("--ignore-existing", action="store_true", help="When updating test results only consider results from the logfiles provided, not existing expectations.")
--- a/testing/web-platform/harness/wptrunner/wptrunner.py
+++ b/testing/web-platform/harness/wptrunner/wptrunner.py
@@ -10,25 +10,23 @@ import os
 import shutil
 import socket
 import sys
 import threading
 import time
 import urlparse
 from Queue import Empty
 from StringIO import StringIO
-from abc import ABCMeta, abstractmethod
-from collections import defaultdict, OrderedDict
+
 from multiprocessing import Queue
 
 from mozlog.structured import commandline, stdadapter
 
-import manifestexpected
-import manifestinclude
 import products
+import testloader
 import wptcommandline
 import wpttest
 from testrunner import ManagerGroup
 
 here = os.path.split(__file__)[0]
 
 
 """Runner for web-platform-tests
@@ -63,90 +61,120 @@ def setup_logging(args, defaults):
     return logger
 
 
 def setup_stdlib_logger():
     logging.root.handlers = []
     logging.root = stdadapter.std_logging_adapter(logging.root)
 
 
-def do_test_relative_imports(test_root):
+def do_delayed_imports(serve_root):
     global serve, manifest
 
-    sys.path.insert(0, os.path.join(test_root))
-    sys.path.insert(0, os.path.join(test_root, "tools", "scripts"))
+    sys.path.insert(0, os.path.join(serve_root))
+    sys.path.insert(0, os.path.join(serve_root, "tools", "scripts"))
     failed = None
     try:
         import serve
     except ImportError:
         failed = "serve"
     try:
         import manifest
     except ImportError:
         failed = "manifest"
 
     if failed:
         logger.critical(
             "Failed to import %s. Ensure that tests path %s contains web-platform-tests" %
-            (failed, test_root))
+            (failed, serve_root))
         sys.exit(1)
 
+
 class TestEnvironmentError(Exception):
     pass
 
 
 class TestEnvironment(object):
-    def __init__(self, test_path, options):
+    def __init__(self, serve_path, test_paths, options):
         """Context manager that owns the test environment i.e. the http and
         websockets servers"""
-        self.test_path = test_path
+        self.serve_path = serve_path
+        self.test_paths = test_paths
         self.server = None
         self.config = None
         self.external_config = None
         self.test_server_port = options.pop("test_server_port", True)
         self.options = options if options is not None else {}
         self.required_files = options.pop("required_files", [])
         self.files_to_restore = []
 
     def __enter__(self):
         self.copy_required_files()
-
+        self.setup_routes()
         self.config = self.load_config()
         serve.set_computed_defaults(self.config)
 
         serve.logger = serve.default_logger("info")
         self.external_config, self.servers = serve.start(self.config)
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         self.restore_files()
         for scheme, servers in self.servers.iteritems():
             for port, server in servers:
                 server.kill()
 
     def load_config(self):
-        default_config_path = os.path.join(self.test_path, "config.default.json")
+        default_config_path = os.path.join(self.serve_path, "config.default.json")
         local_config_path = os.path.join(here, "config.json")
 
         with open(default_config_path) as f:
             default_config = json.load(f)
 
         with open(local_config_path) as f:
             data = f.read()
             local_config = json.loads(data % self.options)
 
         local_config["external_host"] = self.options.get("external_host", None)
 
-        return serve.merge_json(default_config, local_config)
+        config = serve.merge_json(default_config, local_config)
+        config["doc_root"] = self.serve_path
+
+        return config
+
+    def setup_routes(self):
+        for url, paths in self.test_paths.iteritems():
+            if url == "/":
+                continue
+
+            path = paths["tests_path"]
+            url = "/%s/" % url.strip("/")
+
+            for (method,
+                 suffix,
+                 handler_cls) in [(serve.any_method,
+                                   b"*.py",
+                                   serve.handlers.PythonScriptHandler),
+                                  (b"GET",
+                                   "*.asis",
+                                   serve.handlers.AsIsHandler),
+                                  (b"GET",
+                                   "*",
+                                   serve.handlers.FileHandler)]:
+                route = (method, b"%s%s" % (str(url), str(suffix)), handler_cls(path, url_base=url))
+                serve.routes.insert(-3, route)
+
+        if "/" not in self.test_paths:
+            serve.routes = serve.routes[:-3]
 
     def copy_required_files(self):
         logger.info("Placing required files in server environment.")
         for source, destination, copy_if_exists in self.required_files:
             source_path = os.path.join(here, source)
-            dest_path = os.path.join(self.test_path, destination, os.path.split(source)[1])
+            dest_path = os.path.join(self.serve_path, destination, os.path.split(source)[1])
             dest_exists = os.path.exists(dest_path)
             if not dest_exists or copy_if_exists:
                 if dest_exists:
                     backup_path = dest_path + ".orig"
                     logger.info("Backing up %s to %s" % (dest_path, backup_path))
                     self.files_to_restore.append(dest_path)
                     shutil.copy2(dest_path, backup_path)
                 logger.info("Copying %s to %s" % (source_path, dest_path))
@@ -172,379 +200,16 @@ class TestEnvironment(object):
 
     def restore_files(self):
         for path in self.files_to_restore:
             os.unlink(path)
             if os.path.exists(path + ".orig"):
                 os.rename(path + ".orig", path)
 
 
-class TestChunker(object):
-    def __init__(self, total_chunks, chunk_number):
-        self.total_chunks = total_chunks
-        self.chunk_number = chunk_number
-        assert self.chunk_number <= self.total_chunks
-
-    def __call__(self, manifest):
-        raise NotImplementedError
-
-
-class Unchunked(TestChunker):
-    def __init__(self, *args, **kwargs):
-        TestChunker.__init__(self, *args, **kwargs)
-        assert self.total_chunks == 1
-
-    def __call__(self, manifest):
-        for item in manifest:
-            yield item
-
-
-class HashChunker(TestChunker):
-    def __call__(self):
-        chunk_index = self.chunk_number - 1
-        for test_path, tests in manifest:
-            if hash(test_path) % self.total_chunks == chunk_index:
-                yield test_path, tests
-
-
-class EqualTimeChunker(TestChunker):
-    """Chunker that uses the test timeout as a proxy for the running time of the test"""
-
-    def _get_chunk(self, manifest_items):
-        # For each directory containing tests, calculate the maximum execution time after running all
-        # the tests in that directory. Then work out the index into the manifest corresponding to the
-        # directories at fractions of m/N of the running time where m=1..N-1 and N is the total number
-        # of chunks. Return an array of these indicies
-
-        total_time = 0
-        by_dir = OrderedDict()
-
-        class PathData(object):
-            def __init__(self, path):
-                self.path = path
-                self.time = 0
-                self.tests = []
-
-        class Chunk(object):
-            def __init__(self):
-                self.paths = []
-                self.tests = []
-                self.time = 0
-
-            def append(self, path_data):
-                self.paths.append(path_data.path)
-                self.tests.extend(path_data.tests)
-                self.time += path_data.time
-
-        class ChunkList(object):
-            def __init__(self, total_time, n_chunks):
-                self.total_time = total_time
-                self.n_chunks = n_chunks
-
-                self.remaining_chunks = n_chunks
-
-                self.chunks = []
-
-                self.update_time_per_chunk()
-
-            def __iter__(self):
-                for item in self.chunks:
-                    yield item
-
-            def __getitem__(self, i):
-                return self.chunks[i]
-
-            def sort_chunks(self):
-                self.chunks = sorted(self.chunks, key=lambda x:x.paths[0])
-
-            def get_tests(self, chunk_number):
-                return self[chunk_number - 1].tests
-
-            def append(self, chunk):
-                if len(self.chunks) == self.n_chunks:
-                    raise ValueError("Tried to create more than %n chunks" % self.n_chunks)
-                self.chunks.append(chunk)
-                self.remaining_chunks -= 1
-
-            @property
-            def current_chunk(self):
-                if self.chunks:
-                    return self.chunks[-1]
-
-            def update_time_per_chunk(self):
-                self.time_per_chunk = (self.total_time - sum(item.time for item in self)) / self.remaining_chunks
-
-            def create(self):
-                rv = Chunk()
-                self.append(rv)
-                return rv
-
-            def add_path(self, path_data):
-                sum_time = self.current_chunk.time + path_data.time
-                if sum_time > self.time_per_chunk and self.remaining_chunks > 0:
-                    overshoot = sum_time - self.time_per_chunk
-                    undershoot = self.time_per_chunk - self.current_chunk.time
-                    if overshoot < undershoot:
-                        self.create()
-                        self.current_chunk.append(path_data)
-                    else:
-                        self.current_chunk.append(path_data)
-                        self.create()
-                else:
-                    self.current_chunk.append(path_data)
-
-        for i, (test_path, tests) in enumerate(manifest_items):
-            test_dir = tuple(os.path.split(test_path)[0].split(os.path.sep)[:3])
-
-            if not test_dir in by_dir:
-                by_dir[test_dir] = PathData(test_dir)
-
-            data = by_dir[test_dir]
-            time = sum(wpttest.DEFAULT_TIMEOUT if test.timeout !=
-                       "long" else wpttest.LONG_TIMEOUT for test in tests)
-            data.time += time
-            data.tests.append((test_path, tests))
-
-            total_time += time
-
-        chunk_list = ChunkList(total_time, self.total_chunks)
-
-        if len(by_dir) < self.total_chunks:
-            raise ValueError("Tried to split into %i chunks, but only %i subdirectories included" % (
-                self.total_chunks, len(by_dir)))
-
-        # Put any individual dirs with a time greater than the time per chunk into their own
-        # chunk
-        while True:
-            to_remove = []
-            for path_data in by_dir.itervalues():
-                if path_data.time > chunk_list.time_per_chunk:
-                    to_remove.append(path_data)
-            if to_remove:
-                for path_data in to_remove:
-                    chunk = chunk_list.create()
-                    chunk.append(path_data)
-                    del by_dir[path_data.path]
-                chunk_list.update_time_per_chunk()
-            else:
-                break
-
-        chunk = chunk_list.create()
-        for path_data in by_dir.itervalues():
-            chunk_list.add_path(path_data)
-
-        assert len(chunk_list.chunks) == self.total_chunks, len(chunk_list.chunks)
-        assert sum(item.time for item in chunk_list) == chunk_list.total_time
-
-        chunk_list.sort_chunks()
-
-        return chunk_list.get_tests(self.chunk_number)
-
-    def __call__(self, manifest_iter):
-        manifest = list(manifest_iter)
-        tests = self._get_chunk(manifest)
-        for item in tests:
-            yield item
-
-
-class TestFilter(object):
-    def __init__(self, include=None, exclude=None, manifest_path=None):
-        if manifest_path is not None and include is None:
-            self.manifest = manifestinclude.get_manifest(manifest_path)
-        else:
-            self.manifest = manifestinclude.IncludeManifest.create()
-
-        if include is not None:
-            self.manifest.set("skip", "true")
-            for item in include:
-                self.manifest.add_include(item)
-
-        if exclude is not None:
-            for item in exclude:
-                self.manifest.add_exclude(item)
-
-    def __call__(self, manifest_iter):
-        for test_path, tests in manifest_iter:
-            include_tests = set()
-            for test in tests:
-                if self.manifest.include(test):
-                    include_tests.add(test)
-
-            if include_tests:
-                yield test_path, include_tests
-
-
-class TestLoader(object):
-    def __init__(self, tests_root, metadata_root, test_types, test_filter, run_info,
-                 chunk_type="none", total_chunks=1, chunk_number=1):
-        self.tests_root = tests_root
-        self.metadata_root = metadata_root
-        self.test_types = test_types
-        self.test_filter = test_filter
-        self.run_info = run_info
-        self.manifest_path = os.path.join(self.metadata_root, "MANIFEST.json")
-        self.manifest = self.load_manifest()
-        self.tests = None
-        self.disabled_tests = None
-
-        self.chunk_type = chunk_type
-        self.total_chunks = total_chunks
-        self.chunk_number = chunk_number
-
-        self.chunker = {"none": Unchunked,
-                        "hash": HashChunker,
-                        "equal_time": EqualTimeChunker}[chunk_type](total_chunks,
-                                                                    chunk_number)
-
-        self._test_ids = None
-        self._load_tests()
-
-    @property
-    def test_ids(self):
-        if self._test_ids is None:
-            self._test_ids = []
-            for test_dict in [self.disabled_tests, self.tests]:
-                for test_type in self.test_types:
-                    self._test_ids += [item.id for item in test_dict[test_type]]
-        return self._test_ids
-
-    def create_manifest(self):
-        logger.info("Creating test manifest")
-        manifest.setup_git(self.tests_root)
-        manifest_file = manifest.Manifest(None)
-        manifest.update(manifest_file)
-        manifest.write(manifest_file, self.manifest_path)
-
-    def load_manifest(self):
-        if not os.path.exists(self.manifest_path):
-            self.create_manifest()
-        return manifest.load(self.manifest_path)
-
-    def get_test(self, manifest_test, expected_file):
-        if expected_file is not None:
-            expected = expected_file.get_test(manifest_test.id)
-        else:
-            expected = None
-        return wpttest.from_manifest(manifest_test, expected)
-
-    def load_expected_manifest(self, test_path):
-        return manifestexpected.get_manifest(self.metadata_root, test_path, self.run_info)
-
-    def iter_tests(self):
-        manifest_items = self.test_filter(self.manifest.itertypes(*self.test_types))
-
-        if self.chunker is not None:
-            manifest_items = self.chunker(manifest_items)
-
-        for test_path, tests in manifest_items:
-            expected_file = self.load_expected_manifest(test_path)
-            for manifest_test in tests:
-                test = self.get_test(manifest_test, expected_file)
-                test_type = manifest_test.item_type
-                yield test_path, test_type, test
-
-    def _load_tests(self):
-        """Read in the tests from the manifest file and add them to a queue"""
-        tests = {"enabled":defaultdict(list),
-                 "disabled":defaultdict(list)}
-
-        for test_path, test_type, test in self.iter_tests():
-            key = "enabled" if not test.disabled() else "disabled"
-            tests[key][test_type].append(test)
-
-        self.tests = tests["enabled"]
-        self.disabled_tests = tests["disabled"]
-
-    def groups(self, test_types, chunk_type="none", total_chunks=1, chunk_number=1):
-        groups = set()
-
-        for test_type in test_types:
-            for test in self.tests[test_type]:
-                group = test.url.split("/")[1]
-                groups.add(group)
-
-        return groups
-
-
-class TestSource(object):
-    __metaclass__ = ABCMeta
-
-    @abstractmethod
-    def queue_tests(self, test_queue):
-        pass
-
-    @abstractmethod
-    def requeue_test(self, test):
-        pass
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *args, **kwargs):
-        pass
-
-
-class SingleTestSource(TestSource):
-    def __init__(self, test_queue):
-        self.test_queue = test_queue
-
-    @classmethod
-    def queue_tests(cls, test_queue, test_type, tests):
-        for test in tests[test_type]:
-            test_queue.put(test)
-
-    def get_queue(self):
-        if self.test_queue.empty():
-            return None
-        return self.test_queue
-
-    def requeue_test(self, test):
-        self.test_queue.put(test)
-
-class PathGroupedSource(TestSource):
-    def __init__(self, test_queue):
-        self.test_queue = test_queue
-        self.current_queue = None
-
-    @classmethod
-    def queue_tests(cls, test_queue, test_type, tests, depth=None):
-        if depth is True:
-            depth = None
-
-        prev_path = None
-        group = None
-
-        for test in tests[test_type]:
-            path = urlparse.urlsplit(test.url).path.split("/")[1:-1][:depth]
-            if path != prev_path:
-                group = []
-                test_queue.put(group)
-                prev_path = path
-
-            group.append(test)
-
-    def get_queue(self):
-        if not self.current_queue or self.current_queue.empty():
-            try:
-                data = self.test_queue.get(block=True, timeout=1)
-                self.current_queue = Queue()
-                for item in data:
-                    self.current_queue.put(item)
-            except Empty:
-                return None
-
-        return self.current_queue
-
-    def requeue_test(self, test):
-        self.current_queue.put(test)
-
-    def __exit__(self, *args, **kwargs):
-        if self.current_queue:
-            self.current_queue.close()
-
 class LogThread(threading.Thread):
     def __init__(self, queue, logger, level):
         self.queue = queue
         self.log_func = getattr(logger, level)
         threading.Thread.__init__(self, name="Thread-Log")
         self.daemon = True
 
     def run(self):
@@ -579,97 +244,104 @@ class LoggingWrapper(StringIO):
             return
         if self.prefix is not None:
             data = "%s: %s" % (self.prefix, data)
         self.queue.put(data)
 
     def flush(self):
         pass
 
+def list_test_groups(serve_root, test_paths, test_types, product, **kwargs):
 
-def list_test_groups(tests_root, metadata_root, test_types, product, **kwargs):
-    do_test_relative_imports(tests_root)
+    do_delayed_imports(serve_root)
 
-    run_info = wpttest.get_run_info(metadata_root, product, debug=False)
-    test_filter = TestFilter(include=kwargs["include"], exclude=kwargs["exclude"],
-                             manifest_path=kwargs["include_manifest"])
-    test_loader = TestLoader(tests_root, metadata_root, test_types, test_filter, run_info)
+    run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=False)
+    test_filter = testloader.TestFilter(include=kwargs["include"],
+                                        exclude=kwargs["exclude"],
+                                        manifest_path=kwargs["include_manifest"])
+    test_loader = testloader.TestLoader(test_paths,
+                                        test_types,
+                                        test_filter,
+                                        run_info)
 
-    for item in sorted(test_loader.groups()):
+    for item in sorted(test_loader.groups(test_types)):
         print item
 
-
-def list_disabled(tests_root, metadata_root, test_types, product, **kwargs):
-    do_test_relative_imports(tests_root)
-
+def list_disabled(serve_root, test_paths, test_types, product, **kwargs):
+    do_delayed_imports(serve_root)
     rv = []
-    run_info = wpttest.get_run_info(metadata_root, product, debug=False)
-    test_loader = TestLoader(tests_root, metadata_root, test_types, TestFilter(), run_info)
+    run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=False)
+    test_loader = testloader.TestLoader(test_paths,
+                                        test_types,
+                                        testloader.TestFilter(),
+                                        run_info)
 
     for test_type, tests in test_loader.disabled_tests.iteritems():
         for test in tests:
             rv.append({"test": test.id, "reason": test.disabled()})
     print json.dumps(rv, indent=2)
 
 
-def run_tests(config, tests_root, metadata_root, product, **kwargs):
+def run_tests(config, serve_root, test_paths, product, **kwargs):
     logging_queue = None
     logging_thread = None
     original_stdio = (sys.stdout, sys.stderr)
     test_queues = None
 
     try:
         if not kwargs["no_capture_stdio"]:
             logging_queue = Queue()
             logging_thread = LogThread(logging_queue, logger, "info")
             sys.stdout = LoggingWrapper(logging_queue, prefix="STDOUT")
             sys.stderr = LoggingWrapper(logging_queue, prefix="STDERR")
             logging_thread.start()
 
-        do_test_relative_imports(tests_root)
+        do_delayed_imports(serve_root)
 
-        run_info = wpttest.get_run_info(metadata_root, product, debug=False)
+        run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=False)
 
         (check_args,
          browser_cls, get_browser_kwargs,
          executor_classes, get_executor_kwargs,
          env_options) = products.load_product(config, product)
 
         check_args(**kwargs)
 
         browser_kwargs = get_browser_kwargs(**kwargs)
 
         unexpected_total = 0
 
         if "test_loader" in kwargs:
             test_loader = kwargs["test_loader"]
         else:
-            test_filter = TestFilter(include=kwargs["include"],
-                                     exclude=kwargs["exclude"],
-                                     manifest_path=kwargs["include_manifest"])
-            test_loader = TestLoader(tests_root,
-                                     metadata_root,
-                                     kwargs["test_types"],
-                                     test_filter,
-                                     run_info,
-                                     kwargs["chunk_type"],
-                                     kwargs["total_chunks"],
-                                     kwargs["this_chunk"])
+            test_filter = testloader.TestFilter(include=kwargs["include"],
+                                                exclude=kwargs["exclude"],
+                                                manifest_path=kwargs["include_manifest"])
+            test_loader = testloader.TestLoader(test_paths,
+                                                kwargs["test_types"],
+                                                test_filter,
+                                                run_info,
+                                                kwargs["chunk_type"],
+                                                kwargs["total_chunks"],
+                                                kwargs["this_chunk"],
+                                                kwargs["manifest_update"])
 
         if kwargs["run_by_dir"] is False:
-            test_source_cls = SingleTestSource
+            test_source_cls = testloader.SingleTestSource
             test_source_kwargs = {}
         else:
             # A value of None indicates infinite depth
-            test_source_cls = PathGroupedSource
+            test_source_cls = testloader.PathGroupedSource
             test_source_kwargs = {"depth": kwargs["run_by_dir"]}
 
         logger.info("Using %i client processes" % kwargs["processes"])
 
-        with TestEnvironment(tests_root, env_options) as test_environment:
+        with TestEnvironment(serve_root,
+                             test_paths,
+                             env_options) as test_environment:
             try:
                 test_environment.ensure_started()
             except TestEnvironmentError as e:
                 logger.critical("Error starting test environment: %s" % e.message)
                 raise
 
             base_server = "http://%s:%i" % (test_environment.external_config["host"],
                                             test_environment.external_config["ports"]["http"][0])