Bug 1497898 [wpt PR 13618] - [Gecko Bug 1497898] Use testfile mtimes to pre-filter files iterated over for the manifest update, a=testonly
authorAhilya Sinha <ahilyasinha99@gmail.com>
Thu, 22 Nov 2018 10:34:00 +0000
changeset 507118 aca5ae992b39290a8795af5a6c92a12fa544e177
parent 507117 7433f8fac49ab00d576e5ddb4b6d529a67b59478
child 507119 8446b6af9d52efb57bbf3965897970353a3a9a70
push id1905
push userffxbld-merge
push dateMon, 21 Jan 2019 12:33:13 +0000
treeherdermozilla-release@c2fca1944d8c [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1497898, 13618
milestone65.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1497898 [wpt PR 13618] - [Gecko Bug 1497898] Use testfile mtimes to pre-filter files iterated over for the manifest update, a=testonly Automatic update from web-platform-testsUse testfile mtimes to pre-filter files iterated over for the manifest update Depends on D11667 Differential Revision: https://phabricator.services.mozilla.com/D8221 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: aa24577bdeae449b6b51f1184cc6aec809fc3659 gecko-integration-branch: autoland gecko-reviewers: ato -- Update the .gitignore file Depends on D8221 Differential Revision: https://phabricator.services.mozilla.com/D8222 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 284749dd9ee9d5e693cd86887934092a8889e1fc gecko-integration-branch: autoland gecko-reviewers: ato -- Allow the gitignore filter to work on name components only We end up with a lot of rules like (?:.*)/.*\.ext which are basically trying to find the last component in a path and match against that. These are rather slow to run so the easiest thing tdo is just pass in the last component of the path when we know that's the only thing the rule can match. The changes to surrounding code to use this API will be made in future commits. Depends on D8222 Differential Revision: https://phabricator.services.mozilla.com/D8223 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 46c50835b9972454afc33df9df569afd657430ca gecko-integration-branch: autoland gecko-reviewers: ato -- Update the gitignore implementation to work as an iterator filter This updates the gitignore implemenation to take input like os.walk but with additional stat data for the files. It also makes several useful optimistaions: * Avoid using regex when just matching a literal * Identify patterns that can only match the final component of a path and run those against that component rather than the full path. * Add the possibility of providing a dictionary of paths to gitignore statuses as a cache. This dramatically reduces the amount of time we spend in gitignore processing when updating the manifest. Depends on D8223 Differential Revision: https://phabricator.services.mozilla.com/D8224 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 215f0ae6129888a0865d151835a86aee34368156 gecko-integration-branch: autoland gecko-reviewers: ato -- Update the lint to the new gitignore API Depends on D8224 Differential Revision: https://phabricator.services.mozilla.com/D8225 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 166466f3a5fe137e02a710012babe8a14756afee gecko-integration-branch: autoland gecko-reviewers: ato -- Add manifest caches for the mtime and gitignore rules When processing the manifest using the worktree, instead of reading all files to see if the content changed, instead only process files where the mtime has been updated since the previous run. Also cache the result of running gitignore, so we can save a couple of seconds processing the gitignore rules. Depends on D8225 Differential Revision: https://phabricator.services.mozilla.com/D8226 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 9d61715037440d58adf237622245bb847b9a898c gecko-integration-branch: autoland gecko-reviewers: ato -- Add a custom implementation of os.walk Compared to the normal os.walk this has a couple of differences: * It returns lists of (name, stat) for filenames and directories, allowing callers to reuse the stat data without going back to the system to re-request it. * Directories are always returned as paths relative to the root, and the root itself is returned as the empty string. * It is non-recursive. There are also a few features missing that aren't required for our use cases. Depends on D8226 Differential Revision: https://phabricator.services.mozilla.com/D8227 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 10e90d3295ee7a29b094b42912cdb584dfd47e98 gecko-integration-branch: autoland gecko-reviewers: ato -- Fix the cache lifecycle The caches weren't being invalidated when the manifest itself changed. To fix this the manifest itself has to be written before the cache and the cache has to include data about the manifest that it's associated with (the mtime and path are used for this purpose). To make all this work requires a single method that can load the manifest, update it, write the manifest and write the caches. Therefore we introduce a single load_and_update method that is intended to replace all previous use of the load() or update() methods (and as a bonus handles manifest version mismatches in a single place). Depends on D8227 Differential Revision: https://phabricator.services.mozilla.com/D8228 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: fe2d962a8ed2e722266a417df17f7085aeabad3b gecko-integration-branch: autoland gecko-reviewers: ato -- Fix the .gitignore rules Depends on D8228 Differential Revision: https://phabricator.services.mozilla.com/D8229 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: daf29e8abb1eb0f5685ed9d228afeb6b4a0c4327 gecko-integration-branch: autoland gecko-reviewers: ato -- Pass the manifest file directly into wpttest.from_manifest Depends on D8229 Differential Revision: https://phabricator.services.mozilla.com/D8230 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 7f54376353321e398e7389a1aa3f1165d775faf9 gecko-integration-branch: autoland gecko-reviewers: ato -- Update manifest before tests Depends on D8230 Differential Revision: https://phabricator.services.mozilla.com/D8231 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 750b40bcdad7d60d10ea78f386493d962af89de1 gecko-integration-branch: autoland gecko-reviewers: ato -- Handle WindowsError trying to determine if git exists Depends on D8232 Differential Revision: https://phabricator.services.mozilla.com/D10743 bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1497898 gecko-commit: 3dedddeb0c8d6847d9c7cb7074d7d06186182663 gecko-integration-branch: autoland gecko-reviewers: ato -- Fix tests that didn't merge correctly -- wpt-commits: 1ca46e0bec3da78ea11a824f7e6d07755e1f8399, 21ef9ca8b2dbffbe66d3c4b32b6818f827c2059b, be2adcdf2c6d3073ce65f7d4759a46f02ff1f7c2, 4011c5f3456bd84469d273cd7ba83fc62b05725d, 5bb4ba65d70fb397f11d3daad8474300b6628133, 770229abaa02cd3f359434560ce018703506990f, d21304002567198c4570328564e8ddb7244a579f, 8d520f0037303f3a09e982d7eb17fbc4ffe78489, a1dda976894ebc1fbfd4bba84200f388f10618ac, cb7be1940cfa9c25055093c4c2a8f24548b53cc5, fc9918e97b9e9306c31017441324f84c06ad9f4a, 59455cd58781c760e28f9663708d2b0b1acec742, becd4a72757b16321cddb9c79db9b443831d5d5e wpt-pr: 13618
testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_wpttest.py
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_wpttest.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_wpttest.py
@@ -77,32 +77,33 @@ def test_conditional():
                                                     test_path="a",
                                                     url_base="")
 
     test = tests[1][2].pop()
     test_obj = wpttest.from_manifest(tests, test, [], test_metadata.get_test(test.id))
     assert test_obj.prefs == {"a": "b", "c": "d"}
     assert test_obj.expected() == "FAIL"
 
+
 def test_metadata_lsan_stack_depth():
     tests = make_mock_manifest(("test", "a", 10), ("test", "a/b", 10))
 
     test_metadata = manifestexpected.static.compile(BytesIO(test_2),
                                                     {},
                                                     data_cls_getter=manifestexpected.data_cls_getter,
                                                     test_path="a",
                                                     url_base="")
 
     test = tests[2][2].pop()
-    test_obj = wpttest.from_manifest(test, [], test_metadata.get_test(test.id))
+    test_obj = wpttest.from_manifest(tests, test, [], test_metadata.get_test(test.id))
 
     assert test_obj.lsan_max_stack_depth == 42
 
     test = tests[1][2].pop()
-    test_obj = wpttest.from_manifest(test, [], test_metadata.get_test(test.id))
+    test_obj = wpttest.from_manifest(tests, test, [], test_metadata.get_test(test.id))
 
     assert test_obj.lsan_max_stack_depth is None
 
     test_metadata = manifestexpected.static.compile(BytesIO(test_0),
                                                     {},
                                                     data_cls_getter=manifestexpected.data_cls_getter,
                                                     test_path="a",
                                                     url_base="")
@@ -110,11 +111,11 @@ def test_metadata_lsan_stack_depth():
     inherit_metadata = [
         manifestexpected.static.compile(
             BytesIO(dir_ini_2),
             {},
             data_cls_getter=lambda x,y: manifestexpected.DirectoryManifest)
     ]
 
     test = tests[0][2].pop()
-    test_obj = wpttest.from_manifest(test, inherit_metadata, test_metadata.get_test(test.id))
+    test_obj = wpttest.from_manifest(tests, test, inherit_metadata, test_metadata.get_test(test.id))
 
     assert test_obj.lsan_max_stack_depth == 42