Bug 1354232 [wpt PR 12073] - [Gecko Bug 1354232] Enable LSAN Leak detection in wpt, a=testonly
authormoz-wptsync-bot <wptsync@mozilla.com>
Wed, 25 Jul 2018 21:05:57 +0000
changeset 824158 f271db900a41e996687f5142005df7b7f46a825b
parent 824157 8d16fdfe7e9bcaf8ab00febe1632a04980b6899b
child 824159 5d414c8246b4a8ebbf25782405334d39c14aab9e
push id117837
push userbmo:aryx.bugmail@gmx-topmail.de
push dateMon, 30 Jul 2018 09:42:15 +0000
reviewerstestonly
bugs1354232, 12073
milestone63.0a1
Bug 1354232 [wpt PR 12073] - [Gecko Bug 1354232] Enable LSAN Leak detection in wpt, a=testonly Automatic update from web-platform-testsEnable LSAN Leak detection in wpt bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: e0d106c2112dd6845ca75414258ab74e5ed73d87 gecko-integration-branch: mozilla-inbound gecko-reviewers: ahal, mccr8 -- Allow wpt manifest files to specify LSAN errors to ignore This adds a property lsan-allowed to the expectation manifest files that takes a list of strings. Any entry in the list that matches a frame in an LSAN stack will cause that stack to be regarded as an expected failure. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: b8da0ef2695d8bca1c11b44447a71cef5ce5b3fc gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Log run-by-dir setting bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 5b9ba77b7316c8ee1d0dac117e33685ebdfe22bc gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Support skipping output lines in handlers bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 7c976baa71947b409f92ffd64583440b9a2836b9 gecko-integration-branch: mozilla-inbound gecko-reviewers: ato -- Add group_metadata to test group metadata and pass it through to the browser This initially contains a scope entry which is set to the base directory of the tests being run. Typically this is /, but with run_by_dir, it's the path to the current run_by_dir group e.g. /html/semantics/form_elements/ bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 07049ac67119a90a0708b0d511562695f725dd9c gecko-integration-branch: mozilla-inbound gecko-reviewers: ato -- Add support for updating LSAN data in wpt-update LSAN data differs from existing expectation data because the data is only generated when the browser exits, so the problems reported can happen at any point in the current session. We use the `scope` property in the log message to determine the path to a __dir__.ini file that covers all the tests run in the session, and add the LSAN exclusion rules to there. The rules themselves are generated by taking the topmost frame of any stack that's reported as unexpectedly leaking, and adding that to the list of permitted frames in the lsan-allowed property. We never remove entries from this list since intermittents may be present which won't appear on a specific run. Instead we rely on humans fixing the issues to also clean up the expectation files. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 9e8a66385f66f89f8cf2c9a18edf008496fa89fa gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Always try to process the leak log whenever we stop firefox The `stop` method is always called to shutdown firefox, but the cleanup method is only called at the end of a test run. Therefore we need all the leak processing stuff ot happen in stop(). bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 8cc292a22975583f985da521544b08cf36be6517 gecko-integration-branch: mozilla-inbound gecko-reviewers: ato -- Support creating ConditionalValue objects containing a list bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: c1ebcf01626ac0e0b23cad5c80ad04ac467fd893 gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Support a __dir__.ini file in the metadata root bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 0e161374b583a7637db664473e441a1567e82a5b gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Fix updating assert count when there's an exising value In this case we want to take the existing value into account, and update to 1 more than the new value (in the max-asserts case). bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 0f2a7154454ceec763bd64597ad9e3155c8d091c gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Refactor data storage in metadata.py Previously we were holding a map of test id -> test and test -> expectation data. But this is an unnecessary layer of indirection, and it works perfectly well to map test id to the expectation data directly. This makes the code simpler and may also help make the update a little faster. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 6bf4a20193b801d342ff662b5a9712e516561d9e gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Fix, enable, and add to, the manifest update tests Refactor the tests to make them work with pytest directly rather than also depending on unittest. Fix the helper functions to work with the current state of metadata.py. Add some tests for update of assertion count and lsan data. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 5028820efd9ed3d1dbcb9bde5cbc235414a8d23c gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Don't store the result of update_expected The needs_human feature never worked well and is unused. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: 004b079b76d06c3e9d74210700b28994cdeff886 gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Chunk by dir more for wpt on asan This means that when we run these tests, the LSAN exceptions will have the minimum possible scope. bugzilla-url: https://bugzilla.mozilla.org/show_bug.cgi?id=1354232 gecko-commit: a7dec081ead2ac3b5d20b99b2bb177719358e328 gecko-integration-branch: mozilla-inbound gecko-reviewers: maja_zf -- Fixup flake8 failures -- Fix flake8 issues in master on servodriver -- wpt-commits: 9a57710275e1ac52d4c3d3426b370e3f31c46856, 49193c2e02fc20fc1d28e5957d4f3a6fd52adfd3, 429d1b8353e0e1b601bd34ba98fdff290780b127, aa9bdbf0639326900d232f155449e7950adc0649, 91803e3a25afd8e0c201bc57cd05fffd5d543551, df7e6cdf297c06a05b5a116f51fb203cda295809, d8aaec5ffebb93db1b54ccdafb0d7a2897428427, 4d21dcbb1bd9612715ac9ed55c3ec271b3c4e1ca, aa3a405237c3d45187f76906367161a7e9a486a0, 68c8996844a6774aadfd14da4c02c71bb5d9d342, d4b2146217d338e278e24bb3bc705871b50e9c7e, f814a0f1c23cb76e36b6732165c92fa327d86f40, 5782d6704b66adbcce7398911a74a4abdf323ef7, dcfcadb2c6892a187de6f0be7c42eed09d91fe90, 17796c6272e4fe3d1bba24fdfe261572b1ad757a, 654133c23efb269585009818b6f4547d130cb564 wpt-pr: 12073
testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/servodriver.py
testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executorservodriver.py
testing/web-platform/tests/tools/wptrunner/wptrunner/metadata.py
testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py
testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_update.py
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/servodriver.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/servodriver.py
@@ -1,10 +1,9 @@
 import os
-import shutil
 import subprocess
 import tempfile
 
 from mozprocess import ProcessHandler
 
 from serve.serve import make_hosts_file
 
 from .base import Browser, require_arg, get_free_port, browser_command, ExecutorBrowser
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executorservodriver.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executorservodriver.py
@@ -22,16 +22,17 @@ here = os.path.join(os.path.split(__file
 extra_timeout = 5
 
 
 def do_delayed_imports():
     global webdriver
     import webdriver
 
     global ServoCommandExtensions
+
     class ServoCommandExtensions(object):
         def __init__(self, session):
             self.session = session
 
         @webdriver.client.command
         def get_prefs(self, *prefs):
             body = {"prefs": list(prefs)}
             return self.session.send_session_command("POST", "servo/prefs/get", body)
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/metadata.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/metadata.py
@@ -381,38 +381,28 @@ def create_test_tree(metadata_path, test
         dir_path = os.path.split(test_path)[0].replace(os.path.sep, "/")
         while True:
             if dir_path:
                 dir_id = dir_path + "/__dir__"
             else:
                 dir_id = "__dir__"
             dir_id = (test_manifest.url_base + dir_id).lstrip("/")
             if dir_id not in id_test_map:
-                dir_object = DirObject(dir_id, dir_path)
                 test_file_data = TestFileData(test_manifest,
                                               metadata_path,
                                               dir_id,
                                               [])
                 id_test_map[dir_id] = test_file_data
             if not dir_path or dir_path in id_test_map:
                 break
             dir_path = dir_path.rsplit("/", 1)[0] if "/" in dir_path else ""
 
     return id_test_map
 
 
-class DirObject(object):
-    def __init__(self, id, path):
-        self.id = id
-        self.path = path
-
-    def __hash__(self):
-        return hash(self.id)
-
-
 class TestFileData(object):
     def __init__(self, test_manifest, metadata_path, test_path, tests):
         self.test_manifest = test_manifest
         self.test_path = test_path
         self.metadata_path = metadata_path
         self.tests = tests
         self._expected = None
         self._requires_update = False
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py
@@ -594,17 +594,17 @@ class TestSource(object):
 
     @abstractmethod
     # noqa: N805
     #@classmethod (doesn't compose with @abstractmethod)
     def make_queue(cls, tests, **kwargs):
         pass
 
     @classmethod
-    def group_metadata(self, state):
+    def group_metadata(cls, state):
         return {"scope": "/"}
 
     def group(self):
         if not self.current_group or len(self.current_group) == 0:
             try:
                 self.current_group, self.current_metadata = self.test_queue.get(block=False)
             except Empty:
                 return None, None
--- a/testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_update.py
+++ b/testing/web-platform/tests/tools/wptrunner/wptrunner/tests/test_update.py
@@ -1,15 +1,15 @@
 import json
 import mock
 import os
 import sys
 from io import BytesIO
 
-from .. import metadata, manifestupdate, wptmanifest
+from .. import metadata, manifestupdate
 from mozlog import structuredlog, handlers, formatters
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, os.pardir))
 from manifest import manifest, item as manifest_item
 
 
 def rel_path_to_url(rel_path, url_base="/"):
     assert not os.path.isabs(rel_path)
@@ -206,17 +206,17 @@ def test_update_multiple_0():
 
     log_1 = suite_log([("test_start", {"test": test_id}),
                        ("test_status", {"test": test_id,
                                         "subtest": "test1",
                                         "status": "TIMEOUT",
                                         "expected": "FAIL"}),
                        ("test_end", {"test": test_id,
                                      "status": "OK"})],
-                       run_info={"debug": False, "os": "linux"})
+                      run_info={"debug": False, "os": "linux"})
 
     updated = update(tests, log_0, log_1)
     new_manifest = updated[0][1]
 
     assert not new_manifest.is_empty
     assert new_manifest.get_test(test_id).children[0].get(
         "expected", {"debug": False, "os": "osx"}) == "FAIL"
     assert new_manifest.get_test(test_id).children[0].get(
@@ -339,17 +339,17 @@ def test_update_ignore_existing():
 
     log_0 = suite_log([("test_start", {"test": test_id}),
                        ("test_status", {"test": test_id,
                                         "subtest": "test1",
                                         "status": "FAIL",
                                         "expected": "PASS"}),
                        ("test_end", {"test": test_id,
                                      "status": "OK"})],
-                       run_info={"debug": False, "os": "linux"})
+                      run_info={"debug": False, "os": "linux"})
 
     log_1 = suite_log([("test_start", {"test": test_id}),
                        ("test_status", {"test": test_id,
                                         "subtest": "test1",
                                         "status": "FAIL",
                                         "expected": "PASS"}),
                        ("test_end", {"test": test_id,
                                      "status": "OK"})],