Bug 1468210 [wpt PR 11425] - Order both actual and expected in selftests, a=testonly
authorSimon Pieters <zcorpan@gmail.com>
Fri, 06 Jul 2018 16:27:54 +0000
changeset 425840 94cf8c88b44395001bdd49c43890fd10d17324f9
parent 425839 a591d00c8a2556c8295d88f5d41585814521e955
child 425841 8fdeb9bf3c6caef71010f5e4f10769de69cff69b
push id34267
push userrgurzau@mozilla.com
push dateWed, 11 Jul 2018 22:05:21 +0000
treeherdermozilla-central@3aca103e4915 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerstestonly
bugs1468210, 11425
milestone63.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1468210 [wpt PR 11425] - Order both actual and expected in selftests, a=testonly Automatic update from web-platform-testsOrder both actual and expected in selftests (#11425) -- wpt-commits: 4f741e69cb68750b4404b4cd16cd4f396855d73e wpt-pr: 11425
testing/web-platform/tests/resources/test/conftest.py
--- a/testing/web-platform/tests/resources/test/conftest.py
+++ b/testing/web-platform/tests/resources/test/conftest.py
@@ -136,16 +136,17 @@ class HTMLItem(pytest.Item, pytest.Colle
         actual = driver.execute_async_script('runTest("%s", "foo", arguments[0])' % test_url)
 
         # Test object ordering is not guaranteed. This weak assertion verifies
         # that the indices are unique and sequential
         indices = [test_obj.get('index') for test_obj in actual['tests']]
         self._assert_sequence(indices)
 
         summarized = self._summarize(actual)
+        self.expected[u'summarized_tests'].sort(key=lambda test_obj: test_obj.get('name'))
 
         assert summarized == self.expected
 
     def _summarize(self, actual):
         summarized = {}
 
         summarized[u'summarized_status'] = self._summarize_status(actual['status'])
         summarized[u'summarized_tests'] = [