Bug 1473392 - Fix failure conditions for all mochitest tests. r=jmaher
authorGreg Mierzwinski <gmierz1@live.ca>
Fri, 17 Aug 2018 14:09:42 +0000
changeset 432114 25d58e97db2d7f1bd8fcf5bc2676d7bd77bb6e8d
parent 432113 717fb8632016211ce9cf8a7267be60e52f7e77c8
child 432115 5a465e7bdc7a0f25a305a1e4301587dccd8935cd
push id34460
push userdvarga@mozilla.com
push dateFri, 17 Aug 2018 21:51:39 +0000
treeherdermozilla-central@2f1bbddc826b [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersjmaher
bugs1473392
milestone63.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1473392 - Fix failure conditions for all mochitest tests. r=jmaher This patch changes the failure condition of all mochitest tests to take the number of failures (given by self.countfail). It also adds a condition to check if there are passing and todo counts, and if there are not, the test chunk fails. With this change, mochitest test chunks will fail if the failure count is not 0, or all the counts (pass, todo, and fail) are 0. This fixes a case where the TODO tests were considered as failures when no passing tests were run along side them. Differential Revision: https://phabricator.services.mozilla.com/D3612
testing/mochitest/runtests.py
--- a/testing/mochitest/runtests.py
+++ b/testing/mochitest/runtests.py
@@ -2638,19 +2638,22 @@ toolbar#nav-bar {
         else:
             print("0 INFO TEST-START | Shutdown")
             print("1 INFO Passed:  %s" % self.countpass)
             print("2 INFO Failed:  %s" % self.countfail)
             print("3 INFO Todo:    %s" % self.counttodo)
             print("4 INFO Mode:    %s" % e10s_mode)
             print("5 INFO SimpleTest FINISHED")
 
-        if not result and not self.countpass:
-            # either tests failed or no tests run
-            result = 1
+        if not result:
+            if self.countfail or \
+               not (self.countpass or self.counttodo):
+                # at least one test failed, or
+                # no tests passed, and no tests failed (possibly a crash)
+                result = 1
 
         return result
 
     def doTests(self, options, testsToFilter=None):
         # A call to initializeLooping method is required in case of --run-by-dir or --bisect-chunk
         # since we need to initialize variables for each loop.
         if options.bisectChunk or options.runByManifest:
             self.initializeLooping(options)