Bug 1369941 - Provide more advice about how to write fuzzy() annotations on reftests. r=dholbert
authorL. David Baron <dbaron@dbaron.org>
Wed, 22 Aug 2018 02:14:06 +0000
changeset 481085 4fca1f7a874da0e7824d4337c40f96462d5f1c03
parent 481084 5e7484d3b2e3ad6ec39e7af04601742a3ff7643e
child 481086 865339e90f1ab9115c447c86205c7105bf155e89
push id232
push userfmarier@mozilla.com
push dateWed, 05 Sep 2018 20:45:54 +0000
reviewersdholbert
bugs1369941
milestone63.0a1
Bug 1369941 - Provide more advice about how to write fuzzy() annotations on reftests. r=dholbert Differential Revision: https://phabricator.services.mozilla.com/D3940
layout/tools/reftest/README.txt
--- a/layout/tools/reftest/README.txt
+++ b/layout/tools/reftest/README.txt
@@ -117,20 +117,41 @@ 2. A test item
       fuzzy(minDiff-maxDiff,minPixelCount-maxPixelCount)
           This allows a test to pass if the pixel value differences are between
           minDiff and maxDiff, inclusive, and the total number of different
           pixels is between minPixelCount and maxPixelCount, inclusive.
           It can also be used with '!=' to ensure that the difference is
           outside the specified interval. Note that with '!=' tests the
           minimum bounds of the ranges must be zero.
 
+          Fuzzy tends to be used for two different sorts of cases.  The main
+          case is tests that are expected to be equal, but actually fail in a
+          minor way (e.g., an antialiasing difference), and we want to ensure
+          that the test doesn't regress further so we don't want to mark the
+          test as failing.  For these cases, test annotations should be the
+          tightest bounds possible:  if the behavior is entirely deterministic
+          this means a range like fuzzy(1-1,8-8), and if at all possible, the
+          ranges should not include 0.  In cases where the test only sometimes
+          fails, this unfortunately requires using 0 in both ranges, which
+          means that we won't get reports of an unexpected pass if the problem
+          is fixed (allowing us to remove the fuzzy() annotation and expect
+          the test to pass from then on).
+
+          The second case where fuzzy is used is tests that are supposed
+          to allow some amount of variability (i.e., tests where the
+          specification allows variability such that we can't assert
+          that all pixels are the same).  Such tests should generally be
+          avoided (for example, by covering up the pixels that can vary
+          with another element), but when they are needed, the ranges in
+          the fuzzy() annotation should generally include 0.
+
       fuzzy-if(condition,minDiff-maxDiff,minPixelCount-maxPixelCount)
           If the condition is met, the test is treated as if 'fuzzy' had been
           specified. This is useful if there are differences on particular
-          platforms.
+          platforms.  See fuzzy() above.
 
       require-or(cond1&&cond2&&...,fallback)
           Require some particular setup be performed or environmental
           condition(s) made true (eg setting debug mode) before the test
           is run. If any condition is unknown, unimplemented, or fails,
           revert to the fallback failure-type.
           Example: require-or(debugMode,skip)