Bug 1163740 - Paper over qcms crashes due to NaN values in qcms_transform::matrix. r=dmajor, a=sledru
authorSeth Fowler <mark.seth.fowler@gmail.com>
Tue, 26 May 2015 17:34:21 -0700
changeset 266162 9463b7173dd9
parent 266161 e712de047451
child 266163 5c4d05edc7cf
push id4770
push userryanvm@gmail.com
push date2015-06-02 17:21 +0000
treeherdermozilla-beta@9463b7173dd9 [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersdmajor, sledru
bugs1163740
milestone39.0
Bug 1163740 - Paper over qcms crashes due to NaN values in qcms_transform::matrix. r=dmajor, a=sledru
gfx/qcms/transform.c
--- a/gfx/qcms/transform.c
+++ b/gfx/qcms/transform.c
@@ -1332,16 +1332,26 @@ qcms_transform* qcms_transform_create(
 		out_matrix = build_colorant_matrix(out);
 		out_matrix = matrix_invert(out_matrix);
 		if (out_matrix.invalid) {
 			qcms_transform_release(transform);
 			return NULL;
 		}
 		result = matrix_multiply(out_matrix, in_matrix);
 
+		/* check for NaN values in the matrix and bail if we find any */
+		for (unsigned i = 0 ; i < 3 ; ++i) {
+			for (unsigned j = 0 ; j < 3 ; ++j) {
+				if (result.m[i][j] != result.m[i][j]) {
+					qcms_transform_release(transform);
+					return NULL;
+				}
+			}
+		}
+
 		/* store the results in column major mode
 		 * this makes doing the multiplication with sse easier */
 		transform->matrix[0][0] = result.m[0][0];
 		transform->matrix[1][0] = result.m[0][1];
 		transform->matrix[2][0] = result.m[0][2];
 		transform->matrix[0][1] = result.m[1][0];
 		transform->matrix[1][1] = result.m[1][1];
 		transform->matrix[2][1] = result.m[1][2];