servo: Merge #10899 - Added /etc/ci/chaos_monkey_test.py (from asajeffrey:chaos-monkey-ci); r=aneeshusa
authorAlan Jeffrey <ajeffrey@mozilla.com>
Fri, 29 Apr 2016 16:01:45 -0700
changeset 338667 0f2c1af44a4956a40a0282a105effa47a801366a
parent 338666 be5d2d5fb6f1adcba0bccac5c9db2fd82ffaef12
child 338668 dee3192ac5a4ec39e31f93324a1ed524560c3f2d
push id31307
push usergszorc@mozilla.com
push dateSat, 04 Feb 2017 00:59:06 +0000
treeherdermozilla-central@94079d43835f [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewersaneeshusa
servo: Merge #10899 - Added /etc/ci/chaos_monkey_test.py (from asajeffrey:chaos-monkey-ci); r=aneeshusa Add a test to `/etc/ci` which runs a subset of `test-wpt` with `--random-pipeline-failure-probability=0.2`, and checks to make sure that there's no `CRASH` reports, so the constellation survived the experience, even if a lot of tests failed. IRC conversation at http://logs.glob.uno/?c=mozilla%23servo&s=27+Apr+2016&e=27+Apr+2016#c416510 Fixes #10568. r? @aneeshusa Source-Repo: https://github.com/servo/servo Source-Revision: a7a8eba13e799b707495a851b786009aa9652fea
servo/etc/ci/chaos_monkey_test.py
new file mode 100644
--- /dev/null
+++ b/servo/etc/ci/chaos_monkey_test.py
@@ -0,0 +1,50 @@
+# Copyright 2013 The Servo Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+from __future__ import absolute_import, print_function
+
+import json
+import sys
+from subprocess import Popen, PIPE
+
+
+TEST_CMD = [
+    "./mach",
+    "test-wpt",
+    "--release",
+    "--processes=24",
+    "--binary-arg=--random-pipeline-closure-probability=0.1",
+    "--binary-arg=--random-pipeline-closure-seed=123",
+    "--binary-arg=--multiprocess",
+    "--binary-arg=--soft-fail",
+    "--log-raw=-",
+    # We run the content-security-policy test because it creates
+    # cross-origin iframes, which are good for stress-testing pipelines
+    "content-security-policy"
+]
+
+# Note that there will probably be test failures caused
+# by random pipeline closure, so we ignore the status code
+# returned by the test command (which is why we can't use check_output).
+
+test_results = Popen(TEST_CMD, stdout=PIPE)
+any_crashes = False
+
+for line in test_results.stdout:
+    report = json.loads(line.decode('utf-8'))
+    if report.get("action") == "process_output":
+        print("{} - {}".format(report.get("thread"), report.get("data")))
+    status = report.get("status")
+    if status:
+        print("{} - {} - {}".format(report.get("thread"), status, report.get("test")))
+        if status == "CRASH":
+            any_crashes = True
+
+if any_crashes:
+    sys.exit(1)