<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[174130] trunk/Tools</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/174130">174130</a></dd>
<dt>Author</dt> <dd>commit-queue@webkit.org</dd>
<dt>Date</dt> <dd>2014-09-30 15:58:25 -0700 (Tue, 30 Sep 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>LayoutTestResults and ExpectedFailures should know about the
interrupted flag from the json results file
https://bugs.webkit.org/show_bug.cgi?id=137229
Patch by Jake Nielsen <jacob_nielsen@apple.com> on 2014-09-30
Reviewed by Daniel Bates.
Changes LayoutTestResults to use the interrupted flag instead of
counting failures.
* Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py:
(BuilderTest._install_fetch_build._mock_fetch_build):
(BuilderTest.test_latest_layout_test_results):
* Scripts/webkitpy/common/net/layouttestresults.py:
Removes notion of failure_limit_count, and adds
did_exceed_test_failure_limit.
(LayoutTestResults.results_from_string):
(LayoutTestResults.__init__):
(LayoutTestResults.did_exceed_test_failure_limit):
(LayoutTestResults): Deleted.
(LayoutTestResults.set_failure_limit_count): Deleted.
(LayoutTestResults.failure_limit_count): Deleted.
* Scripts/webkitpy/common/net/layouttestresults_unittest.py:
Removes unit test for failure_limit_count logic.
(LayoutTestResultsTest.test_set_failure_limit_count): Deleted.
* Scripts/webkitpy/common/net/resultsjsonparser.py:
Instead of providing a class method that returns a list of
TestResults objects, resultsjsonparser now provides a class
ParsedJSONResults that serves as an interface between the results.json
file and the rest of webkitpy.
(ParsedJSONResults):
(ParsedJSONResults.__init__):
(ParsedJSONResults.did_exceed_test_failure_limit):
(ParsedJSONResults.test_results):
(ResultsJSONParser): Deleted.
(ResultsJSONParser.parse_results_json): Deleted.
* Scripts/webkitpy/common/net/resultsjsonparser_unittest.py:
Updates the unit test so that it can test the new ResultsJSONParser
class.
(ParsedJSONResultsTest):
(test_basic):
(ResultsJSONParserTest): Deleted.
* Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:
Updates the unit test to instantiate ResultsJSONParser objects rather
than simple lists of TestResult objects.
(MockCommitQueue.test_results):
(FailingTestCommitQueue.test_results):
(test_flaky_test_failure):
(test_failed_archive):
* Scripts/webkitpy/tool/bot/expectedfailures.py:
Updates ExpectedFailures to use the did_exceed_test_failure_limit
method rather than counting the number of failed tests.
(ExpectedFailures._should_trust):
* Scripts/webkitpy/tool/bot/expectedfailures_unittest.py:
Updates MockResults to more closely resemble the updated
LayoutTestResults class.
(MockResults.__init__):
(MockResults.did_exceed_test_failure_limit):
(ExpectedFailuresTest.test_can_trust_results):
(ExpectedFailuresTest.test_unexpected_failures_observed):
(ExpectedFailuresTest.test_unexpected_failures_observed_when_tree_is_hosed):
(MockResults.failure_limit_count): Deleted.
* Scripts/webkitpy/tool/bot/layouttestresultsreader.py:
Removes a FIXME regarding the use of the
NON_INTERACTIVE_FAILURE_LIMIT_COUNT value.
(LayoutTestResultsReader.results):
* Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py:
Updates test to not check the now-nonexistant failure_count_limit
method.
(test_missing_unit_test_results_path):
(test_layout_test_results):
* Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py:
Accounts for name change: ResultsJSONParserTest ->
ParsedJSONParserTest.
(RebaselineTestTest.test_gather_baselines):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonnetbuildbotbuildbot_unittestpy">trunk/Tools/Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonnetlayouttestresultspy">trunk/Tools/Scripts/webkitpy/common/net/layouttestresults.py</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonnetlayouttestresults_unittestpy">trunk/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonnetresultsjsonparserpy">trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser.py</a></li>
<li><a href="#trunkToolsScriptswebkitpycommonnetresultsjsonparser_unittestpy">trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolbotcommitqueuetask_unittestpy">trunk/Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolbotexpectedfailurespy">trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolbotexpectedfailures_unittestpy">trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolbotlayouttestresultsreaderpy">trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolbotlayouttestresultsreader_unittestpy">trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpytoolserversrebaselineserver_unittestpy">trunk/Tools/Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/ChangeLog 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -1,3 +1,90 @@
</span><ins>+2014-09-30 Jake Nielsen <jacob_nielsen@apple.com>
+
+ LayoutTestResults and ExpectedFailures should know about the
+ interrupted flag from the json results file
+ https://bugs.webkit.org/show_bug.cgi?id=137229
+
+ Reviewed by Daniel Bates.
+
+ Changes LayoutTestResults to use the interrupted flag instead of
+ counting failures.
+
+ * Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py:
+ (BuilderTest._install_fetch_build._mock_fetch_build):
+ (BuilderTest.test_latest_layout_test_results):
+ * Scripts/webkitpy/common/net/layouttestresults.py:
+ Removes notion of failure_limit_count, and adds
+ did_exceed_test_failure_limit.
+
+ (LayoutTestResults.results_from_string):
+ (LayoutTestResults.__init__):
+ (LayoutTestResults.did_exceed_test_failure_limit):
+ (LayoutTestResults): Deleted.
+ (LayoutTestResults.set_failure_limit_count): Deleted.
+ (LayoutTestResults.failure_limit_count): Deleted.
+ * Scripts/webkitpy/common/net/layouttestresults_unittest.py:
+ Removes unit test for failure_limit_count logic.
+
+ (LayoutTestResultsTest.test_set_failure_limit_count): Deleted.
+ * Scripts/webkitpy/common/net/resultsjsonparser.py:
+ Instead of providing a class method that returns a list of
+ TestResults objects, resultsjsonparser now provides a class
+ ParsedJSONResults that serves as an interface between the results.json
+ file and the rest of webkitpy.
+
+ (ParsedJSONResults):
+ (ParsedJSONResults.__init__):
+ (ParsedJSONResults.did_exceed_test_failure_limit):
+ (ParsedJSONResults.test_results):
+ (ResultsJSONParser): Deleted.
+ (ResultsJSONParser.parse_results_json): Deleted.
+ * Scripts/webkitpy/common/net/resultsjsonparser_unittest.py:
+ Updates the unit test so that it can test the new ResultsJSONParser
+ class.
+
+ (ParsedJSONResultsTest):
+ (test_basic):
+ (ResultsJSONParserTest): Deleted.
+ * Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:
+ Updates the unit test to instantiate ResultsJSONParser objects rather
+ than simple lists of TestResult objects.
+
+ (MockCommitQueue.test_results):
+ (FailingTestCommitQueue.test_results):
+ (test_flaky_test_failure):
+ (test_failed_archive):
+ * Scripts/webkitpy/tool/bot/expectedfailures.py:
+ Updates ExpectedFailures to use the did_exceed_test_failure_limit
+ method rather than counting the number of failed tests.
+
+ (ExpectedFailures._should_trust):
+ * Scripts/webkitpy/tool/bot/expectedfailures_unittest.py:
+ Updates MockResults to more closely resemble the updated
+ LayoutTestResults class.
+
+ (MockResults.__init__):
+ (MockResults.did_exceed_test_failure_limit):
+ (ExpectedFailuresTest.test_can_trust_results):
+ (ExpectedFailuresTest.test_unexpected_failures_observed):
+ (ExpectedFailuresTest.test_unexpected_failures_observed_when_tree_is_hosed):
+ (MockResults.failure_limit_count): Deleted.
+ * Scripts/webkitpy/tool/bot/layouttestresultsreader.py:
+ Removes a FIXME regarding the use of the
+ NON_INTERACTIVE_FAILURE_LIMIT_COUNT value.
+
+ (LayoutTestResultsReader.results):
+ * Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py:
+ Updates test to not check the now-nonexistant failure_count_limit
+ method.
+
+ (test_missing_unit_test_results_path):
+ (test_layout_test_results):
+ * Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py:
+ Accounts for name change: ResultsJSONParserTest ->
+ ParsedJSONParserTest.
+
+ (RebaselineTestTest.test_gather_baselines):
+
</ins><span class="cx"> 2014-09-30 Gyuyoung Kim <gyuyoung.kim@samsung.com>
</span><span class="cx">
</span><span class="cx"> [EFL] Rename TEST_THEME_DIR macro
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonnetbuildbotbuildbot_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -48,7 +48,7 @@
</span><span class="cx"> is_green=build_number < 4
</span><span class="cx"> )
</span><span class="cx"> results = [self._mock_test_result(testname) for testname in failure(build_number)]
</span><del>- layout_test_results = LayoutTestResults(results)
</del><ins>+ layout_test_results = LayoutTestResults(test_results=results, did_exceed_test_failure_limit=False)
</ins><span class="cx"> def mock_layout_test_results():
</span><span class="cx"> return layout_test_results
</span><span class="cx"> build.layout_test_results = mock_layout_test_results
</span><span class="lines">@@ -61,7 +61,7 @@
</span><span class="cx"> self._install_fetch_build(lambda build_number: ["test1", "test2"])
</span><span class="cx">
</span><span class="cx"> def test_latest_layout_test_results(self):
</span><del>- self.builder.fetch_layout_test_results = lambda results_url: LayoutTestResults([self._mock_test_result(testname) for testname in ["test1", "test2"]])
</del><ins>+ self.builder.fetch_layout_test_results = lambda results_url: LayoutTestResults(test_results=[self._mock_test_result(testname) for testname in ["test1", "test2"]], did_exceed_test_failure_limit=False)
</ins><span class="cx"> self.builder.accumulated_results_url = lambda: "http://dummy_url.org"
</span><span class="cx"> self.assertTrue(self.builder.latest_layout_test_results())
</span><span class="cx">
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonnetlayouttestresultspy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/net/layouttestresults.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/net/layouttestresults.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/common/net/layouttestresults.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -28,7 +28,7 @@
</span><span class="cx">
</span><span class="cx"> import logging
</span><span class="cx">
</span><del>-from webkitpy.common.net.resultsjsonparser import ResultsJSONParser
</del><ins>+from webkitpy.common.net.resultsjsonparser import ParsedJSONResults
</ins><span class="cx"> from webkitpy.thirdparty.BeautifulSoup import BeautifulSoup, SoupStrainer
</span><span class="cx"> from webkitpy.layout_tests.models import test_results
</span><span class="cx"> from webkitpy.layout_tests.models import test_failures
</span><span class="lines">@@ -50,28 +50,17 @@
</span><span class="cx"> def results_from_string(cls, string):
</span><span class="cx"> if not string:
</span><span class="cx"> return None
</span><del>- test_results = ResultsJSONParser.parse_results_json(string)
- if not test_results:
- return None
- return cls(test_results)
</del><ins>+ parsed_results = ParsedJSONResults(string)
+ return cls(parsed_results.test_results(), parsed_results.did_exceed_test_failure_limit())
</ins><span class="cx">
</span><del>- def __init__(self, test_results):
- self._test_results = test_results
- self._failure_limit_count = None
</del><ins>+ def __init__(self, test_results, did_exceed_test_failure_limit):
</ins><span class="cx"> self._unit_test_failures = []
</span><ins>+ self._test_results = test_results
+ self._did_exceed_test_failure_limit = did_exceed_test_failure_limit
</ins><span class="cx">
</span><del>- # FIXME: run-webkit-tests should store the --exit-after-N-failures value
- # (or some indication of early exit) somewhere in the results.json
- # file. Until it does, callers should set the limit to
- # --exit-after-N-failures value used in that run. Consumers of LayoutTestResults
- # may use that value to know if absence from the failure list means PASS.
- # https://bugs.webkit.org/show_bug.cgi?id=58481
- def set_failure_limit_count(self, limit):
- self._failure_limit_count = limit
</del><ins>+ def did_exceed_test_failure_limit(self):
+ return self._did_exceed_test_failure_limit
</ins><span class="cx">
</span><del>- def failure_limit_count(self):
- return self._failure_limit_count
-
</del><span class="cx"> def test_results(self):
</span><span class="cx"> return self._test_results
</span><span class="cx">
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonnetlayouttestresults_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -36,11 +36,6 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> class LayoutTestResultsTest(unittest.TestCase):
</span><del>- def test_set_failure_limit_count(self):
- results = LayoutTestResults([])
- self.assertIsNone(results.failure_limit_count())
- results.set_failure_limit_count(10)
- self.assertEqual(results.failure_limit_count(), 10)
</del><span class="cx">
</span><span class="cx"> def test_results_from_string(self):
</span><span class="cx"> self.assertIsNone(LayoutTestResults.results_from_string(None))
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonnetresultsjsonparserpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -137,9 +137,8 @@
</span><span class="cx"> return test_results.TestResult(self._test_name, self._failures())
</span><span class="cx">
</span><span class="cx">
</span><del>-class ResultsJSONParser(object):
- @classmethod
- def parse_results_json(cls, json_string):
</del><ins>+class ParsedJSONResults(object):
+ def __init__(self, json_string):
</ins><span class="cx"> if not json_results_generator.has_json_wrapper(json_string):
</span><span class="cx"> return None
</span><span class="cx">
</span><span class="lines">@@ -149,7 +148,19 @@
</span><span class="cx"> json_results = []
</span><span class="cx"> for_each_test(json_dict['tests'], lambda test, result: json_results.append(JSONTestResult(test, result)))
</span><span class="cx">
</span><del>- # FIXME: What's the short sexy python way to filter None?
- # I would use [foo.bar() for foo in foos if foo.bar()] but bar() is expensive.
- unexpected_failures = [result.test_result() for result in json_results if not result.did_pass_or_run_as_expected()]
- return filter(lambda a: a, unexpected_failures)
</del><ins>+ unexpected_failures = []
+ for json_result in json_results:
+ if json_result.did_pass_or_run_as_expected():
+ continue
+ test_result = json_result.test_result()
+ if test_result:
+ unexpected_failures.append(test_result)
+
+ self._test_results = unexpected_failures
+ self._did_exceed_test_failure_limit = json_dict["interrupted"]
+
+ def did_exceed_test_failure_limit(self):
+ return self._did_exceed_test_failure_limit
+
+ def test_results(self):
+ return self._test_results
</ins></span></pre></div>
<a id="trunkToolsScriptswebkitpycommonnetresultsjsonparser_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -28,12 +28,12 @@
</span><span class="cx">
</span><span class="cx"> import unittest2 as unittest
</span><span class="cx">
</span><del>-from webkitpy.common.net.resultsjsonparser import ResultsJSONParser
</del><ins>+from webkitpy.common.net.resultsjsonparser import ParsedJSONResults
</ins><span class="cx"> from webkitpy.layout_tests.models import test_results
</span><span class="cx"> from webkitpy.layout_tests.models import test_failures
</span><span class="cx">
</span><span class="cx">
</span><del>-class ResultsJSONParserTest(unittest.TestCase):
</del><ins>+class ParsedJSONResultsTest(unittest.TestCase):
</ins><span class="cx"> # The real files have no whitespace, but newlines make this much more readable.
</span><span class="cx">
</span><span class="cx"> _example_full_results_json = """ADD_RESULTS({
</span><span class="lines">@@ -72,6 +72,7 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx"> },
</span><ins>+ "interrupted": true,
</ins><span class="cx"> "skipped": 450,
</span><span class="cx"> "num_regressions": 15,
</span><span class="cx"> "layout_tests_dir": "\/b\/build\/slave\/Webkit_Mac10_5\/build\/src\/third_party\/WebKit\/LayoutTests",
</span><span class="lines">@@ -83,10 +84,63 @@
</span><span class="cx"> "uses_expectations_file": true
</span><span class="cx"> });"""
</span><span class="cx">
</span><ins>+ _not_interrupted_example_full_results_json = """ADD_RESULTS({
+ "tests": {
+ "fast": {
+ "dom": {
+ "prototype-inheritance.html": {
+ "expected": "PASS",
+ "actual": "FAIL"
+ },
+ "prototype-banana.html": {
+ "expected": "FAIL",
+ "actual": "PASS"
+ },
+ "prototype-taco.html": {
+ "expected": "PASS",
+ "actual": "PASS FAIL"
+ },
+ "prototype-chocolate.html": {
+ "expected": "FAIL",
+ "actual": "FAIL"
+ },
+ "prototype-strawberry.html": {
+ "expected": "PASS",
+ "actual": "FAIL PASS"
+ }
+ }
+ },
+ "svg": {
+ "dynamic-updates": {
+ "SVGFEDropShadowElement-dom-stdDeviation-attr.html": {
+ "expected": "PASS",
+ "actual": "IMAGE",
+ "has_stderr": true
+ }
+ }
+ }
+ },
+ "interrupted": false,
+ "skipped": 450,
+ "num_regressions": 15,
+ "layout_tests_dir": "\/b\/build\/slave\/Webkit_Mac10_5\/build\/src\/third_party\/WebKit\/LayoutTests",
+ "version": 3,
+ "num_passes": 77,
+ "has_pretty_patch": false,
+ "fixable": 1220,
+ "num_flaky": 0,
+ "uses_expectations_file": true
+});"""
+
</ins><span class="cx"> def test_basic(self):
</span><span class="cx"> expected_results = [
</span><span class="cx"> test_results.TestResult("svg/dynamic-updates/SVGFEDropShadowElement-dom-stdDeviation-attr.html", [test_failures.FailureImageHashMismatch()], 0),
</span><span class="cx"> test_results.TestResult("fast/dom/prototype-inheritance.html", [test_failures.FailureTextMismatch(), test_failures.FailureImageHashMismatch(), test_failures.FailureAudioMismatch()], 0),
</span><span class="cx"> ]
</span><del>- results = ResultsJSONParser.parse_results_json(self._example_full_results_json)
- self.assertEqual(expected_results, results)
</del><ins>+ parsed_results = ParsedJSONResults(self._example_full_results_json)
+ self.assertEqual(expected_results, parsed_results.test_results())
+ self.assertTrue(parsed_results.did_exceed_test_failure_limit())
+
+ def test_not_interrupted(self):
+ parsed_results = ParsedJSONResults(self._not_interrupted_example_full_results_json)
+ self.assertFalse(parsed_results.did_exceed_test_failure_limit())
</ins></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolbotcommitqueuetask_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -73,7 +73,7 @@
</span><span class="cx"> return ExpectedFailures()
</span><span class="cx">
</span><span class="cx"> def test_results(self):
</span><del>- return None
</del><ins>+ return LayoutTestResults(test_results=[], did_exceed_test_failure_limit=True)
</ins><span class="cx">
</span><span class="cx"> def report_flaky_tests(self, patch, flaky_results, results_archive):
</span><span class="cx"> flaky_tests = [result.filename for result in flaky_results]
</span><span class="lines">@@ -110,9 +110,7 @@
</span><span class="cx"> # Doesn't make sense to ask for the test_results until the tests have run at least once.
</span><span class="cx"> assert(self._test_run_counter >= 0)
</span><span class="cx"> failures_for_run = self._test_failure_plan[self._test_run_counter]
</span><del>- results = LayoutTestResults(map(self._mock_test_result, failures_for_run))
- # This makes the results trustable by ExpectedFailures.
- results.set_failure_limit_count(10)
</del><ins>+ results = LayoutTestResults(test_results=map(self._mock_test_result, failures_for_run), did_exceed_test_failure_limit=(len(self._test_failure_plan[self._test_run_counter]) >= 10))
</ins><span class="cx"> return results
</span><span class="cx">
</span><span class="cx">
</span><span class="lines">@@ -282,7 +280,7 @@
</span><span class="cx"> ])
</span><span class="cx"> # CommitQueueTask will only report flaky tests if we successfully parsed
</span><span class="cx"> # results.json and returned a LayoutTestResults object, so we fake one.
</span><del>- commit_queue.test_results = lambda: LayoutTestResults([])
</del><ins>+ commit_queue.test_results = lambda: LayoutTestResults(test_results=[], did_exceed_test_failure_limit=True)
</ins><span class="cx"> expected_logs = """run_webkit_patch: ['clean']
</span><span class="cx"> command_passed: success_message='Cleaned working directory' patch='10000'
</span><span class="cx"> run_webkit_patch: ['update']
</span><span class="lines">@@ -313,7 +311,7 @@
</span><span class="cx"> None,
</span><span class="cx"> ScriptError("MOCK tests failure"),
</span><span class="cx"> ])
</span><del>- commit_queue.test_results = lambda: LayoutTestResults([])
</del><ins>+ commit_queue.test_results = lambda: LayoutTestResults(test_results=[], did_exceed_test_failure_limit=True)
</ins><span class="cx"> # It's possible delegate to fail to archive layout tests, don't try to report
</span><span class="cx"> # flaky tests when that happens.
</span><span class="cx"> commit_queue.archive_last_test_results = lambda patch: None
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolbotexpectedfailurespy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -38,7 +38,7 @@
</span><span class="cx">
</span><span class="cx"> @classmethod
</span><span class="cx"> def _should_trust(cls, results):
</span><del>- return bool(cls._has_failures(results) and results.failure_limit_count() and len(results.failing_tests()) < results.failure_limit_count())
</del><ins>+ return bool(results and not results.did_exceed_test_failure_limit())
</ins><span class="cx">
</span><span class="cx"> def failures_were_expected(self, results):
</span><span class="cx"> if not self._is_trustworthy:
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolbotexpectedfailures_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/bot/expectedfailures_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -32,12 +32,12 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> class MockResults(object):
</span><del>- def __init__(self, failing_tests=[], failure_limit=10):
</del><ins>+ def __init__(self, failing_tests=[], did_exceed_test_failure_limit=False):
</ins><span class="cx"> self._failing_tests = failing_tests
</span><del>- self._failure_limit_count = failure_limit
</del><ins>+ self._did_exceed_test_failure_limit = did_exceed_test_failure_limit
</ins><span class="cx">
</span><del>- def failure_limit_count(self):
- return self._failure_limit_count
</del><ins>+ def did_exceed_test_failure_limit(self):
+ return self._did_exceed_test_failure_limit
</ins><span class="cx">
</span><span class="cx"> def failing_tests(self):
</span><span class="cx"> return self._failing_tests
</span><span class="lines">@@ -49,12 +49,12 @@
</span><span class="cx">
</span><span class="cx"> def test_can_trust_results(self):
</span><span class="cx"> self._assert_can_trust(None, False)
</span><del>- self._assert_can_trust(MockResults(failing_tests=[], failure_limit=None), False)
- self._assert_can_trust(MockResults(failing_tests=[], failure_limit=10), False)
- self._assert_can_trust(MockResults(failing_tests=[1], failure_limit=None), False)
- self._assert_can_trust(MockResults(failing_tests=[1], failure_limit=2), True)
- self._assert_can_trust(MockResults(failing_tests=[1], failure_limit=1), False)
- self._assert_can_trust(MockResults(failing_tests=[1, 2], failure_limit=1), False)
</del><ins>+ self._assert_can_trust(MockResults(failing_tests=[], did_exceed_test_failure_limit=False), True)
+ self._assert_can_trust(MockResults(failing_tests=[], did_exceed_test_failure_limit=True), False)
+ self._assert_can_trust(MockResults(failing_tests=[1], did_exceed_test_failure_limit=False), True)
+ self._assert_can_trust(MockResults(failing_tests=[1], did_exceed_test_failure_limit=True), False)
+ self._assert_can_trust(MockResults(failing_tests=[1, 2], did_exceed_test_failure_limit=False), True)
+ self._assert_can_trust(MockResults(failing_tests=[1, 2], did_exceed_test_failure_limit=True), False)
</ins><span class="cx">
</span><span class="cx"> def _assert_expected(self, expected_failures, failures, expected):
</span><span class="cx"> self.assertEqual(expected_failures.failures_were_expected(MockResults(failures)), expected)
</span><span class="lines">@@ -79,17 +79,17 @@
</span><span class="cx"> failures.update(MockResults(['foo.html']))
</span><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(MockResults(['foo.html', 'bar.html'])), set(['bar.html']))
</span><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(MockResults(['baz.html'])), set(['baz.html']))
</span><del>- unbounded_results = MockResults(['baz.html', 'qux.html', 'taco.html'], failure_limit=3)
</del><ins>+ unbounded_results = MockResults(['baz.html', 'qux.html', 'taco.html'], did_exceed_test_failure_limit=True)
</ins><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(unbounded_results), set(['baz.html', 'qux.html', 'taco.html']))
</span><del>- unbounded_results_with_existing_failure = MockResults(['foo.html', 'baz.html', 'qux.html', 'taco.html'], failure_limit=4)
</del><ins>+ unbounded_results_with_existing_failure = MockResults(['foo.html', 'baz.html', 'qux.html', 'taco.html'], did_exceed_test_failure_limit=True)
</ins><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(unbounded_results_with_existing_failure), set(['baz.html', 'qux.html', 'taco.html']))
</span><span class="cx">
</span><span class="cx"> def test_unexpected_failures_observed_when_tree_is_hosed(self):
</span><span class="cx"> failures = ExpectedFailures()
</span><del>- failures.update(MockResults(['foo.html', 'banana.html'], failure_limit=2))
</del><ins>+ failures.update(MockResults(['foo.html', 'banana.html'], did_exceed_test_failure_limit=True))
</ins><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(MockResults(['foo.html', 'bar.html'])), None)
</span><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(MockResults(['baz.html'])), None)
</span><del>- unbounded_results = MockResults(['baz.html', 'qux.html', 'taco.html'], failure_limit=3)
</del><ins>+ unbounded_results = MockResults(['baz.html', 'qux.html', 'taco.html'], did_exceed_test_failure_limit=True)
</ins><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(unbounded_results), None)
</span><del>- unbounded_results_with_existing_failure = MockResults(['foo.html', 'baz.html', 'qux.html', 'taco.html'], failure_limit=4)
</del><ins>+ unbounded_results_with_existing_failure = MockResults(['foo.html', 'baz.html', 'qux.html', 'taco.html'], did_exceed_test_failure_limit=True)
</ins><span class="cx"> self.assertEqual(failures.unexpected_failures_observed(unbounded_results_with_existing_failure), None)
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolbotlayouttestresultsreaderpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -70,10 +70,6 @@
</span><span class="cx"> layout_test_results = self._create_layout_test_results()
</span><span class="cx"> unit_test_results = self._create_unit_test_results()
</span><span class="cx"> if layout_test_results:
</span><del>- # FIXME: This is used to detect if we had N failures due to
- # N tests failing, or if we hit the "exit-after-n-failures" limit.
- # These days we could just check for the "interrupted" key in results.json instead!
- layout_test_results.set_failure_limit_count(RunTests.NON_INTERACTIVE_FAILURE_LIMIT_COUNT)
</del><span class="cx"> if unit_test_results:
</span><span class="cx"> layout_test_results.add_unit_test_failures(unit_test_results)
</span><span class="cx"> return layout_test_results
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolbotlayouttestresultsreader_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/bot/layouttestresultsreader_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -68,7 +68,7 @@
</span><span class="cx"> def test_missing_unit_test_results_path(self):
</span><span class="cx"> host = MockHost()
</span><span class="cx"> reader = LayoutTestResultsReader(host, "/mock-results", "/var/logs")
</span><del>- reader._create_layout_test_results = lambda: LayoutTestResults([])
</del><ins>+ reader._create_layout_test_results = lambda: LayoutTestResults(test_results=[], did_exceed_test_failure_limit=False)
</ins><span class="cx"> reader._create_unit_test_results = lambda: None
</span><span class="cx"> # layout_test_results shouldn't raise even if the unit tests xml file is missing.
</span><span class="cx"> self.assertIsNotNone(reader.results(), None)
</span><span class="lines">@@ -81,10 +81,9 @@
</span><span class="cx"> self.assertIsNone(reader.results())
</span><span class="cx"> reader._read_file_contents = lambda path: ""
</span><span class="cx"> self.assertIsNone(reader.results())
</span><del>- reader._create_layout_test_results = lambda: LayoutTestResults([])
</del><ins>+ reader._create_layout_test_results = lambda: LayoutTestResults(test_results=[], did_exceed_test_failure_limit=False)
</ins><span class="cx"> results = reader.results()
</span><span class="cx"> self.assertIsNotNone(results)
</span><del>- self.assertEqual(results.failure_limit_count(), 30) # This value matches RunTests.NON_INTERACTIVE_FAILURE_LIMIT_COUNT
</del><span class="cx">
</span><span class="cx"> def test_archive_last_layout_test_results(self):
</span><span class="cx"> host = MockHost()
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpytoolserversrebaselineserver_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py (174129 => 174130)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py 2014-09-30 22:56:08 UTC (rev 174129)
+++ trunk/Tools/Scripts/webkitpy/tool/servers/rebaselineserver_unittest.py 2014-09-30 22:58:25 UTC (rev 174130)
</span><span class="lines">@@ -205,7 +205,7 @@
</span><span class="cx"> ])
</span><span class="cx">
</span><span class="cx"> def test_gather_baselines(self):
</span><del>- example_json = resultsjsonparser_unittest.ResultsJSONParserTest._example_full_results_json
</del><ins>+ example_json = resultsjsonparser_unittest.ParsedJSONResultsTest._example_full_results_json
</ins><span class="cx"> results_json = json.loads(strip_json_wrapper(example_json))
</span><span class="cx"> server = RebaselineServer()
</span><span class="cx"> server._test_config = get_test_config()
</span></span></pre>
</div>
</div>
</body>
</html>