<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[284866] trunk/Tools</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/284866">284866</a></dd>
<dt>Author</dt> <dd>simon.fraser@apple.com</dd>
<dt>Date</dt> <dd>2021-10-26 08:04:13 -0700 (Tue, 26 Oct 2021)</dd>
</dl>

<h3>Log Message</h3>
<pre>Don't run ImageDiff a second time to generate diff images
https://bugs.webkit.org/show_bug.cgi?id=232288

Reviewed by Martin Robinson.

Currently, for a ref test failure (which is always run with tolerance=0), we run ImageDiff a
second time in FailureReftestMismatch.write_failure() with the intent of generating a diff
image with zero tolerance.

Fix by storing the ImageDiffResult in FailureReftestMismatch and FailureImageHashMismatch so
we already have the diff image. We only regenerate it when the first diff was run with a
non-zero tolerance (only relevant for pixel tests). To faciliate this, store the tolerance
that was used inside ImageDiffResult too.

* ImageDiff/ImageDiff.cpp:
(main): Show tolerance in verbose logging.
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner._compare_image):
(SingleTestRunner._compare_output_with_reference):
* Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py:
(TestResultWriterTest.test_reftest_diff_image.ImageDiffTestPort.diff_image):
(TestResultWriterTest):
(TestResultWriterTest.test_reftest_diff_image):
* Scripts/webkitpy/layout_tests/models/test_failures.py:
(FailureImageHashMismatch.__init__):
(FailureReftestMismatch.__init__):
(FailureReftestMismatch.write_failure):
* Scripts/webkitpy/layout_tests/models/test_run_results.py:
(_interpret_test_failures):
* Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py:
(InterpretTestFailuresTest.test_interpret_test_failures):
* Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(RunTest.test_tolerance.ImageDiffTestPort.diff_image):
* Scripts/webkitpy/port/image_diff.py:
(ImageDiffResult.__init__):
(ImageDiffResult.__repr__):
(ImageDiffer._read):
* Scripts/webkitpy/port/port_testcase.py:
(PortTestCase.test_diff_image):</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsImageDiffImageDiffcpp">trunk/Tools/ImageDiff/ImageDiff.cpp</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testscontrollerssingle_test_runnerpy">trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testscontrollerstest_result_writer_unittestpy">trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testsmodelstest_failurespy">trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testsmodelstest_run_resultspy">trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testsmodelstest_run_results_unittestpy">trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpylayout_testsrun_webkit_tests_integrationtestpy">trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyportimage_diffpy">trunk/Tools/Scripts/webkitpy/port/image_diff.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyportport_testcasepy">trunk/Tools/Scripts/webkitpy/port/port_testcase.py</a></li>
<li><a href="#trunkToolsScriptswebkitpyporttestpy">trunk/Tools/Scripts/webkitpy/port/test.py</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog    2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/ChangeLog       2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -1,3 +1,45 @@
</span><ins>+2021-10-26  Simon Fraser  <simon.fraser@apple.com>
+
+        Don't run ImageDiff a second time to generate diff images
+        https://bugs.webkit.org/show_bug.cgi?id=232288
+
+        Reviewed by Martin Robinson.
+
+        Currently, for a ref test failure (which is always run with tolerance=0), we run ImageDiff a
+        second time in FailureReftestMismatch.write_failure() with the intent of generating a diff
+        image with zero tolerance.
+
+        Fix by storing the ImageDiffResult in FailureReftestMismatch and FailureImageHashMismatch so
+        we already have the diff image. We only regenerate it when the first diff was run with a
+        non-zero tolerance (only relevant for pixel tests). To faciliate this, store the tolerance
+        that was used inside ImageDiffResult too.
+
+        * ImageDiff/ImageDiff.cpp:
+        (main): Show tolerance in verbose logging.
+        * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
+        (SingleTestRunner._compare_image):
+        (SingleTestRunner._compare_output_with_reference):
+        * Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py:
+        (TestResultWriterTest.test_reftest_diff_image.ImageDiffTestPort.diff_image):
+        (TestResultWriterTest):
+        (TestResultWriterTest.test_reftest_diff_image):
+        * Scripts/webkitpy/layout_tests/models/test_failures.py:
+        (FailureImageHashMismatch.__init__):
+        (FailureReftestMismatch.__init__):
+        (FailureReftestMismatch.write_failure):
+        * Scripts/webkitpy/layout_tests/models/test_run_results.py:
+        (_interpret_test_failures):
+        * Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py:
+        (InterpretTestFailuresTest.test_interpret_test_failures):
+        * Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
+        (RunTest.test_tolerance.ImageDiffTestPort.diff_image):
+        * Scripts/webkitpy/port/image_diff.py:
+        (ImageDiffResult.__init__):
+        (ImageDiffResult.__repr__):
+        (ImageDiffer._read):
+        * Scripts/webkitpy/port/port_testcase.py:
+        (PortTestCase.test_diff_image):
+
</ins><span class="cx"> 2021-10-25  Ryan Haddad  <ryanhaddad@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Change default iOS simulator to one with a larger screen size
</span></span></pre></div>
<a id="trunkToolsImageDiffImageDiffcpp"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ImageDiff/ImageDiff.cpp (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ImageDiff/ImageDiff.cpp      2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/ImageDiff/ImageDiff.cpp 2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -217,7 +217,7 @@
</span><span class="cx"> 
</span><span class="cx">         if (actualImage && baselineImage) {
</span><span class="cx">             if (verbose)
</span><del>-                fprintf(stderr, "ImageDiff: processing images\n");
</del><ins>+                fprintf(stderr, "ImageDiff: processing images with tolerance %01.2f%%\n", tolerance);
</ins><span class="cx">             auto result = processImages(std::exchange(actualImage, { }), std::exchange(baselineImage, { }), tolerance, printDifference);
</span><span class="cx">             if (result != EXIT_SUCCESS)
</span><span class="cx">                 return result;
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testscontrollerssingle_test_runnerpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py      2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py 2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -295,7 +295,7 @@
</span><span class="cx">         elif driver_output.image_hash != expected_driver_output.image_hash:
</span><span class="cx">             diff_result = self._port.diff_image(expected_driver_output.image, driver_output.image)
</span><span class="cx">             if not diff_result.passed:
</span><del>-                failures.append(test_failures.FailureImageHashMismatch(diff_result.diff_percent))
</del><ins>+                failures.append(test_failures.FailureImageHashMismatch(diff_result))
</ins><span class="cx">                 if diff_result.error_string:
</span><span class="cx">                     _log.warning('  %s : %s' % (self._test_name, diff_result.error_string))
</span><span class="cx">                     driver_output.error = (driver_output.error or '') + diff_result.error_string
</span><span class="lines">@@ -356,7 +356,7 @@
</span><span class="cx">             # ImageDiff has a hard coded color distance threshold even though tolerance=0 is specified.
</span><span class="cx">             diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
</span><span class="cx">             if not diff_result.passed:
</span><del>-                failures.append(test_failures.FailureReftestMismatch(reference_filename))
</del><ins>+                failures.append(test_failures.FailureReftestMismatch(reference_filename, diff_result))
</ins><span class="cx">                 if diff_result.error_string:
</span><span class="cx">                     _log.warning('  %s : %s' % (self._test_name, diff_result.error_string))
</span><span class="cx">                     actual_driver_output.error = (actual_driver_output.error or '') + diff_result.error_string
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testscontrollerstest_result_writer_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py     2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py        2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -40,9 +40,9 @@
</span><span class="cx">         used_tolerance_values = []
</span><span class="cx"> 
</span><span class="cx">         class ImageDiffTestPort(TestPort):
</span><del>-            def diff_image(self, expected_contents, actual_contents, tolerance=None):
</del><ins>+            def diff_image(self, expected_contents, actual_contents, tolerance):
</ins><span class="cx">                 used_tolerance_values.append(tolerance)
</span><del>-                return ImageDiffResult(passed=False, diff_image=b'', difference=1)
</del><ins>+                return ImageDiffResult(passed=False, diff_image=b'', difference=1, tolerance=tolerance)
</ins><span class="cx"> 
</span><span class="cx">         host = MockHost()
</span><span class="cx">         port = ImageDiffTestPort(host)
</span><span class="lines">@@ -50,7 +50,7 @@
</span><span class="cx">         test_reference_file = host.filesystem.join(port.layout_tests_dir(), 'failures/unexpected/reftest-expected.html')
</span><span class="cx">         driver_output1 = DriverOutput('text1', 'image1', 'imagehash1', 'audio1')
</span><span class="cx">         driver_output2 = DriverOutput('text2', 'image2', 'imagehash2', 'audio2')
</span><del>-        failures = [test_failures.FailureReftestMismatch(test_reference_file)]
</del><ins>+        failures = [test_failures.FailureReftestMismatch(test_reference_file, ImageDiffResult(passed=False, diff_image=b'', difference=1, tolerance=1))]
</ins><span class="cx">         test_result_writer.write_test_result(host.filesystem, ImageDiffTestPort(host), port.results_directory(), test_name,
</span><span class="cx">                                              driver_output1, driver_output2, failures)
</span><span class="cx">         self.assertEqual([0], used_tolerance_values)
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testsmodelstest_failurespy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py        2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py   2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -201,9 +201,9 @@
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx"> class FailureImageHashMismatch(TestFailure):
</span><del>-    def __init__(self, diff_percent=0):
</del><ins>+    def __init__(self, image_diff_result=None):
</ins><span class="cx">         super(FailureImageHashMismatch, self).__init__()
</span><del>-        self.diff_percent = diff_percent
</del><ins>+        self.image_diff_result = image_diff_result
</ins><span class="cx"> 
</span><span class="cx">     def message(self):
</span><span class="cx">         return "image diff"
</span><span class="lines">@@ -219,10 +219,10 @@
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx"> class FailureReftestMismatch(TestFailure):
</span><del>-    def __init__(self, reference_filename=None):
</del><ins>+    def __init__(self, reference_filename=None, image_diff_result=None):
</ins><span class="cx">         super(FailureReftestMismatch, self).__init__()
</span><span class="cx">         self.reference_filename = reference_filename
</span><del>-        self.diff_percent = None
</del><ins>+        self.image_diff_result = image_diff_result
</ins><span class="cx"> 
</span><span class="cx">     def message(self):
</span><span class="cx">         return "reference mismatch"
</span><span class="lines">@@ -229,15 +229,16 @@
</span><span class="cx"> 
</span><span class="cx">     def write_failure(self, writer, driver_output, expected_driver_output, port):
</span><span class="cx">         writer.write_image_files(driver_output.image, expected_driver_output.image)
</span><del>-        # FIXME: This work should be done earlier in the pipeline (e.g., when we compare images for non-ref tests).
-        # FIXME: We should always have 2 images here.
-        if driver_output.image and expected_driver_output.image:
-            diff_result = port.diff_image(expected_driver_output.image, driver_output.image, tolerance=0)
-            if diff_result.diff_image:
-                writer.write_image_diff_files(diff_result.diff_image)
-                self.diff_percent = diff_result.diff_percent
</del><ins>+        if self.image_diff_result:
+            # If the ref test was run with non-zero tolerance, generate the image diff again with zero tolerance.
+            if self.image_diff_result.tolerance != 0:
+                diff_image = port.diff_image(expected_driver_output.image, driver_output.image, tolerance=0).diff_image
</ins><span class="cx">             else:
</span><del>-                _log.warn('ref test mismatch did not produce an image diff.')
</del><ins>+                diff_image = self.image_diff_result.diff_image
+
+            writer.write_image_diff_files(diff_image)
+        else:
+            _log.warn('ref test mismatch did not produce an image diff.')
</ins><span class="cx">         writer.write_reftest(self.reference_filename)
</span><span class="cx"> 
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testsmodelstest_run_resultspy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py     2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py        2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -213,7 +213,7 @@
</span><span class="cx">     if 'image_diff_percent' not in test_dict:
</span><span class="cx">         for failure in failures:
</span><span class="cx">             if isinstance(failure, test_failures.FailureImageHashMismatch) or isinstance(failure, test_failures.FailureReftestMismatch):
</span><del>-                test_dict['image_diff_percent'] = failure.diff_percent
</del><ins>+                test_dict['image_diff_percent'] = failure.image_diff_result.diff_percent
</ins><span class="cx"> 
</span><span class="cx">     return test_dict
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testsmodelstest_run_results_unittestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py    2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py       2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -33,6 +33,7 @@
</span><span class="cx"> from webkitpy.layout_tests.models import test_failures
</span><span class="cx"> from webkitpy.layout_tests.models import test_results
</span><span class="cx"> from webkitpy.layout_tests.models import test_run_results
</span><ins>+from webkitpy.port.image_diff import ImageDiffResult
</ins><span class="cx"> from webkitpy.tool.mocktool import MockOptions
</span><span class="cx"> 
</span><span class="cx"> from webkitcorepy import OutputCapture
</span><span class="lines">@@ -124,10 +125,10 @@
</span><span class="cx">         self.port = host.port_factory.get(port_name='test')
</span><span class="cx"> 
</span><span class="cx">     def test_interpret_test_failures(self):
</span><del>-        test_dict = test_run_results._interpret_test_failures([test_failures.FailureImageHashMismatch(diff_percent=0.42)])
</del><ins>+        test_dict = test_run_results._interpret_test_failures([test_failures.FailureImageHashMismatch(ImageDiffResult(passed=False, diff_image=b'', difference=0.42))])
</ins><span class="cx">         self.assertEqual(test_dict['image_diff_percent'], 0.42)
</span><span class="cx"> 
</span><del>-        test_dict = test_run_results._interpret_test_failures([test_failures.FailureReftestMismatch(self.port.abspath_for_test('foo/reftest-expected.html'))])
</del><ins>+        test_dict = test_run_results._interpret_test_failures([test_failures.FailureReftestMismatch(self.port.abspath_for_test('foo/reftest-expected.html'), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))])
</ins><span class="cx">         self.assertIn('image_diff_percent', test_dict)
</span><span class="cx"> 
</span><span class="cx">         test_dict = test_run_results._interpret_test_failures([test_failures.FailureReftestMismatchDidNotOccur(self.port.abspath_for_test('foo/reftest-expected-mismatch.html'))])
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpylayout_testsrun_webkit_tests_integrationtestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py    2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py       2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -752,7 +752,7 @@
</span><span class="cx">         class ImageDiffTestPort(test.TestPort):
</span><span class="cx">             def diff_image(self, expected_contents, actual_contents, tolerance=None):
</span><span class="cx">                 self.tolerance_used_for_diff_image = self._options.tolerance
</span><del>-                return ImageDiffResult(passed=False, diff_image=b'', difference=1)
</del><ins>+                return ImageDiffResult(passed=False, diff_image=b'', difference=1, tolerance=self._options.tolerance or 0)
</ins><span class="cx"> 
</span><span class="cx">         def get_port_for_run(args):
</span><span class="cx">             options, parsed_args = run_webkit_tests.parse_args(args)
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyportimage_diffpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/port/image_diff.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/port/image_diff.py  2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/port/image_diff.py     2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -40,10 +40,11 @@
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx"> class ImageDiffResult(object):
</span><del>-    def __init__(self, passed, diff_image, difference, error_string=None):
</del><ins>+    def __init__(self, passed, diff_image, difference, tolerance=0, error_string=None):
</ins><span class="cx">         self.passed = passed
</span><span class="cx">         self.diff_image = diff_image
</span><span class="cx">         self.diff_percent = difference
</span><ins>+        self.tolerance = tolerance
</ins><span class="cx">         self.error_string = error_string
</span><span class="cx"> 
</span><span class="cx">     def __eq__(self, other):
</span><span class="lines">@@ -51,6 +52,7 @@
</span><span class="cx">             return (self.passed == other.passed and
</span><span class="cx">                     self.diff_image == other.diff_image and
</span><span class="cx">                     self.diff_percent == other.diff_percent and
</span><ins>+                    self.tolerance == other.tolerance and
</ins><span class="cx">                     self.error_string == other.error_string)
</span><span class="cx"> 
</span><span class="cx">         return False
</span><span class="lines">@@ -59,7 +61,7 @@
</span><span class="cx">         return not self.__eq__(other)
</span><span class="cx"> 
</span><span class="cx">     def __repr__(self):
</span><del>-        return 'ImageDiffResult(Passed {} {} {} {})'.format(self.passed, self.diff_image, self.diff_percent, self.error_string)
</del><ins>+        return 'ImageDiffResult(Passed {} {} diff {} tolerance {} {})'.format(self.passed, self.diff_image, self.diff_percent, self.tolerance, self.error_string)
</ins><span class="cx"> 
</span><span class="cx"> class ImageDiffer(object):
</span><span class="cx">     def __init__(self, port):
</span><span class="lines">@@ -133,7 +135,7 @@
</span><span class="cx">                 return ImageDiffResult(passed=True, diff_image=None, difference=0)
</span><span class="cx">             diff_percent = float(string_utils.decode(m.group(1), target_type=str))
</span><span class="cx"> 
</span><del>-        return ImageDiffResult(passed=False, diff_image=output_image, difference=diff_percent, error_string=err_str or None)
</del><ins>+        return ImageDiffResult(passed=False, diff_image=output_image, difference=diff_percent, tolerance=self._tolerance, error_string=err_str or None)
</ins><span class="cx"> 
</span><span class="cx">     def stop(self):
</span><span class="cx">         if self._process:
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyportport_testcasepy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/port/port_testcase.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/port/port_testcase.py       2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/port/port_testcase.py  2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -297,11 +297,13 @@
</span><span class="cx">         # First test the case of not using the JHBuild wrapper.
</span><span class="cx">         self.assertFalse(port._should_use_jhbuild())
</span><span class="cx"> 
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=100.0, tolerance=0.1))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, [port._path_to_image_diff(), "--tolerance", "0.1"])
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar', None), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+
+        self.assertEqual(port.diff_image(b'foo', b'bar', tolerance=None), ImageDiffResult(passed=False, diff_image=b'', difference=100.0, tolerance=0.1))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, [port._path_to_image_diff(), "--tolerance", "0.1"])
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar', 0), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+
+        self.assertEqual(port.diff_image(b'foo', b'bar', tolerance=0), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, [port._path_to_image_diff(), "--tolerance", "0"])
</span><span class="cx"> 
</span><span class="cx">         # Now test the case of using JHBuild wrapper.
</span><span class="lines">@@ -308,11 +310,13 @@
</span><span class="cx">         port._filesystem.maybe_make_directory(port.path_from_webkit_base('WebKitBuild', 'Dependencies%s' % port.port_name.upper()))
</span><span class="cx">         self.assertTrue(port._should_use_jhbuild())
</span><span class="cx"> 
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=100.0, tolerance=0.1))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, port._jhbuild_wrapper + [port._path_to_image_diff(), "--tolerance", "0.1"])
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar', None), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+
+        self.assertEqual(port.diff_image(b'foo', b'bar', tolerance=None), ImageDiffResult(passed=False, diff_image=b'', difference=100.0, tolerance=0.1))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, port._jhbuild_wrapper + [port._path_to_image_diff(), "--tolerance", "0.1"])
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar', 0), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+
+        self.assertEqual(port.diff_image(b'foo', b'bar', tolerance=0), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</ins><span class="cx">         self.assertEqual(self.proc.cmd, port._jhbuild_wrapper + [port._path_to_image_diff(), "--tolerance", "0"])
</span><span class="cx"> 
</span><span class="cx">         port.clean_up_test_run()
</span><span class="lines">@@ -323,13 +327,13 @@
</span><span class="cx">         port = self.make_port()
</span><span class="cx">         port._server_process_constructor = lambda port, nm, cmd, env, crash_message=None: MockServerProcess(lines=['diff: 0% passed\n'])
</span><span class="cx">         image_differ = ImageDiffer(port)
</span><del>-        self.assertEqual(image_differ.diff_image(b'foo', b'bar', 0.1), ImageDiffResult(passed=True, diff_image=None, difference=0))
</del><ins>+        self.assertEqual(image_differ.diff_image(b'foo', b'bar', tolerance=0.1), ImageDiffResult(passed=True, diff_image=None, difference=0))
</ins><span class="cx"> 
</span><span class="cx">     def test_diff_image_failed(self):
</span><span class="cx">         port = self.make_port()
</span><span class="cx">         port._server_process_constructor = lambda port, nm, cmd, env, crash_message=None: MockServerProcess(lines=['diff: 100% failed\n'])
</span><span class="cx">         image_differ = ImageDiffer(port)
</span><del>-        self.assertEqual(image_differ.diff_image(b'foo', b'bar', 0.1), ImageDiffResult(passed=False, diff_image=b'', difference=100.0))
</del><ins>+        self.assertEqual(image_differ.diff_image(b'foo', b'bar', tolerance=0.1), ImageDiffResult(passed=False, diff_image=b'', difference=100.0, tolerance=0.1))
</ins><span class="cx"> 
</span><span class="cx">     def test_diff_image_crashed(self):
</span><span class="cx">         port = self.make_port()
</span><span class="lines">@@ -345,7 +349,7 @@
</span><span class="cx"> 
</span><span class="cx">         port._server_process_constructor = make_proc
</span><span class="cx">         port.setup_test_run()
</span><del>-        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=0, error_string='ImageDiff crashed\n'))
</del><ins>+        self.assertEqual(port.diff_image(b'foo', b'bar'), ImageDiffResult(passed=False, diff_image=b'', difference=0, tolerance=0.1, error_string='ImageDiff crashed\n'))
</ins><span class="cx">         port.clean_up_test_run()
</span><span class="cx"> 
</span><span class="cx">     @slow
</span></span></pre></div>
<a id="trunkToolsScriptswebkitpyporttestpy"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/webkitpy/port/test.py (284865 => 284866)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/webkitpy/port/test.py        2021-10-26 14:55:58 UTC (rev 284865)
+++ trunk/Tools/Scripts/webkitpy/port/test.py   2021-10-26 15:04:13 UTC (rev 284866)
</span><span class="lines">@@ -408,10 +408,10 @@
</span><span class="cx">         actual_contents = string_utils.encode(actual_contents)
</span><span class="cx">         diffed = actual_contents != expected_contents
</span><span class="cx">         if not actual_contents and not expected_contents:
</span><del>-            return ImageDiffResult(passed=True, diff_image=None, difference=0)
</del><ins>+            return ImageDiffResult(passed=True, diff_image=None, difference=0, tolerance=tolerance or 0)
</ins><span class="cx"> 
</span><span class="cx">         if not actual_contents or not expected_contents:
</span><del>-            return ImageDiffResult(passed=False, diff_image=b'', difference=0)
</del><ins>+            return ImageDiffResult(passed=False, diff_image=b'', difference=0, tolerance=tolerance or 0)
</ins><span class="cx"> 
</span><span class="cx">         if b'ref' in expected_contents:
</span><span class="cx">             assert tolerance == 0
</span><span class="lines">@@ -422,9 +422,10 @@
</span><span class="cx">                     string_utils.decode(expected_contents, target_type=str),
</span><span class="cx">                     string_utils.decode(actual_contents, target_type=str),
</span><span class="cx">                 ),
</span><del>-                difference=1)
</del><ins>+                difference=1,
+                tolerance=tolerance or 0)
</ins><span class="cx"> 
</span><del>-        return ImageDiffResult(passed=True, diff_image=None, difference=0)
</del><ins>+        return ImageDiffResult(passed=True, diff_image=None, difference=0, tolerance=tolerance or 0)
</ins><span class="cx"> 
</span><span class="cx">     def layout_tests_dir(self):
</span><span class="cx">         return LAYOUT_TEST_DIR
</span></span></pre>
</div>
</div>

</body>
</html>