[Webkit-unassigned] [Bug 229788] New: [webkitpy] Occasional exception thrown in change_result_to_failure when repeating flaky layout tests

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Wed Sep 1 18:35:26 PDT 2021


https://bugs.webkit.org/show_bug.cgi?id=229788

            Bug ID: 229788
           Summary: [webkitpy] Occasional exception thrown in
                    change_result_to_failure when repeating flaky layout
                    tests
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: lmoura at igalia.com

It's hard to reproduce with run-webkit-tests, but sometimes, when running multiple iterations of the same layout test, the run is aborted with the following backtrace (from running a single test with --iterations=100):

KeyError raised: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html'                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main                                                                                                                                                
    run_details = run(port, options, args, stderr)                                                                                                                                                                                            
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 494, in run                                                                                                                                                
    run_details = manager.run(args)                                                                                                                                                                                                           
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run                                                                                                                                             
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)                                                                                                 
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset                                                                                                                                
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)                                                          
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests               
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)                                                                                                                       
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests     
    pool.do(                                                                                                                                                                                                                                  
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 397, in do                                                                                                                                          
    result = function(*args, **kwargs)                                                                                                                                                                                                        
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard                                                                                                                             
    return Worker.instance.run_tests(shard)                                                                                                                                                                                                   
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests                                                                                                                            
    Worker.instance.run_test(input, shard.name)                                                                                                                                                                                               
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 353, in run_test                                                                                                                             
    TaskPool.Process.queue.send(TaskPool.Task(                                                                                                                                                                                                
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 165, in send                 
    object(None)                                                                                                       
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 56, in __call__              
    return self.function(*self.args, **self.kwargs)                                                                    
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 81, in handle_finished_test                                                                                                                  
    LayoutTestRunner.instance.update_summary_with_result(result)                                                       
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 241, in update_summary_with_result                                                                                                           
    self._current_run_results.change_result_to_failure(existing, result, existing_expectation, expected)               
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py", line 102, in change_result_to_failure                                                                                                                    
    self.tests_by_expectation[existing_result.type].remove(existing_result.test_name)                                                                                                                                                         
KeyError: 'fast/mediastream/RTCPeerConnection-iceconnectionstatechange-event.html'

When this happens, a common sequence of events is:

* Unexpected results appears (e.g. Failure in an expected Timeout)
* Another kind of unexpected results appears (e.g. Pass in an expected Timeout)
* The same kind of unexpected results appears (e.g. Pass in an expected Timeout)
* EXCEPTION 

This seems to happen due to the `TestResults` instance stored in `TestRunResults.result_by_name` not being updated from Failure to Pass, as `TestResults.convert_to_failure` (from r235467) just extends the list of failures (IIUC, to keep existing failures information across test runs).

This causes the Failure test result to be removed from TestRunResults.test_by_expectation[Failure] inside change_result_to_failure, but still stored as the current result. In the next Pass result, the code tries to remove the Failure it again, raising KeyError.

Tentative patch in the next comment.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20210902/4a8de535/attachment-0001.htm>


More information about the webkit-unassigned mailing list