[webkit-changes] [WebKit/WebKit] b3d269: [Static Analyzer CI] Reported results do not refle...

[Brianna Fan]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: b3d26961a782dcdcc7fa912ffa4de9c4943e944f
      https://github.com/WebKit/WebKit/commit/b3d26961a782dcdcc7fa912ffa4de9c4943e944f
  Author: Brianna Fan <bfan2 at apple.com>
  Date:   2025-02-19 (Wed, 19 Feb 2025)

  Changed paths:
    M Tools/CISupport/ews-build/factories.py
    M Tools/CISupport/ews-build/factories_unittest.py
    M Tools/CISupport/ews-build/steps.py
    M Tools/CISupport/ews-build/steps_unittest.py

  Log Message:
  -----------
  [Static Analyzer CI] Reported results do not reflect changes to expectations in PRs
rdar://140374383
https://bugs.webkit.org/show_bug.cgi?id=284085

Reviewed by Aakash Jain.

When expectations are changed in the PR, EWS results should abide by those expectations.
If not, we see inaccurate comments on PRs and miss potential regressions.

This change introduces logic that checks for modified expectations and will filter the
given results to conform to the expectations.
    1. If a file is removed from expectations, it will be treated as an unexpected failure.
       - If it passes, we suppress the pass as expected.
       - If it fails, we surface the failure, even if it was pre-existing.
    2. If a file is added to expectations, it will be treated as an expected failure.
       - If it passes, we surface the unexpected pass.
       - If it fails, we suppress the failure as expected.

* Tools/CISupport/ews-build/factories.py: Add FindModifiedSaferCPPExpectations to the factory.
(SaferCPPStaticAnalyzerFactory.__init__):
* Tools/CISupport/ews-build/factories_unittest.py:
(TestExpectedBuildSteps):

* Tools/CISupport/ews-build/steps.py:
(FindModifiedSaferCPPExpectations):
    - New step that parses the diff of the PR to find which Safer CPP expectations
      were modified by the PR.
    - Tests are saved as 'user_removed_tests' and 'user_added_tests' build properties.
(FindModifiedSaferCPPExpectations.__init__):
(FindModifiedSaferCPPExpectations.run):
(FindModifiedSaferCPPExpectations.getResultSummary):

(FindUnexpectedStaticAnalyzerResults):
(FindUnexpectedStaticAnalyzerResults.run):
    - On this first run, we compare against the expectations in the PR. If the unexpected results
      match with the user modified tests, we can return results and skip a second build/results_db.
    - Otherwise, we check results_db but we ignore filtering tests if they are modified by the user.
(FindUnexpectedStaticAnalyzerResults.decode_results_data):
    - Load the log text in this function.
(FindUnexpectedStaticAnalyzerResults.get_unexpected_tests):
    - Simple function to convert a dictionary of results into a list of test strings.
(FindUnexpectedStaticAnalyzerResults.filter_results_using_results_db):
(FindUnexpectedStaticAnalyzerResults.check_results_db):
    - This is where we check for user modified tests. The logic is as follows:
      If a test is a pre-existing failure or pass after checking resultsdb, we
          1. Keep it as unexpected failure if a user removed the test
          2. Keep it as unexpected pass if a user added the test
          3. Remove it from unexpected if a user did not modify the test (normal behaviour)

(FindUnexpectedStaticAnalyzerResultsWithoutChange):
    - On the second run, we compare the build with PR changes to a build on ToT.
      If there are user modified tests, we filter the unexpected results and regenerate the results
      index if needed.
(FindUnexpectedStaticAnalyzerResultsWithoutChange.run):
(FindUnexpectedStaticAnalyzerResultsWithoutChange.filter_results_by_user_modification):
    - The filtering logic lives here!
    - For tests that the user added to expectations:
           1. If unexpected pass in the 1st run, add to unexpected passes here.
           2. If unexpected fail in the 2nd run, remove from unexpected failures here.
    - For tests that the user removed from expectations:
           1. If unexpected fail in the 1st run, add to unexpected failures here.
           2. If unexpected pass in the 2nd run, remove from unexpected passes here.
    - This filter ensures that the test results reflect the user modified expectations and not
      just which results were new to this PR.

* Tools/CISupport/ews-build/steps_unittest.py:

Canonical link: https://commits.webkit.org/290621@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications