[Webkit-unassigned] [Bug 74762] New: there should be a separate --lint-test-files step on the build bots

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=74762

           Summary: there should be a separate --lint-test-files step on
                    the build bots
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: dpranke at chromium.org
                CC: eric at webkit.org, tony at chromium.org, abarth at webkit.org,
                    ojan at chromium.org, levin at chromium.org,
                    rniwa at webkit.org


I want to do this for two reasons:

We seem to get a fair number of build failures from bad or duplicate expectations on the bots. It may be that our style checker is buggy, but I also suspect a fair number of expectations changes get checked in unreviewed, let alone un-CQ'ed. I'm not sure if webkit-patch land-cowboy tries to run the style checks on this, or if that would catch issues (or if people would just ignore or miss the errors).

Second, rniwa has largely convinced me that having duplicate expectations shouldn't be a fatal error on the  layout_tests step, but I want to make sure that we don't just start ignoring bad expectations files. Unfortunately, buildbot doesn't give us too many ways to express different kinds of failures, so it seems cleanest to me to break checking the expectations out as a separate step.

I think we should be able to tune this so this check is fast and adding a separate step doesn't slow this down by more than a few seconds (I believe linting is slower than this today, so it might need some profiling first). I think if linting the file takes more than (say) 10-15 seconds/run, it probably shouldn't be needed.

We can also probably tweak things so that we only ran this if something under LayoutTests/ changed in this build, if we had to.

What do others think? Is this worth doing, or should we just try to beef up the presubmit checks if we're missing something?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.