[webkit-dev] are fuzzer tests appropriate layout tests?

[Darin Adler]

On Jun 13, 2012, at 12:12 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Wed, Jun 13, 2012 at 12:05 PM, Darin Adler <darin at apple.com> wrote:
>> On Jun 12, 2012, at 5:17 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> 
>>> It's great to use a fuzzer in order to find cases where we're broken and then make reduced layout tests from those.
>> 
>> Generally we do require a test each time we fix a bug. So it’s a strategy for the project to always make reduced tests when we find a bug.
>> 
>> But using a fuzzer to find bugs and then making a regression test for each bug we find will not give us great coverage. We’d like tests that cover lots of the code paths in WebKit, even the ones without bugs.
>> 
>> I’m not saying we should necessarily keep fuzzer-style tests, but to replace them we would need to add tests with good coverage, going beyond regression tests for bugs that existed in the project at one point.
> 
> I have always been under the impression that  LayoutTests were not just intended for preventing regressions to bugfixes, but that we should also be adding tests to establish correctness (and hopefully achieve good coverage) there.

That’s right. Did my words above give an impression to the contrary?

I am trying to say that we should be sure to keep good coverage when we remove a fuzzer-style test, possibly by adding tests that cover the same code in a different way.

I’m not making some kind of global statement about all the tests.

-- Darin