[webkit-dev] any opposition to enabling reftests by default?

[Ojan Vafai]

On Tue, Nov 1, 2011 at 4:41 PM, Simon Fraser <simon.fraser at apple.com> wrote:

> On Nov 1, 2011, at 4:33 PM, Ojan Vafai wrote:
>
> As it is, we only run reftests if you run with --pixel. Is anyone opposed
> to running them even if you don't pass --pixel. It's a bit weird because
> they do ultimately compare images, but they don't have most of the problems
> that cause ports to not run pixel tests by default.
>
> Notably, it was confusing to me:
> https://bugs.webkit.org/show_bug.cgi?id=71158. In that case, we could
> also just print an error if you manually run a reftest without --pixel, but
> it would be great for the project if all ports ran reftests by default.
>
>
> What does this imply? Does enabling reftests turn on some new set of
> tests, does it change the way we run some existing set of tests, or does it
> add additional steps to an existing set of tests? What kind of new failures
> should we expect?
>

It causes a new set of tests to run. There's a dozen or so reftests in the
tree right now. When they fail, you'll see a pixel diff the same way you
would with pixel tests, but we don't compare the rendertrees, so there's
never a text diff. In theory, none of the current reftests have any
platform specific bits, so they should pass for all ports.

Currently, the thing that determines whether something is a reftest is
whether there is a -expected.html file. So, foo/bar/myreftest.html will
have a foo/bar/myreftest-expected.html that should render the same as
myreftest.html pixel-wise.

As we import reftest suites, we may add other ways of making things
reftests (e.g. putting them in a reftests folder?). See
https://bugs.webkit.org/show_bug.cgi?id=66295.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111101/fbfad517/attachment.html>