[webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Ojan Vafai]

On Wed, Apr 11, 2012 at 3:48 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> > On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig <weinig at apple.com> wrote:
> >>
> >> On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote:
> >>
> >> I just did a first pass a greening the Chromium Lion
> >> bot: http://trac.webkit.org/changeset/110096. Of these hundreds of
> tests,
> >>  ~99% of them are perfect candidates for being reftests (e.g. they
> contain
> >> one line of text and a solid box or two under the text), but most of
> them
> >> are in the CSS imported test suites.
> >>
> >> Is it kosher to convert them to reftests or should we leave pixel tests
> from
> >> imported test suites alone?
> >>
> >>
> >> If we want to make these ref tests, it probably makes more sense to do
> that
> >> work with the CSS WG, so that they can be part of the standard test
> suite.
> >> Until then, I think we should keep them regular pixel tests.
> >>
> >
> > Note that this thread (to resurrect it just-after-easter because it's
> > timely) is directly relevant to the note I just sent out about
> > importing test suites.
> >
> > I, at least, would like some clarification ... if we are importing
> > tests that have no accompanying "expected result" (and are expected to
> > be inspected manually for correctness), is it acceptable to write
> > reference html for the tests, or do they have to be imported as pixel
> > tests?
> >
>
> Put differently, we either need to add an -expected html or an
> -expected png. Does it have to be the latter?
>

I strongly prefer adding -expected.html files. While there is a chance the
reference may not cover all the code paths the original test intended, I
believe our overall test coverage will be better with more reftests and
fewer pixel tests, not to mention our project-wide sanity from spending
less time doing expectations file management. Pixel tests accidentally let
bugs through all the time because not all ports run them and because it's
often hard to tell if a new result is correct.

I agree with the sentiment that we should be upstreaming these to the W3C,
but I don't see why we would require upstreaming them first instead of
committing them locally and then upstreaming them. If there are people
willing to do the work, lets allow them do it either way.


> > I personally think it's acceptable, but I understand that there might
> > be a difference of opinion.
> >
> > https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this.
> >
> > -- Dirk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120411/a581ba07/attachment.html>