[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ojan Vafai]

On Tue, Jul 5, 2011 at 5:46 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Tue, Jul 5, 2011 at 5:40 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> > On Tue, Jul 5, 2011 at 5:16 PM, Ojan Vafai <ojan at chromium.org> wrote:
> >> On Tue, Jul 5, 2011 at 4:51 PM, Dirk Pranke <dpranke at chromium.org>
> wrote:
> >>>
> >>> On Tue, Jul 5, 2011 at 1:53 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> >>> > On Jul 5, 2011 1:26 PM, "Dirk Pranke" <dpranke at chromium.org> wrote:
> >>> >> > However, we can do the same with the existing testing framework
> since
> >>> >> > we
> >>> >> > can
> >>> >> > associate a test with a bug by adding a line like this:
> >>> >> > BUGWK????? my-test.html = PASS
> >>> >>
> >>> >> You lost me here ...
> >>> >
> >>> > What I meant is that we can use test_expectations.txt to annotate a
> test
> >>> > with a bug number even if we overrode -expected.txt by adding that
> line.
> >>> > This allows us to track of tests with bad expected results while also
> >>> > allowing us to catch any changes in actual result.
> >>> >
> >>>
> >>> Ah. That makes a lot of sense. Good idea.
> >>>
> >>> On Tue, Jul 5, 2011 at 3:46 PM, Ojan Vafai <ojan at chromium.org> wrote:
> >>> > We could simplify the syntax somewhat to not require the "= PASS" at
> the
> >>> > end. We could also change the bug format to be actual links instead
> >>> > (e.g.
> >>> > webkit.org/b/12345 and crbug.com/12345).
> >>> > webkit.org/b/12345 : fast/canvas/canvastest.html
> >>> > You can also list multiple bug per line:
> >>> > webkit.org/b/12345 webkit.org/b/33333 : fast/canvas/canvasttest.html
> >>> > That seem OK?
> >>>
> >>> I'm not so much a fan of this change. It's more typing and I'm not
> >>> sure if it makes anything much easier for the user (maybe viewing in a
> >>> text editor that will automatically hyperlink the text, I guess).
> >>>
> >>> Note that the current syntax does support multiple bugs per line
> >>> (although not the hyperlink syntax you propose); I'm sure you knew
> >>> that, so perhaps you were just mentioning that for the others on the
> >>> list.
> >>
> >> One of the problems with the current format is that it's excessively
> >> complicated. Avoiding magic syntax and just using URLs makes it so there
> is
> >> less specialized syntax to make sense of. It also has the potential
> >> advantages you mention (e.g. you could copy-paste the text into your
> >> browser's address bar), but that's secondary.
> >>
> >
> > I keep hearing that the syntax is "excessively complicated". It's a
> > pretty simple syntax, but even you think that it is complicated, but
> > in what way is it excessively so, given that we actively use all of
> > the features it supports?
> >
> > Also note that changing BUGXXX to a URL doesn't necessarily make it
> > less complicated; It may actually make it more complicated, since you
> > would either have to parse URLs out from the tokens or decide what
> > sort of URL-like syntax you were going to allow.
>

Most people's interaction with this file is brief and infrequent. So, every
time they encounter it again, they need to figure out how it all works
again.  The bug syntax isn't the most confusing part of the syntax, but it
adds a bit to the confusion.

I think people are more confused with the various failure types. If we move
to a model where we check in failing expectations and just list them with
the bug number, I would be fine with going back to a world without
fine-grained failure types (i.e. we'd just have PASS, FAIL, TIMEOUT and
CRASH).

People are also confused by dealing with the multitude of different
platforms, but I don't have concrete suggestions on how to improve that
situation.

Finally, people are confused by how SLOW works. I'd much rather we just
increase the default timeout and give a shorter timeout for tests that are
listed as having TIMEOUT expectations. That maintains the benefits to bot
cycle time without needing to manually maintain a list of slow tests.

My apologies if this came off sharper than I had intended; I only
>

No worries. Wasn't offended.


> meant to focus on specific suggestions (like your URL change), rather
> than vague comments. I am certainly open to anything that makes things
> simpler, I just don't know how much there is to simplify without also
> losing functionality. I'm also open to losing functionality that isn't
> really that valuable, just that that needs to be weighed in,
> obviously.
>

I think we agree here. I've just become more and more convinced that some
loss of functionality may be worth the benefit of people being able to make
more sense of this and the benefit of just having fewer lines in this file.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110705/f9c7ef47/attachment.html>