[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

On Tue, Jul 5, 2011 at 5:40 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> On Tue, Jul 5, 2011 at 5:16 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> On Tue, Jul 5, 2011 at 4:51 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>
>>> On Tue, Jul 5, 2011 at 1:53 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>> > On Jul 5, 2011 1:26 PM, "Dirk Pranke" <dpranke at chromium.org> wrote:
>>> >> > However, we can do the same with the existing testing framework since
>>> >> > we
>>> >> > can
>>> >> > associate a test with a bug by adding a line like this:
>>> >> > BUGWK????? my-test.html = PASS
>>> >>
>>> >> You lost me here ...
>>> >
>>> > What I meant is that we can use test_expectations.txt to annotate a test
>>> > with a bug number even if we overrode -expected.txt by adding that line.
>>> > This allows us to track of tests with bad expected results while also
>>> > allowing us to catch any changes in actual result.
>>> >
>>>
>>> Ah. That makes a lot of sense. Good idea.
>>>
>>> On Tue, Jul 5, 2011 at 3:46 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>> > We could simplify the syntax somewhat to not require the "= PASS" at the
>>> > end. We could also change the bug format to be actual links instead
>>> > (e.g.
>>> > webkit.org/b/12345 and crbug.com/12345).
>>> > webkit.org/b/12345 : fast/canvas/canvastest.html
>>> > You can also list multiple bug per line:
>>> > webkit.org/b/12345 webkit.org/b/33333 : fast/canvas/canvasttest.html
>>> > That seem OK?
>>>
>>> I'm not so much a fan of this change. It's more typing and I'm not
>>> sure if it makes anything much easier for the user (maybe viewing in a
>>> text editor that will automatically hyperlink the text, I guess).
>>>
>>> Note that the current syntax does support multiple bugs per line
>>> (although not the hyperlink syntax you propose); I'm sure you knew
>>> that, so perhaps you were just mentioning that for the others on the
>>> list.
>>
>> One of the problems with the current format is that it's excessively
>> complicated. Avoiding magic syntax and just using URLs makes it so there is
>> less specialized syntax to make sense of. It also has the potential
>> advantages you mention (e.g. you could copy-paste the text into your
>> browser's address bar), but that's secondary.
>>
>
> I keep hearing that the syntax is "excessively complicated". It's a
> pretty simple syntax, but even you think that it is complicated, but
> in what way is it excessively so, given that we actively use all of
> the features it supports?
>
> Also note that changing BUGXXX to a URL doesn't necessarily make it
> less complicated; It may actually make it more complicated, since you
> would either have to parse URLs out from the tokens or decide what
> sort of URL-like syntax you were going to allow.
>

My apologies if this came off sharper than I had intended; I only
meant to focus on specific suggestions (like your URL change), rather
than vague comments. I am certainly open to anything that makes things
simpler, I just don't know how much there is to simplify without also
losing functionality. I'm also open to losing functionality that isn't
really that valuable, just that that needs to be weighed in,
obviously.

-- Dirk