<div class="gmail_quote">On Fri, Nov 4, 2011 at 5:52 PM, Ryosuke Niwa <span dir="ltr"><<a href="mailto:rniwa@webkit.org">rniwa@webkit.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><span style="background-color:transparent">On Fri, Nov 4, 2011 at 4:01 PM, Ojan Vafai </span><span dir="ltr" style="background-color:transparent"><<a href="mailto:ojan@chromium.org" target="_blank">ojan@chromium.org</a>></span><span style="background-color:transparent"> wrote:</span><br>
</div><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I don't see any need for manifest files.</blockquote><div><span style="background-color:transparent"><br>
</span></div></div><div><span style="background-color:transparent">W3C's build step auto-generates them.</span></div></div></blockquote><div><br></div><div>I thought you were arguing that we should use manifest files for all reftests because having multiple ways to do things is confusing.</div>
<div><span style="background-color: transparent; "> </span></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>On Fri, Nov 4, 2011 at 2:00 PM, Ryosuke Niwa <span dir="ltr"><<a href="mailto:rniwa@webkit.org" target="_blank">rniwa@webkit.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div>On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke <span dir="ltr"><<a href="mailto:dpranke@chromium.org" target="_blank">dpranke@chromium.org</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's unclear how much of a perf impact there would be but that's<br>
easy enough to determine - I would expect it to be minimal compared to<br>
the time of actually rendering a page.<br></blockquote><div><br></div></div><div>Since I expect w3c to end up having hundreds of thousands of tests, I see any performance implication to be a serious threat.</div></div></blockquote>
<div><br></div></div><div>There is no inherent performance problem that I can see. We just need to structure things in a way that avoids performance problems.</div></div></blockquote><div><br></div></div><div>But we don't need to structure them. They're organized in W3C's repo, and we're just going to import them in some directory; e.g. LayoutTests/w3c/. </div>
</div></blockquote><div><br></div><div>I'm reluctantly OK with doing this exclusively for test suites we import that come with a manifest file, since the maintenance cost is lower. Although, I'd prefer if we limited the number of ways you could mark a test as a reftest. If we're going to do this though, I think that you should not be able to mix a manifest file and non-manifested tests in the same directory.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div>If we're adding new tests that we intend to contribute back to W3C, then those tests should live outside of this directory. <span style="background-color:transparent">The preferred approach, however, is to add tests to W3C repo first, then import them back to WebKit.</span></div>
</div></blockquote><div><br></div><div>I don't think this is realistic. For example, for the new flexbox tests, we're writing them against WebKit's code and they're changing as we go. In addition, all the properties are webkit prefixed. We're not writing test suites in isolation. We're writing them as we implement the feature or fix the appropriate bugs.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><span style="background-color: transparent; ">I don't see what's special about reftests that we'd run them on a separate bot. We might decide to shard test running across different bots in some way, but sharding by test type seems unhelpful.</span></div>
</div></blockquote></div></div><div><br></div>I'm not talking about ref-tests. I'm talking about tests imported from the W3C test suite.</blockquote><div><br></div><div>Even then, I don't see any benefit to not running them as part of our main test suite.</div>
</div>