Chromium skips it (and if I remember correctly, they commissioned it?) Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something? -eric
10.08.2010, в 11:51, Eric Seidel написал(а):
Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something?
Just like any test suite, it catches regressions. We saw that it was effective when we introduced certain JSC regressions that it could have prevented. Since some of the errors Sputnik catches are platform specific (32 bit vs. 64 bit, JIT vs. interpreter etc), it is particularly important to have it on build bots. Some subtests are less interesting (e.g. many Unicode tests basically check ICU behavior), so we have the slowest of those disabled already. One possible way to speed up Sputnik tests would be to run them as part of run-javascriptcore-tests, avoiding web page overhead. Someone proficient in scripting languages just needs to improve run-javascriptcore-tests to the point when it can do that. - WBR, Alexey Proskuryakov
On Tue, Aug 10, 2010 at 12:16 PM, Alexey Proskuryakov <ap@webkit.org> wrote:
10.08.2010, в 11:51, Eric Seidel написал(а):
Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something?
Just like any test suite, it catches regressions. We saw that it was effective when we introduced certain JSC regressions that it could have prevented. Since some of the errors Sputnik catches are platform specific (32 bit vs. 64 bit, JIT vs. interpreter etc), it is particularly important to have it on build bots.
Some subtests are less interesting (e.g. many Unicode tests basically check ICU behavior), so we have the slowest of those disabled already.
One possible way to speed up Sputnik tests would be to run them as part of run-javascriptcore-tests, avoiding web page overhead. Someone proficient in scripting languages just needs to improve run-javascriptcore-tests to the point when it can do that.
FWIW, a Sputnik test caught an error in the new HTML parser (related to the interaction between \r\n collapsing and line numbering in JavaScript) that otherwise might have gotten through. A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel. Adam
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch). - WBR, Alexey Proskuryakov
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool. As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only. Regards, Maciej
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.) -eric On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe... "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people
working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and
according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in
WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
The V8 buildbots are running the sputnik tests and they do catch regressions for us. In the V8 repository we have a list of V8 test expectations for the sputnik tests. http://build.chromium.org/buildbot/v8/waterfall -- Mads On Wed, Aug 11, 2010 at 7:32 PM, Ojan Vafai <ojan@chromium.org> wrote:
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe... that "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Clearly I was misinformed. My apologies for using the webkit-dev list to track down this chromium-specific issue. On Wed, Aug 11, 2010 at 11:09 AM, Mads Sig Ager <ager@chromium.org> wrote:
The V8 buildbots are running the sputnik tests and they do catch regressions for us. In the V8 repository we have a list of V8 test expectations for the sputnik tests.
http://build.chromium.org/buildbot/v8/waterfall
-- Mads
On Wed, Aug 11, 2010 at 7:32 PM, Ojan Vafai <ojan@chromium.org> wrote:
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe... that "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
No apologies necessary! This was illuminating to me too, even though I _might have_ been the guy who put the "we probably don't want to run these" comment in :P :DG< On Wed, Aug 11, 2010 at 11:19 AM, Eric Seidel <eric@webkit.org> wrote:
Clearly I was misinformed. My apologies for using the webkit-dev list to track down this chromium-specific issue.
On Wed, Aug 11, 2010 at 11:09 AM, Mads Sig Ager <ager@chromium.org> wrote:
The V8 buildbots are running the sputnik tests and they do catch regressions for us. In the V8 repository we have a list of V8 test expectations for the sputnik tests.
http://build.chromium.org/buildbot/v8/waterfall
-- Mads
On Wed, Aug 11, 2010 at 7:32 PM, Ojan Vafai <ojan@chromium.org> wrote:
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe... that "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
> A better long-term fix might be to finish new-run-webkit-tests so we > can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Maybe a comment in the text_expectations file to point out where they are run might be a good idea. Avi On Wed, Aug 11, 2010 at 2:27 PM, Dimitri Glazkov <dglazkov@chromium.org>wrote:
No apologies necessary! This was illuminating to me too, even though I _might have_ been the guy who put the "we probably don't want to run these" comment in :P
:DG<
On Wed, Aug 11, 2010 at 11:19 AM, Eric Seidel <eric@webkit.org> wrote:
Clearly I was misinformed. My apologies for using the webkit-dev list to track down this chromium-specific issue.
On Wed, Aug 11, 2010 at 11:09 AM, Mads Sig Ager <ager@chromium.org> wrote:
The V8 buildbots are running the sputnik tests and they do catch regressions for us. In the V8 repository we have a list of V8 test expectations for the sputnik tests.
http://build.chromium.org/buildbot/v8/waterfall
-- Mads
On Wed, Aug 11, 2010 at 7:32 PM, Ojan Vafai <ojan@chromium.org> wrote:
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in
http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe...
that "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com>
wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
> > 10.08.2010, в 14:00, Adam Barth написал(а): > >> A better long-term fix might be to finish new-run-webkit-tests so
we
>> can run the tests in parallel. > > > One reason to move the tests to run-javascriptcore-tests is that people > working on JS run these more often (sometimes not even building WebCore > until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Given Maciej's and Adam's comments about these tests helping find non-JS-engine-specific regressions, why not always run them? On Wed, Aug 11, 2010 at 11:09 AM, Mads Sig Ager <ager@chromium.org> wrote:
The V8 buildbots are running the sputnik tests and they do catch regressions for us. In the V8 repository we have a list of V8 test expectations for the sputnik tests.
http://build.chromium.org/buildbot/v8/waterfall
-- Mads
On Wed, Aug 11, 2010 at 7:32 PM, Ojan Vafai <ojan@chromium.org> wrote:
Just looking at the history, it looks like when the tests were originally checked in, they had no expected results. So someone on the chromium team skipped them all. There's a comment in http://trac.webkit.org/browser/trunk/LayoutTests/platform/chromium/test_expe... that "we probably don't want to run these", but I think that comment is just wrong. On Wed, Aug 11, 2010 at 9:31 AM, Eric Seidel <eric@webkit.org> wrote:
If these tests are actually valuable, then maybe my question then belongs: Why is Chromium no longer running these tests? (Assuming my source is correct.)
-eric
On Wed, Aug 11, 2010 at 7:10 AM, Maciej Stachowiak <mjs@apple.com> wrote:
On Aug 10, 2010, at 2:26 PM, Alexey Proskuryakov wrote:
10.08.2010, в 14:00, Adam Barth написал(а):
A better long-term fix might be to finish new-run-webkit-tests so we can run the tests in parallel.
One reason to move the tests to run-javascriptcore-tests is that people working on JS run these more often (sometimes not even building WebCore until ready to submit a patch).
If these tests can catch regressions from non-JS-engine changes (and according to Adam's message, they have done so at least once), then we need to run them in the full browser engine context, even if we also have a version that runs in a JS-only command-line tool.
As another data point, some of the Sputnik tests are currently failing in WebKit2 on Mac, so they are detecting a problem that they wouldn't be able to if they ran JS-only.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
I thought Sputnik came from Microsoft? -- Dirk On Tue, Aug 10, 2010 at 11:51 AM, Eric Seidel <eric@webkit.org> wrote:
Chromium skips it (and if I remember correctly, they commissioned it?)
Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something?
-eric _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Nope, it's from Chromium: http://blog.chromium.org/2009/06/launching-sputnik-into-orbit.html http://blog.chromium.org/2010/03/does-your-browser-behave.html Mihai On Tue, Aug 10, 2010 at 1:30 PM, Dirk Pranke <dpranke@chromium.org> wrote:
I thought Sputnik came from Microsoft?
-- Dirk
On Tue, Aug 10, 2010 at 11:51 AM, Eric Seidel <eric@webkit.org> wrote:
Chromium skips it (and if I remember correctly, they commissioned it?)
Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something?
-eric _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Eric Seidel írta:
Chromium skips it (and if I remember correctly, they commissioned it?)
Why do we want to be running these 6000 tests and slowing down our builds. I was talking with jamesr, and he seemed to think it adds little value to run it every time? (It was supposedly written as more of a development tool for V8?) But maybe I'm missing something?
Sputnik tests are very fast. All test cases (15642) run in 394s, and 5468 Sputnik tests run in 41s on Qt buildbot, it takes only 10% of the testing time. Alexey Proskuryakov írta:
One possible way to speed up Sputnik tests would be to run them as part of run-javascriptcore-tests, avoiding web page overhead. Someone proficient in scripting languages just needs to improve run-javascriptcore-tests to the point when it can do that.
1127 JavaScriptCore tests (run-javasriptcore-tests) run in 11s, 5468 Sputnik tests run in 41s on Qt buildbot. I don't think we can speed up Sputnik tests significantly. br, Ossy
participants (12)
-
Adam Barth
-
Alexey Proskuryakov
-
Avi Drissman
-
Dimitri Glazkov
-
Dirk Pranke
-
Eric Seidel
-
Evan Martin
-
Maciej Stachowiak
-
Mads Sig Ager
-
Mihai Parparita
-
Ojan Vafai
-
Osztrogonac Csaba