Tests that separate JS and HTML
On multiple occasions, I've been noticing that separating JavaScript for tests into its own file has been causing me pain. There are several disadvantages that I know of: 1. One doesn't see test content when a tool (such as run-webkit-tests) sends them to a failing test. If I want to see it in browser, I need to copy/paste script src, or manually edit the address in address bar. 2. The proliferation of script-tests prompts some contributors to unnecessarily complicate their tests to fit the format. Sometimes, the DOM that could just be in the HTML content is dynamically created. Other times, complicated schemes for async tests are used, instead of directly calling notifyDone(). 3. These tests cannot be attached to Bugzilla bugs in a working form (e.g. if one wants to file a bug against another browser). 4. The test does not live next to its expected results, so one has to keep two folders open when working on the test. On the other hand, I can't think of any clear advantages. Some potential advantages could be: 1. Ease of boilerplate modification. I don't see why we would want to modify boilerplate in existing tests - it's more likely that such changes would break them than improve them. 2. Ease of test writing. Maybe people who use script tests often would object, but for me, a step that I must do any time I start working on a test - at a very specific moment - is a problem. Usually, I start with a simple test that I keep on desktop (or in Bugzilla), and only move it to LayoutTests later. 3. Higher quality of boilerplate. The boilerplate is quite straightforward, all the logic is in external files. 4. Tests become more concise. This is true, but I don't think it has practical benefits - it's quite easy to focus on test code even if everything is in one file. Of course, there is the special case of fast/js tests, which we (I think) still hope to run without a WebView one day. For that, keeping JS separate is obviously desirable. We have lots of imported tests in WebKit regression test suite, as well as tests employing various custom runners developed by WebKit contributors. So to a degree, the manner in which a test is written will always remain a matter of personal preference, we can't and shouldn't standardize this. However, as I was personally seeing more script-tests not just being added, but living their active lives, I started feeling frustration towards them. When reviewing patches, I recommend against using tests that have their main logic in files other than the main HTML. - WBR, Alexey Proskuryakov
On Fri, Aug 13, 2010 at 11:17 AM, Alexey Proskuryakov <ap@webkit.org> wrote:
On multiple occasions, I've been noticing that separating JavaScript for tests into its own file has been causing me pain. There are several disadvantages that I know of:
1. One doesn't see test content when a tool (such as run-webkit-tests) sends them to a failing test. If I want to see it in browser, I need to copy/paste script src, or manually edit the address in address bar. 2. The proliferation of script-tests prompts some contributors to unnecessarily complicate their tests to fit the format. Sometimes, the DOM that could just be in the HTML content is dynamically created. Other times, complicated schemes for async tests are used, instead of directly calling notifyDone(). 3. These tests cannot be attached to Bugzilla bugs in a working form (e.g. if one wants to file a bug against another browser). 4. The test does not live next to its expected results, so one has to keep two folders open when working on the test.
On the other hand, I can't think of any clear advantages. Some potential advantages could be:
1. Ease of boilerplate modification. I don't see why we would want to modify boilerplate in existing tests - it's more likely that such changes would break them than improve them. 2. Ease of test writing. Maybe people who use script tests often would object, but for me, a step that I must do any time I start working on a test - at a very specific moment - is a problem. Usually, I start with a simple test that I keep on desktop (or in Bugzilla), and only move it to LayoutTests later. 3. Higher quality of boilerplate. The boilerplate is quite straightforward, all the logic is in external files. 4. Tests become more concise. This is true, but I don't think it has practical benefits - it's quite easy to focus on test code even if everything is in one file.
Of course, there is the special case of fast/js tests, which we (I think) still hope to run without a WebView one day. For that, keeping JS separate is obviously desirable.
We have lots of imported tests in WebKit regression test suite, as well as tests employing various custom runners developed by WebKit contributors. So to a degree, the manner in which a test is written will always remain a matter of personal preference, we can't and shouldn't standardize this.
However, as I was personally seeing more script-tests not just being added, but living their active lives, I started feeling frustration towards them. When reviewing patches, I recommend against using tests that have their main logic in files other than the main HTML.
I don't feel nearly as strongly as Alexey, but I think he makes some good arguments. The main "advantage" of the script tests style tests is the common code (the function "shouldBe" for example), but there's no reason we can't still include "fast/js/resources/js-test-pre.js" (or other files) in self-contained tests. J
On Aug 13, 2010, at 3:17 AM, Alexey Proskuryakov wrote:
Of course, there is the special case of fast/js tests, which we (I think) still hope to run without a WebView one day. For that, keeping JS separate is obviously desirable.
Absolutely agreed - and these can already be run in a command-line shell, by using the LayoutTests/fast/js/resources/standalone-(pre|post).js scripts. This applies to many of the tests in fast/js, as well as the ietestcenter/Javascript suite. Sadly we don't yet have a script to run all these test for you, we should add one at some point. Also, it would also be great to port over the other js-only tests in fast/js (and any elsewhere, e.g. fast/regex). G.
I agree that dealing with the script to generate tests and having the actual test content be in a different file is a significant maintenance overhead. But I also think that having standard testing code across many tests reduces the amount of effort it takes to understand a test. We discussed this at https://lists.webkit.org/pipermail/webkit-dev/2010-July/013673.html. I think Maciej's solution sounds ideal, although I'm not convinced it's worth including doctype for news tests. If we use the load event, then we can drop the post script entirely and then it's just about including a single script line at the top. FWIW, this is how the dump-as-markup.js tests and the proposed perf tests work (https://bugs.webkit.org/show_bug.cgi?id=43997). Ojan On Fri, Aug 13, 2010 at 3:17 AM, Alexey Proskuryakov <ap@webkit.org> wrote:
On multiple occasions, I've been noticing that separating JavaScript for tests into its own file has been causing me pain. There are several disadvantages that I know of:
1. One doesn't see test content when a tool (such as run-webkit-tests) sends them to a failing test. If I want to see it in browser, I need to copy/paste script src, or manually edit the address in address bar. 2. The proliferation of script-tests prompts some contributors to unnecessarily complicate their tests to fit the format. Sometimes, the DOM that could just be in the HTML content is dynamically created. Other times, complicated schemes for async tests are used, instead of directly calling notifyDone(). 3. These tests cannot be attached to Bugzilla bugs in a working form (e.g. if one wants to file a bug against another browser). 4. The test does not live next to its expected results, so one has to keep two folders open when working on the test.
On the other hand, I can't think of any clear advantages. Some potential advantages could be:
1. Ease of boilerplate modification. I don't see why we would want to modify boilerplate in existing tests - it's more likely that such changes would break them than improve them. 2. Ease of test writing. Maybe people who use script tests often would object, but for me, a step that I must do any time I start working on a test - at a very specific moment - is a problem. Usually, I start with a simple test that I keep on desktop (or in Bugzilla), and only move it to LayoutTests later. 3. Higher quality of boilerplate. The boilerplate is quite straightforward, all the logic is in external files. 4. Tests become more concise. This is true, but I don't think it has practical benefits - it's quite easy to focus on test code even if everything is in one file.
Of course, there is the special case of fast/js tests, which we (I think) still hope to run without a WebView one day. For that, keeping JS separate is obviously desirable.
We have lots of imported tests in WebKit regression test suite, as well as tests employing various custom runners developed by WebKit contributors. So to a degree, the manner in which a test is written will always remain a matter of personal preference, we can't and shouldn't standardize this.
However, as I was personally seeing more script-tests not just being added, but living their active lives, I started feeling frustration towards them. When reviewing patches, I recommend against using tests that have their main logic in files other than the main HTML.
- WBR, Alexey Proskuryakov
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
On Aug 16, 2010, at 12:44 PM, Ojan Vafai wrote:
I agree that dealing with the script to generate tests and having the actual test content be in a different file is a significant maintenance overhead. But I also think that having standard testing code across many tests reduces the amount of effort it takes to understand a test.
We discussed this at https://lists.webkit.org/pipermail/webkit-dev/2010-July/013673.html. I think Maciej's solution sounds ideal, although I'm not convinced it's worth including doctype for news tests.
I think it is. Tests should be in strict mode by default, unless you specifically have a goal of testing quirks mode behavior. Quirks more is more likely to have weird pitfalls or to change in unexpected ways over time. No doctype == quirksmode, for HTML anyway. XML-based tests (including XHTML, MathML and SVG) don't need a doctype. And really, "<!DOCTYPE html>" is hardly a burden on either the reader or the writer. The short, memorable doctype is one of my favorite things about HTML5. Regards, Maciej
participants (5)
-
Alexey Proskuryakov
-
Gavin Barraclough
-
Jeremy Orlow
-
Maciej Stachowiak
-
Ojan Vafai