[webkit-dev] Tests that separate JS and HTML

[Alexey Proskuryakov]

On multiple occasions, I've been noticing that separating JavaScript for tests into its own file has been causing me pain. There are several disadvantages that I know of:

1. One doesn't see test content when a tool (such as run-webkit-tests) sends them to a failing test. If I want to see it in browser, I need to copy/paste script src, or manually edit the address in address bar.
2. The proliferation of script-tests prompts some contributors to unnecessarily complicate their tests to fit the format. Sometimes, the DOM that could just be in the HTML content is dynamically created. Other times, complicated schemes for async tests are used, instead of directly calling notifyDone().
3. These tests cannot be attached to Bugzilla bugs in a working form (e.g. if one wants to file a bug against another browser).
4. The test does not live next to its expected results, so one has to keep two folders open when working on the test.

On the other hand, I can't think of any clear advantages. Some potential advantages could be:

1. Ease of boilerplate modification. I don't see why we would want to modify boilerplate in existing tests - it's more likely that such changes would break them than improve them.
2. Ease of test writing. Maybe people who use script tests often would object, but for me, a step that I must do any time I start working on a test - at a very specific moment - is a problem. Usually, I start with a simple test that I keep on desktop (or in Bugzilla), and only move it to LayoutTests later.
3. Higher quality of boilerplate. The boilerplate is quite straightforward, all the logic is in external files.
4. Tests become more concise. This is true, but I don't think it has practical benefits - it's quite easy to focus on test code even if everything is in one file.

Of course, there is the special case of fast/js tests, which we (I think) still hope to run without a WebView one day. For that, keeping JS separate is obviously desirable.

We have lots of imported tests in WebKit regression test suite, as well as tests employing various custom runners developed by WebKit contributors. So to a degree, the manner in which a test is written will always remain a matter of personal preference, we can't and shouldn't standardize this.

However, as I was personally seeing more script-tests not just being added, but living their active lives, I started feeling frustration towards them. When reviewing patches, I recommend against using tests that have their main logic in files other than the main HTML.

- WBR, Alexey Proskuryakov