[webkit-dev] Adding window.layoutTestInspector

[Hajime Morita]

Hi Maciej, thanks much for sharing your thought.
Overall, it totally makes sense.
And we need an action as Ojan mentioned.

Here is some ideas;

> (1) It seems a little odd that we'll end up with two different objects that have similar names and a very similar purpose, but just differ in how they are implemented. Maybe there's a way to define layoutTestController in WebCore and have DumpRenderTree extend it.

It looks fine. A prototype patch will come shortly.

> (2) It does seem like for some test-specific methods, implementing then in WebCore would be simpler and would save the work of plumbing them through the WebKit layers.

Yeah, it's the main point of this proposal!

> (3) On the other hand, LayoutTestController seems like it has too much stuff in it. Originally DumpRenderTree exposed a very modest set of functionality, mostly to control output (dumpAsText) or to emulate things that you could do by hand when running the test in the browser (waitUntilDone, eventSender). Nowadays, there are dozens of methods. A lot of them are used in only one or two tests. And in many cases, the methods have no interactive equivalent, so a lot of our tests are not runnable in the browser at all. Those seem like bad trends. Maybe instead of making it easier to add to LayoutTestController, we should look at whether we can consolidate functionality, factor it into more objects, and find ways to test things that don't require quite so much custom functionality.

Looks several points here and I think we can tackle them separetely:
- (3-1): LayoutTestController is too large and need to be factored out
- (3-2): We shoud allow our tests to run interactively in the browser.

For (3-1), once we can implement test objects inside WebCore, it will
become easier to do so because we can use our internal mechanism like
IDL-based binding generation.

As an example, defining WebSettings JS object to expose Settings
parameter would be fine. 16 of 120 LayoutTestController APIs are
Settings/WebPreferences setters. WebView and WebFrame also have
several wrapper APIs on LayoutTestController.

Tackling (3-2) looks harder than for (3-1).
As WebKit grows, no single browser cannot (and doesn't need to) access
all of WebKit features. Each of them uses different subset of features.
Browsers themselves also behave differently.
There are over 20 flags on LayoutTestController to control its
behavior, which implies how our browsers collaborate WebKit in many
different ways.

So problem (3-2) is not always of LayoutTestController.
It may just reflect the fact of, say, divergence of our codebase and use-cases.

On the other hand, I agree that we need to run our tests interactively.
One possible idea is: Adding a switch to the inspector that make
LayoutTestController (and associated objects) accessible from the page context.
Once we have LayoutTestController inside WebCore,
these objects will be available without DRT.
It might not solve the root problem.
But it would provide the way to workround.

--
morita

On Tue, Jul 20, 2010 at 2:51 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> Darin and I discussed this proposal, and we had a few thoughts to share:
>
> (1) It seems a little odd that we'll end up with two different objects that have similar names and a very similar purpose, but just differ in how they are implemented. Maybe there's a way to define layoutTestController in WebCore and have DumpRenderTree extend it.
>
> (2) It does seem like for some test-specific methods, implementing then in WebCore would be simpler and would save the work of plumbing them through the WebKit layers.
>
> (3) On the other hand, LayoutTestController seems like it has too much stuff in it. Originally DumpRenderTree exposed a very modest set of functionality, mostly to control output (dumpAsText) or to emulate things that you could do by hand when running the test in the browser (waitUntilDone, eventSender). Nowadays, there are dozens of methods. A lot of them are used in only one or two tests. And in many cases, the methods have no interactive equivalent, so a lot of our tests are not runnable in the browser at all. Those seem like bad trends. Maybe instead of making it easier to add to LayoutTestController, we should look at whether we can consolidate functionality, factor it into more objects, and find ways to test things that don't require quite so much custom functionality.
>
> I'll add on my own behalf that "layoutTestInspector" doesn't seem like a great name and doesn't express the relationship to layoutTestController. It's not used to examine layout tests.
>
> Regards,
> Maciej
>
> On Jul 14, 2010, at 10:16 PM, Hajime Morita wrote:
>
> > Hi WebKit folks,
> >
> > I'm planning to add "window.layoutTestInspector" or something like that to DRT.
> > And I'd like to hear your opinions.
> >
> > Background:
> >
> > Adding new method to LayoutTestController is hard. It
> > - requires to add new WebKit API to each ports, when the method is to
> > access WebCore.
> > - requires to export extra WebCore symbols to access it from WebKit
> > API implementation.
> > - cannot use WebIDL so we need to write binding code manually.
> >
> > In some case, these steps are unavoidable.
> > But in some other case, especially when we just want to access WebCore
> > from the test, we might be able to skip these steps.
> >
> > A concrete example (my first motivation) is to test DocumentMarker
> > for http://webkit.org/b/41423.
> > DocumentMarker is WebCore's internal state and cannot access neither
> > from DOM nor LayoutTestController.
> >
> > To test it,
> > - the first idea is to use a pixel test.
> >  But it has some shortcomings as you know well.
> > - The second idea is to extend render tree's dump format to
> >  include markers. But it is also platform-specific,
> >  and hard to interpret for humans.
> > - The third idea is to add an API to LayoutTestController.
> >  But it is hard as I mentioned above.
> >
> > Is there another way? DocumentMarker is
> > - WebCore's internal state,
> > - so we don't need to inspect it except for testing purpose,
> > - so it's better to avoid an extra WebKit API for that.
> >
> > I think there are similar demands other than for DocumentMarker,
> > and it might be worth to invest a common way to handle them.
> >
> > Plans:
> >
> > To deal with such cases, we can add a test-specific object named
> > LayoutTestInspector to window object. (The name is tentative.)
> > With this object, We'll be able to write a LayoutTest like:
> >
> > if (window.layoutTestInspector) {
> >   var target = document.getElementById("target")
> >   var markerStr = layoutTestInspector.nodeMarkerString(target);
> >   if (markerStr == "Spelling:0:6")
> >      log("PASS");
> >   else
> >      log("FAIL");
> > }
> >
> > Here is a plan to do this:
> >
> > - LayoutTestInspector will be defined in WebCore,
> >  and implemented as a usual DOM object using WebIDL.
> >  (placed under, for example, WebCore/page/LayoutTestInspector.{idl,h,cpp})
> > - window object will expose a non-enumerable
> > windows.layoutTestInspector property
> >  for that.
> > - Settings::m_enableLayoutTestInspector will control windows.layoutTestInspector
> >  availability. This flag should be true only on DRT.
> >
> > Tests with LayoutTestInspector would have several advantages:
> >
> > - Compared to LayoutTestController,
> >  we don't need to add new APIs to WebKit layer for test purpose.
> > - Compared to LayoutTestController,
> >  we don't need to export extra WebCore APIs to WebKit layer.
> > - Compared to Render-tree dump,
> >  the test can be more portable, focused and understandable.
> >
> > But there are some concerns:
> >
> > - WebCore need to have a test-specific code, that might be a waste of space.
> >  Test-specific WebKit APIs would have a same problem, though.
> > - LayoutTestInspector may introduce some potential security risks.
> >  I have no idea about this area.
> >
> > Do you have any other use-cases or better approaches?
> > Are there concerns I've missed? Do we have similar discussions in the past?
> > Any ideas/suggestions are welcome.
> > If there are no strong objections, I'll start to work on this.
> >
> > Regards.
> >
> > --
> > morita
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev at lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>



--
morita