[webkit-changes] [WebKit/WebKit] e00498: Add a way for WKWebView clients to simulate a clic...

[Wenson Hsieh]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: e004985d096c19792336fb1ef1fd9f0ce825e02a
      https://github.com/WebKit/WebKit/commit/e004985d096c19792336fb1ef1fd9f0ce825e02a
  Author: Wenson Hsieh <wenson_hsieh at apple.com>
  Date:   2024-07-15 (Mon, 15 Jul 2024)

  Changed paths:
    M Source/WebCore/dom/BoundaryPoint.cpp
    M Source/WebCore/dom/BoundaryPoint.h
    M Source/WebCore/dom/Node.h
    M Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm
    M Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h
    M Source/WebKit/UIProcess/WebPageProxy.cpp
    M Source/WebKit/UIProcess/WebPageProxy.h
    M Source/WebKit/WebProcess/WebPage/WebPage.cpp
    M Source/WebKit/WebProcess/WebPage/WebPage.h
    M Source/WebKit/WebProcess/WebPage/WebPage.messages.in
    M Tools/TestWebKitAPI/SourcesCocoa.txt
    M Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
    A Tools/TestWebKitAPI/Tests/WebKitCocoa/SimulateClickOverText.mm
    A Tools/TestWebKitAPI/Tests/WebKitCocoa/click-targets.html

  Log Message:
  -----------
  Add a way for WKWebView clients to simulate a click over visible text in the page
https://bugs.webkit.org/show_bug.cgi?id=276613
rdar://131758473

Reviewed by Ryosuke Niwa.

Add a way to simulate mouse events over clickable elements on the webpage, based on matches of found
text. See below for more details.

Tests:  SimulateClickOverText.ClickTargets
        SimulateClickOverText.ClickTargetsAfterScrolling

* Source/WebCore/dom/BoundaryPoint.cpp:
(WebCore::treeOrderInternal):
(WebCore::treeOrder):

Export only `treeOrder<ComposedTree>(const BoundaryPoint&, const BoundaryPoint&)`.

(WebCore::treeOrder<ComposedTree>):
* Source/WebCore/dom/BoundaryPoint.h:
* Source/WebCore/dom/Node.h:

Add `WEBCORE_EXPORT`s to several methods and helper functions.

* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _pauseNowPlayingMediaSession:]):
(-[WKWebView _simulateClickOverFirstMatchingTextInViewportWithUserInteraction:completionHandler:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewPrivate.h:
* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::simulateClickOverFirstMatchingTextInViewportWithUserInteraction):

Add plumbing from `WKWebView` -> `WebPageProxy` -> `WebPage`.

* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::simulateClickOverFirstMatchingTextInViewportWithUserInteraction):

Implement the main heuristic here. At a high level, this is how it works:

1.  Use `findPlainText` to search for the target text in the main document.
2.  Filter out any matches that aren't within links, or otherwise clickable containers by walking up
    the composed tree.
3.  Compute the first text rect (in content coordinates) for each matching range; filter out any
    matches that are outside of the visual viewport.
4.  Perform one final hit-test to filter out potential targets that are obscured.
5.  If there are 2 or more candidates after the above filtering, exit early and don't attempt to
    click anything (we may need a more sophisticated strategy to deal with ambiguous targets in the
    future).
6.  Otherwise, if there is exactly one match, create and send synthetic click events.

* Source/WebKit/WebProcess/WebPage/WebPage.h:
* Source/WebKit/WebProcess/WebPage/WebPage.messages.in:
* Tools/TestWebKitAPI/SourcesCocoa.txt:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKitCocoa/SimulateClickOverText.mm: Added.
(-[WKWebView simulateClickOverText:]):
(TestWebKitAPI::TEST(SimulateClickOverText, ClickTargets)):
(TestWebKitAPI::TEST(SimulateClickOverText, ClickTargetsAfterScrolling)):

Add a couple of API tests to exercise the new functionality by verifying that mouse and click events
are dispatched on several types of visible (and clickable) DOM elements when simulating clicks based
on found text.

* Tools/TestWebKitAPI/Tests/WebKitCocoa/click-targets.html: Added.

Canonical link: https://commits.webkit.org/280991@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications