[webkit-changes] [WebKit/WebKit] 3e29d3: [WebDriver] If Safari window is not focused, WebDr...

[Qianlang Chen]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 3e29d37be2b6cd98183fd643223f0070f55c913e
      https://github.com/WebKit/WebKit/commit/3e29d37be2b6cd98183fd643223f0070f55c913e
  Author: Qianlang Chen <qianlangchen at apple.com>
  Date:   2024-06-14 (Fri, 14 Jun 2024)

  Changed paths:
    M Source/WebKit/UIProcess/Automation/WebAutomationSession.h
    M Source/WebKit/UIProcess/Automation/mac/WebAutomationSessionMac.mm
    M Source/WebKit/UIProcess/mac/WebPageProxyMac.mm
    M Source/WebKit/WebProcess/WebPage/WebFrame.cpp
    M Tools/TestWebKitAPI/Tests/mac/AcceptsFirstMouse.mm

  Log Message:
  -----------
  [WebDriver] If Safari window is not focused, WebDriver-automated mouse events do not get sent to the page
rdar://117035696
https://bugs.webkit.org/show_bug.cgi?id=275326

Reviewed by Patrick Angle.

When the Safari window is not currently focused, AKA not the key window
using AppKit's terminology, safaridriver-automated mouse events get
sent to the window but not to the webpage within it.

The patch https://github.com/WebKit/WebKit/commit/f41c94d11e5a0b8900cf768adeb76d94ef681444#diff-8eae0c0334ee2210a82eef8240463d484f408aceba25aef9f96093c9ca042863
attempted to make the window key, but according to AppKit engineers,
making the window key requires the application to currently be active.
Unfortunately, neither `[NSApplication.sharedApplication activate]` nor
`[NSApplication.sharedApplication activateIgnoringOtherApps:YES]`
actually guarantee activating the app immediately. This results in us
failing to ensure that the window is key before sending the mouse
events, so the mouse events could not get delivered.

   - For `mousemove` events, there is the patch https://github.com/WebKit/WebKit/pull/1431
     that make the delivery bypass the window level and sent directly
     into the web view. However, a more recent enhancement https://github.com/WebKit/WebKit/pull/17145
     deliberately prevented `mousemove` events from being sent to the
     web page if the page is not active, which effectively nullified the
     former patch for `mousemove` events.

To work around that, a simple solution I found was to give automated
mouse events privileges to bypass certain filtering checks, rather than
always forcing the window to steal input focus. Automated mouse events
are assigned a special, "magic" event number (0), so we can recognize
them in the places below in the code.

I have tested the code
   (1) Manually with Python code using Selenium launching the newly
       compiled safaridriver and seeing that mouse events always get
       delivered successfully
   (2) With `wpt run safari webdriver/tests/classic/element-click webdriver/tests/classic/element_send_keys`
       and seeing that the result are the same before and after my
       changes with the same passes and failures.

* Source/WebKit/UIProcess/Automation/WebAutomationSession.h:
* Source/WebKit/UIProcess/Automation/mac/WebAutomationSessionMac.mm:
(WebKit::WebAutomationSession::wasEventSynthesizedForAutomation):
(WebKit::WebAutomationSession::platformSimulateMouseInteraction):
   - Make the magic event number accessible in other places needing to
     check if an event was synthesized.

* Source/WebKit/UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::acceptsFirstMouse):
   - If the mouse event is automated, bypass the check to let the window
     always accept it.
   - I don't fully understand why this works -- this is a suggestion
     from an AppKit engineer, and I thought this function was meant to
     return whether the mouse event should be deemed as "activating the
     window" versus "activating the window and sent to the contents",
     but apparently it's just whether the event gets sent to the
     contents because automated events still don't activate the window
     regardless the function's return value.

* Source/WebKit/WebProcess/WebPage/WebFrame.cpp:
(WebKit::WebFrame::handleMouseEvent):
   - If the mouse event is automated, don't use the enhancement of
     sending the event straight to the scrollbars and still send it
     to the web page.

* Tools/TestWebKitAPI/Tests/mac/AcceptsFirstMouse.mm:
(TestWebKitAPI::AcceptsFirstMouse::runTest):
   - Creating the event with eventNumber:0 for testing resulted in the
     event being considered as synthesized and acceptsFirstMouse always
     returned YES. Use a different eventNumber instead as a workaround.

Canonical link: https://commits.webkit.org/280047@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications