[webkit-changes] [WebKit/WebKit] 03b876: [macOS] Add additional heuristics for context menu...

[Aditya Keerthi]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 03b876ebc5479fe38e607b80d1f09a43bccd2a8c
      https://github.com/WebKit/WebKit/commit/03b876ebc5479fe38e607b80d1f09a43bccd2a8c
  Author: Aditya Keerthi <akeerthi at apple.com>
  Date:   2023-03-01 (Wed, 01 Mar 2023)

  Changed paths:
    M Source/WebCore/page/ContextMenuContext.cpp
    M Source/WebCore/page/ContextMenuContext.h
    M Source/WebCore/page/ContextMenuController.cpp
    M Source/WebKit/Shared/ContextMenuContextData.cpp
    M Source/WebKit/Shared/ContextMenuContextData.h
    M Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm
    M Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm
    M Tools/TestWebKitAPI/cocoa/TestUIDelegate.h
    M Tools/TestWebKitAPI/cocoa/TestUIDelegate.mm
    M Tools/TestWebKitAPI/cocoa/TestWKWebView.h
    M Tools/TestWebKitAPI/cocoa/TestWKWebView.mm

  Log Message:
  -----------
  [macOS] Add additional heuristics for context menu QR code detection
https://bugs.webkit.org/show_bug.cgi?id=252972
rdar://105953041

Reviewed by Wenson Hsieh.

260897 at main added initial support for QR code detection when showing context
menus. However, websites have markup-driven ways of constructing QR codes that
do not involve images. In these cases, the `HitTestResult` obtained prior to
presenting a context menu does not contain an image, and the UI process has
nothing to analyze.

This patch ports Safari heuristics to detect markup-drive QR codes into WebKit.
Specifically, <svg>, <canvas>, and <table> QR codes are now supported, with
additional logic to account for zooming, transforms, and partially obscured
content.

* Source/WebCore/page/ContextMenuContext.cpp:
(WebCore::ContextMenuContext::ContextMenuContext):
* Source/WebCore/page/ContextMenuContext.h:
(WebCore::ContextMenuContext::setPotentialQRCodeNodeSnapshotImage):
(WebCore::ContextMenuContext::potentialQRCodeNodeSnapshotImage const):
(WebCore::ContextMenuContext::setPotentialQRCodeViewportSnapshotImage):
(WebCore::ContextMenuContext::potentialQRCodeViewportSnapshotImage const):
* Source/WebCore/page/ContextMenuController.cpp:
(WebCore::prepareContextForQRCode):

Add a node and viewport snapshot of the element that may be a QR code, to be
sent to the UI process for analysis. These snapshots are only taken for certain
elements that are likely to contain a QR code, and if the `HitTestResult` is
missing a link and image.

Node snapshotting is unreliable, since it does not take transforms into account.
Consequently, a viewport-level snapshot is also taken. However, a viewport-level
snaphot is insufficient for partially obscured elements. Hence, two snapshots
are taken for analysis.

This approach mirrors Safari's implementation and limits the frequency of spurious
analysis. Note that taking the viewport snapshot only if QR code detection fails
for the node snapshot was considered, but this would require another IPC
roundtrip and would complicate the context menu presentation logic. The approach
used in this patch was chosen since this logic does not run frequently.

(WebCore::ContextMenuController::maybeCreateContextMenu):
* Source/WebKit/Shared/ContextMenuContextData.cpp:

Encode and decode the snapshot images as ShareableBitmaps for use in the
UI process.

(WebKit::ContextMenuContextData::ContextMenuContextData):

Drive-by cleanup. `RefPtr` does not need to be explicitly initialized.

(WebKit::ContextMenuContextData::setImage):

Drive-by cleanup to align with modern idiom used below.

(WebKit::ContextMenuContextData::setPotentialQRCodeNodeSnapshotImage):
(WebKit::ContextMenuContextData::setPotentialQRCodeViewportSnapshotImage):
(WebKit::ContextMenuContextData::encode const):
(WebKit::ContextMenuContextData::decode):
* Source/WebKit/Shared/ContextMenuContextData.h:
(WebKit::ContextMenuContextData::potentialQRCodeNodeSnapshotImage const):
(WebKit::ContextMenuContextData::potentialQRCodeViewportSnapshotImage const):
* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm:
(WebKit::WebContextMenuProxyMac::showAfterPostProcessingContextData):

Analyze the snapshot images, if the `WebHitTestResultData` is missing an image
and a link.

* Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm:
(TestWebKitAPI::qrCodeSVGString):
(TestWebKitAPI::TEST):

Add tests to exercise the various heuristics.

* Tools/TestWebKitAPI/cocoa/TestUIDelegate.h:

Add helper methods to wait for context menu presentation and extract
element information.

* Tools/TestWebKitAPI/cocoa/TestUIDelegate.mm:
(-[TestUIDelegate waitForContextMenu]):
(-[WKWebView _test_waitForContextMenu]):
* Tools/TestWebKitAPI/cocoa/TestWKWebView.h:
* Tools/TestWebKitAPI/cocoa/TestWKWebView.mm:
(-[TestWKWebView rightClickAtPoint:]):

Implement a helper method to perform a right click.

Canonical link: https://commits.webkit.org/261019@main