[webkit-changes] [WebKit/WebKit] b8d16f: [macOS] Add support for QR code detection when sho...

[Aditya Keerthi]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: b8d16fc06260ef10522f25e883994e7242554ecf
      https://github.com/WebKit/WebKit/commit/b8d16fc06260ef10522f25e883994e7242554ecf
  Author: Aditya Keerthi <akeerthi at apple.com>
  Date:   2023-02-27 (Mon, 27 Feb 2023)

  Changed paths:
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WTF/wtf/PlatformEnableCocoa.h
    M Source/WTF/wtf/PlatformHave.h
    M Source/WebCore/PAL/PAL.xcodeproj/project.pbxproj
    M Source/WebCore/PAL/pal/PlatformMac.cmake
    A Source/WebCore/PAL/pal/cocoa/VisionSoftLink.h
    A Source/WebCore/PAL/pal/cocoa/VisionSoftLink.mm
    M Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.h
    M Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm
    M Source/WebKit/Shared/ContextMenuContextData.h
    M Source/WebKit/UIProcess/API/APIContextMenuClient.h
    M Source/WebKit/UIProcess/API/APIContextMenuElementInfoMac.h
    M Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm
    M Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfiguration.mm
    M Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfigurationPrivate.h
    M Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.h
    M Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.mm
    M Source/WebKit/UIProcess/Cocoa/UIDelegate.h
    M Source/WebKit/UIProcess/Cocoa/UIDelegate.mm
    M Source/WebKit/UIProcess/WebContextMenuProxy.h
    M Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.h
    M Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm
    M Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
    A Tools/TestWebKitAPI/Tests/WebKit/qr-code.png
    M Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm

  Log Message:
  -----------
  [macOS] Add support for QR code detection when showing a context menu
https://bugs.webkit.org/show_bug.cgi?id=252971
rdar://105953011

Reviewed by Wenson Hsieh.

Safari currently implements support for QR code detection when showing a context
menu using an injected bundle. This patch works towards two goals by adding
equivalent support at the WebKit layer:

1. Reducing the surface of Safari's injected bundle.

2. Enabling iokit blocking in the Web Process. Safari's QR code detection is
   implemented using the Vision framework in the Web Process. With iokit blocking
   enabled, this approach will fail, as Vision fails to create `CIImage`s.

This patch adds initial support for QR code detection when showing a context menu
in WebKit. Support is implemented by using the Vision framework in the UIProcess
and is controlled by new SPI on `WKWebViewConfiguration`, as QR code detection is
not inexpensive.

Note that Safari's heuristics for detecting QR codes outside of <img> elements
are currently unsupported. These heuristics will be implemented in a subsequent
patch.

* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:

Add an off-by-default setting for QR code detection when showing a context menu.

* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WTF/wtf/PlatformHave.h:

The Vision framework is available on all Cocoa platforms other than watchOS.

* Source/WebCore/PAL/PAL.xcodeproj/project.pbxproj:
* Source/WebCore/PAL/pal/PlatformMac.cmake:
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.h: Added.
* Source/WebCore/PAL/pal/cocoa/VisionSoftLink.mm: Added.
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.h:
* Source/WebKit/Platform/cocoa/ImageAnalysisUtilities.mm:
(WebKit::imageFilledWithWhiteBackground):

Adjust the image used to perform QR code detection to have a white background
to support transparent images.

(WebKit::requestPayloadForQRCode):

Add a helper method to obtain QR code data from a `CGImageRef`.

* Source/WebKit/Shared/ContextMenuContextData.h:
(WebKit::ContextMenuContextData::qrCodePayloadString const):
(WebKit::ContextMenuContextData::setQRCodePayloadString):
* Source/WebKit/UIProcess/API/APIContextMenuClient.h:
(API::ContextMenuClient::menuFromProposedMenu):

Adjust the method to take a `ContextMenuContextData` rather than a `WebHitTestResultData`
to include additional context.

* Source/WebKit/UIProcess/API/APIContextMenuElementInfoMac.h:
* Source/WebKit/UIProcess/API/Cocoa/WKWebView.mm:
(-[WKWebView _setupPageConfiguration:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfiguration.mm:
(-[WKWebViewConfiguration init]):
(-[WKWebViewConfiguration copyWithZone:]):
(-[WKWebViewConfiguration _contextMenuQRCodeDetectionEnabled]):
(-[WKWebViewConfiguration _setContextMenuQRCodeDetectionEnabled:]):
* Source/WebKit/UIProcess/API/Cocoa/WKWebViewConfigurationPrivate.h:

Add SPI to control whether QR code detection is performed when showing a context menu.
Detection is not performed by default.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.h:

Expose the detection result via SPI on `_WKContextMenuElementInfo`.

* Source/WebKit/UIProcess/API/Cocoa/_WKContextMenuElementInfo.mm:
(-[_WKContextMenuElementInfo qrCodePayloadString]):
* Source/WebKit/UIProcess/Cocoa/UIDelegate.h:
* Source/WebKit/UIProcess/Cocoa/UIDelegate.mm:
(WebKit::UIDelegate::ContextMenuClient::menuFromProposedMenu):
* Source/WebKit/UIProcess/WebContextMenuProxy.h:

Remove the `const` qualifier from the `ContextMenuContextData` member, as it can
be modified once QR code detection is completed.

* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.h:
* Source/WebKit/UIProcess/mac/WebContextMenuProxyMac.mm:
(WebKit::WebContextMenuProxyMac::show):

Display the context menu asynchronously if any post-processing (QR code detection)
needs to be performed.

(WebKit::WebContextMenuProxyMac::showAfterPostProcessingContextData):

QR code detection is only performed if the setting is enabled, and there is no
link information in the hit test result. This mirrors the existing behavior in
Safari.

(WebKit::WebContextMenuProxyMac::useContextMenuItems):
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebKit/qr-code.png: Added.
* Tools/TestWebKitAPI/Tests/mac/ContextMenuTests.mm:
(TestWebKitAPI::TEST):

Added tests for QR code detection when showing a context menu.

Canonical link: https://commits.webkit.org/260897@main