[webkit-changes] [WebKit/WebKit] ce99b4: [Shape Detection] [Cocoa] Implement first draft of...

[Myles C. Maxfield]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: ce99b4d5dd33bd28ffb58899e356841cb5ed7285
      https://github.com/WebKit/WebKit/commit/ce99b4d5dd33bd28ffb58899e356841cb5ed7285
  Author: Myles C. Maxfield <mmaxfield at apple.com>
  Date:   2023-04-25 (Tue, 25 Apr 2023)

  Changed paths:
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WTF/wtf/PlatformHave.h
    M Source/WTF/wtf/PlatformUse.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.mm
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.mm
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/TextDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/TextDetectorImplementation.mm
    A Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/VisionUtilities.h
    A Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/VisionUtilities.mm
    M Source/WebCore/Modules/async-clipboard/ClipboardImageReader.h
    M Source/WebCore/PAL/pal/spi/cocoa/SpeechSPI.h
    M Source/WebCore/SourcesCocoa.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj

  Log Message:
  -----------
  [Shape Detection] [Cocoa] Implement first draft of the shape detection API using the Vision framework
https://bugs.webkit.org/show_bug.cgi?id=255803
rdar://108380805

Reviewed by Mike Wyrzykowski.

This is the first draft implementation of the shape detection API. This patch makes WebCore.framework link
with Vision.framework. The shape detection API is very very similar to Vision's API, so the implementation
is almost trivial: each call to detect() just turns into the creation of a VNRequest and a VNRequestHandler,
and calls performRequests() on the request handler.

VNRequest has a callback-based codepath, where you can say -[VNRequest initWithCompletionHandler:], but this
isn't really suitable for our purposes, at least right out-of-the-box, for 2 reasons:
1. Even if you set up the VNRequest this way, the call to -[VNImageRequestHandler performRequest:error:] is
       still synchronous. It blocks until all the requests are complete, and AFAICT there's no way to make
       it not block. So, using the callback doesn't actually free up the main thread to do other work. (The
       purpose of the completion handler is when you pass many requests to a single performRequests: call,
       you can be notified as the requests complete, one by one - *not* so that the call to performRequests:
       can be asynchronous. If you want the call to performRequests: to be asynchronous, you're supposed to
       call it from a background thread yourself.)
2. The callbacks are called on another thread internal to Vision, so if we wanted to use them, we'd have to
       immediately bounce back to the RemoteRenderingBackend's worker thread in WebKit2, or the main thread
       in WebKitLegacy. This means the Shape Detection objects would have to have more plumbing to take a
       generic handler which can schedule work on the right thread.
At some point, it probably would make sense to do this plumbing work + calling Vision on a background
thread, but it isn't really necessary for this first-draft implementation. I haven't profiled these calls
yet to know if they're even expensive enough to be worth calling on a background thread or not.

This patch also sets the usesCPUOnly flag, because otherwise the API wouldn't work in WebKit 2, because the
GPU Process's sandbox is blocking the connection to aned on Apple Silicon devices (I haven't tried on Intel
devices, maybe it works using the GPU there?). I'm working with Per Arne to see what we can do about this.
If you manually disable the sandbox, this flag isn't necessary.

The tests for this patch are at
https://github.com/web-platform-tests/wpt/commit/b5a2b1e1b0bd51817dca49b49eb3e8825833b216. I will import
them in a follow-up patch.

This patch marks the shape detection as "testable."

* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.h:
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.mm:
(WebCore::ShapeDetection::convertSymbology):
(WebCore::ShapeDetection::convertBarcodeFormat):
(WebCore::ShapeDetection::convertRequestedBarcodeFormatSet):
(WebCore::ShapeDetection::BarcodeDetectorImpl::BarcodeDetectorImpl):
(WebCore::ShapeDetection::request):
(WebCore::ShapeDetection::BarcodeDetectorImpl::getSupportedFormats):
(WebCore::ShapeDetection::BarcodeDetectorImpl::detect):
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.h:
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.mm:
(WebCore::ShapeDetection::FaceDetectorImpl::FaceDetectorImpl):
(WebCore::ShapeDetection::convertLandmark):
(WebCore::ShapeDetection::convertLandmarks):
(WebCore::ShapeDetection::FaceDetectorImpl::detect):
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/TextDetectorImplementation.mm:
(WebCore::ShapeDetection::TextDetectorImpl::detect):
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:

Canonical link: https://commits.webkit.org/263397@main