[webkit-changes] [WebKit/WebKit] 5ecd84: Plumb shape detection API to the GPU Process

Tue Apr 18 21:26:27 PDT 2023

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 5ecd8455a80d9c5becbddee087062f82891fccea
      https://github.com/WebKit/WebKit/commit/5ecd8455a80d9c5becbddee087062f82891fccea
  Author: Myles C. Maxfield <mmaxfield at apple.com>
  Date:   2023-04-18 (Tue, 18 Apr 2023)

  Changed paths:
    M Source/WTF/wtf/PlatformHave.h
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Modules/ShapeDetection/BarcodeDetector.cpp
    M Source/WebCore/Modules/ShapeDetection/BarcodeDetector.h
    M Source/WebCore/Modules/ShapeDetection/BarcodeDetector.idl
    M Source/WebCore/Modules/ShapeDetection/FaceDetector.cpp
    M Source/WebCore/Modules/ShapeDetection/FaceDetector.h
    M Source/WebCore/Modules/ShapeDetection/FaceDetector.idl
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/TextDetectorImplementation.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/BarcodeDetectorOptionsInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/BarcodeFormatInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedBarcodeInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedFaceInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedTextInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/LandmarkInterface.h
    M Source/WebCore/Modules/ShapeDetection/Interfaces/LandmarkTypeInterface.h
    M Source/WebCore/Modules/ShapeDetection/TextDetector.cpp
    M Source/WebCore/Modules/ShapeDetection/TextDetector.h
    M Source/WebCore/Modules/ShapeDetection/TextDetector.idl
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/page/Chrome.cpp
    M Source/WebCore/page/Chrome.h
    M Source/WebCore/page/ChromeClient.h
    M Source/WebKit/CMakeLists.txt
    M Source/WebKit/DerivedSources-input.xcfilelist
    M Source/WebKit/DerivedSources-output.xcfilelist
    M Source/WebKit/DerivedSources.make
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.cpp
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.h
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.messages.in
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.cpp
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.h
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.messages.in
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.cpp
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.h
    A Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.messages.in
    A Source/WebKit/GPUProcess/ShapeDetection/ShapeDetectionObjectHeap.cpp
    A Source/WebKit/GPUProcess/ShapeDetection/ShapeDetectionObjectHeap.h
    M Source/WebKit/GPUProcess/graphics/RemoteDisplayListRecorder.cpp
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.cpp
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.h
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.messages.in
    M Source/WebKit/GPUProcess/webrtc/RemoteAudioMediaStreamTrackRendererInternalUnitManager.cpp
    M Source/WebKit/Scripts/webkit/messages.py
    M Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp
    M Source/WebKit/Scripts/webkit/tests/MessageNames.cpp
    M Source/WebKit/Scripts/webkit/tests/MessageNames.h
    M Source/WebKit/Scripts/webkit/tests/TestWithCVPixelBufferMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithCVPixelBufferMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithIfMessageMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithIfMessageMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithImageDataMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithImageDataMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithLegacyReceiverMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithLegacyReceiverMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithSemaphoreMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithSemaphoreMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamBatchedMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamBatchedMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamBufferMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamBufferMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithStreamMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithSuperclassMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithSuperclassMessages.h
    M Source/WebKit/Scripts/webkit/tests/TestWithoutAttributesMessageReceiver.cpp
    M Source/WebKit/Scripts/webkit/tests/TestWithoutAttributesMessages.h
    A Source/WebKit/Shared/ShapeDetectionIdentifier.h
    M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
    M Source/WebKit/Sources.txt
    M Source/WebKit/WebKit.xcodeproj/project.pbxproj
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteBarcodeDetectorProxy.cpp
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteBarcodeDetectorProxy.h
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteFaceDetectorProxy.cpp
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteFaceDetectorProxy.h
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteTextDetectorProxy.cpp
    A Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteTextDetectorProxy.h
    M Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp
    M Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h
    M Source/WebKit/WebProcess/Inspector/WebInspectorUIExtensionController.cpp
    M Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.cpp
    M Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.h
    M Source/WebKit/WebProcess/WebPage/RemoteLayerTree/GraphicsLayerCARemote.cpp
    M Source/WebKit/WebProcess/WebPage/WebURLSchemeTaskProxy.cpp
    M Source/WebKitLegacy/mac/WebCoreSupport/WebChromeClient.h
    M Source/WebKitLegacy/mac/WebCoreSupport/WebChromeClient.mm

  Log Message:
  -----------
  Plumb shape detection API to the GPU Process
https://bugs.webkit.org/show_bug.cgi?id=255473
rdar://108070184

Reviewed by Mike Wyrzykowski.

This patch is really just plumbing, though there are a few pieces to it.

1. The shape detection API is already behind a runtime flag, which guards the JavaScript entry points.
       So, the code is getting compiled everywhere. However, eventually something somewhere has to
       actually implement the API itself (for Cocoa ports, that's the Vision framework), and not all
       ports have this support. So, there also needs to be a build-time flag,
       HAVE(SHAPE_DETECTION_API_IMPLEMENTATION), which _just_ guards the calls to the final
       implementation. This is just to make the code compile on all ports.
2. WebCore doesn't know anything about IPC, so the object creation methods need to be plumbed up
       through ChromeClient, so WebKitLegacy or WebKit2 can choose whether to back the objects by
       WebKit::ShapeDetection::RemoteFooProxy objects, or by WebCore::ShapeDetection::FooImpl objects.
       Because these object creation methods are free functions in JavaScript, they need to have the
       [CallWith=CurrentScriptExecutionContext] annotation, to get the context they need to talk to
       the Page and ChromeClient.
3. WebCore::ShapeDetection::FooImpl objects need to be exported from WebCore, so WebKit and
       WebKitLegacy can create them
4. Create RemoteFooProxy and RemoteFoo classes. RemoteFooProxy classes live in the web process, and
       forward all their method calls to IPC send(). RemoteFoo classes live in the GPU process, and own
       an instance of the corresponding WebCore::ShapeDetection::FooImpl object. There are messages
       defined in RemoteFoo.messages.in files.
5. Handle object lifetime in the GPU process. When an object is created in the web process, that
       corresponds to a RemoteRenderingBackend::CreateFoo() message. The RemoteRenderingBackend owns a
       WebKit::ShapeDetection::ObjectHeap, which retains all the objects in the GPU process that get
       created. The objects that get created start listening on the preexisting StreamConnection of the
       RemoteRenderingBackend - I didn't want to give each shape detection object its own totally new
       StreamConnection (like how WebGPU does) because these shape detection objects are so small and
       only have 1 or 2 methods on them. Object destructors in the web process send a
       RemoteRenderingBackend::ReleaseFoo() message.
6. Add serialization information for the relevant structs to WebCoreArgumentCoders.serialization.in

The last thing that's worth mentioning is I haven't hooked up the ImageBitmapSource parameters to the
detect() functions, because those sources can be in many different forms, and the logic to handle all
the forms is quite large, and outside the scope of this simple plumbing patch. My next patch will
implement these parameters.

No tests because there is no behavior change - I haven't actually hooked up the Vision framework to
actually implement any of the calls yet.

* Source/WTF/wtf/PlatformHave.h:
* Source/WebCore/Modules/ShapeDetection/BarcodeDetector.cpp:
(WebCore::BarcodeDetector::create):
(WebCore::BarcodeDetector::getSupportedFormats):
* Source/WebCore/Modules/ShapeDetection/BarcodeDetector.h:
* Source/WebCore/Modules/ShapeDetection/BarcodeDetector.idl:
* Source/WebCore/Modules/ShapeDetection/FaceDetector.cpp:
(WebCore::FaceDetector::create):
* Source/WebCore/Modules/ShapeDetection/FaceDetector.h:
* Source/WebCore/Modules/ShapeDetection/FaceDetector.idl:
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.h:
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/FaceDetectorImplementation.h:
* Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/TextDetectorImplementation.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/BarcodeDetectorOptionsInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/BarcodeFormatInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedBarcodeInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedFaceInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/DetectedTextInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/LandmarkInterface.h:
* Source/WebCore/Modules/ShapeDetection/Interfaces/LandmarkTypeInterface.h:
* Source/WebCore/Modules/ShapeDetection/TextDetector.cpp:
(WebCore::TextDetector::create):
* Source/WebCore/Modules/ShapeDetection/TextDetector.h:
* Source/WebCore/Modules/ShapeDetection/TextDetector.idl:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/page/Chrome.cpp:
(WebCore::Chrome::createBarcodeDetector const):
(WebCore::Chrome::getBarcodeDetectorSupportedFormats const):
(WebCore::Chrome::createFaceDetector const):
(WebCore::Chrome::createTextDetector const):
* Source/WebCore/page/Chrome.h:
* Source/WebCore/page/ChromeClient.h:
(WebCore::ChromeClient::createBarcodeDetector const):
(WebCore::ChromeClient::getBarcodeDetectorSupportedFormats const):
(WebCore::ChromeClient::createFaceDetector const):
(WebCore::ChromeClient::createTextDetector const):
* Source/WebKit/CMakeLists.txt:
* Source/WebKit/DerivedSources-input.xcfilelist:
* Source/WebKit/DerivedSources-output.xcfilelist:
* Source/WebKit/DerivedSources.make:
* Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.cpp: Copied from Source/WebCore/Modules/ShapeDetection/TextDetector.h.
(WebKit::RemoteBarcodeDetector::RemoteBarcodeDetector):
(WebKit::RemoteBarcodeDetector::detect):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.h: Added.
(WebKit::RemoteBarcodeDetector::create):
(WebKit::RemoteBarcodeDetector::backing):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteBarcodeDetector.messages.in: Added.
* Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.cpp: Copied from Source/WebCore/Modules/ShapeDetection/TextDetector.h.
(WebKit::RemoteFaceDetector::RemoteFaceDetector):
(WebKit::RemoteFaceDetector::detect):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.h: Added.
(WebKit::RemoteFaceDetector::create):
(WebKit::RemoteFaceDetector::backing):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteFaceDetector.messages.in: Added.
* Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.cpp: Copied from Source/WebCore/Modules/ShapeDetection/TextDetector.h.
(WebKit::RemoteTextDetector::RemoteTextDetector):
(WebKit::RemoteTextDetector::detect):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.h: Added.
(WebKit::RemoteTextDetector::create):
(WebKit::RemoteTextDetector::backing):
* Source/WebKit/GPUProcess/ShapeDetection/RemoteTextDetector.messages.in: Added.
* Source/WebKit/GPUProcess/ShapeDetection/ShapeDetectionObjectHeap.cpp: Copied from Source/WebCore/Modules/ShapeDetection/TextDetector.cpp.
(WebKit::ShapeDetection::ObjectHeap::addObject):
(WebKit::ShapeDetection::ObjectHeap::removeObject):
(WebKit::ShapeDetection::ObjectHeap::clear):
* Source/WebKit/GPUProcess/ShapeDetection/ShapeDetectionObjectHeap.h: Copied from Source/WebCore/Modules/ShapeDetection/Implementation/Cocoa/BarcodeDetectorImplementation.h.
* Source/WebKit/GPUProcess/graphics/RemoteDisplayListRecorder.cpp:
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.cpp:
(WebKit::RemoteRenderingBackend::RemoteRenderingBackend):
(WebKit::RemoteRenderingBackend::createRemoteBarcodeDetector):
(WebKit::RemoteRenderingBackend::releaseRemoteBarcodeDetector):
(WebKit::RemoteRenderingBackend::getRemoteBarcodeDetectorSupportedFormats):
(WebKit::RemoteRenderingBackend::createRemoteFaceDetector):
(WebKit::RemoteRenderingBackend::releaseRemoteFaceDetector):
(WebKit::RemoteRenderingBackend::createRemoteTextDetector):
(WebKit::RemoteRenderingBackend::releaseRemoteTextDetector):
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.h:
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.messages.in:
* Source/WebKit/GPUProcess/webrtc/RemoteAudioMediaStreamTrackRendererInternalUnitManager.cpp:
(WebKit::RemoteAudioMediaStreamTrackRendererInternalUnitManager::Unit::captureUnitHasStopped):
* Source/WebKit/Scripts/webkit/messages.py:
(serialized_identifiers):
(headers_for_type):
* Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp:
(IPC::serializedIdentifiers):
* Source/WebKit/Shared/ShapeDetectionIdentifier.h: Copied from Source/WebCore/Modules/ShapeDetection/Interfaces/BarcodeDetectorOptionsInterface.h.
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/Sources.txt:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteBarcodeDetectorProxy.cpp: Added.
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::create):
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::RemoteBarcodeDetectorProxy):
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::~RemoteBarcodeDetectorProxy):
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::getSupportedFormats):
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::detect):
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteBarcodeDetectorProxy.h: Added.
(WebKit::ShapeDetection::RemoteBarcodeDetectorProxy::backing const):
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteFaceDetectorProxy.cpp: Added.
(WebKit::ShapeDetection::RemoteFaceDetectorProxy::create):
(WebKit::ShapeDetection::RemoteFaceDetectorProxy::RemoteFaceDetectorProxy):
(WebKit::ShapeDetection::RemoteFaceDetectorProxy::~RemoteFaceDetectorProxy):
(WebKit::ShapeDetection::RemoteFaceDetectorProxy::detect):
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteFaceDetectorProxy.h: Added.
(WebKit::ShapeDetection::RemoteFaceDetectorProxy::backing const):
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteTextDetectorProxy.cpp: Added.
(WebKit::ShapeDetection::RemoteTextDetectorProxy::create):
(WebKit::ShapeDetection::RemoteTextDetectorProxy::RemoteTextDetectorProxy):
(WebKit::ShapeDetection::RemoteTextDetectorProxy::~RemoteTextDetectorProxy):
(WebKit::ShapeDetection::RemoteTextDetectorProxy::detect):
* Source/WebKit/WebProcess/GPU/ShapeDetection/RemoteTextDetectorProxy.h: Added.
(WebKit::ShapeDetection::RemoteTextDetectorProxy::backing const):
* Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp:
* Source/WebKit/WebProcess/GPU/graphics/WebGPU/RemoteGPUProxy.h:
* Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h:
* Source/WebKit/WebProcess/Inspector/WebInspectorUIExtensionController.cpp:
* Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::createBarcodeDetector const):
(WebKit::WebChromeClient::getBarcodeDetectorSupportedFormats const):
(WebKit::WebChromeClient::createFaceDetector const):
(WebKit::WebChromeClient::createTextDetector const):
* Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.h:
* Source/WebKit/WebProcess/WebPage/RemoteLayerTree/GraphicsLayerCARemote.cpp:
* Source/WebKitLegacy/mac/WebCoreSupport/WebChromeClient.h:
* Source/WebKitLegacy/mac/WebCoreSupport/WebChromeClient.mm:
(WebChromeClient::createBarcodeDetector const):
(WebChromeClient::getBarcodeDetectorSupportedFormats const):
(WebChromeClient::createFaceDetector const):
(WebChromeClient::createTextDetector const):

Canonical link: https://commits.webkit.org/263115@main