[webkit-changes] [WebKit/WebKit] 2a8263: Use SerializedImageBuffer to pass ImageBuffers bet...

mattwoodrow noreply at github.com
Fri Dec 16 00:14:25 PST 2022 

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 2a8263edbfe035e846cb103a41d14a4df3a8d357
      https://github.com/WebKit/WebKit/commit/2a8263edbfe035e846cb103a41d14a4df3a8d357
  Author: Matt Woodrow <mattwoodrow at apple.com>
  Date:   2022-12-16 (Fri, 16 Dec 2022)

  Changed paths:
    R LayoutTests/platform/ios/imported/w3c/web-platform-tests/html/canvas/offscreen/manual/the-offscreen-canvas/offscreencanvas.getcontext.worker-expected.txt
    M Source/WebCore/bindings/js/SerializedScriptValue.cpp
    M Source/WebCore/html/ImageBitmap.cpp
    M Source/WebCore/html/ImageBitmapBacking.cpp
    M Source/WebCore/html/ImageBitmapBacking.h
    M Source/WebCore/html/OffscreenCanvas.cpp
    M Source/WebCore/html/OffscreenCanvas.h
    M Source/WebCore/page/Chrome.cpp
    M Source/WebCore/page/Chrome.h
    M Source/WebCore/page/ChromeClient.cpp
    M Source/WebCore/page/ChromeClient.h
    M Source/WebCore/platform/GraphicsClient.h
    M Source/WebCore/platform/graphics/ImageBuffer.cpp
    M Source/WebCore/platform/graphics/ImageBuffer.h
    M Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp
    M Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h
    M Source/WebKit/GPUProcess/GPUConnectionToWebProcess.messages.in
    M Source/WebKit/GPUProcess/graphics/RemoteImageBuffer.cpp
    M Source/WebKit/GPUProcess/graphics/RemoteImageBuffer.h
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.cpp
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.h
    M Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.messages.in
    M Source/WebKit/Scripts/webkit/messages.py
    M Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp
    M Source/WebKit/WebKit.xcodeproj/project.pbxproj
    M Source/WebKit/WebProcess/GPU/graphics/RemoteDisplayListRecorderProxy.cpp
    M Source/WebKit/WebProcess/GPU/graphics/RemoteDisplayListRecorderProxy.h
    M Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.cpp
    M Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.h
    M Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.cpp
    M Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.h
    M Source/WebKit/WebProcess/GPU/graphics/RemoteResourceCacheProxy.cpp
    M Source/WebKit/WebProcess/GPU/graphics/RemoteResourceCacheProxy.h
    A Source/WebKit/WebProcess/GPU/graphics/RemoteSerializedImageBufferIdentifier.h
    M Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.cpp
    M Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.h
    M Source/WebKit/WebProcess/WebCoreSupport/WebWorkerClient.cpp
    M Source/WebKit/WebProcess/WebCoreSupport/WebWorkerClient.h

  Log Message:
  -----------
  Use SerializedImageBuffer to pass ImageBuffers between workers and the main thread.
https://bugs.webkit.org/show_bug.cgi?id=244830
<rdar://99857231>

Reviewed by Kimmo Kinnunen.

Create SerializeImageBuffer to transfer ImageBuffers between threads.

Problem: RemoteImageBufferProxy objects cannot be used from any thread other than the one they were
created one. The current code makes a clone into a non-remote ImageBuffer at the locations where
we transfer to/from a worker thread.

The JS Web API here is that when you postMessage an ImageBitmap to a different thread, the existing
object becomes invalid. Thus it is impossible to have concurrent references to an image from multiple
threads.

One option would be to add a way to re-initialize an ImageBuffer object when it is received on another thread.
That's difficult since it's hard to ensure that all members and nested objects are handled correctly.

A variant of that would be to re-create the ImageBuffer when it's received on the other thread. That's maybe
ok, but still weird that we have a pointer to an object that we're not allowed to touch. We'd need to post it
back to it's original thread when we're done, what if that thread no longer exists?

This takes a slightly different approach of creating a serialization object, and making ImageBuffer non-threadsafe refcounted.
For RemoteImageBufferProxy, we only serialize the identifiers for the remote object, and (via runtime asserts) ensure that the old
object will be consumed in the process.

This is definitely move heavyweight, but I like that it keeps the per-thread instances where they belong. It doesn't avoid the
complexity of needing to post between threads for failure cases etc.

* Source/WebCore/page/Chrome.cpp:
(WebCore::Chrome::sinkIntoImageBuffer):
* Source/WebCore/page/Chrome.h:
* Source/WebCore/page/ChromeClient.cpp:
(WebCore::ChromeClient::sinkIntoImageBuffer):
* Source/WebCore/page/ChromeClient.h:
* Source/WebCore/platform/GraphicsClient.h:
* Source/WebCore/platform/graphics/ImageBuffer.cpp:
(WebCore::SerializedImageBuffer::sinkIntoImageBuffer):
(WebCore::DefaultSerializedImageBuffer::DefaultSerializedImageBuffer):
(WebCore::ImageBuffer::sinkIntoSerializedImageBuffer):
(WebCore::ImageBuffer::takeBackend):
* Source/WebCore/platform/graphics/ImageBuffer.h:
(WebCore::ImageBuffer::backendInfo):
(WebCore::SerializedImageBuffer::isRemoteSerializedImageBufferProxy const):
(WebCore::SerializedImageBuffer::isSerializedRemoteImageBuffer const):
* Source/WebKit/WebProcess/GPU/graphics/RemoteDisplayListRecorderProxy.cpp:
(WebKit::RemoteDisplayListRecorderProxy::disconnect):
(WebKit::RemoteDisplayListRecorderProxy::connect):
* Source/WebKit/WebProcess/GPU/graphics/RemoteDisplayListRecorderProxy.h:
* Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.cpp:
(WebKit::RemoteImageBufferProxy::RemoteImageBufferProxy):
(WebKit::RemoteImageBufferProxy::sinkIntoSerializedImageBuffer):
(WebKit::RemoteSerializedImageBufferProxy::sinkIntoImageBuffer):
(WebKit::RemoteSerializedImageBufferProxy::~RemoteSerializedImageBufferProxy):
* Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.h:
(WebKit::RemoteImageBufferProxy::create):
(WebKit::RemoteSerializedImageBufferProxy::renderingResourceIdentifier):
(WebKit::RemoteSerializedImageBufferProxy::renderingBackendIdentifier const):
(WebKit::RemoteSerializedImageBufferProxy::RemoteSerializedImageBufferProxy):
(isType):
* Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.cpp:
(WebKit::RemoteRenderingBackendProxy::~RemoteRenderingBackendProxy):
(WebKit::RemoteRenderingBackendProxy::createRemoteImageBuffer):
(WebKit::RemoteRenderingBackendProxy::transferImageBuffer):
* Source/WebKit/WebProcess/GPU/graphics/RemoteResourceCacheProxy.cpp:
(WebKit::RemoteResourceCacheProxy::clear):
(WebKit::RemoteResourceCacheProxy::releaseImageBuffer):
(WebKit::RemoteResourceCacheProxy::forgetImageBuffer):
* Source/WebKit/WebProcess/GPU/graphics/RemoteResourceCacheProxy.h:
* Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::sinkIntoImageBuffer):
* Source/WebKit/WebProcess/WebCoreSupport/WebChromeClient.h:
* Source/WebKit/WebProcess/WebCoreSupport/WebWorkerClient.cpp:
(WebKit::WebWorkerClient::sinkIntoImageBuffer):
* Source/WebKit/WebProcess/WebCoreSupport/WebWorkerClient.h:

Use ThreadSafeObjectHeap to transfer RemoteImageBuffers between WorkQueues

We're currently dispatching messages between WorkQueue, which has indeterminate order WRT IPC messages.
Using ThreadSafeObjectHeap should let us store and retrieve the references directly, without waiting in queues.

* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::releaseRenderingResource):
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h:
(WebKit::GPUConnectionToWebProcess::RemoteImageBufferData::RemoteImageBufferData):
(WebKit::GPUConnectionToWebProcess::serializedImageHeap):
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.messages.in:
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.cpp:
(WebKit::RemoteRenderingBackend::moveToSerializedBuffer):
(WebKit::RemoteRenderingBackend::moveToImageBuffer):
(WebKit::RemoteRenderingBackend::transferImageBuffer): Deleted.
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.h:
* Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.messages.in:
* Source/WebKit/Scripts/webkit/messages.py:
(serialized_identifiers):
(types_that_cannot_be_forward_declared):
(headers_for_type):
* Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp:
(IPC::serializedIdentifiers):
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.cpp:
(WebKit::RemoteImageBufferProxy::sinkIntoSerializedImageBuffer):
(WebKit::RemoteSerializedImageBufferProxy::RemoteSerializedImageBufferProxy):
(WebKit::RemoteSerializedImageBufferProxy::sinkIntoImageBuffer):
(WebKit::RemoteSerializedImageBufferProxy::~RemoteSerializedImageBufferProxy):
* Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.h:
(WebKit::RemoteSerializedImageBufferProxy::renderingBackendIdentifier const): Deleted.
(WebKit::RemoteSerializedImageBufferProxy::RemoteSerializedImageBufferProxy): Deleted.
(WebKit::RemoteSerializedImageBufferProxy::sinkIntoImageBufferCompleted): Deleted.
* Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.cpp:
(WebKit::RemoteRenderingBackendProxy::moveToSerializedBuffer):
(WebKit::RemoteRenderingBackendProxy::moveToImageBuffer):
(WebKit::RemoteRenderingBackendProxy::transferImageBuffer): Deleted.
* Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.h:
* Source/WebKit/WebProcess/GPU/graphics/RemoteSerializedImageBufferIdentifier.h: Added.

Use SerializedImageBuffer to pass ImageBuffers between workers and the main thread.

* Source/WebCore/bindings/js/SerializedScriptValue.cpp:
(WebCore::CloneDeserializer::readTransferredImageBitmap):
(WebCore::CloneDeserializer::readOffscreenCanvas):
(WebCore::SerializedScriptValue::create):
* Source/WebCore/html/ImageBitmap.cpp:
(WebCore::ImageBitmap::detachBitmaps):
* Source/WebCore/html/ImageBitmapBacking.cpp:
(WebCore::ImageBitmapBacking::disconnect):
(WebCore::ImageBitmapBacking::connect):
* Source/WebCore/html/ImageBitmapBacking.h:
* Source/WebCore/html/OffscreenCanvas.cpp:
(WebCore::DetachedOffscreenCanvas::DetachedOffscreenCanvas):
(WebCore::DetachedOffscreenCanvas::takeImageBuffer):
(WebCore::OffscreenCanvas::create):
(WebCore::OffscreenCanvas::pushBufferToPlaceholder):
(WebCore::OffscreenCanvas::commitToPlaceholderCanvas):
(WebCore::OffscreenCanvas::takeImageBuffer const):
* Source/WebCore/html/OffscreenCanvas.h:

Canonical link: https://commits.webkit.org/257979@main

More information about the webkit-changes mailing list