[webkit-changes] [WebKit/WebKit] b2e420: Refactor VideoMediaSampleRenderer to support enque...

[Jer Noble]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: b2e420a36ecb45db729eddb8dd38ec1d26b1e439
      https://github.com/WebKit/WebKit/commit/b2e420a36ecb45db729eddb8dd38ec1d26b1e439
  Author: Jer Noble <jer.noble at apple.com>
  Date:   2024-10-31 (Thu, 31 Oct 2024)

  Changed paths:
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm
    M Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h
    M Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm
    M Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h
    M Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm
    M Source/WebCore/platform/mediastream/libwebrtc/VideoFrameLibWebRTC.cpp

  Log Message:
  -----------
  Refactor VideoMediaSampleRenderer to support enqueuing decoded samples
https://bugs.webkit.org/show_bug.cgi?id=281497
rdar://137964106

Reviewed by Jean-Yves Avenard.

Currently, WebCoreDecompressionSession maintains a queue of decoded images. However,
when WebCoreDecompressionSession is used in a mode where those decoded images are
delivered to its clients, the queue is bypassed. VideoMediaSampleRenderer is a more
natural place for such a cue to belong, and eventually, all queueing behavior should
be migrated from WebCoreDecompressionSession to VideoMediaSampleRenderer. As a first
step, expose the promise-based WebCoreDecompressionSession::enqueueSample() function,
and expose a static method to create a CoreMedia buffer queue. Use these primitives
inside VideoMediaSampleRenderer to enqueue decoded images for retrieval by clients
later.

In this mode of operation, a VideoMediaSampleRenderer will feed decoded images to
its associated AVSampleBufferDisplayLayer or AVSampleBufferVideoRenderer, and store
those images in a queue for retrieval via copyDisplayedPixelBuffer(). This requires
a CMTimebaseRef, and to ensure the main thread doesn't excessively block these
operations, move most operations to a WorkQueue.

In the future, VideoMediaSampleRenderer can be extended to support having multiple
AVSBDLs or AVSBVRs simultaneously, allowing reconfigurations of layers or renderers
without having to tear down and re-build decoder pipelines.

* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:
(WebCore::SourceBufferPrivateAVFObjC::enqueueSampleBuffer):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::enqueueSample):
(WebCore::MediaPlayerPrivateWebM::setVideoRenderer):
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h:
(WebCore::VideoMediaSampleRenderer::prefersDecompressionSession):
(WebCore::VideoMediaSampleRenderer::timebase const):
(WebCore::VideoMediaSampleRenderer::setResourceOwner): Deleted.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm:
(WebCore::VideoMediaSampleRenderer::VideoMediaSampleRenderer):
(WebCore::VideoMediaSampleRenderer::~VideoMediaSampleRenderer):
(WebCore::VideoMediaSampleRenderer::isReadyForMoreMediaData const):
(WebCore::VideoMediaSampleRenderer::stopRequestingMediaData):
(WebCore::VideoMediaSampleRenderer::setPrefersDecompressionSession):
(WebCore::VideoMediaSampleRenderer::setPrefersDecompressionSessionInternal):
(WebCore::VideoMediaSampleRenderer::setTimebase):
(WebCore::VideoMediaSampleRenderer::setTimebaseInternal):
(WebCore::VideoMediaSampleRenderer::enqueueSample):
(WebCore::VideoMediaSampleRenderer::initializeDecompressionSession):
(WebCore::VideoMediaSampleRenderer::decodedFrameAvailable):
(WebCore::VideoMediaSampleRenderer::purgeDecodedSampleQueue):
(WebCore::VideoMediaSampleRenderer::ensureDecodedSampleQueue):
(WebCore::VideoMediaSampleRenderer::flush):
(WebCore::VideoMediaSampleRenderer::resetReadyForMoreSample):
(WebCore::VideoMediaSampleRenderer::expectMinimumUpcomingSampleBufferPresentationTime):
(WebCore::VideoMediaSampleRenderer::copyDisplayedPixelBuffer):
(WebCore::VideoMediaSampleRenderer::totalVideoFrames const):
(WebCore::VideoMediaSampleRenderer::droppedVideoFrames const):
(WebCore::VideoMediaSampleRenderer::corruptedVideoFrames const):
(WebCore::VideoMediaSampleRenderer::totalFrameDelay const):
(WebCore::VideoMediaSampleRenderer::setResourceOwner):
(WebCore::VideoMediaSampleRenderer::copyDisplayedPixelBuffer const): Deleted.
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h:
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm:
(WebCore::WebCoreDecompressionSession::createBufferQueue):
(WebCore::WebCoreDecompressionSession::enqueueSample):

Canonical link: https://commits.webkit.org/285965@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications