[webkit-changes] [WebKit/WebKit] 709636: [iOS] WebM video does not display

[Jean-Yves Avenard]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 7096369deae13feb0406190c08315fb99d5ae28d
      https://github.com/WebKit/WebKit/commit/7096369deae13feb0406190c08315fb99d5ae28d
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-01-08 (Mon, 08 Jan 2024)

  Changed paths:
    M Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp
    M Source/WebCore/SourcesCocoa.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/platform/PlatformMediaError.h
    M Source/WebCore/platform/VideoDecoder.cpp
    M Source/WebCore/platform/VideoDecoder.h
    M Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm
    A Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h
    A Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm
    M Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h
    M Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm
    M Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.cpp
    M Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.h
    M Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.cpp
    M Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.h
    M Source/WebKit/WebProcess/GPU/media/RemoteVideoCodecFactory.cpp

  Log Message:
  -----------
  [iOS] WebM video does not display
https://bugs.webkit.org/show_bug.cgi?id=266748
rdar://119493652

Reviewed by Youenn Fablet.

We registered the VP8 decoder in both the content and GPU process but on iOS decoding was done
in mediaserverd.
So instead we decode in the GPU process before passing the decoded frames to the AVSBDL.
This is achieved with a new VideoMediaSampleRenderer AVSBDL wrapper which will first decode
as necessary.

For VP8 and VP9 (where hardware decoding isn't available), on iOS we use the WebCodec's VideoDecoder
as the decoders aren't available in mediaserverd process.
To not have to rely on WebCodec being conditionally enabled, we make VideoDecoder available
even when ENABLE_WEB_CODECS isn't set.

Add VideoDecoder config option to use a CVPixelBufferPool as CoreMedia requires the use of
IOSurface backed CVPixelBuffer. We don't make this option the default to prevent unexpected
regressions that would impact WebCodecs.

Manually tested on iOS as currently all webm tests are disabled on iOS.

* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::createVideoDecoderConfig):
* Source/WebCore/SourcesCocoa.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/platform/PlatformMediaError.h:
* Source/WebCore/platform/VideoDecoder.cpp:
(WebCore::VideoDecoder::fourCCToCodecString):
(WebCore::VideoDecoder::createLocalDecoder):
* Source/WebCore/platform/VideoDecoder.h:
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:
(WebCore::SourceBufferPrivateAVFObjC::destroyRenderers):
(WebCore::SourceBufferPrivateAVFObjC::setCDMInstance):
(WebCore::SourceBufferPrivateAVFObjC::flushIfNeeded):
(WebCore::SourceBufferPrivateAVFObjC::outputObscuredDueToInsufficientExternalProtectionChanged):
(WebCore::SourceBufferPrivateAVFObjC::layerRequiresFlushToResumeDecodingChanged):
(WebCore::SourceBufferPrivateAVFObjC::layerReadyForDisplayChanged):
(WebCore::SourceBufferPrivateAVFObjC::flushVideo):
(WebCore::SourceBufferPrivateAVFObjC::canEnqueueSample):
(WebCore::SourceBufferPrivateAVFObjC::enqueueSample):
(WebCore::SourceBufferPrivateAVFObjC::enqueueSampleBuffer):
(WebCore::SourceBufferPrivateAVFObjC::isReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::didBecomeReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::notifyClientWhenReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::canSetMinimumUpcomingPresentationTime const):
(WebCore::SourceBufferPrivateAVFObjC::setMinimumUpcomingPresentationTime):
(WebCore::SourceBufferPrivateAVFObjC::clearMinimumUpcomingPresentationTime):
(WebCore::SourceBufferPrivateAVFObjC::setVideoLayer):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::platformLayer const):
(WebCore::MediaPlayerPrivateWebM::updateLastPixelBuffer):
(WebCore::MediaPlayerPrivateWebM::shouldEnsureLayer const):
(WebCore::MediaPlayerPrivateWebM::setPresentationSize):
(WebCore::MediaPlayerPrivateWebM::enqueueSample):
(WebCore::MediaPlayerPrivateWebM::notifyClientWhenReadyForMoreSamples):
(WebCore::MediaPlayerPrivateWebM::setMinimumUpcomingPresentationTime):
(WebCore::MediaPlayerPrivateWebM::clearMinimumUpcomingPresentationTime):
(WebCore::MediaPlayerPrivateWebM::isReadyForMoreSamples):
(WebCore::MediaPlayerPrivateWebM::didBecomeReadyForMoreSamples):
(WebCore::MediaPlayerPrivateWebM::provideMediaData):
(WebCore::MediaPlayerPrivateWebM::flushIfNeeded):
(WebCore::MediaPlayerPrivateWebM::flushTrack):
(WebCore::MediaPlayerPrivateWebM::flushVideo):
(WebCore::MediaPlayerPrivateWebM::ensureLayer):
(WebCore::MediaPlayerPrivateWebM::destroyLayer):
(WebCore::MediaPlayerPrivateWebM::canSetMinimumUpcomingPresentationTime const): Deleted.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h: Added.
(WebCore::VideoMediaSampleRenderer::create):
(WebCore::VideoMediaSampleRenderer::displayLayer const):
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm: Added.
(WebCore::VideoMediaSampleRenderer::VideoMediaSampleRenderer):
(WebCore::VideoMediaSampleRenderer::~VideoMediaSampleRenderer):
(WebCore::VideoMediaSampleRenderer::isReadyForMoreMediaData const):
(WebCore::VideoMediaSampleRenderer::stopRequestingMediaData):
(WebCore::VideoMediaSampleRenderer::enqueueSample):
(WebCore::VideoMediaSampleRenderer::initializeDecompressionSession):
(WebCore::VideoMediaSampleRenderer::flush):
(WebCore::VideoMediaSampleRenderer::requestMediaDataWhenReady):
(WebCore::VideoMediaSampleRenderer::resetReadyForMoreSample):
(WebCore::VideoMediaSampleRenderer::expectMinimumUpcomingSampleBufferPresentationTime):
(WebCore::VideoMediaSampleRenderer::resetUpcomingSampleBufferPresentationTimeExpectations):
(WebCore::VideoMediaSampleRenderer::copyDisplayedPixelBuffer const):
(WebCore::VideoMediaSampleRenderer::bounds const):
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h:
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm:
(WebCore::WebCoreDecompressionSession::invalidate):
(WebCore::WebCoreDecompressionSession::setTimebase):
(WebCore::WebCoreDecompressionSession::setTimebaseWithLockHeld):
(WebCore::WebCoreDecompressionSession::timebase const):
(WebCore::WebCoreDecompressionSession::maybeBecomeReadyForMoreMediaData):
(WebCore::WebCoreDecompressionSession::enqueueSample):
(WebCore::WebCoreDecompressionSession::shouldDecodeSample):
(WebCore::WebCoreDecompressionSession::ensureDecompressionSessionForSample):
(WebCore::WebCoreDecompressionSession::enqueueCompressedSample):
(WebCore::WebCoreDecompressionSession::maybeDecodeNextSample):
(WebCore::WebCoreDecompressionSession::decodeSample):
(WebCore::WebCoreDecompressionSession::setErrorListener):
(WebCore::WebCoreDecompressionSession::removeErrorListener):
(WebCore::WebCoreDecompressionSession::finishCurrentDecodingAndReportError):
(WebCore::WebCoreDecompressionSession::finishCurrentDecodingAndDecodeNextSample):
(WebCore::WebCoreDecompressionSession::decodeSampleSync):
(WebCore::WebCoreDecompressionSession::handleDecompressionOutput):
(WebCore::WebCoreDecompressionSession::automaticDequeue):
(WebCore::WebCoreDecompressionSession::enqueueDecodedSample):
(WebCore::WebCoreDecompressionSession::isReadyForMoreMediaData const):
(WebCore::WebCoreDecompressionSession::requestMediaDataWhenReady):
(WebCore::WebCoreDecompressionSession::decodedFrameWhenAvailable):
(WebCore::WebCoreDecompressionSession::stopRequestingMediaData):
(WebCore::WebCoreDecompressionSession::notifyWhenHasAvailableVideoFrame):
(WebCore::WebCoreDecompressionSession::imageForTime):
(WebCore::WebCoreDecompressionSession::flush):
(WebCore::WebCoreDecompressionSession::resetQosTier):
(WebCore::WebCoreDecompressionSession::increaseQosTier):
(WebCore::WebCoreDecompressionSession::decreaseQosTier):
(WebCore::WebCoreDecompressionSession::updateQosWithDecodeTimeStatistics):
(WebCore::WebCoreDecompressionSession::initializeVideoDecoder):
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.cpp:
* Source/WebCore/platform/graphics/gstreamer/VideoDecoderGStreamer.h:
* Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.cpp:
(WebCore::LibWebRTCVPXInternalVideoDecoder::create):
(WebCore::LibWebRTCVPXVideoDecoder::create):
(WebCore::LibWebRTCVPXVideoDecoder::LibWebRTCVPXVideoDecoder):
(WebCore::LibWebRTCVPXInternalVideoDecoder::LibWebRTCVPXInternalVideoDecoder):
(WebCore::LibWebRTCVPXInternalVideoDecoder::pixelBufferPool):
(WebCore::LibWebRTCVPXInternalVideoDecoder::Decoded):
* Source/WebCore/platform/libwebrtc/LibWebRTCVPXVideoDecoder.h:

Canonical link: https://commits.webkit.org/272760@main