[webkit-changes] [WebKit/WebKit] 7372c4: requestVideoFrameCallback is never called when vid...

[Jean-Yves Avenard]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 7372c4293daa59212c8c34bc27da8ec9f6d3251b
      https://github.com/WebKit/WebKit/commit/7372c4293daa59212c8c34bc27da8ec9f6d3251b
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-12-02 (Mon, 02 Dec 2024)

  Changed paths:
    M LayoutTests/http/tests/media/media-source/mediasource-rvfc.html
    M LayoutTests/media/media-rvfc-playing-webm.html
    A LayoutTests/media/media-source/media-source-rvfc-paused-expected.txt
    A LayoutTests/media/media-source/media-source-rvfc-paused-offscreen-expected.txt
    A LayoutTests/media/media-source/media-source-rvfc-paused-offscreen.html
    A LayoutTests/media/media-source/media-source-rvfc-paused.html
    A LayoutTests/media/media-source/media-source-rvfc-playing-expected.txt
    A LayoutTests/media/media-source/media-source-rvfc-playing-offscreen-expected.txt
    A LayoutTests/media/media-source/media-source-rvfc-playing-offscreen.html
    A LayoutTests/media/media-source/media-source-rvfc-playing.html
    M LayoutTests/media/media-source/media-source-vp8-webm-error-offscreen.html
    M LayoutTests/platform/glib/TestExpectations
    M LayoutTests/platform/mac/TestExpectations
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.h
    M Source/WebCore/platform/graphics/MediaPlayerPrivate.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm
    M Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h
    M Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp

  Log Message:
  -----------
  requestVideoFrameCallback is never called when video element is paused or non-visible
https://bugs.webkit.org/show_bug.cgi?id=282782
rdar://139473124

Reviewed by Jer Noble.

In continuation of bug 283360, and similar to the work done on MediaPlayerPrivateWebM we stop using
directly a WebCoreDecompressionSession whenever the video element isn't displayed or is not in the DOM.
Instead we will use the new capabilities of the VideoMediaSampleRenderer to track and provide the
currently displayed video frames as needed.

We introduce two new preferences (both disabled by default) to tune this behaviour.
- MediaSourcePrefersDecompressionSession: if set, the VideoMediaSampleRenderer will always use a WebCoreDecompressionSession
to perform the decoding if it is possible (e.g content isn't encrypted). With this is turned on, the
VideoMediaSampleRenderer will first decode the video frame and then enqueue it to the AVSampleBufferDisplayLayer.

- MediaSourceCanFallbackToDecompressionSession: if set, the VideoMediaSampleRenderer will only use a WebCoreDecompressionSession
if it's used in combination with the requestVideoFrameCallback API. As soon as the HTMLMediaElement requests to capture the video
frame metadata, we will turn on decoding independently of the AVSampleBufferDisplayLayer.

With those preferences off, the behaviour is close to identical to the previous one: just like we did with
the old WebCoreDecompressionSession, we will tear down the existing AVSampleBufferDisplayRenderer or AVSampleBufferVideoRenderer (visionOS)
whenever the video element is not or removed from the DOM.

Added new tests checking how rVFC behaves with a MSE element, being paused or playing, visible or offscreen.
Also re-enabling tests that used to fail.
Covered by other existing tests.

* LayoutTests/http/tests/media/media-source/mediasource-rvfc.html:
* LayoutTests/media/media-rvfc-playing-webm.html:
* LayoutTests/media/media-source/media-source-rvfc-paused-expected.txt: Added.
* LayoutTests/media/media-source/media-source-rvfc-paused-offscreen-expected.txt: Added.
* LayoutTests/media/media-source/media-source-rvfc-paused-offscreen.html: Added.
* LayoutTests/media/media-source/media-source-rvfc-paused.html: Added.
* LayoutTests/media/media-source/media-source-rvfc-playing-expected.txt: Added.
* LayoutTests/media/media-source/media-source-rvfc-playing-offscreen-expected.txt: Added.
* LayoutTests/media/media-source/media-source-rvfc-playing-offscreen.html: Copied from LayoutTests/media/media-rvfc-playing-webm.html.
* LayoutTests/media/media-source/media-source-rvfc-playing.html: Copied from LayoutTests/media/media-rvfc-playing-webm.html.
* LayoutTests/media/media-source/media-source-vp8-webm-error-offscreen.html:
* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/platform/mac/TestExpectations:
* LayoutTests/platform/ios/TestExpectations:
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WebCore/html/HTMLMediaElement.cpp:
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::setDecompressionSessionPreferences):
(WebCore::MediaPlayer::loadWithNextMediaEngine):
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerPrivate.h:
(WebCore::MediaPlayerPrivateInterface::setDecompressionSessionPreferences):
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::decompressionSession const): Deleted.
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::hasVideoRenderer const): Deleted.
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::~MediaPlayerPrivateMediaSourceAVFObjC):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::load):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::updateLastPixelBuffer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::shouldEnsureLayerOrVideoRenderer const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::setPresentationSize):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::setVideoLayerSizeFenced):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::acceleratedRenderingStateChanged):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::updateDisplayLayer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::videoPlaybackQualityMetrics):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::destroyLayer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::destroyVideoRenderer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::ensureLayerOrVideoRenderer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::destroyLayerOrVideoRendererAndCreateRenderlessVideoMediaSampleRenderer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::configureLayerOrVideoRenderer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::createVideoMediaSampleRendererForRendererer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::canUseDecompressionSession const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::isUsingDecompressionSession const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::willUseDecompressionSessionIfNeeded const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::cdmInstanceAttached):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::cdmInstanceDetached):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::needsVideoLayerChanged):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::startVideoFrameMetadataGathering):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::checkNewVideoFrameMetadata):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::stopVideoFrameMetadataGathering):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::playerContentBoxRectChanged):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::hasVideoRenderer const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::layerOrVideoRenderer const):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::setVideoTarget):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::isInFullscreenOrPictureInPictureChanged):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::updateDisplayLayerAndDecompressionSession): Deleted.
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::ensureDecompressionSession): Deleted.
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::destroyDecompressionSession): Deleted.
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::destroyLayerOrVideoRenderer): Deleted.
* Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.mm:
(WebCore::MediaSourcePrivateAVFObjC::setVideoRenderer):
(WebCore::MediaSourcePrivateAVFObjC::stageVideoRenderer):
(WebCore::MediaSourcePrivateAVFObjC::videoRendererWillReconfigure):
(WebCore::MediaSourcePrivateAVFObjC::videoRendererDidReconfigure):
(WebCore::MediaSourcePrivateAVFObjC::setSourceBufferWithSelectedVideo):
(WebCore::MediaSourcePrivateAVFObjC::setDecompressionSession): Deleted.
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/SourceBufferPrivateAVFObjC.mm:
(WebCore::SourceBufferPrivateAVFObjC::destroyRenderers):
(WebCore::SourceBufferPrivateAVFObjC::trackDidChangeSelected):
(WebCore::SourceBufferPrivateAVFObjC::setCDMInstance):
(WebCore::SourceBufferPrivateAVFObjC::flushIfNeeded):
(WebCore::SourceBufferPrivateAVFObjC::flushVideo):
(WebCore::SourceBufferPrivateAVFObjC::enqueueSample):
(WebCore::SourceBufferPrivateAVFObjC::enqueueSampleBuffer):
(WebCore::SourceBufferPrivateAVFObjC::isReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::didBecomeReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::notifyClientWhenReadyForMoreSamples):
(WebCore::SourceBufferPrivateAVFObjC::configureVideoRenderer):
(WebCore::SourceBufferPrivateAVFObjC::invalidateVideoRenderer):
(WebCore::SourceBufferPrivateAVFObjC::setVideoRenderer):
(WebCore::SourceBufferPrivateAVFObjC::stageVideoRenderer):
(WebCore::SourceBufferPrivateAVFObjC::videoRendererWillReconfigure):
(WebCore::SourceBufferPrivateAVFObjC::videoRendererDidReconfigure):
(WebCore::SourceBufferPrivateAVFObjC::setDecompressionSession): Deleted.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h:
(WebCore::VideoMediaSampleRenderer::prefersDecompressionSession const):
(WebCore::VideoMediaSampleRenderer::prefersDecompressionSession): Deleted.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm:
(WebCore::VideoMediaSampleRenderer::isUsingDecompressionSession const):
(WebCore::VideoMediaSampleRenderer::displayLayer const): Deleted.
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::loadMediaSource):

Canonical link: https://commits.webkit.org/287259@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications