[webkit-changes] [WebKit/WebKit] aedef0: RealtimeOutgoingVideoSource needs to trigger resol...

[youennf]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: aedef001a9cb8b3efeb97d9d7afb83c27c12c74c
      https://github.com/WebKit/WebKit/commit/aedef001a9cb8b3efeb97d9d7afb83c27c12c74c
  Author: Youenn Fablet <youennf at gmail.com>
  Date:   2024-01-18 (Thu, 18 Jan 2024)

  Changed paths:
    M LayoutTests/platform/glib/TestExpectations
    A LayoutTests/webrtc/captureCanvas-webrtc-with-video-scaling-adaptation-expected.txt
    A LayoutTests/webrtc/captureCanvas-webrtc-with-video-scaling-adaptation.html
    A LayoutTests/webrtc/video-maxBitrate-expected.txt
    A LayoutTests/webrtc/video-maxBitrate-vp8-expected.txt
    A LayoutTests/webrtc/video-maxBitrate-vp8.html
    A LayoutTests/webrtc/video-maxBitrate.html
    M Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/native/src/objc_frame_buffer.h
    M Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/native/src/objc_frame_buffer.mm
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCMediaEndpoint.cpp
    M Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.cpp
    M Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.h
    M Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp
    M Tools/DumpRenderTree/TestOptions.cpp
    M Tools/WebKitTestRunner/TestOptions.cpp

  Log Message:
  -----------
  RealtimeOutgoingVideoSource needs to trigger resolution scaling in case of WebRTC maintain-framerate degradationPreference
rdar://121041723
https://bugs.webkit.org/show_bug.cgi?id=267577

Reviewed by Eric Carlson.

Our WebRTC backend is applying degradation preference but it relies on WebKit to react upon resolution decrease signal,
like done for frame rate, via the max_pixel_count value.

WebKit is downsampling sent video frames based on max_pixel_count.
We only do this update based on a runtime flag, which is off by default, since this may introduce flakinesses in tests.
We may as a follow-up enable this adaptation in tests by default and disable it in affecting tests based on bot results.

To do this update, WebKit is computing the updated width and height based on max_pixel_count.
It reuses the 3/4-2/3 approach used by Chromium to select lower resolutions based on the original frame resolution.

To limit the cost of downsampling, we add support for an adaptation layer in ObjCFrameBuffer.
ObjCFrameBuffer takes a pixel buffer (or pixel buffer getter) and a width and height.
The width and height may not match with the underlying pixel buffer.
Downsampling happens when trying to grab the actual pixel buffer, in ToI420.
We implement an optimized Scale routine that is only updating the width and height of the ObjCFrameBuffer without creating the corresponding pixel buffer.

Since we are now sharing the underlying pixel buffer in several different ObjCFrameBuffers, scaled ObjCFrameBuffers keeps a reference to the original ObjCFrameBuffer.
This code path is done for libvpx encoders.
For remote VTB encoders, they directly do the downsampling as part of the encoding process.
We thus do not need to do the downsampling before sending the video frames.

We cover both code paths by doing the same test with H264 and VP8.

* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/webrtc/video-maxBitrate-expected.txt: Added.
* LayoutTests/webrtc/video-maxBitrate-vp8-expected.txt: Added.
* LayoutTests/webrtc/video-maxBitrate-vp8.html: Added.
* LayoutTests/webrtc/video-maxBitrate.html: Added.
* Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/native/src/objc_frame_buffer.h:
* Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/native/src/objc_frame_buffer.mm:
(webrtc::ObjCFrameBuffer::ObjCFrameBuffer):
(webrtc::ObjCFrameBuffer::ToI420):
(webrtc::ObjCFrameBuffer::CropAndScale):
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WebCore/Modules/mediastream/libwebrtc/LibWebRTCMediaEndpoint.cpp:
(WebCore::LibWebRTCMediaEndpoint::addTrack):
* Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.cpp:
(WebCore::RealtimeOutgoingVideoSource::updateFramesSending):
(WebCore::RealtimeOutgoingVideoSource::sourceMutedChanged):
(WebCore::RealtimeOutgoingVideoSource::sourceEnabledChanged):
(WebCore::RealtimeOutgoingVideoSource::initializeFromSource):
(WebCore::RealtimeOutgoingVideoSource::AddOrUpdateSink):
(WebCore::RealtimeOutgoingVideoSource::updateBlackFramesSending): Deleted.
* Source/WebCore/platform/mediastream/RealtimeOutgoingVideoSource.h:
(WebCore::RealtimeOutgoingVideoSource::disableVideoScaling):
(WebCore::RealtimeOutgoingVideoSource::videoFrameScaling const):
* Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.cpp:
(WebCore::RealtimeOutgoingVideoSourceCocoa::videoFrameAvailable):
* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::resetToConsistentState):

Canonical link: https://commits.webkit.org/273172@main