[webkit-changes] [WebKit/WebKit] 149d8d: Add support for getDisplayMedia video track clone ...

[youennf]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 149d8d7a3cfd0970497eb25ebdf41baecd09e968
      https://github.com/WebKit/WebKit/commit/149d8d7a3cfd0970497eb25ebdf41baecd09e968
  Author: Youenn Fablet <youennf at gmail.com>
  Date:   2023-05-06 (Sat, 06 May 2023)

  Changed paths:
    M LayoutTests/fast/mediastream/constraint-intrinsic-size.html
    M LayoutTests/fast/mediastream/getDisplayMedia-frame-rate-expected.txt
    M LayoutTests/fast/mediastream/getDisplayMedia-frame-rate.html
    M LayoutTests/fast/mediastream/getDisplayMedia-size-expected.txt
    A LayoutTests/fast/mediastream/getUserMedia-frame-rate-expected.txt
    A LayoutTests/fast/mediastream/getUserMedia-frame-rate.html
    M LayoutTests/platform/glib/TestExpectations
    M LayoutTests/platform/mac-wk2/TestExpectations
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Sources.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/platform/mediastream/MediaConstraints.cpp
    M Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp
    M Source/WebCore/platform/mediastream/RealtimeMediaSource.h
    M Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp
    M Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h
    R Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp
    R Source/WebCore/platform/mediastream/RealtimeVideoSource.h
    M Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp
    M Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.h
    M Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp
    M Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm
    M Source/WebCore/platform/mediastream/mac/MockRealtimeVideoSourceMac.mm
    M Source/WebCore/platform/mock/MockMediaDevice.h
    M Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp
    M Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp
    M Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp
    M Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.h
    M Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.cpp

  Log Message:
  -----------
  Add support for getDisplayMedia video track clone resizing
https://bugs.webkit.org/show_bug.cgi?id=256250
rdar://problem/108834643

Reviewed by Eric Carlson.

Remove RealtimeVideoSource which only handles preset based video capture sources.
Instead, we make RealtimeMediaSource::VideoFrameObserver able to state which size and frame rate it wants.
RealtimeMediaSource will then update video frame sending accordingly.

We enable DisplayCaptureSourceCocoa on iOS as well to be able to run tests on simulator properly.

On WebProcess side, all VideoFrameObservers will not provide desired size/frame rates since they only consume video frames.
On GPUProcess side, VideoFrameObservers created by UserMediaCaptureManagerProxy will provide desired size/frame rates from provided constraints.

The computation of desired size/frame rates from constraints is done in two steps:
- VideoFrameObservers will provide ideal width/height/frameRate values from the MediaConstraints using RealtimeMediaSource::extractVideoFrameSizeConstraints.
- RealtimeMediaSource will compute the desired sizes based on RealtimeMediaSource::computeResizedVideoFrameSize.

Camera sources will allow aspect ratio changes, see RealtimeMediaSource::computeResizedVideoFrameSize.
Display sources will not allow aspect ratio changes, see DisplayCaptureSourceCocoa::computeResizedVideoFrameSize.

We start changing how aspectRatio is handled as well.
It is used for camera sources when validating constraints and when computing the width/height ideal values.
In the future, RealtimeMediaSource may not need to store aspectRatio at all.
For now, if aspectRatio constraint is provided, we will not set the default height constraint to 480, since it can be computed from the width and aspectRatio.

By using size/frame rates as part of VideoFrameObservers, observers can receive video frames with different size/frame rate than the source is exposing via getSettings().
To keep consistent settings values, UserMediaCaptureManagerProxy is overriding source width/height/frameRate based on size/frame rates passed to the VideoFrameObserver.

In the future, we might be able to remove RealtimeMediaSource size and only keep intrinsicSize, which will simplify the class.
The display source is now capturing at the native size and resizing is done as a follow-up step when sending video frame to observers.

Covered by existing tests for getUserMedia and by new tests for getDisplayMedia.

* LayoutTests/fast/mediastream/constraint-intrinsic-size.html:
* LayoutTests/fast/mediastream/getDisplayMedia-frame-rate-expected.txt:
* LayoutTests/fast/mediastream/getDisplayMedia-frame-rate.html:
* LayoutTests/fast/mediastream/getDisplayMedia-size-expected.txt:
* LayoutTests/fast/mediastream/getUserMedia-frame-rate-expected.txt: Added.
* LayoutTests/fast/mediastream/getUserMedia-frame-rate.html: Added.
* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/platform/mac-wk2/TestExpectations:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/platform/mediastream/MediaConstraints.cpp:
(WebCore::addDefaultVideoConstraints):
(WebCore::MediaConstraints::setDefaultVideoConstraints):
* Source/WebCore/platform/mediastream/RealtimeMediaSource.cpp:
(WebCore::VideoFrameAdaptor::VideoFrameAdaptor):
(WebCore::RealtimeMediaSource::~RealtimeMediaSource):
(WebCore::RealtimeMediaSource::addVideoFrameObserver):
(WebCore::RealtimeMediaSource::removeVideoFrameObserver):
(WebCore::RealtimeMediaSource::forEachVideoFrameObserver):
(WebCore::RealtimeMediaSource::videoFrameRotation const):
(WebCore::adaptVideoFrame):
(WebCore::RealtimeMediaSource::computeResizedVideoFrameSize):
(WebCore::RealtimeMediaSource::videoFrameAvailable):
(WebCore::RealtimeMediaSource::extractVideoFrameSizeConstraints):
(WebCore::RealtimeMediaSource::applyConstraints):
* Source/WebCore/platform/mediastream/RealtimeMediaSource.h:
* Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp:
(WebCore::RealtimeVideoCaptureSource::updateCapabilities):
(WebCore::RealtimeVideoCaptureSource::bestSupportedSizeFrameRateAndZoom):
(WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeFrameRateAndZoom):
(WebCore::RealtimeVideoCaptureSource::setSizeFrameRateAndZoom):
(WebCore::RealtimeVideoCaptureSource::videoFrameRotation const): Deleted.
* Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h:
* Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp: Removed.
* Source/WebCore/platform/mediastream/RealtimeVideoSource.h: Removed.
* Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.cpp:
(WebCore::DisplayCaptureSourceCocoa::computeResizedVideoFrameSize):
(WebCore::DisplayCaptureSourceCocoa::setSizeFrameRateAndZoom):
(WebCore::DisplayCaptureSourceCocoa::emitFrame):
(WebCore::DisplayCaptureSourceCocoa::updateFrameSize): Deleted.
* Source/WebCore/platform/mediastream/cocoa/DisplayCaptureSourceCocoa.h:
* Source/WebCore/platform/mediastream/gstreamer/MockRealtimeVideoSourceGStreamer.cpp:
(WebCore::MockRealtimeVideoSource::create):
(WebCore::MockDisplayCaptureSourceGStreamer::create):
* Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:
(WebCore::AVVideoCaptureSource::create):
* Source/WebCore/platform/mediastream/mac/MockRealtimeVideoSourceMac.mm:
(WebCore::MockRealtimeVideoSource::create):
* Source/WebCore/platform/mock/MockMediaDevice.h:
* Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp:
* Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp:
* Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.cpp:
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::applyConstraints):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::settings):
(WebKit::UserMediaCaptureManagerProxy::SourceProxy::copySettings):
(WebKit::UserMediaCaptureManagerProxy::createCameraSource):
(WebKit::UserMediaCaptureManagerProxy::createMediaSourceForCaptureDeviceWithConstraints):
(WebKit::UserMediaCaptureManagerProxy::applyConstraints):
(WebKit::UserMediaCaptureManagerProxy::clone):
* Source/WebKit/UIProcess/Cocoa/UserMediaCaptureManagerProxy.h:
* Source/WebKit/WebProcess/cocoa/UserMediaCaptureManager.cpp:

Canonical link: https://commits.webkit.org/263750@main