[webkit-changes] [WebKit/WebKit] 72be95: All MediaRecorderPrivate should be running in the ...

Sat Oct 26 03:23:25 PDT 2024

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 72be95a9c52756b4412dacc18062120d5c784889
      https://github.com/WebKit/WebKit/commit/72be95a9c52756b4412dacc18062120d5c784889
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-10-26 (Sat, 26 Oct 2024)

  Changed paths:
    M LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable-gpuprocess.html
    M LayoutTests/http/wpt/mediarecorder/MediaRecorder-first-frame-expected.txt
    M LayoutTests/http/wpt/mediarecorder/MediaRecorder-first-frame.html
    M LayoutTests/http/wpt/mediarecorder/MediaRecorder-requestData.html
    M LayoutTests/http/wpt/mediarecorder/mimeType.html
    M LayoutTests/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-mimetype-expected.txt
    M LayoutTests/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-mimetype.html
    R LayoutTests/ipc/shared-video-frame-size-expected.txt
    R LayoutTests/ipc/shared-video-frame-size.html
    M LayoutTests/platform/ios/TestExpectations
    M LayoutTests/platform/mac-wk1/TestExpectations
    M LayoutTests/platform/mac-wk2/TestExpectations
    M LayoutTests/platform/mac/TestExpectations
    M LayoutTests/platform/wk2/TestExpectations
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WTF/wtf/PlatformEnableCocoa.h
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp
    M Source/WebCore/Modules/mediarecorder/MediaRecorder.h
    R Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.cpp
    R Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.h
    M Source/WebCore/PlatformMac.cmake
    M Source/WebCore/SaferCPPExpectations/UncountedCallArgsCheckerExpectations
    M Source/WebCore/Sources.txt
    M Source/WebCore/SourcesCocoa.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/loader/EmptyClients.cpp
    M Source/WebCore/page/Page.cpp
    M Source/WebCore/page/Page.h
    M Source/WebCore/page/PageConfiguration.cpp
    M Source/WebCore/page/PageConfiguration.h
    M Source/WebCore/platform/MediaSample.h
    M Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp
    M Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h
    A Source/WebCore/platform/mediarecorder/MediaRecorderPrivateEncoder.cpp
    A Source/WebCore/platform/mediarecorder/MediaRecorderPrivateEncoder.h
    M Source/WebCore/platform/mediarecorder/MediaRecorderPrivateWriter.cpp
    M Source/WebCore/platform/mediarecorder/MediaRecorderPrivateWriter.h
    A Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterAVFObjC.h
    A Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterAVFObjC.mm
    R Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h
    R Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm
    A Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.cpp
    M Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.h
    R Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.mm
    M Source/WebCore/testing/Internals.cpp
    M Source/WebKit/CMakeLists.txt
    M Source/WebKit/DerivedSources-input.xcfilelist
    M Source/WebKit/DerivedSources-output.xcfilelist
    M Source/WebKit/DerivedSources.make
    M Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp
    M Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.cpp
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.h
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.messages.in
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.cpp
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.h
    R Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.messages.in
    M Source/WebKit/SaferCPPExpectations/UncountedCallArgsCheckerExpectations
    M Source/WebKit/Scripts/webkit/messages.py
    M Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp
    M Source/WebKit/Shared/Cocoa/WebCoreArgumentCodersCocoa.serialization.in
    M Source/WebKit/Shared/WTFArgumentCoders.serialization.in
    M Source/WebKit/Sources.txt
    M Source/WebKit/SourcesCocoa.txt
    M Source/WebKit/WebKit.xcodeproj/project.pbxproj
    R Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderIdentifier.h
    R Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp
    R Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h
    R Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.cpp
    R Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.h
    M Source/WebKit/WebProcess/WebPage/WebPage.cpp
    M Source/WebKitLegacy/mac/WebView/WebView.mm

  Log Message:
  -----------
  All MediaRecorderPrivate should be running in the web content process.
https://bugs.webkit.org/show_bug.cgi?id=281816
rdar://138239763

Reviewed by Jer Noble.

We extend on the work started with the MediaRecorderPrivateWriterWebM introduced in 285061 at main.
We add a new MediaRecorderPrivateEncoder that performs the encoding, encoded frames interleaving management
and writing through the specialised MediaRecorderPrivateWriter.

This new MediaRecorderPrivate can encode into any audio codec supported by AVFoundation (though only opus and AAC is enabled for now)
and all video codecs supported by VideoEncoder with webm (vp8, vp9) and mp4 (h264, h265, av1).

The code runs in its dedicated WorkQueue.

One of the key change which added a significant amount of technical complexity is that while we will generate 2s GOP
a call to MediaRecorder.requestData() will return fully playable media segment, formed such as each new segment starts
with a keyframe. The generated media is MSE compatible.
This way, the page fully controls the duration of the media segments generated by either setting
a time slice when creating a MediaRecorder or calling requestData() at regular interval.

When a new segment is required by the page, we flush all pending frames received to that point. As the audio encoder
has a certain latency, we may need to wait for a future audio frame to come which would allow the audio for the
current time to be available. When we are streaming, this is not a problem ; however if the recording has been paused
no audio will be coming. So instead will fully drain the encoder. As a side effect, resuming the recording
may cause an audible silence for the duration of the codec delay (55ms with Opus content).
In practice, this may not be noticeable as pausing and resuming a recording would produce noticeable
artifacts with audio.

Another technical change with the previous webm recorder is that the video time is now set to use
the start time of the last appended audio frame rather than the end. This too is also a consequence of the audio encoder
having a set latency and allowed for some simplifications in the code.

The new MediaRecorder add supports for the codec to be set and the mimetype for the content generated will
be mostly accurate (we use static values with common profiles rather than attempting to determing accurately
things like the codec level and profile.

The RemoteMediaRecorder being no longer used is fully removed along the related files.
The class MediaRecorderProvider and its overrides are removed.

Tests amended and corrected to reflect the more accurate behaviors.

* LayoutTests/http/wpt/mediarecorder/MediaRecorder-first-frame-expected.txt:
* LayoutTests/http/wpt/mediarecorder/MediaRecorder-first-frame.html:
* LayoutTests/http/wpt/mediarecorder/MediaRecorder-requestData.html:
* LayoutTests/http/wpt/mediarecorder/mimeType.html:
* LayoutTests/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-mimetype-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-mimetype.html:
* LayoutTests/ipc/shared-video-frame-size-expected.txt: Removed.
* LayoutTests/ipc/shared-video-frame-size.html: Removed.
* LayoutTests/platform/ios/TestExpectations:
* LayoutTests/platform/mac/TestExpectations:
* LayoutTests/platform/mac-wk1/TestExpectations: Amend test, was failing before, just failed a different way. There's no MediaStream support on WK1
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp:
(WebCore::MediaRecorder::isTypeSupported):
(WebCore::MediaRecorder::createMediaRecorderPrivate):
(WebCore::MediaRecorder::startRecording):
* Source/WebCore/Modules/mediarecorder/MediaRecorder.h:
* Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.cpp: Removed.
* Source/WebCore/Modules/mediarecorder/MediaRecorderProvider.h: Removed.
* Source/WebCore/PlatformMac.cmake:
* Source/WebCore/SaferCPPExpectations/UncountedCallArgsCheckerExpectations:
* Source/WebCore/Sources.txt:
* Source/WebCore/SourcesCocoa.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/loader/EmptyClients.cpp:
(WebCore::pageConfigurationWithEmptyClients):
* Source/WebCore/page/Page.cpp:
(WebCore::Page::Page):
* Source/WebCore/page/Page.h:
(WebCore::Page::socketProvider):
(WebCore::Page::mediaRecorderProvider): Deleted.
* Source/WebCore/page/PageConfiguration.cpp:
(WebCore::PageConfiguration::PageConfiguration):
* Source/WebCore/page/PageConfiguration.h:
* Source/WebCore/platform/MediaSample.h:
(WebCore::MediaSample::presentationEndTime const):
* Source/WebCore/platform/graphics/cocoa/CMUtilities.mm:
(WebCore::createFormatDescriptionFromTrackInfo):
(WebCore::toCMSampleBuffer):
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
(WebCore::MediaRecorderPrivateAVFImpl::isTypeSupported):
(WebCore::MediaRecorderPrivateAVFImpl::create):
(WebCore::MediaRecorderPrivateAVFImpl::MediaRecorderPrivateAVFImpl):
(WebCore::MediaRecorderPrivateAVFImpl::~MediaRecorderPrivateAVFImpl):
(WebCore::MediaRecorderPrivateAVFImpl::startRecording):
(WebCore::MediaRecorderPrivateAVFImpl::videoFrameAvailable):
(WebCore::MediaRecorderPrivateAVFImpl::audioSamplesAvailable):
(WebCore::MediaRecorderPrivateAVFImpl::stopRecording):
(WebCore::MediaRecorderPrivateAVFImpl::pauseRecording):
(WebCore::MediaRecorderPrivateAVFImpl::resumeRecording):
(WebCore::MediaRecorderPrivateAVFImpl::fetchData):
(WebCore::MediaRecorderPrivateAVFImpl::mimeType const):
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateEncoder.cpp: Added.
(WebCore::MediaRecorderPrivateEncoder::Listener::create):
(WebCore::MediaRecorderPrivateEncoder::Listener::Listener):
(WebCore::MediaRecorderPrivateEncoder::create):
(WebCore::MediaRecorderPrivateEncoder::queueSingleton):
(WebCore::MediaRecorderPrivateEncoder::MediaRecorderPrivateEncoder):
(WebCore::codecStringForMediaVideoCodecId):
(WebCore::MediaRecorderPrivateEncoder::compressedAudioOutputBufferCallback):
(WebCore::MediaRecorderPrivateEncoder::initialize):
(WebCore::MediaRecorderPrivateEncoder::generateMIMEType):
(WebCore::MediaRecorderPrivateEncoder::pause):
(WebCore::MediaRecorderPrivateEncoder::resume):
(WebCore::MediaRecorderPrivateEncoder::close):
(WebCore::MediaRecorderPrivateEncoder::mimeType const):
(WebCore::MediaRecorderPrivateEncoder::audioBitRate const):
(WebCore::MediaRecorderPrivateEncoder::videoBitRate const):
(WebCore::MediaRecorderPrivateEncoder::appendAudioSampleBuffer):
(WebCore::MediaRecorderPrivateEncoder::appendVideoFrame):
(WebCore::MediaRecorderPrivateEncoder::nextVideoFrameTime):
(WebCore::MediaRecorderPrivateEncoder::appendData):
(WebCore::MediaRecorderPrivateEncoder::flushDataBuffer):
(WebCore::MediaRecorderPrivateEncoder::hasMuxedDataSinceEndSegment const):
(WebCore::MediaRecorderPrivateEncoder::takeData):
(WebCore::MediaRecorderPrivateEncoder::enqueueCompressedAudioSampleBuffers):
(WebCore::MediaRecorderPrivateEncoder::maybeStartWriter):
(WebCore::MediaRecorderPrivateEncoder::encodePendingVideoFrames):
(WebCore::MediaRecorderPrivateEncoder::processVideoEncoderActiveConfiguration):
(WebCore::MediaRecorderPrivateEncoder::enqueueCompressedVideoFrame):
(WebCore::MediaRecorderPrivateEncoder::partiallyFlushEncodedQueues):
(WebCore::MediaRecorderPrivateEncoder::waitForMatchingAudio):
(WebCore::MediaRecorderPrivateEncoder::flushToEndSegment):
(WebCore::MediaRecorderPrivateEncoder::flushAllEncodedQueues):
(WebCore::MediaRecorderPrivateEncoder::muxNextFrame):
(WebCore::MediaRecorderPrivateEncoder::stopRecording):
(WebCore::MediaRecorderPrivateEncoder::fetchData):
(WebCore::MediaRecorderPrivateEncoder::flushPendingData):
(WebCore::MediaRecorderPrivateEncoder::currentTime const):
(WebCore::MediaRecorderPrivateEncoder::currentEndTime const):
(WebCore::MediaRecorderPrivateEncoder::audioCompressor const):
(WebCore::MediaRecorderPrivateEncoder::listener):
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateEncoder.h: Added.
(WebCore::MediaRecorderPrivateEncoder::hasAudio const):
(WebCore::MediaRecorderPrivateEncoder::hasVideo const):
(WebCore::MediaRecorderPrivateEncoder::WTF_GUARDED_BY_CAPABILITY):
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateWriter.cpp:
(WebCore::MediaRecorderPrivateWriter::create):
* Source/WebCore/platform/mediarecorder/MediaRecorderPrivateWriter.h:
(): Deleted.
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterAVFObjC.h: Copied from Source/WebCore/platform/mediarecorder/MediaRecorderPrivateWriter.h.
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterAVFObjC.mm: Added.
(-[WebAVAssetWriterDelegate initWithWriter:]):
(-[WebAVAssetWriterDelegate assetWriter:didOutputSegmentData:segmentType:]):
(WebCore::MediaRecorderPrivateWriterAVFObjC::create):
(WebCore::MediaRecorderPrivateWriterAVFObjC::MediaRecorderPrivateWriterAVFObjC):
(WebCore::MediaRecorderPrivateWriterAVFObjC::addAudioTrack):
(WebCore::MediaRecorderPrivateWriterAVFObjC::addVideoTrack):
(WebCore::MediaRecorderPrivateWriterAVFObjC::allTracksAdded):
(WebCore::MediaRecorderPrivateWriterAVFObjC::muxFrame):
(WebCore::appendEndsPreviousSampleDurationMarker):
(WebCore::MediaRecorderPrivateWriterAVFObjC::forceNewSegment):
(WebCore::MediaRecorderPrivateWriterAVFObjC::close):
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h: Removed.
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm: Removed.
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.cpp: Added.
(WebCore::MediaRecorderPrivateWriterWebMDelegate::MediaRecorderPrivateWriterWebMDelegate):
(WebCore::MediaRecorderPrivateWriterWebMDelegate::addAudioTrack):
(WebCore::MediaRecorderPrivateWriterWebMDelegate::addVideoTrack):
(WebCore::MediaRecorderPrivateWriterWebMDelegate::addFrame):
(WebCore::MediaRecorderPrivateWriterWebMDelegate::forceNewClusterOnNextFrame):
(WebCore::MediaRecorderPrivateWriterWebMDelegate::finalize):
(WebCore::MediaRecorderPrivateWriterWebM::create):
(WebCore::mkvCodeIcForMediaVideoCodecId):
(WebCore::MediaRecorderPrivateWriterWebM::MediaRecorderPrivateWriterWebM):
(WebCore::MediaRecorderPrivateWriterWebM::addAudioTrack):
(WebCore::MediaRecorderPrivateWriterWebM::addVideoTrack):
(WebCore::MediaRecorderPrivateWriterWebM::muxFrame):
(WebCore::MediaRecorderPrivateWriterWebM::forceNewSegment):
(WebCore::MediaRecorderPrivateWriterWebM::close):
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.h:
* Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterWebM.mm: Removed.
* Source/WebCore/testing/Internals.cpp:
* Source/WebKit/CMakeLists.txt:
* Source/WebKit/DerivedSources-input.xcfilelist:
* Source/WebKit/DerivedSources-output.xcfilelist:
* Source/WebKit/DerivedSources.make:
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::allowsExitUnderMemoryPressure const):
(WebKit::GPUConnectionToWebProcess::dispatchMessage):
(WebKit::GPUConnectionToWebProcess::mediaRecorderManager): Deleted.
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h:
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.cpp: Removed.
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.h: Removed.
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorder.messages.in: Removed.
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.cpp: Removed.
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.h: Removed.
* Source/WebKit/GPUProcess/webrtc/RemoteMediaRecorderManager.messages.in: Removed.
* Source/WebKit/SaferCPPExpectations/UncountedCallArgsCheckerExpectations:
* Source/WebKit/Scripts/webkit/messages.py:
(serialized_identifiers):
* Source/WebKit/Scripts/webkit/tests/MessageArgumentDescriptions.cpp:
(IPC::serializedIdentifiers):
* Source/WebKit/Shared/Cocoa/WebCoreArgumentCodersCocoa.serialization.in:
* Source/WebKit/Shared/WTFArgumentCoders.serialization.in:
* Source/WebKit/Sources.txt:
* Source/WebKit/SourcesCocoa.txt:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderIdentifier.h: Removed.
* Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp: Removed.
* Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderPrivate.h: Removed.
* Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.cpp: Removed.
* Source/WebKit/WebProcess/GPU/webrtc/MediaRecorderProvider.h: Removed.
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::m_textAnimationController):
* Source/WebKitLegacy/mac/WebView/WebView.mm:
(-[WebView _commonInitializationWithFrameName:groupName:]):
(-[WebView initSimpleHTMLDocumentWithStyle:frame:preferences:groupName:]):

Canonical link: https://commits.webkit.org/285726@main

To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications