[webkit-changes] [WebKit/WebKit] 90086f: [MSE] Run MediaSourcePrivateRemote and SourceBuffe...

Fri Feb 9 04:01:02 PST 2024

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 90086fffd11ed100b343d615c03cf446a59734e0
      https://github.com/WebKit/WebKit/commit/90086fffd11ed100b343d615c03cf446a59734e0
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-02-09 (Fri, 09 Feb 2024)

  Changed paths:
    M Source/WTF/wtf/Forward.h
    M Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp
    M Source/WebCore/Modules/mediasource/ManagedMediaSource.h
    M Source/WebCore/Modules/mediasource/MediaSource.cpp
    M Source/WebCore/Modules/mediasource/MediaSource.h
    M Source/WebCore/Modules/mediasource/SourceBuffer.cpp
    M Source/WebCore/Modules/mediasource/SourceBuffer.h
    M Source/WebCore/platform/graphics/MediaSourcePrivate.cpp
    M Source/WebCore/platform/graphics/MediaSourcePrivate.h
    M Source/WebCore/platform/graphics/MediaSourcePrivateClient.h
    M Source/WebCore/platform/graphics/SourceBufferPrivate.cpp
    M Source/WebCore/platform/graphics/SourceBufferPrivate.h
    M Source/WebCore/platform/graphics/SourceBufferPrivateClient.h
    M Source/WebKit/CMakeLists.txt
    M Source/WebKit/DerivedSources-input.xcfilelist
    M Source/WebKit/DerivedSources-output.xcfilelist
    M Source/WebKit/DerivedSources.make
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h
    M Source/WebKit/GPUProcess/media/RemoteMediaSourceProxy.cpp
    M Source/WebKit/GPUProcess/media/RemoteSourceBufferProxy.cpp
    M Source/WebKit/WebKit.xcodeproj/project.pbxproj
    M Source/WebKit/WebProcess/GPU/GPUProcessConnection.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.messages.in
    M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h
    R Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.messages.in
    A Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemoteMessageReceiver.h
    A Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemoteMessageReceiver.messages.in
    M Source/WebKit/WebProcess/GPU/media/RemoteMediaPlayerState.h
    M Source/WebKit/WebProcess/GPU/media/RemoteMediaPlayerState.serialization.in
    M Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.h
    R Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.messages.in
    A Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemoteMessageReceiver.h
    A Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemoteMessageReceiver.messages.in

  Log Message:
  -----------
  [MSE] Run MediaSourcePrivateRemote and SourceBufferPrivateRemote in a dedicated work queue
https://bugs.webkit.org/show_bug.cgi?id=265982
rdar://119296319

Reviewed by Jer Noble.

Make the MediaSourcePrivateRemote and SourceBufferPrivateRemote use a WorkQueueMessageReceiver
with a dedicated WorkQueue.
The threading model in the GPU process remains unchanged as the current implementations
and relationship between SourceBufferPrivate and MediaPlayerPrivate rely
heavily on the objects being main-thread only (in particular the extensive use of
AVSampleBufferDisplayerLayer).
>From an observer point of view in the content process, the various MSE objects operations
are now fully parallelised.

The threading flow as such is now:

Content Process:
MediaSource (main thread) -> MediaSourcePrivateRemote (workqueue)
SourceBuffer (main thread) -> SourceBufferPrivateRemote (workqueue)

GPU Process:
RemoteMediaSourceProxy (main thread) -> MediaSourcePrivate (main thread)
RemoteSourceBufferProxy (main thread) -> SourceBufferPrivate (main thread)

The MediaPlayerPrivate/MediaPlayerPrivateRemote runs on the main thread.

Some MediaSourcePrivate methods had to be made thread-safe to more easily synchronise member attributes.

Covered by existing tests, no change in observable behaviour.

* Source/WTF/wtf/Forward.h:
Add SerialFunctionDispatcher and RefCountedSerialFunctionDispatcher forward declaration.
* Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp:
(WebCore::ManagedMediaSource::elementDetached):
(WebCore::ManagedMediaSource::setStreaming):
(WebCore::ManagedMediaSource::monitorSourceBuffers):
* Source/WebCore/Modules/mediasource/ManagedMediaSource.h:
* Source/WebCore/Modules/mediasource/MediaSource.cpp:
(WebCore::MediaSource::MediaSource):
(WebCore::MediaSource::waitForTarget):
(WebCore::MediaSource::completeSeek):
(WebCore::MediaSource::seekToTime):
(WebCore::MediaSource::seekable):
(WebCore::MediaSource::setLiveSeekableRange):
(WebCore::MediaSource::clearLiveSeekableRange):
(WebCore::MediaSource::hasBufferedTime):
(WebCore::MediaSource::hasFutureTime):
(WebCore::MediaSource::monitorSourceBuffers):
(WebCore::MediaSource::setDuration):
(WebCore::MediaSource::setDurationInternal):
(WebCore::MediaSource::setReadyState):
(WebCore::MediaSource::endOfStream):
(WebCore::MediaSource::streamEndedWithError):
(WebCore::MediaSource::addSourceBuffer):
(WebCore::MediaSource::detachFromElement):
(WebCore::MediaSource::onReadyStateChange):
(WebCore::MediaSource::activeRanges const):
(WebCore::MediaSource::regenerateActiveSourceBuffers):
(WebCore::MediaSource::notifyElementUpdateMediaState const):
(WebCore::MediaSource::sourceBufferBufferedChanged):
(WebCore::MediaSource::updateBufferedIfNeeded):
(WebCore::MediaSource::failedToCreateRenderer):
(WebCore::MediaSource::sourceBufferReceivedFirstInitializationSegmentChanged):
(WebCore::MediaSource::sourceBufferActiveTrackFlagChanged):
(WebCore::MediaSource::setMediaPlayerReadyState):
(WebCore::MediaSource::memoryPressure):
(WebCore::MediaSource::ensureWeakOnDispatcher const):
* Source/WebCore/Modules/mediasource/MediaSource.h:
(WebCore::MediaSource::elementDetached):
(WebCore::MediaSource::promisedWeakOnDispatcher const):
* Source/WebCore/Modules/mediasource/SourceBuffer.cpp:
(WebCore::m_logIdentifier):
(WebCore::SourceBuffer::~SourceBuffer):
(WebCore::SourceBuffer::rangeRemoval):
(WebCore::SourceBuffer::removedFromMediaSource):
(WebCore::SourceBuffer::seekToTime):
(WebCore::SourceBuffer::appendBufferInternal):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveRenderingError):
(WebCore::SourceBuffer::maximumBufferSize const):
(WebCore::SourceBuffer::setActive):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveInitializationSegment):
(WebCore::SourceBuffer::sourceBufferPrivateDurationChanged):
(WebCore::SourceBuffer::sourceBufferPrivateHighestPresentationTimestampChanged):
(WebCore::SourceBuffer::sourceBufferPrivateDidDropSample):
(WebCore::SourceBuffer::sourceBufferPrivateBufferedChanged):
(WebCore::SourceBuffer::ensureWeakOnDispatcher const):
(WebCore::SourceBuffer::promisedWeakOnDispatcher const):
* Source/WebCore/Modules/mediasource/SourceBuffer.h:
* Source/WebCore/platform/graphics/MediaSourcePrivate.cpp:
(WebCore::MediaSourcePrivate::MediaSourcePrivate):
(WebCore::MediaSourcePrivate::duration const):
(WebCore::MediaSourcePrivate::removeSourceBuffer):
(WebCore::MediaSourcePrivate::sourceBufferPrivateDidChangeActiveState):
(WebCore::MediaSourcePrivate::hasAudio const):
(WebCore::MediaSourcePrivate::hasVideo const):
(WebCore::MediaSourcePrivate::durationChanged):
(WebCore::MediaSourcePrivate::bufferedChanged):
(WebCore::MediaSourcePrivate::buffered const):
(WebCore::MediaSourcePrivate::setCDMSession):
(WebCore::MediaSourcePrivate::ensureOnDispatcher const):
* Source/WebCore/platform/graphics/MediaSourcePrivate.h:
* Source/WebCore/platform/graphics/MediaSourcePrivateClient.h:
* Source/WebCore/platform/graphics/SourceBufferPrivate.cpp:
(WebCore::SourceBufferPrivate::SourceBufferPrivate):
(WebCore::SourceBufferPrivate::removedFromMediaSource):
(WebCore::SourceBufferPrivate::setClient):
(WebCore::SourceBufferPrivate::updateBuffered):
(WebCore::SourceBufferPrivate::seekToTime):
(WebCore::SourceBufferPrivate::removeCodedFrames):
(WebCore::SourceBufferPrivate::updateTrackIds):
(WebCore::SourceBufferPrivate::setAllTrackBuffersNeedRandomAccess):
(WebCore::SourceBufferPrivate::didReceiveInitializationSegment):
(WebCore::SourceBufferPrivate::didUpdateFormatDescriptionForTrackId):
(WebCore::SourceBufferPrivate::append):
(WebCore::SourceBufferPrivate::resetParserState):
(WebCore::SourceBufferPrivate::memoryPressure):
(WebCore::SourceBufferPrivate::ensureOnDispatcher const):
(WebCore::SourceBufferPrivate::detach): Deleted.
* Source/WebCore/platform/graphics/SourceBufferPrivate.h:
* Source/WebCore/platform/graphics/SourceBufferPrivateClient.h:
* Source/WebKit/CMakeLists.txt:
* Source/WebKit/DerivedSources-input.xcfilelist:
* Source/WebKit/DerivedSources-output.xcfilelist:
* Source/WebKit/DerivedSources.make:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::commitAllTransactions):
(WebKit::RemoteMediaPlayerProxy::mediaPlayerNetworkStateChanged):
(WebKit::RemoteMediaPlayerProxy::mediaPlayerReadyStateChanged):
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h:
* Source/WebKit/GPUProcess/media/RemoteMediaSourceProxy.cpp:
(WebKit::RemoteMediaSourceProxy::waitForTarget):
(WebKit::RemoteMediaSourceProxy::seekToTime):
(WebKit::RemoteMediaSourceProxy::shutdown):
* Source/WebKit/GPUProcess/media/RemoteSourceBufferProxy.cpp:
(WebKit::RemoteSourceBufferProxy::~RemoteSourceBufferProxy):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateDidReceiveInitializationSegment):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateHighestPresentationTimestampChanged):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateDurationChanged):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateBufferedChanged):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateDidDropSample):
(WebKit::RemoteSourceBufferProxy::sourceBufferPrivateDidReceiveRenderingError):
(WebKit::RemoteSourceBufferProxy::append):
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/GPU/GPUProcessConnection.cpp:
(WebKit::GPUProcessConnection::dispatchMessage):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
(WebKit::MediaPlayerPrivateRemote::setReadyState):
(WebKit::MediaPlayerPrivateRemote::readyStateChanged):
(WebKit::MediaPlayerPrivateRemote::updateCachedState):
(WebKit::MediaPlayerPrivateRemote::addRemoteAudioTrack):
(WebKit::MediaPlayerPrivateRemote::removeRemoteAudioTrack):
(WebKit::MediaPlayerPrivateRemote::remoteAudioTrackConfigurationChanged):
(WebKit::MediaPlayerPrivateRemote::addRemoteTextTrack):
(WebKit::MediaPlayerPrivateRemote::removeRemoteTextTrack):
(WebKit::MediaPlayerPrivateRemote::remoteTextTrackConfigurationChanged):
(WebKit::MediaPlayerPrivateRemote::parseWebVTTFileHeader):
(WebKit::MediaPlayerPrivateRemote::parseWebVTTCueData):
(WebKit::MediaPlayerPrivateRemote::parseWebVTTCueDataStruct):
(WebKit::MediaPlayerPrivateRemote::addDataCue):
(WebKit::MediaPlayerPrivateRemote::addDataCueWithType):
(WebKit::MediaPlayerPrivateRemote::updateDataCue):
(WebKit::MediaPlayerPrivateRemote::removeDataCue):
(WebKit::MediaPlayerPrivateRemote::addGenericCue):
(WebKit::MediaPlayerPrivateRemote::updateGenericCue):
(WebKit::MediaPlayerPrivateRemote::removeGenericCue):
(WebKit::MediaPlayerPrivateRemote::addRemoteVideoTrack):
(WebKit::MediaPlayerPrivateRemote::removeRemoteVideoTrack):
(WebKit::MediaPlayerPrivateRemote::remoteVideoTrackConfigurationChanged):
(WebKit::MediaPlayerPrivateRemote::requestResource):
(WebKit::MediaPlayerPrivateRemote::removeResource):
(WebKit::MediaPlayerPrivateRemote::audioTrackPrivateRemote const):
(WebKit::MediaPlayerPrivateRemote::videoTrackPrivateRemote const):
(WebKit::MediaPlayerPrivateRemote::textTrackPrivateRemote const):
(WebKit::MediaPlayerPrivateRemote::commitAllTransactions):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.messages.in:
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp:
(WebKit::MediaSourcePrivateRemote::queue):
(WebKit::MediaSourcePrivateRemote::ensureOnDispatcherSync const):
(WebKit::MediaSourcePrivateRemote::MediaSourcePrivateRemote):
(WebKit::MediaSourcePrivateRemote::~MediaSourcePrivateRemote):
(WebKit::MediaSourcePrivateRemote::addSourceBuffer):
(WebKit::MediaSourcePrivateRemote::durationChanged):
Directly notify the MediaPlayerPrivateRemote that the duration as changed
rather than go through two round-trips to the GPU process.
(WebKit::MediaSourcePrivateRemote::bufferedChanged):
(WebKit::MediaSourcePrivateRemote::markEndOfStream):
(WebKit::MediaSourcePrivateRemote::unmarkEndOfStream):
(WebKit::MediaSourcePrivateRemote::setMediaPlayerReadyState):
Directly notify the MediaPlayerPrivateRemote that the readyState as changed
rather than go through two round-trips to the GPU process.
(WebKit::MediaSourcePrivateRemote::setTimeFudgeFactor):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::client const):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::proxyWaitForTarget):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::proxySeekToTime):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::mediaSourcePrivateShuttingDown):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::MessageReceiver):
(WebKit::MediaSourcePrivateRemote::proxyWaitForTarget): Deleted.
(WebKit::MediaSourcePrivateRemote::proxySeekToTime): Deleted.
(WebKit::MediaSourcePrivateRemote::mediaSourcePrivateShuttingDown): Deleted.
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h:
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemoteMessageReceiver.h: Copied from Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.messages.in.
The current infrastructure doesn't allow to define the IPC protocol in a private sub-class, so add an alias.
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemoteMessageReceiver.messages.in: Copied from Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.messages.in.
* Source/WebKit/WebProcess/GPU/media/RemoteMediaPlayerState.h:
Remove readyState from being state caching structure. As the readyState was updated alongside other attributes, it was possible
for the readyState to be upgraded with a GPU's stale value before updated to the new value from the content process. Leading to intermittent
failures, that those changes made more likely.
* Source/WebKit/WebProcess/GPU/media/RemoteMediaPlayerState.serialization.in:
* Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.cpp:
(WebKit::SourceBufferPrivateRemote::append):
(WebKit::SourceBufferPrivateRemote::isActive const):
(WebKit::SourceBufferPrivateRemote::setActive):
(WebKit::SourceBufferPrivateRemote::setMediaSourceEnded):
(WebKit::SourceBufferPrivateRemote::queue):
(WebKit::SourceBufferPrivateRemote::ensureOnDispatcherSync):
(WebKit::SourceBufferPrivateRemote::SourceBufferPrivateRemote):
(WebKit::SourceBufferPrivateRemote::~SourceBufferPrivateRemote):
(WebKit::SourceBufferPrivateRemote::append):
(WebKit::SourceBufferPrivateRemote::abort):
(WebKit::SourceBufferPrivateRemote::resetParserState):
(WebKit::SourceBufferPrivateRemote::removedFromMediaSource):
(WebKit::SourceBufferPrivateRemote::canSwitchToType):
(WebKit::SourceBufferPrivateRemote::setMediaSourceEnded):
(WebKit::SourceBufferPrivateRemote::setMode):
(WebKit::SourceBufferPrivateRemote::removeCodedFrames):
(WebKit::SourceBufferPrivateRemote::evictCodedFrames):
(WebKit::SourceBufferPrivateRemote::addTrackBuffer):
(WebKit::SourceBufferPrivateRemote::resetTrackBuffers):
(WebKit::SourceBufferPrivateRemote::clearTrackBuffers):
(WebKit::SourceBufferPrivateRemote::setAllTrackBuffersNeedRandomAccess):
(WebKit::SourceBufferPrivateRemote::setGroupStartTimestamp):
(WebKit::SourceBufferPrivateRemote::setGroupStartTimestampToEndTimestamp):
(WebKit::SourceBufferPrivateRemote::setShouldGenerateTimestamps):
(WebKit::SourceBufferPrivateRemote::reenqueueMediaIfNeeded):
(WebKit::SourceBufferPrivateRemote::resetTimestampOffsetInTrackBuffers):
(WebKit::SourceBufferPrivateRemote::startChangingType):
(WebKit::SourceBufferPrivateRemote::timestampOffset const):
(WebKit::SourceBufferPrivateRemote::setTimestampOffset):
(WebKit::SourceBufferPrivateRemote::setAppendWindowStart):
(WebKit::SourceBufferPrivateRemote::setAppendWindowEnd):
(WebKit::SourceBufferPrivateRemote::computeSeekTime):
(WebKit::SourceBufferPrivateRemote::seekToTime):
(WebKit::SourceBufferPrivateRemote::updateTrackIds):
(WebKit::SourceBufferPrivateRemote::bufferedSamplesForTrackId):
(WebKit::SourceBufferPrivateRemote::enqueuedSamplesForTrackID):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::client const):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::takeOwnershipOfMemory):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateDidReceiveInitializationSegment):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateHighestPresentationTimestampChanged):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateDurationChanged):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateBufferedChanged):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateDidDropSample):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::sourceBufferPrivateDidReceiveRenderingError):
(WebKit::SourceBufferPrivateRemote::memoryPressure):
(WebKit::SourceBufferPrivateRemote::minimumUpcomingPresentationTimeForTrackID):
(WebKit::SourceBufferPrivateRemote::setMaximumQueueDepthForTrackID):
(WebKit::SourceBufferPrivateRemote::MessageReceiver::MessageReceiver):
(WebKit::SourceBufferPrivateRemote::takeOwnershipOfMemory): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateDidReceiveInitializationSegment): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateHighestPresentationTimestampChanged): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateDurationChanged): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateBufferedChanged): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateDidDropSample): Deleted.
(WebKit::SourceBufferPrivateRemote::sourceBufferPrivateDidReceiveRenderingError): Deleted.
* Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.h:
* Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemoteMessageReceiver.h: Renamed from Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.messages.in.
The current infrastructure doesn't allow to define the IPC protocol in a private sub-class, so add an alias.
* Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemoteMessageReceiver.messages.in: Renamed from Source/WebKit/WebProcess/GPU/media/SourceBufferPrivateRemote.messages.in.

Canonical link: https://commits.webkit.org/274356@main