[webkit-changes] [WebKit/WebKit] 7db823: [MSE] Add MediaSourceHandle object and update to M...

Jean-Yves Avenard noreply at github.com
Fri Feb 16 10:27:02 PST 2024 

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 7db823fea8a1af033195fe8ddcaad6d7fae675b5
      https://github.com/WebKit/WebKit/commit/7db823fea8a1af033195fe8ddcaad6d7fae675b5
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-02-16 (Fri, 16 Feb 2024)

  Changed paths:
    M Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml
    M Source/WTF/wtf/PlatformEnable.h
    M Source/WTF/wtf/PlatformEnableCocoa.h
    M Source/WebCore/CMakeLists.txt
    M Source/WebCore/DerivedSources-input.xcfilelist
    M Source/WebCore/DerivedSources-output.xcfilelist
    M Source/WebCore/DerivedSources.make
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Modules/mediasource/DOMURLMediaSource.cpp
    M Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp
    M Source/WebCore/Modules/mediasource/ManagedMediaSource.idl
    M Source/WebCore/Modules/mediasource/MediaSource.cpp
    M Source/WebCore/Modules/mediasource/MediaSource.h
    M Source/WebCore/Modules/mediasource/MediaSource.idl
    A Source/WebCore/Modules/mediasource/MediaSourceHandle.cpp
    A Source/WebCore/Modules/mediasource/MediaSourceHandle.h
    A Source/WebCore/Modules/mediasource/MediaSourceHandle.idl
    A Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.cpp
    A Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.h
    A Source/WebCore/Modules/mediasource/MediaSourceInterfaceProxy.h
    A Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.cpp
    A Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.h
    M Source/WebCore/Modules/mediasource/SourceBuffer.cpp
    M Source/WebCore/Modules/mediasource/SourceBuffer.h
    M Source/WebCore/Modules/mediasource/SourceBuffer.idl
    M Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp
    M Source/WebCore/Sources.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/bindings/js/SerializedScriptValue.cpp
    M Source/WebCore/bindings/js/SerializedScriptValue.h
    M Source/WebCore/bindings/js/WebCoreBuiltinNames.h
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/html/HTMLMediaElement.h
    M Source/WebCore/html/HTMLMediaElement.idl
    M Source/WebCore/html/track/AudioTrack.cpp
    M Source/WebCore/html/track/TrackListBase.cpp
    M Source/WebCore/html/track/TrackListBase.h
    M Source/WebCore/html/track/VideoTrack.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.cpp
    M Source/WebCore/platform/graphics/MediaSourcePrivate.cpp
    M Source/WebCore/platform/graphics/MediaSourcePrivate.h
    M Source/WebCore/platform/graphics/MediaSourcePrivateClient.h
    M Source/WebCore/platform/graphics/SourceBufferPrivate.cpp
    M Source/WebCore/platform/graphics/SourceBufferPrivateClient.h
    M Source/WebCore/platform/graphics/TrackPrivateBase.h
    M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
    M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h

  Log Message:
  -----------
  [MSE] Add MediaSourceHandle object and update to MSE's latest spec (including MSE in worker)
https://bugs.webkit.org/show_bug.cgi?id=269461
rdar://problem/123006531

Reviewed by Youenn Fablet.

The MSE spec now defines that a MediaSource can be constructed in a DedicatedWorker
and attached to a HTMLMediaElement running in the main thread through a MediaSourceHandle
object.
To determine if a MediaSource can be constructed in a DedicatedWorker a new attribute
has been added.
This commit adds support for all those latest functionalities and is placed behind a MediaSourceInWorker
pref currently off by default.

Some architectural explanations:

The MediaSource and SourceBuffer were ThreadSafeRefCounted objects which was
problematic as we couldn't guarantee which thread those objets would be destructed on.
To solve this, we make MediaSource and SourceBuffer be RefCounted instead and
use a ThreadSafeRefCounted {MediaSource|SourceBuffer}PrivateClient each providing
a function dispatcher used for the {MediaSource|SourceBuffer}Private to communicate
with their client.
The function dispatcher guarantees that:
1- The function will always be run on the MediaSource's thread.
2- That the MediaSource hasn't been destructed in-between calls
3- That the MediaSource's readyState is not "Closed".

As such, {MediaSource|SourceBuffer} methods are only ever called on the correct thread,
it is no longer the responsibility of either the MediaSource or the SourceBuffer to
run on the correct thread, so we can remove the now redundant `assertIsCurrent(m_dispatcher)`
and the m_dispatcher member.
Guarantees 2 and 3 allows to remove the checks made in each method, allowing to simplify the code.

One of the existing complexity, was that the media element directly interacted with the MediaSource
through both sync and asynchronous methods which is no longer a possibility if the MediaSource
now exists in the DedicatedWorker.
We introduce the MediaSourceInterfaceProxy virtual interface so that the media element
can continue to interact with the MediaSource with the minimum amount of changes.
Two implementation are provided.
MediaSourceInterfaceMainThread, which performs a simple, passthrough connection
and the MediaSourceInterfaceWorker which is constructed from the MediaSourceHandle.

The MediaSourceHandle implementation contains the minimum information so that the
MediaSourceInterfaceProxy can be established.

While asynchronous calls can be easily achieved with the MediaSourceHandle's
function dispatcher, synchronous calls can't (you can't dispatch synchronously
to a worker)
As such, all information needed to be synchronously retrieved
by the HTMLMediaElement from the MediaSource are now moved to the MediaSourcePrivate
instead, this includes:
- readyState
- duration
- buffered range
- seekable range
- live seekable range.

In order to avoid duplicating such handling with all MediaSourcePrivate implementations:
(AVF, Mock, GStreamer, GPU) we place it in the MediaSourcePrivate base class.

Various small changes had to be performed for things to operate when a worker is in use.
- Tracks validity where being checked using main thread's only AtomString, we change the
code use litteral strings for now.
- Some calls from the MediaSource (worker) are made directly to the MediaPlayer (which
lives in the main thread), rather than making the code thread-safe, we instead
dispatch synchronously to the main thread.

Present issues:
- The handling of Tracks and TrackList in the worker is currently broken as the
MSE spec do not cater yet for necessary changes (such as {Audio|Video|Text}Track
objects are currently only exposed to the `Document` (see https://github.com/w3c/media-source/issues/280)
- Media Logging in the Worker is currently disabled.
- There appear to still be some unnecessary hops via the main thread while a MediaSource is in
operation in the work, which results in some janks still occurring when the main thread is
under heavy load.
- When in the main thread, it is possible via dynamic preferences to limit the codec's supported
configuration. As Settings aren't available in the Worker, this functionality is disabled.

MediaSource in a Worker functionality is currently turn off by default.
Tests will be added at a later stage.

No changes in observable behaviour, currently covered by extensive existing tests.

* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
* Source/WTF/wtf/PlatformEnable.h:
* Source/WTF/wtf/PlatformEnableCocoa.h:
* Source/WebCore/CMakeLists.txt:
* Source/WebCore/DerivedSources-input.xcfilelist:
* Source/WebCore/DerivedSources-output.xcfilelist:
* Source/WebCore/DerivedSources.make:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/mediasource/DOMURLMediaSource.cpp:
(WebCore::DOMURLMediaSource::createObjectURL):
* Source/WebCore/Modules/mediasource/ManagedMediaSource.cpp:
(WebCore::ManagedMediaSource::setStreaming):
(WebCore::ManagedMediaSource::ensurePrefsRead):
(WebCore::ManagedMediaSource::monitorSourceBuffers):
(WebCore::ManagedMediaSource::streamingTimerFired):
* Source/WebCore/Modules/mediasource/ManagedMediaSource.idl:
* Source/WebCore/Modules/mediasource/MediaSource.cpp:
(WebCore::MediaSource::setRegistry):
(WebCore::MediaSource::MediaSource):
(WebCore::MediaSource::logger):
(WebCore::MediaSource::didLogMessage):
(WebCore::MediaSource::setPrivateAndOpen):
(WebCore::MediaSource::currentTime const):
(WebCore::MediaSource::buffered const):
(WebCore::MediaSource::waitForTarget):
(WebCore::MediaSource::completeSeek):
(WebCore::MediaSource::seekToTime):
(WebCore::MediaSource::seekable):
(WebCore::MediaSource::setLiveSeekableRange):
(WebCore::MediaSource::clearLiveSeekableRange):
(WebCore::MediaSource::hasBufferedTime):
(WebCore::MediaSource::hasFutureTime):
(WebCore::MediaSource::monitorSourceBuffers):
(WebCore::MediaSource::setDuration):
(WebCore::MediaSource::setDurationInternal):
(WebCore::MediaSource::setReadyState):
(WebCore::MediaSource::endOfStream):
(WebCore::MediaSource::streamEndedWithError):
(WebCore::MediaSource::addSourceBuffer):
(WebCore::MediaSource::removeSourceBuffer):
(WebCore::MediaSource::isTypeSupported):
(WebCore::MediaSource::isOpen const):
(WebCore::MediaSource::isClosed const):
(WebCore::MediaSource::isEnded const):
(WebCore::MediaSource::detachFromElement):
(WebCore::MediaSource::attachToElement):
(WebCore::MediaSource::openIfInEndedState):
(WebCore::MediaSource::openIfDeferredOpen):
(WebCore::MediaSource::stop):
(WebCore::MediaSource::readyState const):
(WebCore::MediaSource::onReadyStateChange):
(WebCore::MediaSource::activeRanges const):
(WebCore::MediaSource::regenerateActiveSourceBuffers):
(WebCore::MediaSource::notifyElementUpdateMediaState const):
(WebCore::MediaSource::ensureWeakOnHTMLMediaElementContext const):
(WebCore::MediaSource::sourceBufferBufferedChanged):
(WebCore::MediaSource::updateBufferedIfNeeded):
(WebCore::MediaSource::failedToCreateRenderer):
(WebCore::MediaSource::sourceBufferReceivedFirstInitializationSegmentChanged):
(WebCore::MediaSource::sourceBufferActiveTrackFlagChanged):
(WebCore::MediaSource::setMediaPlayerReadyState):
(WebCore::MediaSource::incrementDroppedFrameCount):
(WebCore::MediaSource::addAudioTrackToElement):
(WebCore::MediaSource::addTextTrackToElement):
(WebCore::MediaSource::addVideoTrackToElement):
(WebCore::MediaSource::addAudioTrackMirrorToElement):
(WebCore::MediaSource::addTextTrackMirrorToElement):
(WebCore::MediaSource::addVideoTrackMirrorToElement):
(WebCore::MediaSource::memoryPressure):
(WebCore::MediaSource::client const):
(WebCore::MediaSource::enabledForContext):
(WebCore::MediaSource::handle):
(WebCore::MediaSource::canConstructInDedicatedWorker):
(WebCore::MediaSource::settings const): Deleted.
(WebCore::MediaSource::ensureWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/MediaSource.h:
(WebCore::MediaSource::promisedWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/MediaSource.idl:
* Source/WebCore/Modules/mediasource/MediaSourceHandle.cpp: Added.
(WebCore::MediaSourceHandle::create):
(WebCore::MediaSourceHandle::MediaSourceHandle):
(WebCore::MediaSourceHandle::canDetach const):
(WebCore::MediaSourceHandle::setHasEverBeenAssignedAsSrcObject):
(WebCore::MediaSourceHandle::hasEverBeenAssignedAsSrcObject const):
(WebCore::MediaSourceHandle::isManaged const):
(WebCore::MediaSourceHandle::ensureOnDispatcher const):
(WebCore::MediaSourceHandle::detach):
(WebCore::MediaSourceHandle::mediaSourcePrivateClient const):
(WebCore::MediaSourceHandle::mediaSourceDidOpen):
(WebCore::MediaSourceHandle::mediaSourcePrivate const):
* Source/WebCore/Modules/mediasource/MediaSourceHandle.h: Added.
(WebCore::MediaSourceHandle::isDetached const):
(WebCore::MediaSourceHandle::setDetached):
* Source/WebCore/Modules/mediasource/MediaSourceHandle.idl: Copied from Source/WebCore/Modules/mediasource/ManagedMediaSource.idl.
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.cpp: Added.
(WebCore::MediaSourceInterfaceMainThread::MediaSourceInterfaceMainThread):
(WebCore::MediaSourceInterfaceMainThread::client const):
(WebCore::MediaSourceInterfaceMainThread::monitorSourceBuffers):
(WebCore::MediaSourceInterfaceMainThread::isClosed const):
(WebCore::MediaSourceInterfaceMainThread::duration const):
(WebCore::MediaSourceInterfaceMainThread::buffered const):
(WebCore::MediaSourceInterfaceMainThread::seekable const):
(WebCore::MediaSourceInterfaceMainThread::isStreamingContent const):
(WebCore::MediaSourceInterfaceMainThread::attachToElement):
(WebCore::MediaSourceInterfaceMainThread::detachFromElement):
(WebCore::MediaSourceInterfaceMainThread::openIfDeferredOpen):
(WebCore::MediaSourceInterfaceMainThread::isManaged const):
(WebCore::MediaSourceInterfaceMainThread::setAsSrcObject):
(WebCore::MediaSourceInterfaceMainThread::memoryPressure):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceMainThread.h: Copied from Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
(WebCore::MediaSourceInterfaceMainThread::create):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceProxy.h: Copied from Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.cpp: Added.
(WebCore::MediaSourceInterfaceWorker::MediaSourceInterfaceWorker):
(WebCore::MediaSourceInterfaceWorker::client const):
(WebCore::MediaSourceInterfaceWorker::monitorSourceBuffers):
(WebCore::MediaSourceInterfaceWorker::isClosed const):
(WebCore::MediaSourceInterfaceWorker::duration const):
(WebCore::MediaSourceInterfaceWorker::buffered const):
(WebCore::MediaSourceInterfaceWorker::seekable const):
(WebCore::MediaSourceInterfaceWorker::isStreamingContent const):
(WebCore::MediaSourceInterfaceWorker::attachToElement):
(WebCore::MediaSourceInterfaceWorker::detachFromElement):
(WebCore::MediaSourceInterfaceWorker::openIfDeferredOpen):
(WebCore::MediaSourceInterfaceWorker::isManaged const):
(WebCore::MediaSourceInterfaceWorker::setAsSrcObject):
(WebCore::MediaSourceInterfaceWorker::memoryPressure):
* Source/WebCore/Modules/mediasource/MediaSourceInterfaceWorker.h: Copied from Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp.
(WebCore::MediaSourceInterfaceWorker::create):
* Source/WebCore/Modules/mediasource/SourceBuffer.cpp:
(WebCore::SourceBuffer::SourceBuffer):
(WebCore::m_logIdentifier):
(WebCore::SourceBuffer::~SourceBuffer):
(WebCore::SourceBuffer::abort):
(WebCore::SourceBuffer::rangeRemoval):
(WebCore::SourceBuffer::changeType):
(WebCore::SourceBuffer::abortIfUpdating):
(WebCore::SourceBuffer::removedFromMediaSource):
(WebCore::SourceBuffer::seekToTime):
(WebCore::SourceBuffer::appendBufferInternal):
(WebCore::SourceBuffer::sourceBufferPrivateAppendComplete):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveRenderingError):
(WebCore::SourceBuffer::maximumBufferSize const):
(WebCore::SourceBuffer::setActive):
(WebCore::SourceBuffer::sourceBufferPrivateDidReceiveInitializationSegment):
(WebCore::SourceBuffer::validateInitializationSegment):
(WebCore::SourceBuffer::sourceBufferPrivateDurationChanged):
(WebCore::SourceBuffer::sourceBufferPrivateHighestPresentationTimestampChanged):
(WebCore::SourceBuffer::sourceBufferPrivateDidDropSample):
(WebCore::SourceBuffer::bufferedSamplesForTrackId):
(WebCore::SourceBuffer::enqueuedSamplesForTrackID):
(WebCore::SourceBuffer::sourceBufferPrivateBufferedChanged):
(WebCore::SourceBuffer::enabledForContext):
(WebCore::SourceBuffer::settings const): Deleted.
(WebCore::SourceBuffer::ensureWeakOnDispatcher const): Deleted.
(WebCore::SourceBuffer::promisedWeakOnDispatcher const): Deleted.
* Source/WebCore/Modules/mediasource/SourceBuffer.h:
* Source/WebCore/Modules/mediasource/SourceBuffer.idl:
* Source/WebCore/Modules/pictureinpicture/PictureInPictureWindow.cpp:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/bindings/js/SerializedScriptValue.cpp:
(WebCore::name):
(WebCore::isTypeExposedToGlobalObject):
(WebCore::CloneSerializer::serialize):
(WebCore::CloneSerializer::CloneSerializer):
(WebCore::CloneSerializer::dumpMediaSourceHandle):
(WebCore::CloneSerializer::dumpIfTerminal):
(WebCore::CloneDeserializer::deserialize):
(WebCore::CloneDeserializer::CloneDeserializer):
(WebCore::CloneDeserializer::takeDetachedMediaSourceHandles):
(WebCore::CloneDeserializer::readMediaSourceHandle):
(WebCore::CloneDeserializer::readTerminal):
(WebCore::validateSerializedResult):
(WebCore::SerializedScriptValue::SerializedScriptValue):
(WebCore::canDetachMediaSourceHandles):
(WebCore::SerializedScriptValue::create):
(WebCore::SerializedScriptValue::deserialize):
* Source/WebCore/bindings/js/SerializedScriptValue.h:
* Source/WebCore/bindings/js/WebCoreBuiltinNames.h:
* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::selectMediaResource):
(WebCore::HTMLMediaElement::loadResource):
(WebCore::HTMLMediaElement::hasManagedMediaSource const):
(WebCore::HTMLMediaElement::detachMediaSource):
(WebCore::HTMLMediaElement::mediaState const):
* Source/WebCore/html/HTMLMediaElement.h:
* Source/WebCore/html/HTMLMediaElement.idl:
* Source/WebCore/html/track/AudioTrack.cpp:
(WebCore::AudioTrack::isValidKind const):
(WebCore::AudioTrack::updateKindFromPrivate):
* Source/WebCore/html/track/VideoTrack.cpp:
(WebCore::VideoTrack::isValidKind const):
(WebCore::VideoTrack::updateKindFromPrivate):
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::applicationOctetStream):
* Source/WebCore/platform/graphics/MediaSourcePrivate.cpp:
(WebCore::MediaSourcePrivate::MediaSourcePrivate):
(WebCore::MediaSourcePrivate::seekable const):
(WebCore::MediaSourcePrivate::setLiveSeekableRange):
(WebCore::MediaSourcePrivate::clearLiveSeekableRange):
(WebCore::MediaSourcePrivate::liveSeekableRange const):
* Source/WebCore/platform/graphics/MediaSourcePrivate.h:
* Source/WebCore/platform/graphics/MediaSourcePrivateClient.h:
(WebCore::MediaSourcePrivateClient::logger const):
* Source/WebCore/platform/graphics/SourceBufferPrivate.cpp:
(WebCore::SourceBufferPrivate::setClient):
* Source/WebCore/platform/graphics/SourceBufferPrivateClient.h:
* Source/WebCore/platform/graphics/TrackPrivateBase.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.cpp:
(WebKit::MediaSourcePrivateRemote::MediaSourcePrivateRemote):
(WebKit::MediaSourcePrivateRemote::mediaPlayerReadyState const):
(WebKit::MediaSourcePrivateRemote::setMediaPlayerReadyState):
(WebKit::MediaSourcePrivateRemote::MessageReceiver::mediaSourcePrivateShuttingDown):
* Source/WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h:

Canonical link: https://commits.webkit.org/274853@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications

More information about the webkit-changes mailing list