[webkit-changes] [WebKit/WebKit] b75003: [GStreamer] Set audio track ID to stream value in ...

[Vivienne]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: b750030983005416b9601a3ab58b3b3886dbbd90
      https://github.com/WebKit/WebKit/commit/b750030983005416b9601a3ab58b3b3886dbbd90
  Author: Vivienne Watermeier <vwatermeier at igalia.com>
  Date:   2023-09-20 (Wed, 20 Sep 2023)

  Changed paths:
    M LayoutTests/media/track/audio/audio-track-mkv-vorbis-language-expected.txt
    M LayoutTests/media/track/audio/audio-track-mkv-vorbis-language.html
    M LayoutTests/media/track/video/video-track-mkv-theora-language-expected.txt
    M LayoutTests/media/track/video/video-track-mkv-theora-language.html
    M Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp
    M Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h
    M Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp
    M Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h
    M Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp
    M Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h
    M Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp
    M Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h

  Log Message:
  -----------
  [GStreamer] Set audio track ID to stream value in regular playback
https://bugs.webkit.org/show_bug.cgi?id=260520

Reviewed by Xabier Rodriguez-Calvar.

This change only affects playbin2 implementations (regular playback),
where tracks are based on pads instead of in streams.

The proper track IDs are only actually available when the stream-start
bus message is received. The track creation is delayed until that moment.
At that point, all the audio and video pads handled by GStreamer are
processed and the corresponding audio/video tracks are created. The proper
track IDs are present at that moment. This means that the old
audio/videoChanged callbacks aren't needed anymore.

After that, we listen to changes in caps and tags in the pad and trigger
the needed updates, which will complete the information available for the
corresponding track.

Some approaches that were tried and resulted to be unsuccessful:

Trying to create the track the first time (on video-changed or
audio-changed), already with the definitive id isn't a valid approach,
since the sticky stream-start event isn't available at that moment.
Creating the track with a synthesized id and then updating it when the
stream-start event is available isn't a good idea, because the only way
to update the track is by removing it from audio/videoTracks and readding
it, and that triggers spureous events visible from JavaScript. Therefore,
the proper way was to delay track creation until stream-start comes.

Trying to install a probe on the audio/video pads and detecting the
stream-start event from there (instead of relying on the bus message),
but calling the track manipulation code from a non-main thread would be
unsafe and would require deferring the manipulation to the main thread.
Even in that case, there's no guarantee for the stream-start sticky event
to be available when the manipulation code runs in the main thread,
because that sticky event is stored after probe processing and there's
a race.

Some layout tests had to be changed to refer to the tracks by index
instead of by track id. The GStreamer based ports now return the native
id embedded in the stream container, while Apple ports still use synthesized
ids (A1, A2, V1...). That was the best way to deal with the differences
without duplicating the tests for different platforms.

Co-authored by Enrique Ocaña González <eocanha at igalia.com>

* LayoutTests/media/track/audio/audio-track-mkv-vorbis-language-expected.txt: Changed expected output.
* LayoutTests/media/track/audio/audio-track-mkv-vorbis-language.html: Refer to the tracks by index instead of by id.
* LayoutTests/media/track/video/video-track-mkv-theora-language-expected.txt: Changed expected output.
* LayoutTests/media/track/video/video-track-mkv-theora-language.html: Refer to the tracks by index instead of by id.
* Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp:
(WebCore::AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer): Install configuration change handlers. Use move semantics.
(WebCore::AudioTrackPrivateGStreamer::capsChanged): Updated signature, use move semantics.
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromTags): Updated signature.
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromCaps): Ditto.
* Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h: Declare configuration change handlers as overridding the base implementation. Updated some signatures to use r-value references.
* Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfTrack): Set track IDs (now always valid and final at this point, unlike before) and only notify the media engine if actual changes have been made. This is because this method can now be called even if no tracks of the given
type have appeared.
(WebCore::MediaPlayerPrivateGStreamer::handleMessage): Listen to stream-start bus message and unconditionally notify for audio/video track creation. notifyPlayerOfTrack() will know if tracks of each type have been created or not.
(WebCore::MediaPlayerPrivateGStreamer::createGSTPlayBin): Don't listen audio/video/text-changed anymore. We now have everything we need when stream-start comes.
(WebCore::MediaPlayerPrivateGStreamer::audioChangedCallback): Deleted.
(WebCore::MediaPlayerPrivateGStreamer::textChangedCallback): Deleted.
(WebCore::MediaPlayerPrivateGStreamer::videoChangedCallback): Deleted.
* Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h: Deleted audio/video/textChangedCallback().
* Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
(WebCore::findBestUpstreamPad): Added protection against null pad.
(WebCore::TrackPrivateBaseGStreamer::TrackPrivateBaseGStreamer): Generate ID from pad stream-start or synthesize one. This factors in common code and avoids repetition.
(WebCore::TrackPrivateBaseGStreamer::setPad): Protect against null m_bestUpstreamPad and reuse preexistent streamId from track.
(WebCore::TrackPrivateBaseGStreamer::tagsChanged): Initialize tags list to avoid uninitialized pointer in case parsing fails. Use move semantics.
(WebCore::TrackPrivateBaseGStreamer::capsChanged): Use move semantics.
(WebCore::TrackPrivateBaseGStreamer::notifyTrackOfTagsChanged): Avoid leak. Use move semantics.
(WebCore::TrackPrivateBaseGStreamer::notifyTrackOfStreamChanged): Protect against null pad.
(WebCore::TrackPrivateBaseGStreamer::installUpdateConfigurationHandlers): Listen to caps and tags changes and update configuration accordingly by calling the abstract handlers (to be implemented by subclasses as needed, empty by default). This installation works for pad-based (playbin2) as well as stream-based (playbin3) variants.
(WebCore::TrackPrivateBaseGStreamer::updateConfigurationFromCaps): Use r-value references.
(WebCore::TrackPrivateBaseGStreamer::updateConfigurationFromTags): Ditto.
(WebCore::TrackPrivateBaseGStreamer::trackIdFromPadStreamStartOrUniqueID): Extract the track ID name from the pad, or synthesize an ID if it can't be extracted.
(WebCore::TrackPrivateBaseGStreamer::getAllTags): Utility function to merge multiple tags stored in a pad as a sticky tags event.
* Source/WebCore/platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h: Factored in common code into trackIdFromPadStreamStartOrUniqueID(). Added default empty abstract implementations of configuration change handlers and common method to install them, for those subclasses that chose to do so. Use r-value references in some signatures.
* Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h: Declare configuration change handlers as overridding the base implementation.
* Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:
(WebCore::VideoTrackPrivateGStreamer::VideoTrackPrivateGStreamer): Install configuration change handlers.
(WebCore::VideoTrackPrivateGStreamer::capsChanged): Use move semantics.
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromTags): Ditto.
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromCaps): Ditto.

Canonical link: https://commits.webkit.org/268194@main