<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[287180] trunk</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/287180">287180</a></dd>
<dt>Author</dt> <dd>commit-queue@webkit.org</dd>
<dt>Date</dt> <dd>2021-12-17 04:47:56 -0800 (Fri, 17 Dec 2021)</dd>
</dl>

<h3>Log Message</h3>
<pre>[GStreamer][WebRTC] Huge memory leak
https://bugs.webkit.org/show_bug.cgi?id=234134

Patch by Philippe Normand <pnormand@igalia.com> on 2021-12-17
Reviewed by Xabier Rodriguez-Calvar.

Source/WebCore:

The main issue was RealtimeOutgoingVideoSourceLibWebRTC leaking GstSamples. Fixing this lead
me to further clean-ups in the GstSample<->LibWebRTCVideoFrame handling. Native frames
should not consume GstSamples, but reference them, otherwise there are crashes where the
dangling GstSamples would be passed to the libwebrtc video decoder...

Also the video decoders were not reporting their initialization status correctly,
WEBRTC_VIDEO_CODEC_OK is 0 hence false...

* platform/graphics/gstreamer/MediaSampleGStreamer.cpp:
(WebCore::MediaSampleGStreamer::MediaSampleGStreamer):
(WebCore::MediaSampleGStreamer::initializeFromBuffer):
* platform/graphics/gstreamer/MediaSampleGStreamer.h:
(WebCore::MediaSampleGStreamer::createWrappedSample):
* platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp:
(WebCore::GStreamerVideoDecoder::pullSample):
* platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp:
* platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp:
(WebCore::convertLibWebRTCVideoFrameToGStreamerSample):
(WebCore::convertGStreamerSampleToLibWebRTCVideoFrame):
(WebCore::GStreamerVideoFrameLibWebRTC::create):
(WebCore::GStreamerSampleFromLibWebRTCVideoFrame): Deleted.
(WebCore::LibWebRTCVideoFrameFromGStreamerSample): Deleted.
* platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h:
(WebCore::GStreamerVideoFrameLibWebRTC::GStreamerVideoFrameLibWebRTC):
(WebCore::GStreamerVideoFrameLibWebRTC::getSample const):
(WebCore::GStreamerVideoFrameLibWebRTC::takeSample): Deleted.
* platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
(WebCore::RealtimeIncomingVideoSourceLibWebRTC::OnFrame):
* platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
(WebCore::RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable):

LayoutTests:

* platform/glib/TestExpectations: Update test expectations, #233740 was mostly due to video
decoder initialization wrongly reported as failed.
webrtc/captureCanvas-webrtc-software-h264-baseline.html no longer times out.</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsChangeLog">trunk/LayoutTests/ChangeLog</a></li>
<li><a href="#trunkLayoutTestsplatformglibTestExpectations">trunk/LayoutTests/platform/glib/TestExpectations</a></li>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCoreplatformgraphicsgstreamerMediaSampleGStreamercpp">trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformgraphicsgstreamerMediaSampleGStreamerh">trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoDecoderFactorycpp">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoEncoderFactorycpp">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoFrameLibWebRTCcpp">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoFrameLibWebRTCh">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerRealtimeIncomingVideoSourceLibWebRTCcpp">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerRealtimeOutgoingVideoSourceLibWebRTCcpp">trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/ChangeLog (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/ChangeLog      2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/LayoutTests/ChangeLog 2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -1,3 +1,14 @@
</span><ins>+2021-12-17  Philippe Normand  <pnormand@igalia.com>
+
+        [GStreamer][WebRTC] Huge memory leak
+        https://bugs.webkit.org/show_bug.cgi?id=234134
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        * platform/glib/TestExpectations: Update test expectations, #233740 was mostly due to video
+        decoder initialization wrongly reported as failed.
+        webrtc/captureCanvas-webrtc-software-h264-baseline.html no longer times out.
+
</ins><span class="cx"> 2021-12-16  Tyler Wilcock  <tyler_w@apple.com>
</span><span class="cx"> 
</span><span class="cx">         AX: Make aria-multiline.html, clipped-text-under-element.html, mixed-checkbox.html, and selection-states.html pass in isolated tree mode
</span></span></pre></div>
<a id="trunkLayoutTestsplatformglibTestExpectations"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/platform/glib/TestExpectations (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/platform/glib/TestExpectations 2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/LayoutTests/platform/glib/TestExpectations    2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -1349,8 +1349,7 @@
</span><span class="cx"> 
</span><span class="cx"> webkit.org/b/208125 webrtc/peerconnection-new-candidate-page-cache.html [ Pass Crash ]
</span><span class="cx"> 
</span><del>-# Was Slow Failure before libwebrtc bumped to M96 in r285577
-webkit.org/b/216538 webrtc/captureCanvas-webrtc-software-h264-baseline.html [ Timeout ]
</del><ins>+webkit.org/b/216538 webrtc/captureCanvas-webrtc-software-h264-baseline.html [ Slow Failure ]
</ins><span class="cx"> 
</span><span class="cx"> webkit.org/b/216763 webrtc/captureCanvas-webrtc-software-h264-high.html [ Crash Failure ]
</span><span class="cx"> 
</span><span class="lines">@@ -1370,10 +1369,6 @@
</span><span class="cx"> 
</span><span class="cx"> webkit.org/b/229055 http/wpt/webrtc/sframe-transform-error.html [ Failure ]
</span><span class="cx"> 
</span><del>-webkit.org/b/233740 http/wpt/webrtc/video-script-transform-keyframe-only.html [ Failure ]
-webkit.org/b/233740 webrtc/vp9-svc.html [ Timeout ]
-webkit.org/b/233740 webrtc/vp9.html [ Timeout ]
-
</del><span class="cx"> webkit.org/b/233879 fast/mediastream/get-display-media-settings.html [ Failure ]
</span><span class="cx"> 
</span><span class="cx"> #////////////////////////////////////////////////////////////////////////////////////////
</span></span></pre></div>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/ChangeLog      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -1,3 +1,41 @@
</span><ins>+2021-12-17  Philippe Normand  <pnormand@igalia.com>
+
+        [GStreamer][WebRTC] Huge memory leak
+        https://bugs.webkit.org/show_bug.cgi?id=234134
+
+        Reviewed by Xabier Rodriguez-Calvar.
+
+        The main issue was RealtimeOutgoingVideoSourceLibWebRTC leaking GstSamples. Fixing this lead
+        me to further clean-ups in the GstSample<->LibWebRTCVideoFrame handling. Native frames
+        should not consume GstSamples, but reference them, otherwise there are crashes where the
+        dangling GstSamples would be passed to the libwebrtc video decoder...
+
+        Also the video decoders were not reporting their initialization status correctly,
+        WEBRTC_VIDEO_CODEC_OK is 0 hence false...
+
+        * platform/graphics/gstreamer/MediaSampleGStreamer.cpp:
+        (WebCore::MediaSampleGStreamer::MediaSampleGStreamer):
+        (WebCore::MediaSampleGStreamer::initializeFromBuffer):
+        * platform/graphics/gstreamer/MediaSampleGStreamer.h:
+        (WebCore::MediaSampleGStreamer::createWrappedSample):
+        * platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp:
+        (WebCore::GStreamerVideoDecoder::pullSample):
+        * platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp:
+        * platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp:
+        (WebCore::convertLibWebRTCVideoFrameToGStreamerSample):
+        (WebCore::convertGStreamerSampleToLibWebRTCVideoFrame):
+        (WebCore::GStreamerVideoFrameLibWebRTC::create):
+        (WebCore::GStreamerSampleFromLibWebRTCVideoFrame): Deleted.
+        (WebCore::LibWebRTCVideoFrameFromGStreamerSample): Deleted.
+        * platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h:
+        (WebCore::GStreamerVideoFrameLibWebRTC::GStreamerVideoFrameLibWebRTC):
+        (WebCore::GStreamerVideoFrameLibWebRTC::getSample const):
+        (WebCore::GStreamerVideoFrameLibWebRTC::takeSample): Deleted.
+        * platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp:
+        (WebCore::RealtimeIncomingVideoSourceLibWebRTC::OnFrame):
+        * platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp:
+        (WebCore::RealtimeOutgoingVideoSourceLibWebRTC::videoSampleAvailable):
+
</ins><span class="cx"> 2021-12-17  Antoine Quint  <graouts@webkit.org>
</span><span class="cx"> 
</span><span class="cx">         ActiveDOMObject::suspendIfNeeded() should not be called within constructors
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformgraphicsgstreamerMediaSampleGStreamercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp        2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp   2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -42,7 +42,6 @@
</span><span class="cx">     , m_videoRotation(videoRotation)
</span><span class="cx">     , m_videoMirrored(videoMirrored)
</span><span class="cx"> {
</span><del>-    const GstClockTime minimumDuration = 1000; // 1 us
</del><span class="cx">     ASSERT(sample);
</span><span class="cx">     GstBuffer* buffer = gst_sample_get_buffer(sample.get());
</span><span class="cx">     RELEASE_ASSERT(buffer);
</span><span class="lines">@@ -50,35 +49,10 @@
</span><span class="cx">     if (metadata)
</span><span class="cx">         buffer = webkitGstBufferSetVideoSampleMetadata(buffer, WTFMove(metadata));
</span><span class="cx"> 
</span><del>-    if (GST_BUFFER_PTS_IS_VALID(buffer))
-        m_pts = fromGstClockTime(GST_BUFFER_PTS(buffer));
-    if (GST_BUFFER_DTS_IS_VALID(buffer) || GST_BUFFER_PTS_IS_VALID(buffer))
-        m_dts = fromGstClockTime(GST_BUFFER_DTS_OR_PTS(buffer));
-    if (GST_BUFFER_DURATION_IS_VALID(buffer)) {
-        // Sometimes (albeit rarely, so far seen only at the end of a track)
-        // frames have very small durations, so small that may be under the
-        // precision we are working with and be truncated to zero.
-        // SourceBuffer algorithms are not expecting frames with zero-duration,
-        // so let's use something very small instead in those fringe cases.
-        m_duration = fromGstClockTime(std::max(GST_BUFFER_DURATION(buffer), minimumDuration));
-    } else {
-        // Unfortunately, sometimes samples don't provide a duration. This can never happen in MP4 because of the way
-        // the format is laid out, but it's pretty common in WebM.
-        // The good part is that durations don't matter for playback, just for buffered ranges and coded frame deletion.
-        // We want to pick something small enough to not cause unwanted frame deletion, but big enough to never be
-        // mistaken for a rounding artifact.
-        m_duration = fromGstClockTime(16666667); // 1/60 seconds
-    }
-
-    m_size = gst_buffer_get_size(buffer);
</del><span class="cx">     m_sample = adoptGRef(gst_sample_new(buffer, gst_sample_get_caps(sample.get()), nullptr,
</span><span class="cx">         gst_sample_get_info(sample.get()) ? gst_structure_copy(gst_sample_get_info(sample.get())) : nullptr));
</span><span class="cx"> 
</span><del>-    if (GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT))
-        m_flags = MediaSample::None;
-
-    if (GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DECODE_ONLY))
-        m_flags = static_cast<MediaSample::SampleFlags>(m_flags | MediaSample::IsNonDisplaying);
</del><ins>+    initializeFromBuffer();
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> MediaSampleGStreamer::MediaSampleGStreamer(const FloatSize& presentationSize, const AtomString& trackId)
</span><span class="lines">@@ -90,6 +64,13 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+MediaSampleGStreamer::MediaSampleGStreamer(const GRefPtr<GstSample>& sample, VideoRotation videoRotation)
+    : m_sample(sample)
+    , m_videoRotation(videoRotation)
+{
+    initializeFromBuffer();
+}
+
</ins><span class="cx"> Ref<MediaSampleGStreamer> MediaSampleGStreamer::createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId)
</span><span class="cx"> {
</span><span class="cx">     MediaSampleGStreamer* gstreamerMediaSample = new MediaSampleGStreamer(presentationSize, trackId);
</span><span class="lines">@@ -154,6 +135,41 @@
</span><span class="cx">     return create(WTFMove(sample), FloatSize(width, height), { }, videoRotation, videoMirrored);
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void MediaSampleGStreamer::initializeFromBuffer()
+{
+    const GstClockTime minimumDuration = 1000; // 1 us
+    auto* buffer = gst_sample_get_buffer(m_sample.get());
+    RELEASE_ASSERT(buffer);
+
+    if (GST_BUFFER_PTS_IS_VALID(buffer))
+        m_pts = fromGstClockTime(GST_BUFFER_PTS(buffer));
+    if (GST_BUFFER_DTS_IS_VALID(buffer) || GST_BUFFER_PTS_IS_VALID(buffer))
+        m_dts = fromGstClockTime(GST_BUFFER_DTS_OR_PTS(buffer));
+    if (GST_BUFFER_DURATION_IS_VALID(buffer)) {
+        // Sometimes (albeit rarely, so far seen only at the end of a track)
+        // frames have very small durations, so small that may be under the
+        // precision we are working with and be truncated to zero.
+        // SourceBuffer algorithms are not expecting frames with zero-duration,
+        // so let's use something very small instead in those fringe cases.
+        m_duration = fromGstClockTime(std::max(GST_BUFFER_DURATION(buffer), minimumDuration));
+    } else {
+        // Unfortunately, sometimes samples don't provide a duration. This can never happen in MP4 because of the way
+        // the format is laid out, but it's pretty common in WebM.
+        // The good part is that durations don't matter for playback, just for buffered ranges and coded frame deletion.
+        // We want to pick something small enough to not cause unwanted frame deletion, but big enough to never be
+        // mistaken for a rounding artifact.
+        m_duration = fromGstClockTime(16666667); // 1/60 seconds
+    }
+
+    m_size = gst_buffer_get_size(buffer);
+
+    if (GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DELTA_UNIT))
+        m_flags = MediaSample::None;
+
+    if (GST_BUFFER_FLAG_IS_SET(buffer, GST_BUFFER_FLAG_DECODE_ONLY))
+        m_flags = static_cast<MediaSample::SampleFlags>(m_flags | MediaSample::IsNonDisplaying);
+}
+
</ins><span class="cx"> RefPtr<JSC::Uint8ClampedArray> MediaSampleGStreamer::getRGBAImageData() const
</span><span class="cx"> {
</span><span class="cx">     auto* caps = gst_sample_get_caps(m_sample.get());
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformgraphicsgstreamerMediaSampleGStreamerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h  2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h     2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -40,6 +40,11 @@
</span><span class="cx">         return adoptRef(*new MediaSampleGStreamer(WTFMove(sample), presentationSize, trackId, videoRotation, videoMirrored, WTFMove(metadata)));
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    static Ref<MediaSampleGStreamer> createWrappedSample(const GRefPtr<GstSample>& sample, VideoRotation videoRotation = VideoRotation::None)
+    {
+        return adoptRef(*new MediaSampleGStreamer(sample, videoRotation));
+    }
+
</ins><span class="cx">     static Ref<MediaSampleGStreamer> createFakeSample(GstCaps*, MediaTime pts, MediaTime dts, MediaTime duration, const FloatSize& presentationSize, const AtomString& trackId);
</span><span class="cx">     static Ref<MediaSampleGStreamer> createImageSample(PixelBuffer&&, const IntSize& destinationSize = { }, double frameRate = 1, VideoRotation videoRotation = VideoRotation::None, bool videoMirrored = false, std::optional<VideoSampleMetadata>&& metadata = std::nullopt);
</span><span class="cx"> 
</span><span class="lines">@@ -66,11 +71,14 @@
</span><span class="cx"> 
</span><span class="cx"> protected:
</span><span class="cx">     MediaSampleGStreamer(GRefPtr<GstSample>&&, const FloatSize& presentationSize, const AtomString& trackId, VideoRotation = VideoRotation::None, bool videoMirrored = false, std::optional<VideoSampleMetadata>&& = std::nullopt);
</span><ins>+    MediaSampleGStreamer(const GRefPtr<GstSample>&, VideoRotation = VideoRotation::None);
</ins><span class="cx">     virtual ~MediaSampleGStreamer() = default;
</span><span class="cx"> 
</span><span class="cx"> private:
</span><span class="cx">     MediaSampleGStreamer(const FloatSize& presentationSize, const AtomString& trackId);
</span><span class="cx"> 
</span><ins>+    void initializeFromBuffer();
+
</ins><span class="cx">     MediaTime m_pts;
</span><span class="cx">     MediaTime m_dts;
</span><span class="cx">     MediaTime m_duration;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoDecoderFactorycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoDecoderFactory.cpp      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -150,7 +150,7 @@
</span><span class="cx">             return WEBRTC_VIDEO_CODEC_ERROR;
</span><span class="cx">         }
</span><span class="cx"> 
</span><del>-        return WEBRTC_VIDEO_CODEC_OK;
</del><ins>+        return true;
</ins><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback* callback) override
</span><span class="lines">@@ -240,7 +240,7 @@
</span><span class="cx">         auto timestamps = m_dtsPtsMap[GST_BUFFER_PTS(buffer)];
</span><span class="cx">         m_dtsPtsMap.erase(GST_BUFFER_PTS(buffer));
</span><span class="cx"> 
</span><del>-        auto frame(LibWebRTCVideoFrameFromGStreamerSample(WTFMove(sample), webrtc::kVideoRotation_0,
</del><ins>+        auto frame(convertGStreamerSampleToLibWebRTCVideoFrame(sample, webrtc::kVideoRotation_0,
</ins><span class="cx">             timestamps.timestamp, timestamps.renderTimeMs));
</span><span class="cx"> 
</span><span class="cx">         GST_BUFFER_DTS(buffer) = GST_CLOCK_TIME_NONE;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoEncoderFactorycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoEncoderFactory.cpp      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -228,7 +228,7 @@
</span><span class="cx">             return WEBRTC_VIDEO_CODEC_UNINITIALIZED;
</span><span class="cx">         }
</span><span class="cx"> 
</span><del>-        auto sample = GStreamerSampleFromLibWebRTCVideoFrame(frame);
</del><ins>+        auto sample = convertLibWebRTCVideoFrameToGStreamerSample(frame);
</ins><span class="cx">         auto buffer = gst_sample_get_buffer(sample.get());
</span><span class="cx"> 
</span><span class="cx">         if (!GST_CLOCK_TIME_IS_VALID(m_firstFramePts)) {
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoFrameLibWebRTCcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.cpp      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -27,13 +27,9 @@
</span><span class="cx"> 
</span><span class="cx"> namespace WebCore {
</span><span class="cx"> 
</span><del>-GRefPtr<GstSample> GStreamerSampleFromLibWebRTCVideoFrame(const webrtc::VideoFrame& frame)
</del><ins>+GRefPtr<GstSample> convertLibWebRTCVideoFrameToGStreamerSample(const webrtc::VideoFrame& frame)
</ins><span class="cx"> {
</span><del>-    if (frame.video_frame_buffer()->type() == webrtc::VideoFrameBuffer::Type::kNative) {
-        auto* framebuffer = static_cast<GStreamerVideoFrameLibWebRTC*>(frame.video_frame_buffer().get());
-        return framebuffer->takeSample();
-    }
-
</del><ins>+    RELEASE_ASSERT(frame.video_frame_buffer()->type() != webrtc::VideoFrameBuffer::Type::kNative);
</ins><span class="cx">     auto* i420Buffer = frame.video_frame_buffer()->ToI420().release();
</span><span class="cx">     int height = i420Buffer->height();
</span><span class="cx">     int strides[3] = {
</span><span class="lines">@@ -60,23 +56,22 @@
</span><span class="cx">     return sample;
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-rtc::scoped_refptr<webrtc::VideoFrameBuffer> GStreamerVideoFrameLibWebRTC::create(GRefPtr<GstSample>&& sample)
</del><ins>+std::unique_ptr<webrtc::VideoFrame> convertGStreamerSampleToLibWebRTCVideoFrame(GRefPtr<GstSample>& sample, webrtc::VideoRotation rotation, int64_t timestamp, int64_t renderTimeMs)
</ins><span class="cx"> {
</span><ins>+    auto frameBuffer(GStreamerVideoFrameLibWebRTC::create(sample));
+    return std::unique_ptr<webrtc::VideoFrame>(new webrtc::VideoFrame(frameBuffer, timestamp, renderTimeMs, rotation));
+}
+
+rtc::scoped_refptr<webrtc::VideoFrameBuffer> GStreamerVideoFrameLibWebRTC::create(const GRefPtr<GstSample>& sample)
+{
</ins><span class="cx">     GstVideoInfo info;
</span><span class="cx"> 
</span><span class="cx">     if (!gst_video_info_from_caps(&info, gst_sample_get_caps(sample.get())))
</span><span class="cx">         ASSERT_NOT_REACHED();
</span><span class="cx"> 
</span><del>-    return rtc::scoped_refptr<webrtc::VideoFrameBuffer>(new GStreamerVideoFrameLibWebRTC(WTFMove(sample), info));
</del><ins>+    return rtc::scoped_refptr<webrtc::VideoFrameBuffer>(new GStreamerVideoFrameLibWebRTC(sample, info));
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GRefPtr<GstSample>&& sample, webrtc::VideoRotation rotation,
-    int64_t timestamp, int64_t renderTimeMs)
-{
-    auto frameBuffer(GStreamerVideoFrameLibWebRTC::create(WTFMove(sample)));
-    return std::unique_ptr<webrtc::VideoFrame>(new webrtc::VideoFrame(frameBuffer, timestamp, renderTimeMs, rotation));
-}
-
</del><span class="cx"> rtc::scoped_refptr<webrtc::I420BufferInterface> GStreamerVideoFrameLibWebRTC::ToI420()
</span><span class="cx"> {
</span><span class="cx">     GstMappedFrame inFrame(m_sample, GST_MAP_READ);
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerGStreamerVideoFrameLibWebRTCh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h     2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/GStreamerVideoFrameLibWebRTC.h        2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -31,19 +31,19 @@
</span><span class="cx"> 
</span><span class="cx"> namespace WebCore {
</span><span class="cx"> 
</span><del>-WARN_UNUSED_RETURN GRefPtr<GstSample> GStreamerSampleFromLibWebRTCVideoFrame(const webrtc::VideoFrame&);
</del><ins>+WARN_UNUSED_RETURN GRefPtr<GstSample> convertLibWebRTCVideoFrameToGStreamerSample(const webrtc::VideoFrame&);
</ins><span class="cx"> 
</span><del>-std::unique_ptr<webrtc::VideoFrame> LibWebRTCVideoFrameFromGStreamerSample(GRefPtr<GstSample>&&, webrtc::VideoRotation, int64_t timestamp, int64_t renderTimeMs);
</del><ins>+std::unique_ptr<webrtc::VideoFrame> convertGStreamerSampleToLibWebRTCVideoFrame(GRefPtr<GstSample>&, webrtc::VideoRotation, int64_t timestamp, int64_t renderTimeMs);
</ins><span class="cx"> 
</span><span class="cx"> class GStreamerVideoFrameLibWebRTC : public rtc::RefCountedObject<webrtc::VideoFrameBuffer> {
</span><span class="cx"> public:
</span><del>-    GStreamerVideoFrameLibWebRTC(GRefPtr<GstSample>&& sample, GstVideoInfo info)
-        : m_sample(WTFMove(sample))
</del><ins>+    GStreamerVideoFrameLibWebRTC(const GRefPtr<GstSample>& sample, GstVideoInfo info)
+        : m_sample(sample)
</ins><span class="cx">         , m_info(info) { }
</span><span class="cx"> 
</span><del>-    static rtc::scoped_refptr<webrtc::VideoFrameBuffer> create(GRefPtr<GstSample>&&);
</del><ins>+    static rtc::scoped_refptr<webrtc::VideoFrameBuffer> create(const GRefPtr<GstSample>&);
</ins><span class="cx"> 
</span><del>-    GRefPtr<GstSample>&& takeSample() { return WTFMove(m_sample); }
</del><ins>+    GstSample* getSample() const { return m_sample.get(); }
</ins><span class="cx">     rtc::scoped_refptr<webrtc::I420BufferInterface> ToI420() final;
</span><span class="cx"> 
</span><span class="cx">     int width() const final { return GST_VIDEO_INFO_WIDTH(&m_info); }
</span><span class="lines">@@ -55,6 +55,7 @@
</span><span class="cx">     GRefPtr<GstSample> m_sample;
</span><span class="cx">     GstVideoInfo m_info;
</span><span class="cx"> };
</span><ins>+
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> #endif // USE(GSTREAMER) && USE(LIBWEBRTC)
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerRealtimeIncomingVideoSourceLibWebRTCcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeIncomingVideoSourceLibWebRTC.cpp      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -58,12 +58,14 @@
</span><span class="cx">     if (!isProducingData())
</span><span class="cx">         return;
</span><span class="cx"> 
</span><del>-    callOnMainThread([protectedThis = Ref { *this }, frame] {
-        auto gstSample = GStreamerSampleFromLibWebRTCVideoFrame(frame);
</del><ins>+    if (frame.video_frame_buffer()->type() == webrtc::VideoFrameBuffer::Type::kNative) {
+        auto* framebuffer = static_cast<GStreamerVideoFrameLibWebRTC*>(frame.video_frame_buffer().get());
+        videoSampleAvailable(MediaSampleGStreamer::createWrappedSample(framebuffer->getSample(), static_cast<MediaSample::VideoRotation>(frame.rotation())), { });
+    } else {
+        auto gstSample = convertLibWebRTCVideoFrameToGStreamerSample(frame);
</ins><span class="cx">         auto metadata = std::make_optional(metadataFromVideoFrame(frame));
</span><del>-        auto sample = MediaSampleGStreamer::create(WTFMove(gstSample), { }, { }, static_cast<MediaSample::VideoRotation>(frame.rotation()), false, WTFMove(metadata));
-        protectedThis->videoSampleAvailable(sample.get(), { });
-    });
</del><ins>+        videoSampleAvailable(MediaSampleGStreamer::create(WTFMove(gstSample), { }, { }, static_cast<MediaSample::VideoRotation>(frame.rotation()), false, WTFMove(metadata)), { });
+    }
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> } // namespace WebCore
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamlibwebrtcgstreamerRealtimeOutgoingVideoSourceLibWebRTCcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp (287179 => 287180)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp   2021-12-17 12:41:14 UTC (rev 287179)
+++ trunk/Source/WebCore/platform/mediastream/libwebrtc/gstreamer/RealtimeOutgoingVideoSourceLibWebRTC.cpp      2021-12-17 12:47:56 UTC (rev 287180)
</span><span class="lines">@@ -69,7 +69,7 @@
</span><span class="cx"> 
</span><span class="cx">     ASSERT(sample.platformSample().type == PlatformSample::GStreamerSampleType);
</span><span class="cx">     auto& mediaSample = static_cast<MediaSampleGStreamer&>(sample);
</span><del>-    auto frameBuffer = GStreamerVideoFrameLibWebRTC::create(gst_sample_ref(mediaSample.platformSample().sample.gstSample));
</del><ins>+    auto frameBuffer = GStreamerVideoFrameLibWebRTC::create(mediaSample.platformSample().sample.gstSample);
</ins><span class="cx"> 
</span><span class="cx">     sendFrame(WTFMove(frameBuffer));
</span><span class="cx"> }
</span></span></pre>
</div>
</div>

</body>
</html>