<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[260364] trunk</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/260364">260364</a></dd>
<dt>Author</dt> <dd>youenn@apple.com</dd>
<dt>Date</dt> <dd>2020-04-20 07:37:47 -0700 (Mon, 20 Apr 2020)</dd>
</dl>

<h3>Log Message</h3>
<pre>Safari doesn't apply frameRate limit when request stream from Camera
https://bugs.webkit.org/show_bug.cgi?id=210186
<rdar://problem/61452794>

Reviewed by Eric Carlson.

Source/WebCore:

Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.

Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
Fix mock capture source to update its frame rate when asked by applyConstraints.

Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
       fast/mediastream/mediastreamtrack-video-frameRate-increasing.html

* platform/mediastream/RealtimeVideoCaptureSource.cpp:
(WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
(WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):
* platform/mediastream/RealtimeVideoCaptureSource.h:
(WebCore::RealtimeVideoCaptureSource::observedFrameRate const):
* platform/mediastream/RealtimeVideoSource.cpp:
(WebCore::RealtimeVideoSource::RealtimeVideoSource):
(WebCore::m_source):
(WebCore::RealtimeVideoSource::adaptVideoSample):
(WebCore::RealtimeVideoSource::videoSampleAvailable):
* platform/mediastream/RealtimeVideoSource.h:
* platform/mock/MockRealtimeVideoSource.cpp:
(WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):
* testing/Internals.cpp:
(WebCore::Internals::observeMediaStreamTrack):

LayoutTests:

* fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
* fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
* fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
* fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
* fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
* fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
* webrtc/routines.js:</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsChangeLog">trunk/LayoutTests/ChangeLog</a></li>
<li><a href="#trunkLayoutTestswebrtcroutinesjs">trunk/LayoutTests/webrtc/routines.js</a></li>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamRealtimeVideoCaptureSourcecpp">trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamRealtimeVideoCaptureSourceh">trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamRealtimeVideoSourcecpp">trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamRealtimeVideoSourceh">trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmockMockRealtimeVideoSourcecpp">trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp</a></li>
<li><a href="#trunkSourceWebCoretestingInternalscpp">trunk/Source/WebCore/testing/Internals.cpp</a></li>
</ul>

<h3>Added Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateclonedecreasingexpectedtxt">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateclonedecreasinghtml">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatecloneincreasingexpectedtxt">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatecloneincreasinghtml">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatedecreasingexpectedtxt">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatedecreasinghtml">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateincreasingexpectedtxt">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateincreasinghtml">trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/ChangeLog (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/ChangeLog      2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/LayoutTests/ChangeLog 2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -1,3 +1,21 @@
</span><ins>+2020-04-20  Youenn Fablet  <youenn@apple.com>
+
+        Safari doesn't apply frameRate limit when request stream from Camera
+        https://bugs.webkit.org/show_bug.cgi?id=210186
+        <rdar://problem/61452794>
+
+        Reviewed by Eric Carlson.
+
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt: Added.
+        * fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html: Added.
+        * fast/mediastream/mediastreamtrack-video-framerate-decreasing-expected.txt: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-decreasing.html: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-increasing-expected.txt: added.
+        * fast/mediastream/mediastreamtrack-video-framerate-increasing.html: added.
+        * webrtc/routines.js:
+
</ins><span class="cx"> 2020-04-20  Antoine Quint  <graouts@apple.com>
</span><span class="cx"> 
</span><span class="cx">         WebAnimations API doesn't properly apply keyframe easings to transforms
</span></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateclonedecreasingexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt                                (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing-expected.txt   2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,4 @@
</span><ins>+
+
+PASS clone and applyConstraints to decrease frame rate 
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateclonedecreasinghtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html                                (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html   2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,37 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src="../../resources/testharness.js"></script>
+    <script src="../../resources/testharnessreport.js"></script>
+    <script src ="../../webrtc/routines.js"></script>
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+    const stream2 = stream1.clone();
+
+    await stream2.getVideoTracks()[0].applyConstraints({ width : 160, frameRate : 5 });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000));
+
+    let frameRate = await computeFrameRate(stream1, video);
+
+    // On heavily loaded bots, our mock capture source might not be able to deliver 30 fps. Our test only makes sense if we have sufficient fps.
+    if (frameRate <= 10)
+        return;
+
+    assert_greater_than(frameRate, 10, "stream1 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream1 frame rate below 60");
+
+    frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 1, "stream2 frame rate above 1");
+    assert_less_than(frameRate, 10, "stream2 frame rate below 10");
+}, "clone and applyConstraints to decrease frame rate");
+    </script>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatecloneincreasingexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt                                (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing-expected.txt   2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,4 @@
</span><ins>+
+
+PASS clone and applyConstraints to increase frame rate 
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatecloneincreasinghtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html                                (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html   2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,37 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src="../../resources/testharness.js"></script>
+    <script src="../../resources/testharnessreport.js"></script>
+    <script src ="../../webrtc/routines.js"></script>
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+    const stream2 = stream1.clone();
+
+    await stream2.getVideoTracks()[0].applyConstraints({ width : 720, frameRate : 30 });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000));
+
+    let frameRate = await computeFrameRate(stream2, video);
+
+    // On heavily loaded bots, our mock capture source might not be able to deliver 30 fps. Our test only makes sense if we have sufficient fps.
+    if (frameRate <= 10)
+        return;
+
+    assert_greater_than(frameRate, 10, "stream2 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream2 frame rate below 60");
+
+    frameRate = await computeFrameRate(stream1, video);
+    assert_greater_than(frameRate, 1, "stream1 frame rate above 1");
+    assert_less_than(frameRate, 10, "stream1 frame rate below 10");
+}, "clone and applyConstraints to increase frame rate");
+    </script>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatedecreasingexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt                              (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing-expected.txt 2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,4 @@
</span><ins>+
+
+PASS getUserMedia with decreasing frame rates 
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRatedecreasinghtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html                              (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html 2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,37 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src="../../resources/testharness.js"></script>
+    <script src="../../resources/testharnessreport.js"></script>
+    <script src ="../../webrtc/routines.js"></script>
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+    const stream2 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000));
+
+    if (!stream1.getVideoTracks()[0].muted) {
+        const frameRate = await computeFrameRate(stream1, video);
+
+        // On heavily loaded bots, our mock capture source might not be able to deliver 30 fps. Our test only makes sense if we have sufficient fps.
+        if (frameRate <= 10)
+            return;
+
+        assert_greater_than(frameRate, 10, "stream1 frame rate above 10");
+        assert_less_than(frameRate, 60, "stream1 frame rate below 60");
+    }
+
+    const frameRate = await computeFrameRate(stream2, video);
+    assert_greater_than(frameRate, 1, "stream2 frame rate above 1");
+    assert_less_than(frameRate, 10, "stream2 frame rate below 10");
+}, "getUserMedia with decreasing frame rates");
+    </script>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateincreasingexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt                              (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing-expected.txt 2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,4 @@
</span><ins>+
+
+PASS getUserMedia with increasing frame rates 
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastmediastreammediastreamtrackvideoframeRateincreasinghtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html (0 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html                              (rev 0)
+++ trunk/LayoutTests/fast/mediastream/mediastreamtrack-video-frameRate-increasing.html 2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -0,0 +1,37 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Adapt camera track framerate</title>
+    <script src="../../resources/testharness.js"></script>
+    <script src="../../resources/testharnessreport.js"></script>
+    <script src ="../../webrtc/routines.js"></script>
+</head>
+<body>
+    <video autoplay id="video"></video>
+    <script>
+promise_test(async (t) => {
+    const stream1 = await navigator.mediaDevices.getUserMedia({ video: { width: 160, frameRate : 5 } });
+    const stream2 = await navigator.mediaDevices.getUserMedia({ video: { width: 720, frameRate : 30 } });
+
+    // We adapt frame rate after 1 second of samples.
+    await new Promise(resolve => setTimeout(resolve, 1000));
+
+    const frameRate = await computeFrameRate(stream2, video);
+
+    // On heavily loaded bots, our mock capture source might not be able to deliver 30 fps. Our test only makes sense if we have sufficient fps.
+    if (frameRate <= 10)
+        return;
+
+    assert_greater_than(frameRate, 10, "stream2 frame rate above 10");
+    assert_less_than(frameRate, 60, "stream2 frame rate below 60");
+
+    if (!stream1.getVideoTracks()[0].muted) {
+        const frameRate = await computeFrameRate(stream1, video);
+        assert_greater_than(frameRate, 1, "stream1 frame rate above 1");
+        assert_less_than(frameRate, 10, "stream1 frame rate below 10");
+    }
+}, "getUserMedia with increasing frame rates");
+    </script>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkLayoutTestswebrtcroutinesjs"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/webrtc/routines.js (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/webrtc/routines.js     2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/LayoutTests/webrtc/routines.js        2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -225,3 +225,45 @@
</span><span class="cx">     });
</span><span class="cx">     return stats;
</span><span class="cx"> }
</span><ins>+
+function getReceivedTrackStats(connection)
+{
+    return connection.getStats().then((report) => {
+        var stats;
+        report.forEach((statItem) => {
+            if (statItem.type === "track") {
+                stats = statItem;
+            }
+        });
+        return stats;
+    });
+}
+
+async function computeFrameRate(stream, video)
+{
+    if (window.internals) {
+        internals.observeMediaStreamTrack(stream.getVideoTracks()[0]);
+        await new Promise(resolve => setTimeout(resolve, 1000)); 
+        return internals.trackVideoSampleCount;
+    }
+
+    let connection;
+    video.srcObject = await new Promise((resolve, reject) => {
+        createConnections((firstConnection) => {
+            firstConnection.addTrack(stream.getVideoTracks()[0], stream);
+        }, (secondConnection) => {
+            connection = secondConnection;
+            secondConnection.ontrack = (trackEvent) => {
+                resolve(trackEvent.streams[0]);
+            };
+        });
+        setTimeout(() => reject("Test timed out"), 5000);
+    });
+
+    await video.play();
+
+    const stats1 = await getReceivedTrackStats(connection);
+    await new Promise(resolve => setTimeout(resolve, 1000)); 
+    const stats2 = await getReceivedTrackStats(connection);
+    return (stats2.framesReceived - stats1.framesReceived) * 1000 / (stats2.timestamp - stats1.timestamp);
+}
</ins></span></pre></div>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog   2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/ChangeLog      2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -1,3 +1,39 @@
</span><ins>+2020-04-20  Youenn Fablet  <youenn@apple.com>
+
+        Safari doesn't apply frameRate limit when request stream from Camera
+        https://bugs.webkit.org/show_bug.cgi?id=210186
+        <rdar://problem/61452794>
+
+        Reviewed by Eric Carlson.
+
+        Add support to RealtimeVideoSource to decimate the video samples based on the observed frame rate of its capture source.
+        This allows supporting two tracks using the same capture device, one track being low frame rate and the other one high frame rate.
+
+        Clean-up refactoring to make RealtimeVideoSource directly inherit from RealtimeVideoCaptureSource.
+        Migrate size and format of frame adaptation from RealtimeVideoCaptureSource to RealtimeVideoSource.
+        Fix mock capture source to update its frame rate when asked by applyConstraints.
+
+        Tests: fast/mediastream/mediastreamtrack-video-frameRate-clone-decreasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-clone-increasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-decreasing.html
+               fast/mediastream/mediastreamtrack-video-frameRate-increasing.html
+
+        * platform/mediastream/RealtimeVideoCaptureSource.cpp:
+        (WebCore::RealtimeVideoCaptureSource::dispatchMediaSampleToObservers):
+        (WebCore::RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate):
+        * platform/mediastream/RealtimeVideoCaptureSource.h:
+        (WebCore::RealtimeVideoCaptureSource::observedFrameRate const):
+        * platform/mediastream/RealtimeVideoSource.cpp:
+        (WebCore::RealtimeVideoSource::RealtimeVideoSource):
+        (WebCore::m_source):
+        (WebCore::RealtimeVideoSource::adaptVideoSample):
+        (WebCore::RealtimeVideoSource::videoSampleAvailable):
+        * platform/mediastream/RealtimeVideoSource.h:
+        * platform/mock/MockRealtimeVideoSource.cpp:
+        (WebCore::MockRealtimeVideoSource::setFrameRateWithPreset):
+        * testing/Internals.cpp:
+        (WebCore::Internals::observeMediaStreamTrack):
+
</ins><span class="cx"> 2020-04-20  Antoine Quint  <graouts@apple.com>
</span><span class="cx"> 
</span><span class="cx">         WebAnimations API doesn't properly apply keyframe easings to transforms
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamRealtimeVideoCaptureSourcecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp 2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.cpp    2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -34,10 +34,6 @@
</span><span class="cx"> #include "RemoteVideoSample.h"
</span><span class="cx"> #include <wtf/JSONValues.h>
</span><span class="cx"> 
</span><del>-#if PLATFORM(COCOA)
-#include "ImageTransferSessionVT.h"
-#endif
-
</del><span class="cx"> namespace WebCore {
</span><span class="cx"> 
</span><span class="cx"> RealtimeVideoCaptureSource::RealtimeVideoCaptureSource(String&& name, String&& id, String&& hashSalt)
</span><span class="lines">@@ -376,7 +372,7 @@
</span><span class="cx">     setFrameRate(match->requestedFrameRate);
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-RefPtr<MediaSample> RealtimeVideoCaptureSource::adaptVideoSample(MediaSample& sample)
</del><ins>+void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
</ins><span class="cx"> {
</span><span class="cx">     MediaTime sampleTime = sample.presentationTime();
</span><span class="cx"> 
</span><span class="lines">@@ -390,35 +386,9 @@
</span><span class="cx">     if (interval > 1)
</span><span class="cx">         m_observedFrameRate = (m_observedFrameTimeStamps.size() / interval);
</span><span class="cx"> 
</span><del>-    auto mediaSample = makeRefPtr(&sample);
-
-#if PLATFORM(COCOA)
-    auto size = this->size();
-    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
-
-        if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
-            m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
-
-        ASSERT(m_imageTransferSession);
-        if (m_imageTransferSession) {
-            mediaSample = m_imageTransferSession->convertMediaSample(sample, size);
-            if (!mediaSample) {
-                ASSERT_NOT_REACHED();
-                return nullptr;
-            }
-        }
-    }
-#endif
-
-    return mediaSample.releaseNonNull();
</del><ins>+    videoSampleAvailable(sample);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-void RealtimeVideoCaptureSource::dispatchMediaSampleToObservers(MediaSample& sample)
-{
-    if (auto mediaSample = adaptVideoSample(sample))
-        videoSampleAvailable(*mediaSample);
-}
-
</del><span class="cx"> void RealtimeVideoCaptureSource::clientUpdatedSizeAndFrameRate(Optional<int> width, Optional<int> height, Optional<double> frameRate)
</span><span class="cx"> {
</span><span class="cx">     // FIXME: We only change settings if capture resolution is below requested one. We should get the best preset for all clients.
</span><span class="lines">@@ -427,9 +397,10 @@
</span><span class="cx">         width = { };
</span><span class="cx">     if (height && *height < static_cast<int>(settings.height()))
</span><span class="cx">         height = { };
</span><ins>+    if (frameRate && *frameRate < static_cast<double>(settings.frameRate()))
+        frameRate = { };
</ins><span class="cx"> 
</span><del>-    // FIXME: handle frameRate potential increase.
-    if (!width && !height)
</del><ins>+    if (!width && !height && !frameRate)
</ins><span class="cx">         return;
</span><span class="cx"> 
</span><span class="cx">     auto match = bestSupportedSizeAndFrameRate(width, height, frameRate);
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamRealtimeVideoCaptureSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h   2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoCaptureSource.h      2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -48,6 +48,8 @@
</span><span class="cx">     virtual void generatePresets() = 0;
</span><span class="cx">     virtual MediaSample::VideoRotation sampleRotation() const { return MediaSample::VideoRotation::None; }
</span><span class="cx"> 
</span><ins>+    double observedFrameRate() const { return m_observedFrameRate; }
+
</ins><span class="cx"> protected:
</span><span class="cx">     RealtimeVideoCaptureSource(String&& name, String&& id, String&& hashSalt);
</span><span class="cx"> 
</span><span class="lines">@@ -69,11 +71,8 @@
</span><span class="cx"> 
</span><span class="cx">     void setDefaultSize(const IntSize& size) { m_defaultSize = size; }
</span><span class="cx"> 
</span><del>-    double observedFrameRate() const { return m_observedFrameRate; }
-
</del><span class="cx">     void dispatchMediaSampleToObservers(MediaSample&);
</span><span class="cx">     const Vector<IntSize>& standardVideoSizes();
</span><del>-    RefPtr<MediaSample> adaptVideoSample(MediaSample&);
</del><span class="cx"> 
</span><span class="cx"> private:
</span><span class="cx">     struct CaptureSizeAndFrameRate {
</span><span class="lines">@@ -93,9 +92,6 @@
</span><span class="cx">     Deque<double> m_observedFrameTimeStamps;
</span><span class="cx">     double m_observedFrameRate { 0 };
</span><span class="cx">     IntSize m_defaultSize;
</span><del>-#if PLATFORM(COCOA)
-    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
-#endif
</del><span class="cx"> };
</span><span class="cx"> 
</span><span class="cx"> struct SizeAndFrameRate {
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamRealtimeVideoSourcecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp        2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.cpp   2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -28,15 +28,20 @@
</span><span class="cx"> 
</span><span class="cx"> #if ENABLE(MEDIA_STREAM)
</span><span class="cx"> 
</span><ins>+#if PLATFORM(COCOA)
+#include "ImageTransferSessionVT.h"
+#endif
+
</ins><span class="cx"> namespace WebCore {
</span><span class="cx"> 
</span><span class="cx"> RealtimeVideoSource::RealtimeVideoSource(Ref<RealtimeVideoCaptureSource>&& source)
</span><del>-    : RealtimeVideoCaptureSource(String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
</del><ins>+    : RealtimeMediaSource(Type::Video, String { source->name() }, String { source->persistentID() }, String { source->deviceIDHashSalt() })
</ins><span class="cx">     , m_source(WTFMove(source))
</span><span class="cx"> {
</span><span class="cx">     m_source->addObserver(*this);
</span><span class="cx">     m_currentSettings = m_source->settings();
</span><span class="cx">     setSize(m_source->size());
</span><ins>+    setFrameRate(m_source->frameRate());
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> RealtimeVideoSource::~RealtimeVideoSource()
</span><span class="lines">@@ -142,13 +147,46 @@
</span><span class="cx">     });
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+#if PLATFORM(COCOA)
+RefPtr<MediaSample> RealtimeVideoSource::adaptVideoSample(MediaSample& sample)
+{
+    if (!m_imageTransferSession || m_imageTransferSession->pixelFormat() != sample.videoPixelFormat())
+        m_imageTransferSession = ImageTransferSessionVT::create(sample.videoPixelFormat());
+
+    ASSERT(m_imageTransferSession);
+    if (!m_imageTransferSession)
+        return nullptr;
+
+    auto mediaSample = m_imageTransferSession->convertMediaSample(sample, size());
+    ASSERT(mediaSample);
+
+    return mediaSample;
+}
+#endif
+
</ins><span class="cx"> void RealtimeVideoSource::videoSampleAvailable(MediaSample& sample)
</span><span class="cx"> {
</span><span class="cx">     if (!isProducingData())
</span><span class="cx">         return;
</span><span class="cx"> 
</span><del>-    if (auto mediaSample = adaptVideoSample(sample))
-        RealtimeMediaSource::videoSampleAvailable(*mediaSample);
</del><ins>+    if (m_frameDecimation > 1 && ++m_frameDecimationCounter % m_frameDecimation)
+        return;
+
+    m_frameDecimation = static_cast<size_t>(m_source->observedFrameRate() / frameRate());
+    if (!m_frameDecimation)
+        m_frameDecimation = 1;
+
+#if PLATFORM(COCOA)
+    auto size = this->size();
+    if (!size.isEmpty() && size != expandedIntSize(sample.presentationSize())) {
+        if (auto mediaSample = adaptVideoSample(sample)) {
+            RealtimeMediaSource::videoSampleAvailable(*mediaSample);
+            return;
+        }
+    }
+#endif
+
+    RealtimeMediaSource::videoSampleAvailable(sample);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> Ref<RealtimeMediaSource> RealtimeVideoSource::clone()
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamRealtimeVideoSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h  2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeVideoSource.h     2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -31,8 +31,9 @@
</span><span class="cx"> 
</span><span class="cx"> namespace WebCore {
</span><span class="cx"> 
</span><del>-// FIXME: Make RealtimeVideoSource derive from RealtimeMediaSource directly.
-class RealtimeVideoSource final : public RealtimeVideoCaptureSource, public RealtimeMediaSource::Observer {
</del><ins>+class ImageTransferSessionVT;
+
+class RealtimeVideoSource final : public RealtimeMediaSource, public RealtimeMediaSource::Observer {
</ins><span class="cx"> public:
</span><span class="cx">     static Ref<RealtimeVideoSource> create(Ref<RealtimeVideoCaptureSource>&& source) { return adoptRef(*new RealtimeVideoSource(WTFMove(source))); }
</span><span class="cx"> 
</span><span class="lines">@@ -51,7 +52,6 @@
</span><span class="cx"> 
</span><span class="cx">     const RealtimeMediaSourceCapabilities& capabilities() final { return m_source->capabilities(); }
</span><span class="cx">     const RealtimeMediaSourceSettings& settings() final { return m_currentSettings; }
</span><del>-    void generatePresets() final { m_source->generatePresets(); }
</del><span class="cx">     bool isCaptureSource() const final { return m_source->isCaptureSource(); }
</span><span class="cx">     CaptureDevice::DeviceType deviceType() const final { return m_source->deviceType(); }
</span><span class="cx">     void monitorOrientation(OrientationNotifier& notifier) final { m_source->monitorOrientation(notifier); }
</span><span class="lines">@@ -64,6 +64,10 @@
</span><span class="cx">     bool preventSourceFromStopping() final;
</span><span class="cx">     void videoSampleAvailable(MediaSample&) final;
</span><span class="cx"> 
</span><ins>+#if PLATFORM(COCOA)
+    RefPtr<MediaSample> adaptVideoSample(MediaSample&);
+#endif
+
</ins><span class="cx"> #if !RELEASE_LOG_DISABLED
</span><span class="cx">     void setLogger(const Logger&, const void*) final;
</span><span class="cx"> #endif
</span><span class="lines">@@ -70,6 +74,11 @@
</span><span class="cx"> 
</span><span class="cx">     Ref<RealtimeVideoCaptureSource> m_source;
</span><span class="cx">     RealtimeMediaSourceSettings m_currentSettings;
</span><ins>+#if PLATFORM(COCOA)
+    std::unique_ptr<ImageTransferSessionVT> m_imageTransferSession;
+#endif
+    size_t m_frameDecimation { 1 };
+    size_t m_frameDecimationCounter { 0 };
</ins><span class="cx"> #if !RELEASE_LOG_DISABLED
</span><span class="cx">     uint64_t m_cloneCounter { 0 };
</span><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmockMockRealtimeVideoSourcecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp   2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp      2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -187,11 +187,13 @@
</span><span class="cx">     return m_currentSettings.value();
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void MockRealtimeVideoSource::setFrameRateWithPreset(double, RefPtr<VideoPreset> preset)
</del><ins>+void MockRealtimeVideoSource::setFrameRateWithPreset(double frameRate, RefPtr<VideoPreset> preset)
</ins><span class="cx"> {
</span><span class="cx">     m_preset = WTFMove(preset);
</span><span class="cx">     if (m_preset)
</span><span class="cx">         setIntrinsicSize(m_preset->size);
</span><ins>+    if (isProducingData())
+        m_emitFrameTimer.startRepeating(1_s / frameRate);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> IntSize MockRealtimeVideoSource::captureSize() const
</span></span></pre></div>
<a id="trunkSourceWebCoretestingInternalscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/testing/Internals.cpp (260363 => 260364)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/testing/Internals.cpp       2020-04-20 12:29:23 UTC (rev 260363)
+++ trunk/Source/WebCore/testing/Internals.cpp  2020-04-20 14:37:47 UTC (rev 260364)
</span><span class="lines">@@ -5002,6 +5002,14 @@
</span><span class="cx"> 
</span><span class="cx"> void Internals::observeMediaStreamTrack(MediaStreamTrack& track)
</span><span class="cx"> {
</span><ins>+    if (m_trackSource) {
+        m_trackSource->removeObserver(*this);
+        m_trackSource->removeAudioSampleObserver(*this);
+
+        m_trackAudioSampleCount = 0;
+        m_trackVideoSampleCount = 0;
+    }
+
</ins><span class="cx">     m_trackSource = &track.source();
</span><span class="cx">     m_trackSource->addObserver(*this);
</span><span class="cx">     m_trackSource->addAudioSampleObserver(*this);
</span></span></pre>
</div>
</div>

</body>
</html>