<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[213880] trunk</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/213880">213880</a></dd>
<dt>Author</dt> <dd>eric.carlson@apple.com</dd>
<dt>Date</dt> <dd>2017-03-13 17:30:48 -0700 (Mon, 13 Mar 2017)</dd>
</dl>
<h3>Log Message</h3>
<pre>[MediaStream] Move paintCurrentFrameInContext from RealtimeMediaSources to MediaPlayer
https://bugs.webkit.org/show_bug.cgi?id=169474
<rdar://problem/30976747>
Reviewed by Youenn Fablet.
Source/WebCore:
Every video capture source has extremely similar code to render the current frame to
a graphics context. Because the media player gets every video sample buffer, have it
hang onto the most recent frame so it can implement paintCurrentFrameInContext directly.
Fix an existing race condition that occasionally caused the readyState to advance to
"have enough data" before a video was ready to paint by defining a MediaStreamTrackPrivate
readyState and observing that.
No new tests, covered by existing tests. These changes uncovered a bug in
fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html, which
was updated.
* Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
(WebCore::CanvasCaptureMediaStreamTrack::Source::captureCanvas):
(WebCore::CanvasCaptureMediaStreamTrack::Source::paintCurrentFrameInContext): Deleted.
(WebCore::CanvasCaptureMediaStreamTrack::Source::currentFrameImage): Deleted.
* Modules/mediastream/CanvasCaptureMediaStreamTrack.h:
* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
(-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
Drive-by change - don't pass status to parent callback, it is a property of the layer.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::isAvailable): Drive-by cleanup - we don't
use AVSampleBufferRenderSynchronizer so don't fail if it isn't available.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample): Hang onto new frame,
invalidate cached image, update readyState.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange): No more "updatePausedImage".
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayer): Drive-by cleanup - Add an early
return if there is no need for a layer.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::destroyLayer): renderingModeChanged -> updateRenderingMode.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::currentDisplayMode): Minor cleanup.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode): Renamed from renderingModeChanged,
add a bool return to signal when the mode changes.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::play): No more m_haveEverPlayed. Update display
mode immediately.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::pause): No more paused image.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::currentReadyState): Only return HaveNothing, HaveMetadata,
or HaveEnoughData. Don't return HaveEnoughData until all enabled tracks are providing data and never
drop back to HaveMetadata.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateRenderingMode): Renamed from renderingModeChanged.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::characteristicsChanged): Update intrinsic
size directly.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferUpdated): No more m_hasReceivedMedia.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::readyStateChanged): Ditto.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack): Reset imagePainter
when active video track changes.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage): Deleted.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext): Paint current
frame image.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::CurrentFramePainter::reset): New.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::shouldEnqueueVideoSampleBuffer): Deleted.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updatePausedImage): Deleted.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateIntrinsicSize): Deleted.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::renderingModeChanged): Deleted.
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::audioSamplesAvailable): Deleted.
* platform/mediastream/MediaStreamPrivate.cpp:
(WebCore::MediaStreamPrivate::paintCurrentFrameInContext): Deleted.
(WebCore::MediaStreamPrivate::currentFrameImage): Deleted.
* platform/mediastream/MediaStreamPrivate.h:
* platform/mediastream/MediaStreamTrackPrivate.cpp:
(WebCore::MediaStreamTrackPrivate::MediaStreamTrackPrivate):
(WebCore::MediaStreamTrackPrivate::endTrack): Update readyState.
(WebCore::MediaStreamTrackPrivate::clone): Clone readyState.
(WebCore::MediaStreamTrackPrivate::sourceStopped): Update readyState.
(WebCore::MediaStreamTrackPrivate::videoSampleAvailable): Ditto.
(WebCore::MediaStreamTrackPrivate::audioSamplesAvailable): Ditto.
(WebCore::MediaStreamTrackPrivate::updateReadyState): New, update readyState and notify observers.
(WebCore::MediaStreamTrackPrivate::paintCurrentFrameInContext): Deleted.
* platform/mediastream/MediaStreamTrackPrivate.h:
* platform/mediastream/MediaStreamTrackPrivate.cpp:
(WebCore::MediaStreamTrackPrivate::paintCurrentFrameInContext): Deleted.
* platform/mediastream/RealtimeMediaSource.h:
(WebCore::RealtimeMediaSource::currentFrameImage): Deleted.
(WebCore::RealtimeMediaSource::paintCurrentFrameInContext): Deleted.
* platform/mediastream/mac/AVMediaCaptureSource.mm:
(-[WebCoreAVMediaCaptureSourceObserver disconnect]): Drive-by fix - clear m_callback
after calling removeNotificationObservers.
(-[WebCoreAVMediaCaptureSourceObserver removeNotificationObservers]): Drive-by fix - remove
the correct listener.
(-[WebCoreAVMediaCaptureSourceObserver endSessionInterrupted:]):
* platform/mediastream/mac/AVVideoCaptureSource.h:
* platform/mediastream/mac/AVVideoCaptureSource.mm:
(WebCore::AVVideoCaptureSource::currentFrameImage): Deleted.
(WebCore::AVVideoCaptureSource::currentFrameCGImage): Deleted.
(WebCore::AVVideoCaptureSource::paintCurrentFrameInContext): Deleted.
* platform/mediastream/mac/RealtimeIncomingVideoSource.cpp:
(WebCore::drawImage): Deleted.
(WebCore::RealtimeIncomingVideoSource::currentFrameImage): Deleted.
(WebCore::RealtimeIncomingVideoSource::paintCurrentFrameInContext): Deleted.
* platform/mediastream/mac/RealtimeIncomingVideoSource.h:
* platform/mock/MockRealtimeVideoSource.cpp:
(WebCore::MockRealtimeVideoSource::paintCurrentFrameInContext): Deleted.
(WebCore::MockRealtimeVideoSource::currentFrameImage): Deleted.
* platform/mock/MockRealtimeVideoSource.h:
LayoutTests:
* fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt:
* fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html: Fix
bug uncovered by patch.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsChangeLog">trunk/LayoutTests/ChangeLog</a></li>
<li><a href="#trunkLayoutTestsfastmediastreamMediaStreamvideoelementvideotracksdisabledthenenabledexpectedtxt">trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastmediastreamMediaStreamvideoelementvideotracksdisabledthenenabledhtml">trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html</a></li>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCoreModulesmediastreamCanvasCaptureMediaStreamTrackcpp">trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp</a></li>
<li><a href="#trunkSourceWebCoreModulesmediastreamCanvasCaptureMediaStreamTrackh">trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.h</a></li>
<li><a href="#trunkSourceWebCoreplatformgraphicsavfoundationobjcMediaPlayerPrivateMediaStreamAVFObjCh">trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h</a></li>
<li><a href="#trunkSourceWebCoreplatformgraphicsavfoundationobjcMediaPlayerPrivateMediaStreamAVFObjCmm">trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamMediaStreamPrivatecpp">trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamMediaStreamPrivateh">trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamMediaStreamTrackPrivatecpp">trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamMediaStreamTrackPrivateh">trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreamRealtimeMediaSourceh">trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreammacAVMediaCaptureSourcemm">trunk/Source/WebCore/platform/mediastream/mac/AVMediaCaptureSource.mm</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreammacAVVideoCaptureSourceh">trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreammacAVVideoCaptureSourcemm">trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreammacRealtimeIncomingVideoSourcecpp">trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmediastreammacRealtimeIncomingVideoSourceh">trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.h</a></li>
<li><a href="#trunkSourceWebCoreplatformmockMockRealtimeVideoSourcecpp">trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformmockMockRealtimeVideoSourceh">trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/ChangeLog (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/ChangeLog 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/LayoutTests/ChangeLog 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -1,3 +1,15 @@
</span><ins>+2017-03-13 Eric Carlson <eric.carlson@apple.com>
+
+ [MediaStream] Move paintCurrentFrameInContext from RealtimeMediaSources to MediaPlayer
+ https://bugs.webkit.org/show_bug.cgi?id=169474
+ <rdar://problem/30976747>
+
+ Reviewed by Youenn Fablet.
+
+ * fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt:
+ * fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html: Fix
+ bug uncovered by patch.
+
</ins><span class="cx"> 2017-03-13 Ryan Haddad <ryanhaddad@apple.com>
</span><span class="cx">
</span><span class="cx"> Skip WebGPU tests on ios-simulator.
</span></span></pre></div>
<a id="trunkLayoutTestsfastmediastreamMediaStreamvideoelementvideotracksdisabledthenenabledexpectedtxt"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled-expected.txt 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -7,25 +7,26 @@
</span><span class="cx"> video.src = window.URL.createObjectURL(mediaStream)
</span><span class="cx">
</span><span class="cx"> === beginning round of pixel tests ===
</span><del>-PASS pixel was black
</del><ins>+PASS pixel was black.
</ins><span class="cx">
</span><span class="cx"> === all video tracks disabled ===
</span><span class="cx"> PASS pixel was black.
</span><span class="cx">
</span><del>- === video track reenabled ===
-PASS pixel was white.
</del><ins>+ === video track reenabled, should NOT render current frame ===
+PASS pixel was black.
</ins><span class="cx">
</span><span class="cx"> ===== play video =====
</span><span class="cx"> video.play()
</span><span class="cx">
</span><span class="cx"> === beginning round of pixel tests ===
</span><del>-PASS pixel was black
</del><ins>+PASS pixel was white.
</ins><span class="cx">
</span><span class="cx"> === all video tracks disabled ===
</span><span class="cx"> PASS pixel was black.
</span><span class="cx">
</span><del>- === video track reenabled ===
</del><ins>+ === video track reenabled, should render current frame ===
</ins><span class="cx"> PASS pixel was white.
</span><ins>+
</ins><span class="cx"> PASS successfullyParsed is true
</span><span class="cx">
</span><span class="cx"> TEST COMPLETE
</span></span></pre></div>
<a id="trunkLayoutTestsfastmediastreamMediaStreamvideoelementvideotracksdisabledthenenabledhtml"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/LayoutTests/fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -33,8 +33,13 @@
</span><span class="cx"> return pixel[0] === 255 && pixel[1] === 255 && pixel[2] === 255 && pixel[3] === 255;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- function attempt(numberOfTries, call, callback, successMessage)
</del><ins>+ function canvasShouldBeBlack()
</ins><span class="cx"> {
</span><ins>+ return !(mediaStream.getVideoTracks()[0].enabled && havePlayed);
+ }
+
+ function attempt(numberOfTries, call, callback)
+ {
</ins><span class="cx"> if (numberOfTries <= 0) {
</span><span class="cx"> testFailed('Pixel check did not succeed after multiple tries.');
</span><span class="cx"> return;
</span><span class="lines">@@ -42,13 +47,13 @@
</span><span class="cx">
</span><span class="cx"> let attemptSucceeded = call();
</span><span class="cx"> if (attemptSucceeded) {
</span><del>- testPassed(successMessage);
</del><ins>+ testPassed(canvasShouldBeBlack() ? 'pixel was black.' : 'pixel was white.');
</ins><span class="cx"> callback();
</span><span class="cx">
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- setTimeout(() => { attempt(--numberOfTries, call, callback, successMessage); }, 50);
</del><ins>+ setTimeout(() => { attempt(--numberOfTries, call, callback); }, 50);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function repeatWithVideoPlayingAndFinishTest()
</span><span class="lines">@@ -56,20 +61,23 @@
</span><span class="cx"> if (video.paused) {
</span><span class="cx"> debug('<br> ===== play video =====');
</span><span class="cx"> evalAndLog('video.play()');
</span><ins>+ havePlayed = true;
</ins><span class="cx"> beginTestRound();
</span><del>- } else
</del><ins>+ } else {
+ debug('');
</ins><span class="cx"> video.pause();
</span><span class="cx"> finishJSTest();
</span><ins>+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function reenableTrack()
</span><span class="cx"> {
</span><span class="cx"> mediaStream.getVideoTracks()[0].enabled = true;
</span><del>- debug('<br> === video track reenabled ===');
</del><ins>+ debug(`<br> === video track reenabled, should${havePlayed ? "" : " NOT"} render current frame ===`);
</ins><span class="cx">
</span><span class="cx"> // The video is not guaranteed to render non-black frames before the canvas is drawn to and the pixels are checked.
</span><span class="cx"> // A timeout is used to ensure that the pixel check is done after the video renders non-black frames.
</span><del>- attempt(10, checkPixels, repeatWithVideoPlayingAndFinishTest, 'pixel was white.');
</del><ins>+ attempt(10, checkPixels, repeatWithVideoPlayingAndFinishTest);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function checkPixels()
</span><span class="lines">@@ -76,14 +84,13 @@
</span><span class="cx"> {
</span><span class="cx"> context.clearRect(0, 0, canvas.width, canvas.height);
</span><span class="cx"> buffer = context.getImageData(30, 242, 1, 1).data;
</span><del>- if(!isPixelTransparent(buffer)) {
</del><ins>+ if (!isPixelTransparent(buffer))
</ins><span class="cx"> testFailed('pixel was not transparent after clearing canvas.');
</span><del>- }
</del><span class="cx">
</span><span class="cx"> context.drawImage(video, 0, 0, canvas.width, canvas.height);
</span><span class="cx"> buffer = context.getImageData(30, 242, 1, 1).data;
</span><span class="cx">
</span><del>- if (mediaStream.getVideoTracks()[0].enabled && havePlayed)
</del><ins>+ if (!canvasShouldBeBlack())
</ins><span class="cx"> return isPixelWhite(buffer);
</span><span class="cx"> else
</span><span class="cx"> return isPixelBlack(buffer);
</span><span class="lines">@@ -96,13 +103,13 @@
</span><span class="cx">
</span><span class="cx"> // The video is not guaranteed to render black frames before the canvas is drawn to and the pixels are checked.
</span><span class="cx"> // A timeout is used to ensure that the pixel check is done after the video renders black frames.
</span><del>- attempt(10, checkPixels, reenableTrack, 'pixel was black.');
</del><ins>+ attempt(10, checkPixels, reenableTrack);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function beginTestRound()
</span><span class="cx"> {
</span><span class="cx"> debug('<br> === beginning round of pixel tests ===');
</span><del>- attempt(1, checkPixels, disableAllTracks, 'pixel was black');
</del><ins>+ attempt(10, checkPixels, disableAllTracks);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function canplay()
</span></span></pre></div>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/ChangeLog 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -1,3 +1,112 @@
</span><ins>+2017-03-13 Eric Carlson <eric.carlson@apple.com>
+
+ [MediaStream] Move paintCurrentFrameInContext from RealtimeMediaSources to MediaPlayer
+ https://bugs.webkit.org/show_bug.cgi?id=169474
+ <rdar://problem/30976747>
+
+ Reviewed by Youenn Fablet.
+
+ Every video capture source has extremely similar code to render the current frame to
+ a graphics context. Because the media player gets every video sample buffer, have it
+ hang onto the most recent frame so it can implement paintCurrentFrameInContext directly.
+ Fix an existing race condition that occasionally caused the readyState to advance to
+ "have enough data" before a video was ready to paint by defining a MediaStreamTrackPrivate
+ readyState and observing that.
+
+ No new tests, covered by existing tests. These changes uncovered a bug in
+ fast/mediastream/MediaStream-video-element-video-tracks-disabled-then-enabled.html, which
+ was updated.
+
+ * Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp:
+ (WebCore::CanvasCaptureMediaStreamTrack::Source::captureCanvas):
+ (WebCore::CanvasCaptureMediaStreamTrack::Source::paintCurrentFrameInContext): Deleted.
+ (WebCore::CanvasCaptureMediaStreamTrack::Source::currentFrameImage): Deleted.
+ * Modules/mediastream/CanvasCaptureMediaStreamTrack.h:
+
+ * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
+ * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
+ (-[WebAVSampleBufferStatusChangeListener observeValueForKeyPath:ofObject:change:context:]):
+ Drive-by change - don't pass status to parent callback, it is a property of the layer.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::isAvailable): Drive-by cleanup - we don't
+ use AVSampleBufferRenderSynchronizer so don't fail if it isn't available.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::enqueueVideoSample): Hang onto new frame,
+ invalidate cached image, update readyState.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange): No more "updatePausedImage".
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::ensureLayer): Drive-by cleanup - Add an early
+ return if there is no need for a layer.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::destroyLayer): renderingModeChanged -> updateRenderingMode.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::currentDisplayMode): Minor cleanup.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode): Renamed from renderingModeChanged,
+ add a bool return to signal when the mode changes.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::play): No more m_haveEverPlayed. Update display
+ mode immediately.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::pause): No more paused image.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::currentReadyState): Only return HaveNothing, HaveMetadata,
+ or HaveEnoughData. Don't return HaveEnoughData until all enabled tracks are providing data and never
+ drop back to HaveMetadata.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateRenderingMode): Renamed from renderingModeChanged.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::characteristicsChanged): Update intrinsic
+ size directly.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::sampleBufferUpdated): No more m_hasReceivedMedia.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::readyStateChanged): Ditto.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::checkSelectedVideoTrack): Reset imagePainter
+ when active video track changes.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage): Deleted.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext): Paint current
+ frame image.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::CurrentFramePainter::reset): New.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::shouldEnqueueVideoSampleBuffer): Deleted.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updatePausedImage): Deleted.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateIntrinsicSize): Deleted.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::renderingModeChanged): Deleted.
+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::audioSamplesAvailable): Deleted.
+
+ * platform/mediastream/MediaStreamPrivate.cpp:
+ (WebCore::MediaStreamPrivate::paintCurrentFrameInContext): Deleted.
+ (WebCore::MediaStreamPrivate::currentFrameImage): Deleted.
+ * platform/mediastream/MediaStreamPrivate.h:
+
+ * platform/mediastream/MediaStreamTrackPrivate.cpp:
+ (WebCore::MediaStreamTrackPrivate::MediaStreamTrackPrivate):
+ (WebCore::MediaStreamTrackPrivate::endTrack): Update readyState.
+ (WebCore::MediaStreamTrackPrivate::clone): Clone readyState.
+ (WebCore::MediaStreamTrackPrivate::sourceStopped): Update readyState.
+ (WebCore::MediaStreamTrackPrivate::videoSampleAvailable): Ditto.
+ (WebCore::MediaStreamTrackPrivate::audioSamplesAvailable): Ditto.
+ (WebCore::MediaStreamTrackPrivate::updateReadyState): New, update readyState and notify observers.
+ (WebCore::MediaStreamTrackPrivate::paintCurrentFrameInContext): Deleted.
+ * platform/mediastream/MediaStreamTrackPrivate.h:
+
+ * platform/mediastream/MediaStreamTrackPrivate.cpp:
+ (WebCore::MediaStreamTrackPrivate::paintCurrentFrameInContext): Deleted.
+ * platform/mediastream/RealtimeMediaSource.h:
+ (WebCore::RealtimeMediaSource::currentFrameImage): Deleted.
+ (WebCore::RealtimeMediaSource::paintCurrentFrameInContext): Deleted.
+
+ * platform/mediastream/mac/AVMediaCaptureSource.mm:
+ (-[WebCoreAVMediaCaptureSourceObserver disconnect]): Drive-by fix - clear m_callback
+ after calling removeNotificationObservers.
+ (-[WebCoreAVMediaCaptureSourceObserver removeNotificationObservers]): Drive-by fix - remove
+ the correct listener.
+ (-[WebCoreAVMediaCaptureSourceObserver endSessionInterrupted:]):
+
+ * platform/mediastream/mac/AVVideoCaptureSource.h:
+ * platform/mediastream/mac/AVVideoCaptureSource.mm:
+ (WebCore::AVVideoCaptureSource::currentFrameImage): Deleted.
+ (WebCore::AVVideoCaptureSource::currentFrameCGImage): Deleted.
+ (WebCore::AVVideoCaptureSource::paintCurrentFrameInContext): Deleted.
+
+ * platform/mediastream/mac/RealtimeIncomingVideoSource.cpp:
+ (WebCore::drawImage): Deleted.
+ (WebCore::RealtimeIncomingVideoSource::currentFrameImage): Deleted.
+ (WebCore::RealtimeIncomingVideoSource::paintCurrentFrameInContext): Deleted.
+ * platform/mediastream/mac/RealtimeIncomingVideoSource.h:
+
+ * platform/mock/MockRealtimeVideoSource.cpp:
+ (WebCore::MockRealtimeVideoSource::paintCurrentFrameInContext): Deleted.
+ (WebCore::MockRealtimeVideoSource::currentFrameImage): Deleted.
+ * platform/mock/MockRealtimeVideoSource.h:
+
</ins><span class="cx"> 2017-03-13 Carlos Alberto Lopez Perez <clopez@igalia.com>
</span><span class="cx">
</span><span class="cx"> [GTK][SOUP] Fix build after r213877
</span></span></pre></div>
<a id="trunkSourceWebCoreModulesmediastreamCanvasCaptureMediaStreamTrackcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.cpp 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -142,9 +142,6 @@
</span><span class="cx"> if (!m_canvas->originClean())
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- // FIXME: This is probably not efficient.
- m_currentImage = m_canvas->copiedImage();
-
</del><span class="cx"> auto sample = m_canvas->toMediaSample();
</span><span class="cx"> if (!sample)
</span><span class="cx"> return;
</span><span class="lines">@@ -152,31 +149,6 @@
</span><span class="cx"> videoSampleAvailable(*sample);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void CanvasCaptureMediaStreamTrack::Source::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (!m_canvas)
- return;
-
- if (context.paintingDisabled())
- return;
-
- auto image = currentFrameImage();
- if (!image)
- return;
-
- FloatRect fullRect(0, 0, m_canvas->width(), m_canvas->height());
-
- GraphicsContextStateSaver stateSaver(context);
- context.setImageInterpolationQuality(InterpolationLow);
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- context.drawImage(*image, rect);
</del><span class="cx"> }
</span><span class="cx">
</span><del>-RefPtr<Image> CanvasCaptureMediaStreamTrack::Source::currentFrameImage()
-{
- return m_currentImage;
-}
-
-}
-
</del><span class="cx"> #endif // ENABLE(MEDIA_STREAM)
</span></span></pre></div>
<a id="trunkSourceWebCoreModulesmediastreamCanvasCaptureMediaStreamTrackh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/Modules/mediastream/CanvasCaptureMediaStreamTrack.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -64,8 +64,6 @@
</span><span class="cx"> bool isProducingData() const { return m_isProducingData; }
</span><span class="cx"> RefPtr<RealtimeMediaSourceCapabilities> capabilities() const final { return nullptr; }
</span><span class="cx"> const RealtimeMediaSourceSettings& settings() const final { return m_settings; }
</span><del>- RefPtr<Image> currentFrameImage() final;
- void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) final;
</del><span class="cx"> bool applySize(const IntSize&) final { return true; }
</span><span class="cx">
</span><span class="cx"> void captureCanvas();
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformgraphicsavfoundationobjcMediaPlayerPrivateMediaStreamAVFObjCh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -49,6 +49,7 @@
</span><span class="cx"> class AVVideoCaptureSource;
</span><span class="cx"> class Clock;
</span><span class="cx"> class MediaSourcePrivateClient;
</span><ins>+class PixelBufferConformerCV;
</ins><span class="cx"> class VideoTrackPrivateMediaStream;
</span><span class="cx">
</span><span class="cx"> #if PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE)
</span><span class="lines">@@ -77,7 +78,7 @@
</span><span class="cx"> void ensureLayer();
</span><span class="cx"> void destroyLayer();
</span><span class="cx">
</span><del>- void layerStatusDidChange(AVSampleBufferDisplayLayer*, NSNumber*);
</del><ins>+ void layerStatusDidChange(AVSampleBufferDisplayLayer*);
</ins><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> // MediaPlayerPrivateInterface
</span><span class="lines">@@ -135,7 +136,6 @@
</span><span class="cx"> MediaTime calculateTimelineOffset(const MediaSample&, double);
</span><span class="cx">
</span><span class="cx"> void enqueueVideoSample(MediaStreamTrackPrivate&, MediaSample&);
</span><del>- bool shouldEnqueueVideoSampleBuffer() const;
</del><span class="cx"> void flushAndRemoveVideoSampleBuffers();
</span><span class="cx"> void requestNotificationWhenReadyForVideoData();
</span><span class="cx">
</span><span class="lines">@@ -161,9 +161,8 @@
</span><span class="cx"> MediaPlayer::ReadyState currentReadyState();
</span><span class="cx"> void updateReadyState();
</span><span class="cx">
</span><del>- void updateIntrinsicSize(const FloatSize&);
</del><span class="cx"> void updateTracks();
</span><del>- void renderingModeChanged();
</del><ins>+ void updateRenderingMode();
</ins><span class="cx"> void checkSelectedVideoTrack();
</span><span class="cx">
</span><span class="cx"> void scheduleDeferredTask(Function<void ()>&&);
</span><span class="lines">@@ -175,8 +174,8 @@
</span><span class="cx"> LivePreview,
</span><span class="cx"> };
</span><span class="cx"> DisplayMode currentDisplayMode() const;
</span><del>- void updateDisplayMode();
- void updatePausedImage();
</del><ins>+ bool updateDisplayMode();
+ void updateCurrentFrameImage();
</ins><span class="cx">
</span><span class="cx"> // MediaStreamPrivate::Observer
</span><span class="cx"> void activeStatusChanged() override;
</span><span class="lines">@@ -190,7 +189,7 @@
</span><span class="cx"> void trackSettingsChanged(MediaStreamTrackPrivate&) override { };
</span><span class="cx"> void trackEnabledChanged(MediaStreamTrackPrivate&) override { };
</span><span class="cx"> void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) override;
</span><del>- void audioSamplesAvailable(MediaStreamTrackPrivate&) override;
</del><ins>+ void readyStateChanged(MediaStreamTrackPrivate&) override;
</ins><span class="cx">
</span><span class="cx"> #if PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE)
</span><span class="cx"> void setVideoFullscreenLayer(PlatformLayer*, std::function<void()> completionHandler) override;
</span><span class="lines">@@ -212,8 +211,17 @@
</span><span class="cx"> std::unique_ptr<Clock> m_clock;
</span><span class="cx">
</span><span class="cx"> MediaTime m_pausedTime;
</span><del>- RetainPtr<CGImageRef> m_pausedImage;
</del><span class="cx">
</span><ins>+ struct CurrentFramePainter {
+ CurrentFramePainter() = default;
+ void reset();
+
+ RetainPtr<CGImageRef> cgImage;
+ RefPtr<MediaSample> mediaSample;
+ std::unique_ptr<PixelBufferConformerCV> pixelBufferConformer;
+ };
+ CurrentFramePainter m_imagePainter;
+
</ins><span class="cx"> HashMap<String, RefPtr<AudioTrackPrivateMediaStreamCocoa>> m_audioTrackMap;
</span><span class="cx"> HashMap<String, RefPtr<VideoTrackPrivateMediaStream>> m_videoTrackMap;
</span><span class="cx"> PendingSampleQueue m_pendingVideoSampleQueue;
</span><span class="lines">@@ -220,17 +228,16 @@
</span><span class="cx">
</span><span class="cx"> MediaPlayer::NetworkState m_networkState { MediaPlayer::Empty };
</span><span class="cx"> MediaPlayer::ReadyState m_readyState { MediaPlayer::HaveNothing };
</span><ins>+ MediaPlayer::ReadyState m_previousReadyState { MediaPlayer::HaveNothing };
</ins><span class="cx"> FloatSize m_intrinsicSize;
</span><span class="cx"> float m_volume { 1 };
</span><span class="cx"> DisplayMode m_displayMode { None };
</span><span class="cx"> bool m_playing { false };
</span><span class="cx"> bool m_muted { false };
</span><del>- bool m_haveEverPlayed { false };
</del><span class="cx"> bool m_ended { false };
</span><span class="cx"> bool m_hasEverEnqueuedVideoFrame { false };
</span><del>- bool m_hasReceivedMedia { false };
- bool m_isFrameDisplayed { false };
</del><span class="cx"> bool m_pendingSelectedTrackCheck { false };
</span><ins>+ bool m_shouldDisplayFirstVideoFrame { false };
</ins><span class="cx">
</span><span class="cx"> #if PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE)
</span><span class="cx"> std::unique_ptr<VideoFullscreenLayerManager> m_videoFullscreenLayerManager;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformgraphicsavfoundationobjcMediaPlayerPrivateMediaStreamAVFObjCmm"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -31,11 +31,11 @@
</span><span class="cx"> #import "AVFoundationSPI.h"
</span><span class="cx"> #import "AudioTrackPrivateMediaStreamCocoa.h"
</span><span class="cx"> #import "Clock.h"
</span><del>-#import "CoreMediaSoftLink.h"
-#import "GraphicsContext.h"
</del><ins>+#import "GraphicsContextCG.h"
</ins><span class="cx"> #import "Logging.h"
</span><span class="cx"> #import "MediaStreamPrivate.h"
</span><span class="cx"> #import "MediaTimeAVFoundation.h"
</span><ins>+#import "PixelBufferConformerCV.h"
</ins><span class="cx"> #import "VideoTrackPrivateMediaStream.h"
</span><span class="cx"> #import <AVFoundation/AVSampleBufferDisplayLayer.h>
</span><span class="cx"> #import <QuartzCore/CALayer.h>
</span><span class="lines">@@ -50,14 +50,13 @@
</span><span class="cx">
</span><span class="cx"> #pragma mark - Soft Linking
</span><span class="cx">
</span><ins>+#import "CoreMediaSoftLink.h"
+#import "CoreVideoSoftLink.h"
+
</ins><span class="cx"> SOFT_LINK_FRAMEWORK_OPTIONAL(AVFoundation)
</span><span class="cx">
</span><span class="cx"> SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVSampleBufferDisplayLayer)
</span><del>-SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVSampleBufferRenderSynchronizer)
</del><span class="cx">
</span><del>-SOFT_LINK_CONSTANT(AVFoundation, AVAudioTimePitchAlgorithmSpectral, NSString*)
-SOFT_LINK_CONSTANT(AVFoundation, AVAudioTimePitchAlgorithmVarispeed, NSString*)
-
</del><span class="cx"> #define AVAudioTimePitchAlgorithmSpectral getAVAudioTimePitchAlgorithmSpectral()
</span><span class="cx"> #define AVAudioTimePitchAlgorithmVarispeed getAVAudioTimePitchAlgorithmVarispeed()
</span><span class="cx">
</span><span class="lines">@@ -134,11 +133,11 @@
</span><span class="cx"> ASSERT(_layers.contains(layer.get()));
</span><span class="cx"> ASSERT([keyPath isEqualToString:@"status"]);
</span><span class="cx">
</span><del>- callOnMainThread([protectedSelf = WTFMove(protectedSelf), layer = WTFMove(layer), status = WTFMove(status)] {
</del><ins>+ callOnMainThread([protectedSelf = WTFMove(protectedSelf), layer = WTFMove(layer)] {
</ins><span class="cx"> if (!protectedSelf->_parent)
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- protectedSelf->_parent->layerStatusDidChange(layer.get(), status.get());
</del><ins>+ protectedSelf->_parent->layerStatusDidChange(layer.get());
</ins><span class="cx"> });
</span><span class="cx">
</span><span class="cx"> } else
</span><span class="lines">@@ -201,11 +200,6 @@
</span><span class="cx"> if (!AVFoundationLibrary() || !isCoreMediaFrameworkAvailable() || !getAVSampleBufferDisplayLayerClass())
</span><span class="cx"> return false;
</span><span class="cx">
</span><del>-#if PLATFORM(MAC)
- if (!getAVSampleBufferRenderSynchronizerClass())
- return false;
-#endif
-
</del><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -281,9 +275,14 @@
</span><span class="cx"> if (&track != m_mediaStreamPrivate->activeVideoTrack())
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- m_hasReceivedMedia = true;
- updateReadyState();
- if (m_displayMode != LivePreview || (m_displayMode == PausedImage && m_isFrameDisplayed))
</del><ins>+ if (!m_imagePainter.mediaSample || m_displayMode != PausedImage) {
+ m_imagePainter.mediaSample = &sample;
+ m_imagePainter.cgImage = nullptr;
+ if (m_readyState < MediaPlayer::ReadyState::HaveEnoughData)
+ updateReadyState();
+ }
+
+ if (m_displayMode != LivePreview || (m_displayMode == PausedImage && m_imagePainter.mediaSample))
</ins><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> auto videoTrack = m_videoTrackMap.get(track.id());
</span><span class="lines">@@ -306,11 +305,8 @@
</span><span class="cx"> [m_sampleBufferDisplayLayer enqueueSampleBuffer:sample.platformSample().sample.cmSampleBuffer];
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_isFrameDisplayed = true;
</del><span class="cx"> if (!m_hasEverEnqueuedVideoFrame) {
</span><span class="cx"> m_hasEverEnqueuedVideoFrame = true;
</span><del>- if (m_displayMode == PausedImage)
- updatePausedImage();
</del><span class="cx"> m_player->firstVideoFrameAvailable();
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="lines">@@ -338,11 +334,12 @@
</span><span class="cx"> return nullptr;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange(AVSampleBufferDisplayLayer* layer, NSNumber* status)
</del><ins>+void MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange(AVSampleBufferDisplayLayer* layer)
</ins><span class="cx"> {
</span><del>- if (status.integerValue != AVQueuedSampleBufferRenderingStatusRendering)
</del><ins>+ LOG(Media, "MediaPlayerPrivateMediaStreamAVFObjC::layerStatusDidChange(%p) - status = %d", this, (int)layer.status);
+
+ if (layer.status != AVQueuedSampleBufferRenderingStatusRendering)
</ins><span class="cx"> return;
</span><del>-
</del><span class="cx"> if (!m_sampleBufferDisplayLayer || !m_activeVideoTrack || layer != m_sampleBufferDisplayLayer)
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="lines">@@ -357,21 +354,9 @@
</span><span class="cx"> [m_sampleBufferDisplayLayer flush];
</span><span class="cx"> }
</span><span class="cx">
</span><del>-bool MediaPlayerPrivateMediaStreamAVFObjC::shouldEnqueueVideoSampleBuffer() const
-{
- if (m_displayMode == LivePreview)
- return true;
-
- if (m_displayMode == PausedImage && !m_isFrameDisplayed)
- return true;
-
- return false;
-}
-
</del><span class="cx"> void MediaPlayerPrivateMediaStreamAVFObjC::flushAndRemoveVideoSampleBuffers()
</span><span class="cx"> {
</span><span class="cx"> [m_sampleBufferDisplayLayer flushAndRemoveImage];
</span><del>- m_isFrameDisplayed = false;
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MediaPlayerPrivateMediaStreamAVFObjC::ensureLayer()
</span><span class="lines">@@ -379,7 +364,15 @@
</span><span class="cx"> if (m_sampleBufferDisplayLayer)
</span><span class="cx"> return;
</span><span class="cx">
</span><ins>+ if (!m_mediaStreamPrivate || !m_mediaStreamPrivate->activeVideoTrack() || !m_mediaStreamPrivate->activeVideoTrack()->enabled())
+ return;
+
</ins><span class="cx"> m_sampleBufferDisplayLayer = adoptNS([allocAVSampleBufferDisplayLayerInstance() init]);
</span><ins>+ if (!m_sampleBufferDisplayLayer) {
+ LOG_ERROR("MediaPlayerPrivateMediaStreamAVFObjC::ensureLayers: +[AVSampleBufferDisplayLayer alloc] failed.");
+ return;
+ }
+
</ins><span class="cx"> #ifndef NDEBUG
</span><span class="cx"> [m_sampleBufferDisplayLayer setName:@"MediaPlayerPrivateMediaStreamAVFObjC AVSampleBufferDisplayLayer"];
</span><span class="cx"> #endif
</span><span class="lines">@@ -386,7 +379,7 @@
</span><span class="cx"> m_sampleBufferDisplayLayer.get().backgroundColor = cachedCGColor(Color::black);
</span><span class="cx"> [m_statusChangeListener beginObservingLayer:m_sampleBufferDisplayLayer.get()];
</span><span class="cx">
</span><del>- renderingModeChanged();
</del><ins>+ updateRenderingMode();
</ins><span class="cx">
</span><span class="cx"> #if PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE)
</span><span class="cx"> m_videoFullscreenLayerManager->setVideoLayer(m_sampleBufferDisplayLayer.get(), snappedIntRect(m_player->client().mediaPlayerContentBoxRect()).size());
</span><span class="lines">@@ -405,7 +398,7 @@
</span><span class="cx"> [m_sampleBufferDisplayLayer flush];
</span><span class="cx"> }
</span><span class="cx">
</span><del>- renderingModeChanged();
</del><ins>+ updateRenderingMode();
</ins><span class="cx">
</span><span class="cx"> #if PLATFORM(MAC) && ENABLE(VIDEO_PRESENTATION_MODE)
</span><span class="cx"> m_videoFullscreenLayerManager->didDestroyVideoLayer();
</span><span class="lines">@@ -480,8 +473,10 @@
</span><span class="cx"> if (m_ended || m_intrinsicSize.isEmpty() || !metaDataAvailable() || !m_sampleBufferDisplayLayer)
</span><span class="cx"> return None;
</span><span class="cx">
</span><del>- if (m_mediaStreamPrivate->activeVideoTrack() && !m_mediaStreamPrivate->activeVideoTrack()->enabled())
- return PaintItBlack;
</del><ins>+ if (auto* track = m_mediaStreamPrivate->activeVideoTrack()) {
+ if (!m_shouldDisplayFirstVideoFrame || !track->enabled() || track->muted())
+ return PaintItBlack;
+ }
</ins><span class="cx">
</span><span class="cx"> if (m_playing) {
</span><span class="cx"> if (!m_mediaStreamPrivate->isProducingData())
</span><span class="lines">@@ -492,30 +487,19 @@
</span><span class="cx"> return PausedImage;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode()
</del><ins>+bool MediaPlayerPrivateMediaStreamAVFObjC::updateDisplayMode()
</ins><span class="cx"> {
</span><span class="cx"> DisplayMode displayMode = currentDisplayMode();
</span><span class="cx">
</span><span class="cx"> if (displayMode == m_displayMode)
</span><del>- return;
</del><ins>+ return false;
+
</ins><span class="cx"> m_displayMode = displayMode;
</span><span class="cx">
</span><span class="cx"> if (m_displayMode < PausedImage && m_sampleBufferDisplayLayer)
</span><span class="cx"> flushAndRemoveVideoSampleBuffers();
</span><del>-}
</del><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::updatePausedImage()
-{
- if (m_displayMode < PausedImage)
- return;
-
- RefPtr<Image> image = m_mediaStreamPrivate->currentFrameImage();
- ASSERT(image);
- if (!image)
- return;
-
- m_pausedImage = image->nativeImage();
- ASSERT(m_pausedImage);
</del><ins>+ return true;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MediaPlayerPrivateMediaStreamAVFObjC::play()
</span><span class="lines">@@ -532,9 +516,10 @@
</span><span class="cx"> for (const auto& track : m_audioTrackMap.values())
</span><span class="cx"> track->play();
</span><span class="cx">
</span><del>- m_haveEverPlayed = true;
</del><ins>+ m_shouldDisplayFirstVideoFrame = true;
+ updateDisplayMode();
+
</ins><span class="cx"> scheduleDeferredTask([this] {
</span><del>- updateDisplayMode();
</del><span class="cx"> updateReadyState();
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="lines">@@ -553,7 +538,6 @@
</span><span class="cx"> track->pause();
</span><span class="cx">
</span><span class="cx"> updateDisplayMode();
</span><del>- updatePausedImage();
</del><span class="cx"> flushRenderers();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -632,20 +616,26 @@
</span><span class="cx">
</span><span class="cx"> MediaPlayer::ReadyState MediaPlayerPrivateMediaStreamAVFObjC::currentReadyState()
</span><span class="cx"> {
</span><del>- if (!m_mediaStreamPrivate)
</del><ins>+ if (!m_mediaStreamPrivate || !m_mediaStreamPrivate->active() || !m_mediaStreamPrivate->tracks().size())
</ins><span class="cx"> return MediaPlayer::ReadyState::HaveNothing;
</span><span class="cx">
</span><del>- // https://w3c.github.io/mediacapture-main/ Change 8. from July 4, 2013.
- // FIXME: Only update readyState to HAVE_ENOUGH_DATA when all active tracks have sent a sample buffer.
- if (m_mediaStreamPrivate->active() && m_hasReceivedMedia)
- return MediaPlayer::ReadyState::HaveEnoughData;
</del><ins>+ bool allTracksAreLive = true;
+ for (auto& track : m_mediaStreamPrivate->tracks()) {
+ if (!track->enabled() || track->readyState() != MediaStreamTrackPrivate::ReadyState::Live) {
+ allTracksAreLive = false;
+ break;
+ }
</ins><span class="cx">
</span><del>- updateDisplayMode();
</del><ins>+ if (track == m_mediaStreamPrivate->activeVideoTrack() && !m_imagePainter.mediaSample) {
+ allTracksAreLive = false;
+ break;
+ }
+ }
</ins><span class="cx">
</span><del>- if (m_displayMode == PausedImage)
- return MediaPlayer::ReadyState::HaveCurrentData;
</del><ins>+ if (!allTracksAreLive && m_previousReadyState == MediaPlayer::ReadyState::HaveNothing)
+ return MediaPlayer::ReadyState::HaveMetadata;
</ins><span class="cx">
</span><del>- return MediaPlayer::ReadyState::HaveMetadata;
</del><ins>+ return MediaPlayer::ReadyState::HaveEnoughData;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MediaPlayerPrivateMediaStreamAVFObjC::updateReadyState()
</span><span class="lines">@@ -674,17 +664,11 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::updateIntrinsicSize(const FloatSize& size)
</del><ins>+void MediaPlayerPrivateMediaStreamAVFObjC::updateRenderingMode()
</ins><span class="cx"> {
</span><del>- if (size == m_intrinsicSize)
</del><ins>+ if (!updateDisplayMode())
</ins><span class="cx"> return;
</span><span class="cx">
</span><del>- m_intrinsicSize = size;
-}
-
-void MediaPlayerPrivateMediaStreamAVFObjC::renderingModeChanged()
-{
- updateDisplayMode();
</del><span class="cx"> scheduleDeferredTask([this] {
</span><span class="cx"> if (m_player)
</span><span class="cx"> m_player->client().mediaPlayerRenderingModeChanged(m_player);
</span><span class="lines">@@ -698,7 +682,7 @@
</span><span class="cx">
</span><span class="cx"> FloatSize intrinsicSize = m_mediaStreamPrivate->intrinsicSize();
</span><span class="cx"> if (intrinsicSize.height() != m_intrinsicSize.height() || intrinsicSize.width() != m_intrinsicSize.width()) {
</span><del>- updateIntrinsicSize(intrinsicSize);
</del><ins>+ m_intrinsicSize = intrinsicSize;
</ins><span class="cx"> sizeChanged = true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -734,12 +718,7 @@
</span><span class="cx"> ASSERT(mediaSample.platformSample().type == PlatformSample::CMSampleBufferType);
</span><span class="cx"> ASSERT(m_mediaStreamPrivate);
</span><span class="cx">
</span><del>- if (!m_hasReceivedMedia) {
- m_hasReceivedMedia = true;
- updateReadyState();
- }
-
- if (!m_playing || streamTime().toDouble() < 0)
</del><ins>+ if (streamTime().toDouble() < 0)
</ins><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> switch (track.type()) {
</span><span class="lines">@@ -755,12 +734,8 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::audioSamplesAvailable(MediaStreamTrackPrivate&)
</del><ins>+void MediaPlayerPrivateMediaStreamAVFObjC::readyStateChanged(MediaStreamTrackPrivate&)
</ins><span class="cx"> {
</span><del>- if (m_hasReceivedMedia)
- return;
- m_hasReceivedMedia = true;
-
</del><span class="cx"> scheduleDeferredTask([this] {
</span><span class="cx"> updateReadyState();
</span><span class="cx"> });
</span><span class="lines">@@ -834,6 +809,7 @@
</span><span class="cx">
</span><span class="cx"> m_pendingSelectedTrackCheck = true;
</span><span class="cx"> scheduleDeferredTask([this] {
</span><ins>+ auto oldVideoTrack = m_activeVideoTrack;
</ins><span class="cx"> bool hideVideoLayer = true;
</span><span class="cx"> m_activeVideoTrack = nullptr;
</span><span class="cx"> if (m_mediaStreamPrivate->activeVideoTrack()) {
</span><span class="lines">@@ -847,9 +823,12 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ if (oldVideoTrack != m_activeVideoTrack)
+ m_imagePainter.reset();
</ins><span class="cx"> ensureLayer();
</span><span class="cx"> m_sampleBufferDisplayLayer.get().hidden = hideVideoLayer;
</span><span class="cx"> m_pendingSelectedTrackCheck = false;
</span><ins>+ updateDisplayMode();
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -914,25 +893,40 @@
</span><span class="cx"> paintCurrentFrameInContext(context, rect);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
</del><ins>+void MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage()
</ins><span class="cx"> {
</span><ins>+ if (m_imagePainter.cgImage || !m_imagePainter.mediaSample)
+ return;
+
+ if (!m_imagePainter.pixelBufferConformer)
+ m_imagePainter.pixelBufferConformer = std::make_unique<PixelBufferConformerCV>((CFDictionaryRef)@{ (NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA) });
+
+ ASSERT(m_imagePainter.pixelBufferConformer);
+ if (!m_imagePainter.pixelBufferConformer)
+ return;
+
+ auto pixelBuffer = static_cast<CVPixelBufferRef>(CMSampleBufferGetImageBuffer(m_imagePainter.mediaSample->platformSample().sample.cmSampleBuffer));
+ m_imagePainter.cgImage = m_imagePainter.pixelBufferConformer->createImageFromPixelBuffer(pixelBuffer);
+}
+
+void MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& destRect)
+{
</ins><span class="cx"> if (m_displayMode == None || !metaDataAvailable() || context.paintingDisabled())
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- if (m_displayMode == LivePreview)
- m_mediaStreamPrivate->paintCurrentFrameInContext(context, rect);
- else {
- GraphicsContextStateSaver stateSaver(context);
- context.translate(rect.x(), rect.y() + rect.height());
- context.scale(FloatSize(1, -1));
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- context.setImageInterpolationQuality(InterpolationLow);
</del><ins>+ GraphicsContextStateSaver stateSaver(context);
</ins><span class="cx">
</span><del>- if (m_displayMode == PausedImage && m_pausedImage)
- CGContextDrawImage(context.platformContext(), CGRectMake(0, 0, paintRect.width(), paintRect.height()), m_pausedImage.get());
- else
- context.fillRect(paintRect, Color::black);
</del><ins>+ if (m_displayMode != PaintItBlack && m_imagePainter.mediaSample)
+ updateCurrentFrameImage();
+
+ if (m_displayMode == PaintItBlack || !m_imagePainter.cgImage || !m_imagePainter.mediaSample) {
+ context.fillRect(IntRect(IntPoint(), IntSize(destRect.width(), destRect.height())), Color::black);
+ return;
</ins><span class="cx"> }
</span><ins>+
+ auto image = m_imagePainter.cgImage.get();
+ FloatRect imageRect(0, 0, CGImageGetWidth(image), CGImageGetHeight(image));
+ context.drawNativeImage(image, imageRect.size(), destRect, imageRect);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MediaPlayerPrivateMediaStreamAVFObjC::acceleratedRenderingStateChanged()
</span><span class="lines">@@ -959,6 +953,7 @@
</span><span class="cx"> if (m_readyState == readyState)
</span><span class="cx"> return;
</span><span class="cx">
</span><ins>+ m_previousReadyState = m_readyState;
</ins><span class="cx"> m_readyState = readyState;
</span><span class="cx"> characteristicsChanged();
</span><span class="cx">
</span><span class="lines">@@ -985,6 +980,13 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void MediaPlayerPrivateMediaStreamAVFObjC::CurrentFramePainter::reset()
+{
+ cgImage = nullptr;
+ mediaSample = nullptr;
+ pixelBufferConformer = nullptr;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+}
+
</ins><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamMediaStreamPrivatecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.cpp (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.cpp 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.cpp 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -239,30 +239,6 @@
</span><span class="cx"> return size;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaStreamPrivate::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (context.paintingDisabled())
- return;
-
- if (active() && m_activeVideoTrack)
- m_activeVideoTrack->paintCurrentFrameInContext(context, rect);
- else {
- GraphicsContextStateSaver stateSaver(context);
- context.translate(rect.x(), rect.y() + rect.height());
- context.scale(FloatSize(1, -1));
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- context.fillRect(paintRect, Color::black);
- }
-}
-
-RefPtr<Image> MediaStreamPrivate::currentFrameImage()
-{
- if (!active() || !m_activeVideoTrack)
- return nullptr;
-
- return m_activeVideoTrack->source().currentFrameImage();
-}
-
</del><span class="cx"> void MediaStreamPrivate::updateActiveVideoTrack()
</span><span class="cx"> {
</span><span class="cx"> m_activeVideoTrack = nullptr;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamMediaStreamPrivateh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/MediaStreamPrivate.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -93,9 +93,6 @@
</span><span class="cx"> void stopProducingData();
</span><span class="cx"> bool isProducingData() const;
</span><span class="cx">
</span><del>- RefPtr<Image> currentFrameImage();
- void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);
-
</del><span class="cx"> bool hasVideo() const;
</span><span class="cx"> bool hasAudio() const;
</span><span class="cx"> bool muted() const;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamMediaStreamTrackPrivatecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.cpp 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -49,8 +49,6 @@
</span><span class="cx"> MediaStreamTrackPrivate::MediaStreamTrackPrivate(Ref<RealtimeMediaSource>&& source, String&& id)
</span><span class="cx"> : m_source(WTFMove(source))
</span><span class="cx"> , m_id(WTFMove(id))
</span><del>- , m_isEnabled(true)
- , m_isEnded(false)
</del><span class="cx"> {
</span><span class="cx"> m_source->addObserver(*this);
</span><span class="cx"> }
</span><span class="lines">@@ -115,6 +113,7 @@
</span><span class="cx"> // only track using the source and it does stop, we will only call each observer's
</span><span class="cx"> // trackEnded method once.
</span><span class="cx"> m_isEnded = true;
</span><ins>+ updateReadyState();
</ins><span class="cx">
</span><span class="cx"> m_source->requestStop(this);
</span><span class="cx">
</span><span class="lines">@@ -127,6 +126,7 @@
</span><span class="cx"> auto clonedMediaStreamTrackPrivate = create(m_source.copyRef());
</span><span class="cx"> clonedMediaStreamTrackPrivate->m_isEnabled = this->m_isEnabled;
</span><span class="cx"> clonedMediaStreamTrackPrivate->m_isEnded = this->m_isEnded;
</span><ins>+ clonedMediaStreamTrackPrivate->updateReadyState();
</ins><span class="cx">
</span><span class="cx"> return clonedMediaStreamTrackPrivate;
</span><span class="cx"> }
</span><span class="lines">@@ -146,21 +146,6 @@
</span><span class="cx"> return m_source->capabilities();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MediaStreamTrackPrivate::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (context.paintingDisabled() || m_source->type() != RealtimeMediaSource::Type::Video || ended())
- return;
-
- if (!m_source->muted())
- m_source->paintCurrentFrameInContext(context, rect);
- else {
- GraphicsContextStateSaver stateSaver(context);
- context.translate(rect.x(), rect.y() + rect.height());
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- context.fillRect(paintRect, Color::black);
- }
-}
-
</del><span class="cx"> void MediaStreamTrackPrivate::applyConstraints(const MediaConstraints& constraints, RealtimeMediaSource::SuccessHandler successHandler, RealtimeMediaSource::FailureHandler failureHandler)
</span><span class="cx"> {
</span><span class="cx"> m_source->applyConstraints(constraints, successHandler, failureHandler);
</span><span class="lines">@@ -177,6 +162,7 @@
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> m_isEnded = true;
</span><ins>+ updateReadyState();
</ins><span class="cx">
</span><span class="cx"> for (auto& observer : m_observers)
</span><span class="cx"> observer->trackEnded(*this);
</span><span class="lines">@@ -208,6 +194,11 @@
</span><span class="cx">
</span><span class="cx"> void MediaStreamTrackPrivate::videoSampleAvailable(MediaSample& mediaSample)
</span><span class="cx"> {
</span><ins>+ if (!m_haveProducedData) {
+ m_haveProducedData = true;
+ updateReadyState();
+ }
+
</ins><span class="cx"> mediaSample.setTrackID(id());
</span><span class="cx"> for (auto& observer : m_observers)
</span><span class="cx"> observer->sampleBufferUpdated(*this, mediaSample);
</span><span class="lines">@@ -215,10 +206,33 @@
</span><span class="cx">
</span><span class="cx"> void MediaStreamTrackPrivate::audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)
</span><span class="cx"> {
</span><ins>+ if (!m_haveProducedData) {
+ m_haveProducedData = true;
+ updateReadyState();
+ }
+
</ins><span class="cx"> for (auto& observer : m_observers)
</span><span class="cx"> observer->audioSamplesAvailable(*this);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+
+void MediaStreamTrackPrivate::updateReadyState()
+{
+ ReadyState state = ReadyState::None;
+
+ if (m_isEnded)
+ state = ReadyState::Ended;
+ else if (m_haveProducedData)
+ state = ReadyState::Live;
+
+ if (state == m_readyState)
+ return;
+
+ m_readyState = state;
+ for (auto& observer : m_observers)
+ observer->readyStateChanged(*this);
+}
+
</ins><span class="cx"> } // namespace WebCore
</span><span class="cx">
</span><span class="cx"> #endif // ENABLE(MEDIA_STREAM)
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamMediaStreamTrackPrivateh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/MediaStreamTrackPrivate.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -49,6 +49,7 @@
</span><span class="cx"> virtual void trackEnabledChanged(MediaStreamTrackPrivate&) = 0;
</span><span class="cx"> virtual void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) { };
</span><span class="cx"> virtual void audioSamplesAvailable(MediaStreamTrackPrivate&) { };
</span><ins>+ virtual void readyStateChanged(MediaStreamTrackPrivate&) { };
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> static Ref<MediaStreamTrackPrivate> create(Ref<RealtimeMediaSource>&&);
</span><span class="lines">@@ -93,6 +94,9 @@
</span><span class="cx">
</span><span class="cx"> void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);
</span><span class="cx">
</span><ins>+ enum class ReadyState { None, Live, Ended };
+ ReadyState readyState() const { return m_readyState; }
+
</ins><span class="cx"> private:
</span><span class="cx"> MediaStreamTrackPrivate(Ref<RealtimeMediaSource>&&, String&& id);
</span><span class="cx">
</span><span class="lines">@@ -105,12 +109,16 @@
</span><span class="cx"> void videoSampleAvailable(MediaSample&) final;
</span><span class="cx"> void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
</span><span class="cx">
</span><ins>+ void updateReadyState();
+
</ins><span class="cx"> Vector<Observer*> m_observers;
</span><span class="cx"> Ref<RealtimeMediaSource> m_source;
</span><span class="cx">
</span><span class="cx"> String m_id;
</span><del>- bool m_isEnabled;
- bool m_isEnded;
</del><ins>+ ReadyState m_readyState { ReadyState::None };
+ bool m_isEnabled { true };
+ bool m_isEnded { false };
+ bool m_haveProducedData { false };
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> typedef Vector<RefPtr<MediaStreamTrackPrivate>> MediaStreamTrackPrivateVector;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreamRealtimeMediaSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/RealtimeMediaSource.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -141,9 +141,6 @@
</span><span class="cx">
</span><span class="cx"> virtual AudioSourceProvider* audioSourceProvider() { return nullptr; }
</span><span class="cx">
</span><del>- virtual RefPtr<Image> currentFrameImage() { return nullptr; }
- virtual void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) { }
-
</del><span class="cx"> void setWidth(int);
</span><span class="cx"> void setHeight(int);
</span><span class="cx"> const IntSize& size() const { return m_size; }
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreammacAVMediaCaptureSourcemm"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/mac/AVMediaCaptureSource.mm (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/mac/AVMediaCaptureSource.mm 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/mac/AVMediaCaptureSource.mm 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -347,8 +347,8 @@
</span><span class="cx"> - (void)disconnect
</span><span class="cx"> {
</span><span class="cx"> [NSObject cancelPreviousPerformRequestsWithTarget:self];
</span><del>- m_callback = 0;
</del><span class="cx"> [self removeNotificationObservers];
</span><ins>+ m_callback = nullptr;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> - (void)addNotificationObservers
</span><span class="lines">@@ -368,8 +368,7 @@
</span><span class="cx"> - (void)removeNotificationObservers
</span><span class="cx"> {
</span><span class="cx"> #if PLATFORM(IOS)
</span><del>- ASSERT(m_callback);
- [[NSNotificationCenter defaultCenter] removeObserver:m_callback->session()];
</del><ins>+ [[NSNotificationCenter defaultCenter] removeObserver:self];
</ins><span class="cx"> #endif
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -426,7 +425,7 @@
</span><span class="cx">
</span><span class="cx"> - (void)endSessionInterrupted:(NSNotification*)notification
</span><span class="cx"> {
</span><del>- LOG(Media, "WebCoreAVMediaCaptureSourceObserver::endSessionInterrupted(%p) ", self);
</del><ins>+ LOG(Media, "WebCoreAVMediaCaptureSourceObserver::endSessionInterrupted(%p)", self);
</ins><span class="cx">
</span><span class="cx"> if (m_callback)
</span><span class="cx"> m_callback->captureSessionEndInterruption(notification);
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreammacAVVideoCaptureSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -77,11 +77,6 @@
</span><span class="cx"> void captureOutputDidOutputSampleBufferFromConnection(AVCaptureOutput*, CMSampleBufferRef, AVCaptureConnection*) final;
</span><span class="cx"> void processNewFrame(RetainPtr<CMSampleBufferRef>);
</span><span class="cx">
</span><del>- void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) final;
-
- RetainPtr<CGImageRef> currentFrameCGImage();
- RefPtr<Image> currentFrameImage() final;
-
</del><span class="cx"> RetainPtr<NSString> m_pendingPreset;
</span><span class="cx"> RetainPtr<CMSampleBufferRef> m_buffer;
</span><span class="cx"> RetainPtr<CGImageRef> m_lastImage;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreammacAVVideoCaptureSourcemm"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -103,7 +103,11 @@
</span><span class="cx">
</span><span class="cx"> namespace WebCore {
</span><span class="cx">
</span><ins>+#if PLATFORM(MAC)
</ins><span class="cx"> const OSType videoCaptureFormat = kCVPixelFormatType_420YpCbCr8Planar;
</span><ins>+#else
+const OSType videoCaptureFormat = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange;
+#endif
</ins><span class="cx">
</span><span class="cx"> RefPtr<AVMediaCaptureSource> AVVideoCaptureSource::create(AVCaptureDeviceTypedef* device, const AtomicString& id, const MediaConstraints* constraints, String& invalidConstraint)
</span><span class="cx"> {
</span><span class="lines">@@ -436,58 +440,6 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-RefPtr<Image> AVVideoCaptureSource::currentFrameImage()
-{
- if (!currentFrameCGImage())
- return nullptr;
-
- FloatRect imageRect(0, 0, m_width, m_height);
- std::unique_ptr<ImageBuffer> imageBuffer = ImageBuffer::create(imageRect.size(), Unaccelerated);
-
- if (!imageBuffer)
- return nullptr;
-
- paintCurrentFrameInContext(imageBuffer->context(), imageRect);
-
- return ImageBuffer::sinkIntoImage(WTFMove(imageBuffer));
-}
-
-RetainPtr<CGImageRef> AVVideoCaptureSource::currentFrameCGImage()
-{
- if (m_lastImage)
- return m_lastImage;
-
- if (!m_buffer)
- return nullptr;
-
- CVPixelBufferRef pixelBuffer = static_cast<CVPixelBufferRef>(CMSampleBufferGetImageBuffer(m_buffer.get()));
- ASSERT(CVPixelBufferGetPixelFormatType(pixelBuffer) == videoCaptureFormat);
-
- if (!m_pixelBufferConformer)
- m_pixelBufferConformer = std::make_unique<PixelBufferConformerCV>((CFDictionaryRef)@{ (NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA) });
-
- ASSERT(m_pixelBufferConformer);
- if (!m_pixelBufferConformer)
- return nullptr;
-
- m_lastImage = m_pixelBufferConformer->createImageFromPixelBuffer(pixelBuffer);
-
- return m_lastImage;
-}
-
-void AVVideoCaptureSource::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (context.paintingDisabled() || !currentFrameCGImage())
- return;
-
- GraphicsContextStateSaver stateSaver(context);
- context.translate(rect.x(), rect.y() + rect.height());
- context.scale(FloatSize(1, -1));
- context.setImageInterpolationQuality(InterpolationLow);
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- CGContextDrawImage(context.platformContext(), CGRectMake(0, 0, paintRect.width(), paintRect.height()), m_lastImage.get());
-}
-
</del><span class="cx"> NSString* AVVideoCaptureSource::bestSessionPresetForVideoDimensions(std::optional<int> width, std::optional<int> height) const
</span><span class="cx"> {
</span><span class="cx"> if (!width && !height)
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreammacRealtimeIncomingVideoSourcecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.cpp (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.cpp 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.cpp 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -150,52 +150,6 @@
</span><span class="cx"> videoSampleAvailable(MediaSampleAVFObjC::create(sample));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static inline void drawImage(ImageBuffer& imageBuffer, CGImageRef image, const FloatRect& rect)
-{
- auto& context = imageBuffer.context();
- GraphicsContextStateSaver stateSaver(context);
- context.translate(rect.x(), rect.y() + rect.height());
- context.scale(FloatSize(1, -1));
- context.setImageInterpolationQuality(InterpolationLow);
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- CGContextDrawImage(context.platformContext(), CGRectMake(0, 0, paintRect.width(), paintRect.height()), image);
-}
-
-RefPtr<Image> RealtimeIncomingVideoSource::currentFrameImage()
-{
- if (!m_buffer)
- return nullptr;
-
- FloatRect rect(0, 0, m_currentSettings.width(), m_currentSettings.height());
- auto imageBuffer = ImageBuffer::create(rect.size(), Unaccelerated);
-
- auto pixelBuffer = static_cast<CVPixelBufferRef>(CMSampleBufferGetImageBuffer(m_buffer.get()));
- drawImage(*imageBuffer, m_conformer.createImageFromPixelBuffer(pixelBuffer).get(), rect);
-
- return ImageBuffer::sinkIntoImage(WTFMove(imageBuffer));
-}
-
-void RealtimeIncomingVideoSource::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (context.paintingDisabled())
- return;
-
- if (!m_buffer)
- return;
-
- // FIXME: Can we optimize here the painting?
- FloatRect fullRect(0, 0, m_currentSettings.width(), m_currentSettings.height());
- auto imageBuffer = ImageBuffer::create(fullRect.size(), Unaccelerated);
-
- auto pixelBuffer = static_cast<CVPixelBufferRef>(CMSampleBufferGetImageBuffer(m_buffer.get()));
- drawImage(*imageBuffer, m_conformer.createImageFromPixelBuffer(pixelBuffer).get(), fullRect);
-
- GraphicsContextStateSaver stateSaver(context);
- context.setImageInterpolationQuality(InterpolationLow);
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
- context.drawImage(*imageBuffer->copyImage(DontCopyBackingStore), rect);
-}
-
</del><span class="cx"> RefPtr<RealtimeMediaSourceCapabilities> RealtimeIncomingVideoSource::capabilities() const
</span><span class="cx"> {
</span><span class="cx"> return m_capabilities;
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmediastreammacRealtimeIncomingVideoSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mediastream/mac/RealtimeIncomingVideoSource.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -63,10 +63,7 @@
</span><span class="cx"> RealtimeMediaSourceSupportedConstraints& supportedConstraints();
</span><span class="cx">
</span><span class="cx"> void processNewSample(CMSampleBufferRef, unsigned, unsigned);
</span><del>- RefPtr<Image> currentFrameImage() final;
</del><span class="cx">
</span><del>- void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) final;
-
</del><span class="cx"> bool isProducingData() const final { return m_isProducingData && m_buffer; }
</span><span class="cx"> bool applySize(const IntSize&) final { return true; }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmockMockRealtimeVideoSourcecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.cpp 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -357,26 +357,6 @@
</span><span class="cx"> return m_imageBuffer.get();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MockRealtimeVideoSource::paintCurrentFrameInContext(GraphicsContext& context, const FloatRect& rect)
-{
- if (context.paintingDisabled() || !imageBuffer())
- return;
-
- GraphicsContextStateSaver stateSaver(context);
- context.setImageInterpolationQuality(InterpolationLow);
- IntRect paintRect(IntPoint(0, 0), IntSize(rect.width(), rect.height()));
-
- context.drawImage(*m_imageBuffer->copyImage(DontCopyBackingStore), rect);
-}
-
-RefPtr<Image> MockRealtimeVideoSource::currentFrameImage()
-{
- if (!imageBuffer())
- return nullptr;
-
- return m_imageBuffer->copyImage(DontCopyBackingStore);
-}
-
</del><span class="cx"> } // namespace WebCore
</span><span class="cx">
</span><span class="cx"> #endif // ENABLE(MEDIA_STREAM)
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformmockMockRealtimeVideoSourceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.h (213879 => 213880)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.h 2017-03-14 00:18:16 UTC (rev 213879)
+++ trunk/Source/WebCore/platform/mock/MockRealtimeVideoSource.h 2017-03-14 00:30:48 UTC (rev 213880)
</span><span class="lines">@@ -76,9 +76,6 @@
</span><span class="cx"> bool applyFacingMode(RealtimeMediaSourceSettings::VideoFacingMode) override { return true; }
</span><span class="cx"> bool applyAspectRatio(double) override { return true; }
</span><span class="cx">
</span><del>- RefPtr<Image> currentFrameImage() override;
- void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) override;
-
</del><span class="cx"> void generateFrame();
</span><span class="cx">
</span><span class="cx"> float m_baseFontSize { 0 };
</span></span></pre>
</div>
</div>
</body>
</html>