<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[270158] trunk</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/270158">270158</a></dd>
<dt>Author</dt> <dd>sihui_liu@apple.com</dd>
<dt>Date</dt> <dd>2020-11-21 21:51:10 -0800 (Sat, 21 Nov 2020)</dd>
</dl>

<h3>Log Message</h3>
<pre>Implement audio capture for SpeechRecognition on macOS
https://bugs.webkit.org/show_bug.cgi?id=218855
<rdar://problem/71331001>

Reviewed by Youenn Fablet.

Source/WebCore:

Introduce SpeechRecognizer, which performs audio capture and speech recogntion operations. On start,
SpeechRecognizer creates a SpeechRecognitionCaptureSource and starts audio capturing. On stop, SpeechRecognizer
clears the source and stops recognizing. SpeechRecognizer can only handle one request at a time, so calling
start on already started SpeechRecognizer would cause ongoing request to be aborted.

Tests: fast/speechrecognition/start-recognition-then-stop.html
       fast/speechrecognition/start-second-recognition.html

* Headers.cmake:
* Modules/speech/SpeechRecognitionCaptureSource.cpp: Added.
(WebCore::SpeechRecognitionCaptureSource::SpeechRecognitionCaptureSource):
* Modules/speech/SpeechRecognitionCaptureSource.h: Added.
* Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp: Added. SpeechRecognitionCaptureSourceImpl provides
implementation of SpeechRecognitionCaptureSource on when ENABLE(MEDIA_STREAM) is true.
(WebCore::nextLogIdentifier):
(WebCore::nullLogger):
(WebCore::SpeechRecognitionCaptureSourceImpl::SpeechRecognitionCaptureSourceImpl):
(WebCore::SpeechRecognitionCaptureSourceImpl::~SpeechRecognitionCaptureSourceImpl):
(WebCore::SpeechRecognitionCaptureSourceImpl::audioSamplesAvailable): Push data to buffer, signal main thread to
pull from buffer and invoke data callback.
(WebCore::SpeechRecognitionCaptureSourceImpl::sourceStarted):
(WebCore::SpeechRecognitionCaptureSourceImpl::sourceStopped):
(WebCore::SpeechRecognitionCaptureSourceImpl::sourceMutedChanged):
* Modules/speech/SpeechRecognitionCaptureSourceImpl.h: Added.
* Modules/speech/SpeechRecognizer.cpp: Added.
(WebCore::SpeechRecognizer::SpeechRecognizer):
(WebCore::SpeechRecognizer::reset):
(WebCore::SpeechRecognizer::start):
(WebCore::SpeechRecognizer::startInternal):
(WebCore::SpeechRecognizer::stop):
(WebCore::SpeechRecognizer::stopInternal):
* Modules/speech/SpeechRecognizer.h: Added.
(WebCore::SpeechRecognizer::currentClientIdentifier const):
* Sources.txt:
* SourcesCocoa.txt:
* WebCore.xcodeproj/project.pbxproj:
* platform/cocoa/MediaUtilities.cpp: Added.
(WebCore::createAudioFormatDescription):
(WebCore::createAudioSampleBuffer):
* platform/cocoa/MediaUtilities.h: Added.
* platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm: Move code for creating CMSampleBufferRef to
MediaUtilities.h/cpp so it can shared between SpeechRecognition and UserMedia, as Speech recognition backend
will take CMSampleBufferRef as input.
(WebCore::createAudioFormatDescription): Deleted.
(WebCore::createAudioSampleBuffer): Deleted.

Source/WebKit:

* UIProcess/SpeechRecognitionPermissionManager.cpp:
(WebKit::SpeechRecognitionPermissionManager::startProcessingRequest): Check and enable mock devices based on
preference as SpeechRecognition needs it for testing.
* UIProcess/SpeechRecognitionServer.cpp:
(WebKit::SpeechRecognitionServer::start):
(WebKit::SpeechRecognitionServer::requestPermissionForRequest):
(WebKit::SpeechRecognitionServer::handleRequest):
(WebKit::SpeechRecognitionServer::stop):
(WebKit::SpeechRecognitionServer::abort):
(WebKit::SpeechRecognitionServer::invalidate):
(WebKit::SpeechRecognitionServer::sendUpdate):
(WebKit::SpeechRecognitionServer::stopRequest): Deleted.
(WebKit::SpeechRecognitionServer::abortRequest): Deleted.
* UIProcess/SpeechRecognitionServer.h:
* UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::syncIfMockDevicesEnabledChanged):
* UIProcess/WebPageProxy.h:

LayoutTests:

* TestExpectations:
* fast/speechrecognition/start-recognition-in-removed-iframe.html: mark test as async to avoid flakiness.
* fast/speechrecognition/start-recognition-then-stop-expected.txt: Added.
* fast/speechrecognition/start-recognition-then-stop.html: Added.
* fast/speechrecognition/start-second-recognition-expected.txt: Added.
* fast/speechrecognition/start-second-recognition.html: Added.
* platform/wk2/TestExpectations:</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsChangeLog">trunk/LayoutTests/ChangeLog</a></li>
<li><a href="#trunkLayoutTestsTestExpectations">trunk/LayoutTests/TestExpectations</a></li>
<li><a href="#trunkLayoutTestsfastspeechrecognitionstartrecognitioninremovediframehtml">trunk/LayoutTests/fast/speechrecognition/start-recognition-in-removed-iframe.html</a></li>
<li><a href="#trunkLayoutTestsplatformwk2TestExpectations">trunk/LayoutTests/platform/wk2/TestExpectations</a></li>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCoreHeaderscmake">trunk/Source/WebCore/Headers.cmake</a></li>
<li><a href="#trunkSourceWebCoreSourcestxt">trunk/Source/WebCore/Sources.txt</a></li>
<li><a href="#trunkSourceWebCoreSourcesCocoatxt">trunk/Source/WebCore/SourcesCocoa.txt</a></li>
<li><a href="#trunkSourceWebCoreWebCorexcodeprojprojectpbxproj">trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj</a></li>
<li><a href="#trunkSourceWebCoreplatformmediarecordercocoaMediaRecorderPrivateWriterCocoamm">trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm</a></li>
<li><a href="#trunkSourceWebKitChangeLog">trunk/Source/WebKit/ChangeLog</a></li>
<li><a href="#trunkSourceWebKitUIProcessSpeechRecognitionPermissionManagercpp">trunk/Source/WebKit/UIProcess/SpeechRecognitionPermissionManager.cpp</a></li>
<li><a href="#trunkSourceWebKitUIProcessSpeechRecognitionServercpp">trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.cpp</a></li>
<li><a href="#trunkSourceWebKitUIProcessSpeechRecognitionServerh">trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.h</a></li>
<li><a href="#trunkSourceWebKitUIProcessWebPageProxycpp">trunk/Source/WebKit/UIProcess/WebPageProxy.cpp</a></li>
<li><a href="#trunkSourceWebKitUIProcessWebPageProxyh">trunk/Source/WebKit/UIProcess/WebPageProxy.h</a></li>
</ul>

<h3>Added Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsfastspeechrecognitionstartrecognitionthenstopexpectedtxt">trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastspeechrecognitionstartrecognitionthenstophtml">trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop.html</a></li>
<li><a href="#trunkLayoutTestsfastspeechrecognitionstartsecondrecognitionexpectedtxt">trunk/LayoutTests/fast/speechrecognition/start-second-recognition-expected.txt</a></li>
<li><a href="#trunkLayoutTestsfastspeechrecognitionstartsecondrecognitionhtml">trunk/LayoutTests/fast/speechrecognition/start-second-recognition.html</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourcecpp">trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.cpp</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceh">trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.h</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceImplcpp">trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceImplh">trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.h</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognizercpp">trunk/Source/WebCore/Modules/speech/SpeechRecognizer.cpp</a></li>
<li><a href="#trunkSourceWebCoreModulesspeechSpeechRecognizerh">trunk/Source/WebCore/Modules/speech/SpeechRecognizer.h</a></li>
<li><a href="#trunkSourceWebCoreplatformcocoaMediaUtilitiescpp">trunk/Source/WebCore/platform/cocoa/MediaUtilities.cpp</a></li>
<li><a href="#trunkSourceWebCoreplatformcocoaMediaUtilitiesh">trunk/Source/WebCore/platform/cocoa/MediaUtilities.h</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/ChangeLog (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/ChangeLog      2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/LayoutTests/ChangeLog 2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -1,3 +1,19 @@
</span><ins>+2020-11-21  Sihui Liu  <sihui_liu@apple.com>
+
+        Implement audio capture for SpeechRecognition on macOS
+        https://bugs.webkit.org/show_bug.cgi?id=218855
+        <rdar://problem/71331001>
+
+        Reviewed by Youenn Fablet.
+
+        * TestExpectations:
+        * fast/speechrecognition/start-recognition-in-removed-iframe.html: mark test as async to avoid flakiness.
+        * fast/speechrecognition/start-recognition-then-stop-expected.txt: Added.
+        * fast/speechrecognition/start-recognition-then-stop.html: Added.
+        * fast/speechrecognition/start-second-recognition-expected.txt: Added.
+        * fast/speechrecognition/start-second-recognition.html: Added.
+        * platform/wk2/TestExpectations:
+
</ins><span class="cx"> 2020-11-21  Chris Dumez  <cdumez@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Poor resampling quality when using AudioContext sampleRate parameter
</span></span></pre></div>
<a id="trunkLayoutTestsTestExpectations"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/TestExpectations (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/TestExpectations       2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/LayoutTests/TestExpectations  2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -179,6 +179,8 @@
</span><span class="cx"> fast/forms/call-text-did-change-in-text-field-when-typing.html [ Skip ]
</span><span class="cx"> http/tests/in-app-browser-privacy/ [ Skip ]
</span><span class="cx"> fast/speechrecognition/permission-error.html [ Skip ]
</span><ins>+fast/speechrecognition/start-recognition-then-stop.html [ Skip ]
+fast/speechrecognition/start-second-recognition.html [ Skip ]
</ins><span class="cx"> 
</span><span class="cx"> # Only partial support on Cocoa platforms.
</span><span class="cx"> imported/w3c/web-platform-tests/speech-api/ [ Skip ]
</span></span></pre></div>
<a id="trunkLayoutTestsfastspeechrecognitionstartrecognitioninremovediframehtml"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/fast/speechrecognition/start-recognition-in-removed-iframe.html (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/speechrecognition/start-recognition-in-removed-iframe.html        2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/LayoutTests/fast/speechrecognition/start-recognition-in-removed-iframe.html   2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -5,15 +5,20 @@
</span><span class="cx"> <script>
</span><span class="cx"> description("Verify that process does not crash when starting recognition in a removed iframe.");
</span><span class="cx"> 
</span><ins>+if (window.testRunner) {
+    jsTestIsAsync = true;
+}
+
</ins><span class="cx"> function test()
</span><span class="cx"> {
</span><span class="cx">     iframe = document.getElementsByTagName('iframe')[0];
</span><del>-    shouldNotThrow("iframe.contentWindow.startRecognition()"); 
</del><ins>+    shouldNotThrow("iframe.contentWindow.startRecognition()");
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> function removeFrame()
</span><span class="cx"> {
</span><span class="cx">     shouldNotThrow("iframe.parentNode.removeChild(iframe)");
</span><ins>+    setTimeout(() => finishJSTest(), 0);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> window.addEventListener('load', test, false);
</span></span></pre></div>
<a id="trunkLayoutTestsfastspeechrecognitionstartrecognitionthenstopexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop-expected.txt (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop-expected.txt                                (rev 0)
+++ trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop-expected.txt   2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,16 @@
</span><ins>+Verify that events are received corretly when start and stop recognition normally.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS recognition = new SpeechRecognition(); did not throw exception.
+PASS recognition.start() did not throw exception.
+Received start event
+Received audiostart event
+PASS recognition.stop() did not throw exception.
+Received audioend event
+Received end event
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastspeechrecognitionstartrecognitionthenstophtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop.html (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop.html                                (rev 0)
+++ trunk/LayoutTests/fast/speechrecognition/start-recognition-then-stop.html   2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,37 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<body>
+<script src="../../resources/js-test.js"></script>
+<script>
+description("Verify that events are received corretly when start and stop recognition normally.");
+
+if (window.testRunner) {
+    jsTestIsAsync = true;
+}
+
+shouldNotThrow("recognition = new SpeechRecognition();");
+recognition.onstart = (event) => {
+    debug("Received start event");
+}
+
+recognition.onaudiostart = (event) => {
+    debug("Received audiostart event");
+
+    shouldNotThrow("recognition.stop()");
+}
+
+recognition.onaudioend = (event) => {
+    debug("Received audioend event");
+}
+
+recognition.onend = (event) => {
+    debug("Received end event");
+
+    finishJSTest();
+}
+
+shouldNotThrow("recognition.start()");
+
+</script>
+</body>
+</html>
</ins><span class="cx">\ No newline at end of file
</span></span></pre></div>
<a id="trunkLayoutTestsfastspeechrecognitionstartsecondrecognitionexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/speechrecognition/start-second-recognition-expected.txt (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/speechrecognition/start-second-recognition-expected.txt                           (rev 0)
+++ trunk/LayoutTests/fast/speechrecognition/start-second-recognition-expected.txt      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,16 @@
</span><ins>+Verify that starting a second recognition aborts ongoing one.
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS recognition = new SpeechRecognition(); did not throw exception.
+PASS recognition.start() did not throw exception.
+PASS secondRecognition = new SpeechRecognition(); did not throw exception.
+PASS secondRecognition.start() did not throw exception.
+PASS receivedStart is true
+PASS event.error is "aborted"
+PASS event.message is "Another request is started"
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
</ins></span></pre></div>
<a id="trunkLayoutTestsfastspeechrecognitionstartsecondrecognitionhtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/fast/speechrecognition/start-second-recognition.html (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/fast/speechrecognition/start-second-recognition.html                           (rev 0)
+++ trunk/LayoutTests/fast/speechrecognition/start-second-recognition.html      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,32 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<body>
+<script src="../../resources/js-test.js"></script>
+<script>
+description("Verify that starting a second recognition aborts ongoing one.");
+
+if (window.testRunner) {
+    jsTestIsAsync = true;
+}
+
+shouldNotThrow("recognition = new SpeechRecognition();");
+receivedStart = false;
+recognition.onstart = (event) => {
+    receivedStart = true;
+}
+
+recognition.onerror = (event) => {
+    shouldBeTrue("receivedStart");
+    shouldBeEqualToString("event.error", "aborted");
+    shouldBeEqualToString("event.message", "Another request is started");
+
+    finishJSTest();
+}
+
+shouldNotThrow("recognition.start()");
+shouldNotThrow("secondRecognition = new SpeechRecognition();");
+shouldNotThrow("secondRecognition.start()");
+
+</script>
+</body>
+</html>
</ins><span class="cx">\ No newline at end of file
</span></span></pre></div>
<a id="trunkLayoutTestsplatformwk2TestExpectations"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/platform/wk2/TestExpectations (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/platform/wk2/TestExpectations  2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/LayoutTests/platform/wk2/TestExpectations     2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -804,4 +804,6 @@
</span><span class="cx"> # WebKit2 only.
</span><span class="cx"> js/throw-large-string-oom.html [ Pass ]
</span><span class="cx"> fast/speechrecognition/permission-error.html [ Pass ]
</span><ins>+fast/speechrecognition/start-recognition-then-stop.html [ Pass ]
+fast/speechrecognition/start-second-recognition.html [ Pass ]
</ins><span class="cx"> fullscreen/full-screen-enter-while-exiting.html [ Pass ]
</span></span></pre></div>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog   2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/ChangeLog      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -1,3 +1,57 @@
</span><ins>+2020-11-21  Sihui Liu  <sihui_liu@apple.com>
+
+        Implement audio capture for SpeechRecognition on macOS
+        https://bugs.webkit.org/show_bug.cgi?id=218855
+        <rdar://problem/71331001>
+
+        Reviewed by Youenn Fablet.
+
+        Introduce SpeechRecognizer, which performs audio capture and speech recogntion operations. On start,
+        SpeechRecognizer creates a SpeechRecognitionCaptureSource and starts audio capturing. On stop, SpeechRecognizer 
+        clears the source and stops recognizing. SpeechRecognizer can only handle one request at a time, so calling
+        start on already started SpeechRecognizer would cause ongoing request to be aborted.
+
+        Tests: fast/speechrecognition/start-recognition-then-stop.html
+               fast/speechrecognition/start-second-recognition.html
+
+        * Headers.cmake:
+        * Modules/speech/SpeechRecognitionCaptureSource.cpp: Added.
+        (WebCore::SpeechRecognitionCaptureSource::SpeechRecognitionCaptureSource):
+        * Modules/speech/SpeechRecognitionCaptureSource.h: Added.
+        * Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp: Added. SpeechRecognitionCaptureSourceImpl provides 
+        implementation of SpeechRecognitionCaptureSource on when ENABLE(MEDIA_STREAM) is true.
+        (WebCore::nextLogIdentifier):
+        (WebCore::nullLogger):
+        (WebCore::SpeechRecognitionCaptureSourceImpl::SpeechRecognitionCaptureSourceImpl):
+        (WebCore::SpeechRecognitionCaptureSourceImpl::~SpeechRecognitionCaptureSourceImpl):
+        (WebCore::SpeechRecognitionCaptureSourceImpl::audioSamplesAvailable): Push data to buffer, signal main thread to
+        pull from buffer and invoke data callback.
+        (WebCore::SpeechRecognitionCaptureSourceImpl::sourceStarted):
+        (WebCore::SpeechRecognitionCaptureSourceImpl::sourceStopped):
+        (WebCore::SpeechRecognitionCaptureSourceImpl::sourceMutedChanged):
+        * Modules/speech/SpeechRecognitionCaptureSourceImpl.h: Added.
+        * Modules/speech/SpeechRecognizer.cpp: Added.
+        (WebCore::SpeechRecognizer::SpeechRecognizer):
+        (WebCore::SpeechRecognizer::reset):
+        (WebCore::SpeechRecognizer::start):
+        (WebCore::SpeechRecognizer::startInternal):
+        (WebCore::SpeechRecognizer::stop):
+        (WebCore::SpeechRecognizer::stopInternal):
+        * Modules/speech/SpeechRecognizer.h: Added.
+        (WebCore::SpeechRecognizer::currentClientIdentifier const):
+        * Sources.txt:
+        * SourcesCocoa.txt:
+        * WebCore.xcodeproj/project.pbxproj:
+        * platform/cocoa/MediaUtilities.cpp: Added.
+        (WebCore::createAudioFormatDescription):
+        (WebCore::createAudioSampleBuffer):
+        * platform/cocoa/MediaUtilities.h: Added.
+        * platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm: Move code for creating CMSampleBufferRef to
+        MediaUtilities.h/cpp so it can shared between SpeechRecognition and UserMedia, as Speech recognition backend 
+        will take CMSampleBufferRef as input.
+        (WebCore::createAudioFormatDescription): Deleted.
+        (WebCore::createAudioSampleBuffer): Deleted.
+
</ins><span class="cx"> 2020-11-21  Chris Dumez  <cdumez@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Poor resampling quality when using AudioContext sampleRate parameter
</span></span></pre></div>
<a id="trunkSourceWebCoreHeaderscmake"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/Headers.cmake (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Headers.cmake       2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/Headers.cmake  2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -116,6 +116,8 @@
</span><span class="cx">     Modules/plugins/PluginReplacement.h
</span><span class="cx">     Modules/plugins/YouTubePluginReplacement.h
</span><span class="cx"> 
</span><ins>+    Modules/speech/SpeechRecognitionCaptureSource.h
+    Modules/speech/SpeechRecognitionCaptureSourceImpl.h
</ins><span class="cx">     Modules/speech/SpeechRecognitionConnection.h
</span><span class="cx">     Modules/speech/SpeechRecognitionConnectionClient.h
</span><span class="cx">     Modules/speech/SpeechRecognitionConnectionClientIdentifier.h
</span><span class="lines">@@ -124,6 +126,7 @@
</span><span class="cx">     Modules/speech/SpeechRecognitionRequestInfo.h
</span><span class="cx">     Modules/speech/SpeechRecognitionResultData.h
</span><span class="cx">     Modules/speech/SpeechRecognitionUpdate.h
</span><ins>+    Modules/speech/SpeechRecognizer.h
</ins><span class="cx"> 
</span><span class="cx">     Modules/streams/ReadableStreamChunk.h
</span><span class="cx">     Modules/streams/ReadableStreamSink.h
</span></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourcecpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.cpp (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.cpp                           (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.cpp      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,76 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "SpeechRecognitionCaptureSource.h"
+
+#if ENABLE(MEDIA_STREAM)
+#include "CaptureDeviceManager.h"
+#include "RealtimeMediaSourceCenter.h"
+#include "SpeechRecognitionUpdate.h"
+#endif
+
+namespace WebCore {
+
+SpeechRecognitionCaptureSource::SpeechRecognitionCaptureSource(SpeechRecognitionConnectionClientIdentifier clientIdentifier, DataCallback&& dataCallback, StateUpdateCallback&& stateUpdateCallback)
+{
+#if ENABLE(MEDIA_STREAM)
+    Optional<CaptureDevice> captureDevice;
+    auto devices = RealtimeMediaSourceCenter::singleton().audioCaptureFactory().audioCaptureDeviceManager().captureDevices();
+    for (auto device : devices) {
+        if (!device.enabled())
+            continue;
+
+        if (!captureDevice)
+            captureDevice = device;
+
+        if (device.isDefault()) {
+            captureDevice = device;
+            break;
+        }
+    }
+
+    if (!captureDevice) {
+        auto error = SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, "No device is available for capture" };
+        stateUpdateCallback(SpeechRecognitionUpdate::createError(clientIdentifier, error));
+        return;
+    }
+
+    auto result = RealtimeMediaSourceCenter::singleton().audioCaptureFactory().createAudioCaptureSource(*captureDevice, { }, { });
+    if (!result) {
+        auto error = SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, result.errorMessage };
+        stateUpdateCallback(SpeechRecognitionUpdate::createError(clientIdentifier, error));
+        return;
+    }
+
+    m_impl = makeUnique<SpeechRecognitionCaptureSourceImpl>(clientIdentifier, WTFMove(dataCallback), WTFMove(stateUpdateCallback), result.source());
+#else
+    UNUSED_PARAM(clientIdentifier);
+    UNUSED_PARAM(dataCallback);
+    UNUSED_PARAM(stateUpdateCallback);
+#endif
+}
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceh"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.h (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.h                             (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSource.h        2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,56 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "SpeechRecognitionCaptureSourceImpl.h"
+#include "SpeechRecognitionConnectionClientIdentifier.h"
+
+namespace WTF {
+class MediaTime;
+}
+
+namespace WebCore {
+
+class AudioStreamDescription;
+class PlatformAudioData;
+class SpeechRecognitionCaptureSourceImpl;
+class SpeechRecognitionUpdate;
+
+class SpeechRecognitionCaptureSource {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    ~SpeechRecognitionCaptureSource() = default;
+    using DataCallback = Function<void(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)>;
+    using StateUpdateCallback = Function<void(const SpeechRecognitionUpdate&)>;
+    SpeechRecognitionCaptureSource(SpeechRecognitionConnectionClientIdentifier, DataCallback&&, StateUpdateCallback&&);
+
+private:
+#if ENABLE(MEDIA_STREAM)
+    std::unique_ptr<SpeechRecognitionCaptureSourceImpl> m_impl;
+#endif
+};
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceImplcpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp                               (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp  2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,151 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "SpeechRecognitionCaptureSourceImpl.h"
+
+#if ENABLE(MEDIA_STREAM)
+
+#include "SpeechRecognitionUpdate.h"
+
+#if PLATFORM(COCOA)
+#include "CAAudioStreamDescription.h"
+#include "WebAudioBufferList.h"
+#endif
+
+namespace WebCore {
+
+#if !RELEASE_LOG_DISABLED
+static const void* nextLogIdentifier()
+{
+    static uint64_t logIdentifier = cryptographicallyRandomNumber();
+    return reinterpret_cast<const void*>(++logIdentifier);
+}
+
+static RefPtr<Logger>& nullLogger()
+{
+    static NeverDestroyed<RefPtr<Logger>> logger;
+    return logger;
+}
+#endif
+
+SpeechRecognitionCaptureSourceImpl::SpeechRecognitionCaptureSourceImpl(SpeechRecognitionConnectionClientIdentifier identifier, DataCallback&& dataCallback, StateUpdateCallback&& stateUpdateCallback, Ref<RealtimeMediaSource>&& source)
+    : m_clientIdentifier(identifier)
+    , m_dataCallback(WTFMove(dataCallback))
+    , m_stateUpdateCallback(WTFMove(stateUpdateCallback))
+    , m_source(WTFMove(source))
+{
+    m_source->addAudioSampleObserver(*this);
+    m_source->addObserver(*this);
+    m_source->start();
+
+#if !RELEASE_LOG_DISABLED
+    if (!nullLogger().get()) {
+        nullLogger() = Logger::create(this);
+        nullLogger()->setEnabled(this, false);
+    }
+
+    m_source->setLogger(*nullLogger(), nextLogIdentifier());
+#endif
+
+    auto weakThis = makeWeakPtr(this);
+}
+
+SpeechRecognitionCaptureSourceImpl::~SpeechRecognitionCaptureSourceImpl()
+{
+    m_source->removeAudioSampleObserver(*this);
+    m_source->removeObserver(*this);
+    m_source->stop();
+}
+
+void SpeechRecognitionCaptureSourceImpl::audioSamplesAvailable(const MediaTime& time, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount)
+{
+#if PLATFORM(COCOA)
+    ASSERT(description.platformDescription().type == PlatformDescription::CAAudioStreamBasicType);
+    auto audioDescription = toCAAudioStreamDescription(description);
+    if (!m_dataSource || !m_dataSource->inputDescription() || *m_dataSource->inputDescription() != description) {
+        auto dataSource = AudioSampleDataSource::create(description.sampleRate() * 1, m_source.get());
+        if (dataSource->setInputFormat(audioDescription)) {
+            callOnMainThread([this, weakThis = makeWeakPtr(this)] {
+                if (weakThis)
+                    m_stateUpdateCallback(SpeechRecognitionUpdate::createError(m_clientIdentifier, SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, "Unable to set input format" }));
+            });
+            return;
+        }
+
+        if (dataSource->setOutputFormat(audioDescription)) {
+            callOnMainThread([this, weakThis = makeWeakPtr(this)] {
+                if (weakThis)
+                    m_stateUpdateCallback(SpeechRecognitionUpdate::createError(m_clientIdentifier, SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, "Unable to set output format" }));
+            });
+            return;
+        }
+        
+        if (auto locker = tryHoldLock(m_dataSourceLock))
+            m_dataSource = WTFMove(dataSource);
+        else
+            return;
+    }
+
+    m_dataSource->pushSamples(time, data, sampleCount);
+    callOnMainThread([this, weakThis = makeWeakPtr(this), time, audioDescription, sampleCount] {
+        if (!weakThis)
+            return;
+
+        auto data = WebAudioBufferList { audioDescription, static_cast<uint32_t>(sampleCount) };
+        {
+            auto locker = holdLock(m_dataSourceLock);
+            m_dataSource->pullSamples(*data.list(), sampleCount, time.timeValue(), 0, AudioSampleDataSource::Copy);
+        }
+
+        m_dataCallback(time, data, audioDescription, sampleCount);
+    });
+#else
+    m_dataCallback(time, data, description, sampleCount);
+#endif
+}
+
+void SpeechRecognitionCaptureSourceImpl::sourceStarted()
+{
+    ASSERT(isMainThread());
+    m_stateUpdateCallback(SpeechRecognitionUpdate::create(m_clientIdentifier, SpeechRecognitionUpdateType::AudioStart));
+}
+
+void SpeechRecognitionCaptureSourceImpl::sourceStopped()
+{
+    ASSERT(isMainThread());
+    ASSERT(m_source->captureDidFail());
+    m_stateUpdateCallback(SpeechRecognitionUpdate::createError(m_clientIdentifier, SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, "Source is stopped" }));
+}
+
+void SpeechRecognitionCaptureSourceImpl::sourceMutedChanged()
+{
+    ASSERT(isMainThread());
+    m_stateUpdateCallback(SpeechRecognitionUpdate::createError(m_clientIdentifier, SpeechRecognitionError { SpeechRecognitionErrorType::AudioCapture, "Source is muted" }));
+}
+
+} // namespace WebCore
+
+#endif
</ins></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognitionCaptureSourceImplh"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.h (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.h                         (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognitionCaptureSourceImpl.h    2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,81 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if ENABLE(MEDIA_STREAM)
+
+#include "RealtimeMediaSource.h"
+#include "SpeechRecognitionConnectionClientIdentifier.h"
+
+#if PLATFORM(COCOA)
+#include "AudioSampleDataSource.h"
+#endif
+
+namespace WTF {
+class MediaTime;
+}
+
+namespace WebCore {
+
+class AudioStreamDescription;
+class PlatformAudioData;
+class SpeechRecognitionUpdate;
+enum class SpeechRecognitionUpdateType;
+
+class SpeechRecognitionCaptureSourceImpl
+    : public RealtimeMediaSource::Observer
+    , public RealtimeMediaSource::AudioSampleObserver {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    using DataCallback = Function<void(const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t)>;
+    using StateUpdateCallback = Function<void(const SpeechRecognitionUpdate&)>;
+    SpeechRecognitionCaptureSourceImpl(SpeechRecognitionConnectionClientIdentifier, DataCallback&&, StateUpdateCallback&&, Ref<RealtimeMediaSource>&&);
+    ~SpeechRecognitionCaptureSourceImpl();
+
+private:
+    // RealtimeMediaSource::AudioSampleObserver
+    void audioSamplesAvailable(const MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final;
+
+    // RealtimeMediaSource::Observer
+    void sourceStarted() final;
+    void sourceStopped() final;
+    void sourceMutedChanged() final;
+
+    SpeechRecognitionConnectionClientIdentifier m_clientIdentifier;
+    DataCallback m_dataCallback;
+    StateUpdateCallback m_stateUpdateCallback;
+    Ref<RealtimeMediaSource> m_source;
+
+#if PLATFORM(COCOA)
+    RefPtr<AudioSampleDataSource> m_dataSource;
+    Lock m_dataSourceLock;
+#endif
+};
+
+} // namespace WebCore
+
+#endif
+
</ins></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognizercpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognizer.cpp (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognizer.cpp                         (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognizer.cpp    2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,122 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "SpeechRecognizer.h"
+
+#include "SpeechRecognitionUpdate.h"
+
+#if PLATFORM(COCOA)
+#include "MediaUtilities.h"
+#include <pal/avfoundation/MediaTimeAVFoundation.h>
+#endif
+
+namespace WebCore {
+
+SpeechRecognizer::SpeechRecognizer(DelegateCallback&& callback)
+    : m_delegateCallback(WTFMove(callback))
+{
+}
+
+void SpeechRecognizer::reset()
+{
+    if (!m_clientIdentifier)
+        return;
+
+    if (m_source)
+        m_source = nullptr;
+
+    auto error = SpeechRecognitionError { SpeechRecognitionErrorType::Aborted, "Another request is started" };
+    m_delegateCallback(SpeechRecognitionUpdate::createError(*m_clientIdentifier, error));
+}
+
+void SpeechRecognizer::start(SpeechRecognitionConnectionClientIdentifier identifier)
+{
+    reset();
+
+    m_clientIdentifier = identifier;
+    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::Start));
+
+    startInternal();
+}
+
+void SpeechRecognizer::startInternal()
+{
+    auto dataCallback = [weakThis = makeWeakPtr(this)](const auto& time, const auto& data, const auto& description, auto sampleCount) {
+        if (!weakThis)
+            return;
+
+#if PLATFORM(COCOA)
+        auto buffer = createAudioSampleBuffer(data, description, PAL::toCMTime(time), sampleCount);
+        UNUSED_PARAM(buffer);
+#else
+        UNUSED_PARAM(time);
+        UNUSED_PARAM(data);
+        UNUSED_PARAM(description);
+        UNUSED_PARAM(sampleCount);
+#endif
+    };
+
+    auto stateUpdateCallback = [this, weakThis = makeWeakPtr(this)](const auto& update) {
+        if (!weakThis)
+            return;
+
+        ASSERT(m_clientIdentifier && m_clientIdentifier.value() == update.clientIdentifier());
+        m_delegateCallback(update);
+
+        if (update.type() == SpeechRecognitionUpdateType::Error)
+            m_source = nullptr;
+    };
+
+    m_source = makeUnique<SpeechRecognitionCaptureSource>(*m_clientIdentifier, WTFMove(dataCallback), WTFMove(stateUpdateCallback));
+}
+
+void SpeechRecognizer::stop(ShouldGenerateFinalResult shouldGenerateFinalResult)
+{
+    if (!m_clientIdentifier)
+        return;
+
+    stopInternal();
+
+    if (shouldGenerateFinalResult == ShouldGenerateFinalResult::Yes) {
+        // TODO: generate real result when speech recognition backend is implemented.
+        Vector<SpeechRecognitionResultData> resultDatas;
+        m_delegateCallback(SpeechRecognitionUpdate::createResult(*m_clientIdentifier, resultDatas));
+    }
+
+    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::End));
+    m_clientIdentifier = WTF::nullopt;
+}
+
+void SpeechRecognizer::stopInternal()
+{
+    if (!m_source)
+        return;
+
+    m_source = nullptr;
+    m_delegateCallback(SpeechRecognitionUpdate::create(*m_clientIdentifier, SpeechRecognitionUpdateType::AudioEnd));
+}
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreModulesspeechSpeechRecognizerh"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/Modules/speech/SpeechRecognizer.h (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Modules/speech/SpeechRecognizer.h                           (rev 0)
+++ trunk/Source/WebCore/Modules/speech/SpeechRecognizer.h      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,58 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "SpeechRecognitionCaptureSource.h"
+#include "SpeechRecognitionConnectionClientIdentifier.h"
+
+namespace WebCore {
+
+class SpeechRecognitionUpdate;
+
+class SpeechRecognizer : public CanMakeWeakPtr<SpeechRecognizer> {
+    WTF_MAKE_FAST_ALLOCATED;
+public:
+    using DelegateCallback = Function<void(const SpeechRecognitionUpdate&)>;
+    WEBCORE_EXPORT explicit SpeechRecognizer(DelegateCallback&&);
+    WEBCORE_EXPORT ~SpeechRecognizer() = default;
+
+    WEBCORE_EXPORT void start(SpeechRecognitionConnectionClientIdentifier);
+    enum class ShouldGenerateFinalResult { No, Yes };
+    WEBCORE_EXPORT void stop(ShouldGenerateFinalResult = ShouldGenerateFinalResult::Yes);
+
+    Optional<SpeechRecognitionConnectionClientIdentifier> currentClientIdentifier() const { return m_clientIdentifier; }
+
+private:
+    void reset();
+    void startInternal();
+    void stopInternal();
+
+    Optional<SpeechRecognitionConnectionClientIdentifier> m_clientIdentifier;
+    DelegateCallback m_delegateCallback;
+    std::unique_ptr<SpeechRecognitionCaptureSource> m_source;
+};
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreSourcestxt"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/Sources.txt (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/Sources.txt 2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/Sources.txt    2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -205,6 +205,9 @@
</span><span class="cx"> Modules/speech/SpeechRecognitionResult.cpp
</span><span class="cx"> Modules/speech/SpeechRecognitionResultList.cpp
</span><span class="cx"> Modules/speech/SpeechRecognitionUpdate.cpp
</span><ins>+Modules/speech/SpeechRecognitionCaptureSource.cpp
+Modules/speech/SpeechRecognitionCaptureSourceImpl.cpp
+Modules/speech/SpeechRecognizer.cpp
</ins><span class="cx"> Modules/speech/DOMWindowSpeechSynthesis.cpp
</span><span class="cx"> Modules/speech/SpeechSynthesis.cpp
</span><span class="cx"> Modules/speech/SpeechSynthesisEvent.cpp
</span></span></pre></div>
<a id="trunkSourceWebCoreSourcesCocoatxt"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/SourcesCocoa.txt (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/SourcesCocoa.txt    2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/SourcesCocoa.txt       2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -238,6 +238,7 @@
</span><span class="cx"> platform/cocoa/KeyEventCocoa.mm
</span><span class="cx"> platform/cocoa/LocalizedStringsCocoa.mm
</span><span class="cx"> platform/cocoa/MIMETypeRegistryCocoa.mm
</span><ins>+platform/cocoa/MediaUtilities.cpp
</ins><span class="cx"> platform/cocoa/NetworkExtensionContentFilter.mm
</span><span class="cx"> platform/cocoa/ParentalControlsContentFilter.mm
</span><span class="cx"> platform/cocoa/PasteboardCocoa.mm
</span></span></pre></div>
<a id="trunkSourceWebCoreWebCorexcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj   2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/WebCore.xcodeproj/project.pbxproj      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -120,8 +120,8 @@
</span><span class="cx">          073794FE19F5864E00E5A045 /* RTCNotifiersMock.h in Headers */ = {isa = PBXBuildFile; fileRef = 073794F819F5864E00E5A045 /* RTCNotifiersMock.h */; };
</span><span class="cx">          0738E5EC2499839000DA101C /* AVOutputDeviceMenuControllerTargetPicker.mm in Sources */ = {isa = PBXBuildFile; fileRef = 0738E5EA249968AD00DA101C /* AVOutputDeviceMenuControllerTargetPicker.mm */; };
</span><span class="cx">          073A15542177A42600EA08F2 /* RemoteVideoSample.h in Headers */ = {isa = PBXBuildFile; fileRef = 073A15532177A39A00EA08F2 /* RemoteVideoSample.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><del>-               073B87671E4385AC0071C0EC /* AudioSampleBufferList.h in Headers */ = {isa = PBXBuildFile; fileRef = 073B87631E43859D0071C0EC /* AudioSampleBufferList.h */; };
-               073B87691E4385AC0071C0EC /* AudioSampleDataSource.h in Headers */ = {isa = PBXBuildFile; fileRef = 073B87651E43859D0071C0EC /* AudioSampleDataSource.h */; };
</del><ins>+                073B87671E4385AC0071C0EC /* AudioSampleBufferList.h in Headers */ = {isa = PBXBuildFile; fileRef = 073B87631E43859D0071C0EC /* AudioSampleBufferList.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               073B87691E4385AC0071C0EC /* AudioSampleDataSource.h in Headers */ = {isa = PBXBuildFile; fileRef = 073B87651E43859D0071C0EC /* AudioSampleDataSource.h */; settings = {ATTRIBUTES = (Private, ); }; };
</ins><span class="cx">           074E82BB18A69F0E007EF54C /* PlatformTimeRanges.h in Headers */ = {isa = PBXBuildFile; fileRef = 074E82B918A69F0E007EF54C /* PlatformTimeRanges.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          075033A8252BD36800F70CE3 /* VideoPlaybackQualityMetrics.h in Headers */ = {isa = PBXBuildFile; fileRef = 075033A6252BD36800F70CE3 /* VideoPlaybackQualityMetrics.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          0753860314489E9800B78452 /* CachedTextTrack.h in Headers */ = {isa = PBXBuildFile; fileRef = 0753860114489E9800B78452 /* CachedTextTrack.h */; };
</span><span class="lines">@@ -2795,6 +2795,8 @@
</span><span class="cx">          9393E600151A99F200066F06 /* CSSImageSetValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 9393E5FE151A99F200066F06 /* CSSImageSetValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          939885C408B7E3D100E707C4 /* EventNames.h in Headers */ = {isa = PBXBuildFile; fileRef = 939885C208B7E3D100E707C4 /* EventNames.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          939B02EF0EA2DBC400C54570 /* WidthIterator.h in Headers */ = {isa = PBXBuildFile; fileRef = 939B02ED0EA2DBC400C54570 /* WidthIterator.h */; };
</span><ins>+               939C0D272564E47F00B3211B /* SpeechRecognizer.h in Headers */ = {isa = PBXBuildFile; fileRef = 939C0D2125648C3900B3211B /* SpeechRecognizer.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               939C0D2B2564E7F300B3211B /* MediaUtilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 939C0D292564E7F200B3211B /* MediaUtilities.h */; settings = {ATTRIBUTES = (Private, ); }; };
</ins><span class="cx">           93A0482825495506000AC462 /* SpeechRecognitionProvider.h in Headers */ = {isa = PBXBuildFile; fileRef = 93A0482625495500000AC462 /* SpeechRecognitionProvider.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          93A0482925495511000AC462 /* SpeechRecognitionResultData.h in Headers */ = {isa = PBXBuildFile; fileRef = 93A0481B254954E4000AC462 /* SpeechRecognitionResultData.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          93A0482A25495514000AC462 /* SpeechRecognitionRequest.h in Headers */ = {isa = PBXBuildFile; fileRef = 93A0481F254954E6000AC462 /* SpeechRecognitionRequest.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="lines">@@ -2856,6 +2858,8 @@
</span><span class="cx">          93F1D5BB12D532C400832BEC /* WebGLLoseContext.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F1D5B812D532C400832BEC /* WebGLLoseContext.h */; };
</span><span class="cx">          93F1D5C112D5335600832BEC /* JSWebGLLoseContext.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F1D5BF12D5335600832BEC /* JSWebGLLoseContext.h */; };
</span><span class="cx">          93F2CC932427FB9C005851D8 /* CharacterRange.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F2CC912427FB9A005851D8 /* CharacterRange.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><ins>+               93F6B81F2567A08C00A08488 /* SpeechRecognitionCaptureSourceImpl.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F6B81C25679F7000A08488 /* SpeechRecognitionCaptureSourceImpl.h */; settings = {ATTRIBUTES = (Private, ); }; };
+               93F6B8222567A65600A08488 /* SpeechRecognitionCaptureSource.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F6B81B25679F6F00A08488 /* SpeechRecognitionCaptureSource.h */; settings = {ATTRIBUTES = (Private, ); }; };
</ins><span class="cx">           93F6F1EE127F70B10055CB06 /* WebGLContextEvent.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F6F1EB127F70B10055CB06 /* WebGLContextEvent.h */; };
</span><span class="cx">          93F925430F7EF5B8007E37C9 /* RadioButtonGroups.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F925410F7EF5B8007E37C9 /* RadioButtonGroups.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx">          93F9B6E10BA0FB7200854064 /* JSComment.h in Headers */ = {isa = PBXBuildFile; fileRef = 93F9B6DF0BA0FB7200854064 /* JSComment.h */; };
</span><span class="lines">@@ -11453,6 +11457,10 @@
</span><span class="cx">          939885C208B7E3D100E707C4 /* EventNames.h */ = {isa = PBXFileReference; fileEncoding = 30; indentWidth = 4; lastKnownFileType = sourcecode.c.h; path = EventNames.h; sourceTree = "<group>"; tabWidth = 8; usesTabs = 0; };
</span><span class="cx">          939B02EC0EA2DBC400C54570 /* WidthIterator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WidthIterator.cpp; sourceTree = "<group>"; };
</span><span class="cx">          939B02ED0EA2DBC400C54570 /* WidthIterator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WidthIterator.h; sourceTree = "<group>"; };
</span><ins>+               939C0D2125648C3900B3211B /* SpeechRecognizer.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = SpeechRecognizer.h; sourceTree = "<group>"; };
+               939C0D2325648C4E00B3211B /* SpeechRecognizer.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = SpeechRecognizer.cpp; sourceTree = "<group>"; };
+               939C0D282564E7F200B3211B /* MediaUtilities.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MediaUtilities.cpp; sourceTree = "<group>"; };
+               939C0D292564E7F200B3211B /* MediaUtilities.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MediaUtilities.h; sourceTree = "<group>"; };
</ins><span class="cx">           93A0481B254954E4000AC462 /* SpeechRecognitionResultData.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = SpeechRecognitionResultData.h; sourceTree = "<group>"; };
</span><span class="cx">          93A0481D254954E5000AC462 /* SpeechRecognitionConnection.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = SpeechRecognitionConnection.h; sourceTree = "<group>"; };
</span><span class="cx">          93A0481F254954E6000AC462 /* SpeechRecognitionRequest.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = SpeechRecognitionRequest.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -11539,6 +11547,10 @@
</span><span class="cx">          93F1D5BE12D5335600832BEC /* JSWebGLLoseContext.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JSWebGLLoseContext.cpp; sourceTree = "<group>"; };
</span><span class="cx">          93F1D5BF12D5335600832BEC /* JSWebGLLoseContext.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSWebGLLoseContext.h; sourceTree = "<group>"; };
</span><span class="cx">          93F2CC912427FB9A005851D8 /* CharacterRange.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CharacterRange.h; sourceTree = "<group>"; };
</span><ins>+               93F6B81B25679F6F00A08488 /* SpeechRecognitionCaptureSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpeechRecognitionCaptureSource.h; sourceTree = "<group>"; };
+               93F6B81C25679F7000A08488 /* SpeechRecognitionCaptureSourceImpl.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpeechRecognitionCaptureSourceImpl.h; sourceTree = "<group>"; };
+               93F6B81D25679F7000A08488 /* SpeechRecognitionCaptureSourceImpl.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SpeechRecognitionCaptureSourceImpl.cpp; sourceTree = "<group>"; };
+               93F6B81E25679F7100A08488 /* SpeechRecognitionCaptureSource.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SpeechRecognitionCaptureSource.cpp; sourceTree = "<group>"; };
</ins><span class="cx">           93F6F1EA127F70B10055CB06 /* WebGLContextEvent.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebGLContextEvent.cpp; sourceTree = "<group>"; };
</span><span class="cx">          93F6F1EB127F70B10055CB06 /* WebGLContextEvent.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebGLContextEvent.h; sourceTree = "<group>"; };
</span><span class="cx">          93F6F1EC127F70B10055CB06 /* WebGLContextEvent.idl */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = WebGLContextEvent.idl; sourceTree = "<group>"; };
</span><span class="lines">@@ -23587,6 +23599,13 @@
</span><span class="cx">                  path = cocoa;
</span><span class="cx">                  sourceTree = "<group>";
</span><span class="cx">          };
</span><ins>+               93F6B80E2566EDE100A08488 /* cocoa */ = {
+                       isa = PBXGroup;
+                       children = (
+                       );
+                       path = cocoa;
+                       sourceTree = "<group>";
+               };
</ins><span class="cx">           946D37271D6CB2250077084F /* parser */ = {
</span><span class="cx">                  isa = PBXGroup;
</span><span class="cx">                  children = (
</span><span class="lines">@@ -24423,6 +24442,8 @@
</span><span class="cx">                          A5C974D011485FF10066F2AB /* KeyEventCocoa.mm */,
</span><span class="cx">                          06E81ED60AB5D5E900C87837 /* LocalCurrentGraphicsContext.h */,
</span><span class="cx">                          1A4832B21A953BA6008B4DFE /* LocalizedStringsCocoa.mm */,
</span><ins>+                               939C0D282564E7F200B3211B /* MediaUtilities.cpp */,
+                               939C0D292564E7F200B3211B /* MediaUtilities.h */,
</ins><span class="cx">                           C53D39331C97892D007F3AE9 /* MIMETypeRegistryCocoa.mm */,
</span><span class="cx">                          A19D93491AA11B1E00B46C24 /* NetworkExtensionContentFilter.h */,
</span><span class="cx">                          A19D93481AA11B1E00B46C24 /* NetworkExtensionContentFilter.mm */,
</span><span class="lines">@@ -25615,6 +25636,7 @@
</span><span class="cx">          AA2A5AB716A485A400975A25 /* speech */ = {
</span><span class="cx">                  isa = PBXGroup;
</span><span class="cx">                  children = (
</span><ins>+                               93F6B80E2566EDE100A08488 /* cocoa */,
</ins><span class="cx">                           AA2A5ABA16A485D500975A25 /* DOMWindow+SpeechSynthesis.idl */,
</span><span class="cx">                          AA2A5AB816A485D500975A25 /* DOMWindowSpeechSynthesis.cpp */,
</span><span class="cx">                          AA2A5AB916A485D500975A25 /* DOMWindowSpeechSynthesis.h */,
</span><span class="lines">@@ -25624,6 +25646,10 @@
</span><span class="cx">                          934950B72539434B0099F171 /* SpeechRecognitionAlternative.cpp */,
</span><span class="cx">                          934950BC2539434E0099F171 /* SpeechRecognitionAlternative.h */,
</span><span class="cx">                          934950BB2539434E0099F171 /* SpeechRecognitionAlternative.idl */,
</span><ins>+                               93F6B81E25679F7100A08488 /* SpeechRecognitionCaptureSource.cpp */,
+                               93F6B81B25679F6F00A08488 /* SpeechRecognitionCaptureSource.h */,
+                               93F6B81D25679F7000A08488 /* SpeechRecognitionCaptureSourceImpl.cpp */,
+                               93F6B81C25679F7000A08488 /* SpeechRecognitionCaptureSourceImpl.h */,
</ins><span class="cx">                           93A0481D254954E5000AC462 /* SpeechRecognitionConnection.h */,
</span><span class="cx">                          93A04824254954E9000AC462 /* SpeechRecognitionConnectionClient.h */,
</span><span class="cx">                          93A04820254954E6000AC462 /* SpeechRecognitionConnectionClientIdentifier.h */,
</span><span class="lines">@@ -25648,6 +25674,8 @@
</span><span class="cx">                          934950C5253943530099F171 /* SpeechRecognitionResultList.idl */,
</span><span class="cx">                          93D6B76E254B8E1B0058DD3A /* SpeechRecognitionUpdate.cpp */,
</span><span class="cx">                          93D6B76D254B8E1B0058DD3A /* SpeechRecognitionUpdate.h */,
</span><ins>+                               939C0D2325648C4E00B3211B /* SpeechRecognizer.cpp */,
+                               939C0D2125648C3900B3211B /* SpeechRecognizer.h */,
</ins><span class="cx">                           AA2A5ABD16A485D500975A25 /* SpeechSynthesis.cpp */,
</span><span class="cx">                          AA2A5ABE16A485D500975A25 /* SpeechSynthesis.h */,
</span><span class="cx">                          AA2A5ABF16A485D500975A25 /* SpeechSynthesis.idl */,
</span><span class="lines">@@ -33534,6 +33562,7 @@
</span><span class="cx">                          932CC0B71DFFD158004C0F9F /* MediaTrackConstraints.h in Headers */,
</span><span class="cx">                          07C1C0E21BFB600100BD2256 /* MediaTrackSupportedConstraints.h in Headers */,
</span><span class="cx">                          07611DC12440E59B00D80704 /* MediaUsageInfo.h in Headers */,
</span><ins>+                               939C0D2B2564E7F300B3211B /* MediaUtilities.h in Headers */,
</ins><span class="cx">                           51E1BAC31BD8064E0055D81F /* MemoryBackingStoreTransaction.h in Headers */,
</span><span class="cx">                          BCB16C180979C3BD00467741 /* MemoryCache.h in Headers */,
</span><span class="cx">                          517139081BF64DEF000D5F01 /* MemoryCursor.h in Headers */,
</span><span class="lines">@@ -34426,6 +34455,8 @@
</span><span class="cx">                          626CDE0F1140424C001E5A68 /* SpatialNavigation.h in Headers */,
</span><span class="cx">                          934950CD253943610099F171 /* SpeechRecognition.h in Headers */,
</span><span class="cx">                          934950CE253943650099F171 /* SpeechRecognitionAlternative.h in Headers */,
</span><ins>+                               93F6B8222567A65600A08488 /* SpeechRecognitionCaptureSource.h in Headers */,
+                               93F6B81F2567A08C00A08488 /* SpeechRecognitionCaptureSourceImpl.h in Headers */,
</ins><span class="cx">                           93A0482C25495519000AC462 /* SpeechRecognitionConnection.h in Headers */,
</span><span class="cx">                          93A0482E2549551E000AC462 /* SpeechRecognitionConnectionClient.h in Headers */,
</span><span class="cx">                          93A0482D2549551B000AC462 /* SpeechRecognitionConnectionClientIdentifier.h in Headers */,
</span><span class="lines">@@ -34440,6 +34471,7 @@
</span><span class="cx">                          93A0482925495511000AC462 /* SpeechRecognitionResultData.h in Headers */,
</span><span class="cx">                          934950D6253943810099F171 /* SpeechRecognitionResultList.h in Headers */,
</span><span class="cx">                          93D6B771254BAB450058DD3A /* SpeechRecognitionUpdate.h in Headers */,
</span><ins>+                               939C0D272564E47F00B3211B /* SpeechRecognizer.h in Headers */,
</ins><span class="cx">                           AA2A5AD416A4861100975A25 /* SpeechSynthesis.h in Headers */,
</span><span class="cx">                          C14938072234551A000CD707 /* SpeechSynthesisClient.h in Headers */,
</span><span class="cx">                          AA2A5AD216A4860A00975A25 /* SpeechSynthesisEvent.h in Headers */,
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformcocoaMediaUtilitiescpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/platform/cocoa/MediaUtilities.cpp (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/cocoa/MediaUtilities.cpp                           (rev 0)
+++ trunk/Source/WebCore/platform/cocoa/MediaUtilities.cpp      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,73 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "MediaUtilities.h"
+
+#include "AudioStreamDescription.h"
+#include "WebAudioBufferList.h"
+#include <wtf/SoftLinking.h>
+#include <pal/cf/CoreMediaSoftLink.h>
+
+namespace WebCore {
+
+using namespace PAL;
+
+RetainPtr<CMFormatDescriptionRef> createAudioFormatDescription(const AudioStreamDescription& description)
+{
+    auto basicDescription = WTF::get<const AudioStreamBasicDescription*>(description.platformDescription().description);
+    CMFormatDescriptionRef format = nullptr;
+    auto error = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, basicDescription, 0, nullptr, 0, nullptr, nullptr, &format);
+    if (error) {
+        LOG_ERROR("createAudioFormatDescription failed with %d", error);
+        return nullptr;
+    }
+    return adoptCF(format);
+}
+
+RetainPtr<CMSampleBufferRef> createAudioSampleBuffer(const PlatformAudioData& data, const AudioStreamDescription& description, CMTime time, size_t sampleCount)
+{
+    // FIXME: check if we can reuse the format for multiple sample buffers.
+    auto format = createAudioFormatDescription(description);
+    if (!format)
+        return nullptr;
+
+    CMSampleBufferRef sampleBuffer = nullptr;
+    auto error = CMAudioSampleBufferCreateWithPacketDescriptions(kCFAllocatorDefault, nullptr, false, nullptr, nullptr, format.get(), sampleCount, time, nullptr, &sampleBuffer);
+    if (error) {
+        LOG_ERROR("createAudioSampleBuffer with packet descriptions failed - %d", error);
+        return nullptr;
+    }
+    auto buffer = adoptCF(sampleBuffer);
+
+    error = CMSampleBufferSetDataBufferFromAudioBufferList(buffer.get(), kCFAllocatorDefault, kCFAllocatorDefault, 0, downcast<WebAudioBufferList>(data).list());
+    if (error) {
+        LOG_ERROR("createAudioSampleBuffer from audio buffer list failed - %d", error);
+        return nullptr;
+    }
+    return buffer;
+}
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreplatformcocoaMediaUtilitiesh"></a>
<div class="addfile"><h4>Added: trunk/Source/WebCore/platform/cocoa/MediaUtilities.h (0 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/cocoa/MediaUtilities.h                             (rev 0)
+++ trunk/Source/WebCore/platform/cocoa/MediaUtilities.h        2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -0,0 +1,42 @@
</span><ins>+/*
+ * Copyright (C) 2020 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include <CoreMedia/CMTime.h>
+#include <wtf/RetainPtr.h>
+
+typedef const struct opaqueCMFormatDescription* CMFormatDescriptionRef;
+typedef struct opaqueCMSampleBuffer* CMSampleBufferRef;
+
+namespace WebCore {
+
+class AudioStreamDescription;
+class PlatformAudioData;
+
+RetainPtr<CMFormatDescriptionRef> createAudioFormatDescription(const AudioStreamDescription&);
+RetainPtr<CMSampleBufferRef> createAudioSampleBuffer(const PlatformAudioData&, const AudioStreamDescription&, CMTime, size_t sampleCount);
+
+} // namespace WebCore
</ins></span></pre></div>
<a id="trunkSourceWebCoreplatformmediarecordercocoaMediaRecorderPrivateWriterCocoamm"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm     2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm        2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -34,6 +34,7 @@
</span><span class="cx"> #include "MediaRecorderPrivate.h"
</span><span class="cx"> #include "MediaRecorderPrivateOptions.h"
</span><span class="cx"> #include "MediaStreamTrackPrivate.h"
</span><ins>+#include "MediaUtilities.h"
</ins><span class="cx"> #include "VideoSampleBufferCompressor.h"
</span><span class="cx"> #include "WebAudioBufferList.h"
</span><span class="cx"> #include <AVFoundation/AVAssetWriter.h>
</span><span class="lines">@@ -443,40 +444,6 @@
</span><span class="cx">         m_videoCompressor->addSampleBuffer(bufferWithCurrentTime.get());
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-static inline RetainPtr<CMFormatDescriptionRef> createAudioFormatDescription(const AudioStreamDescription& description)
-{
-    auto basicDescription = WTF::get<const AudioStreamBasicDescription*>(description.platformDescription().description);
-    CMFormatDescriptionRef format = nullptr;
-    auto error = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, basicDescription, 0, NULL, 0, NULL, NULL, &format);
-    if (error) {
-        RELEASE_LOG_ERROR(MediaStream, "MediaRecorderPrivateWriter CMAudioFormatDescriptionCreate failed with %d", error);
-        return nullptr;
-    }
-    return adoptCF(format);
-}
-
-static inline RetainPtr<CMSampleBufferRef> createAudioSampleBuffer(const PlatformAudioData& data, const AudioStreamDescription& description, CMTime time, size_t sampleCount)
-{
-    auto format = createAudioFormatDescription(description);
-    if (!format)
-        return nullptr;
-
-    CMSampleBufferRef sampleBuffer = nullptr;
-    auto error = CMAudioSampleBufferCreateWithPacketDescriptions(kCFAllocatorDefault, NULL, false, NULL, NULL, format.get(), sampleCount, time, NULL, &sampleBuffer);
-    if (error) {
-        RELEASE_LOG_ERROR(MediaStream, "MediaRecorderPrivateWriter createAudioSampleBufferWithPacketDescriptions failed with %d", error);
-        return nullptr;
-    }
-    auto buffer = adoptCF(sampleBuffer);
-
-    error = CMSampleBufferSetDataBufferFromAudioBufferList(buffer.get(), kCFAllocatorDefault, kCFAllocatorDefault, 0, downcast<WebAudioBufferList>(data).list());
-    if (error) {
-        RELEASE_LOG_ERROR(MediaStream, "MediaRecorderPrivateWriter CMSampleBufferSetDataBufferFromAudioBufferList failed with %d", error);
-        return nullptr;
-    }
-    return buffer;
-}
-
</del><span class="cx"> void MediaRecorderPrivateWriter::appendAudioSampleBuffer(const PlatformAudioData& data, const AudioStreamDescription& description, const WTF::MediaTime&, size_t sampleCount)
</span><span class="cx"> {
</span><span class="cx">     if (auto sampleBuffer = createAudioSampleBuffer(data, description, m_currentAudioSampleTime, sampleCount))
</span></span></pre></div>
<a id="trunkSourceWebKitChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/ChangeLog (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/ChangeLog    2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/ChangeLog       2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -1,3 +1,29 @@
</span><ins>+2020-11-21  Sihui Liu  <sihui_liu@apple.com>
+
+        Implement audio capture for SpeechRecognition on macOS
+        https://bugs.webkit.org/show_bug.cgi?id=218855
+        <rdar://problem/71331001>
+
+        Reviewed by Youenn Fablet.
+
+        * UIProcess/SpeechRecognitionPermissionManager.cpp:
+        (WebKit::SpeechRecognitionPermissionManager::startProcessingRequest): Check and enable mock devices based on 
+        preference as SpeechRecognition needs it for testing.
+        * UIProcess/SpeechRecognitionServer.cpp:
+        (WebKit::SpeechRecognitionServer::start):
+        (WebKit::SpeechRecognitionServer::requestPermissionForRequest):
+        (WebKit::SpeechRecognitionServer::handleRequest):
+        (WebKit::SpeechRecognitionServer::stop):
+        (WebKit::SpeechRecognitionServer::abort):
+        (WebKit::SpeechRecognitionServer::invalidate):
+        (WebKit::SpeechRecognitionServer::sendUpdate):
+        (WebKit::SpeechRecognitionServer::stopRequest): Deleted.
+        (WebKit::SpeechRecognitionServer::abortRequest): Deleted.
+        * UIProcess/SpeechRecognitionServer.h:
+        * UIProcess/WebPageProxy.cpp:
+        (WebKit::WebPageProxy::syncIfMockDevicesEnabledChanged):
+        * UIProcess/WebPageProxy.h:
+
</ins><span class="cx"> 2020-11-21  Simon Fraser  <simon.fraser@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Propagate the 'wheelEventGesturesBecomeNonBlocking' setting to the ScrollingTree
</span></span></pre></div>
<a id="trunkSourceWebKitUIProcessSpeechRecognitionPermissionManagercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/UIProcess/SpeechRecognitionPermissionManager.cpp (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/UIProcess/SpeechRecognitionPermissionManager.cpp     2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/UIProcess/SpeechRecognitionPermissionManager.cpp        2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -104,6 +104,7 @@
</span><span class="cx">     m_speechRecognitionServiceCheck = computeSpeechRecognitionServiceAccess();
</span><span class="cx"> 
</span><span class="cx">     if (m_page.preferences().mockCaptureDevicesEnabled()) {
</span><ins>+        m_page.syncIfMockDevicesEnabledChanged();
</ins><span class="cx">         m_microphoneCheck = CheckResult::Granted;
</span><span class="cx">         m_speechRecognitionServiceCheck = CheckResult::Granted;
</span><span class="cx">     }
</span></span></pre></div>
<a id="trunkSourceWebKitUIProcessSpeechRecognitionServercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.cpp (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.cpp        2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.cpp   2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -46,12 +46,11 @@
</span><span class="cx"> void SpeechRecognitionServer::start(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier, String&& lang, bool continuous, bool interimResults, uint64_t maxAlternatives, WebCore::ClientOrigin&& origin)
</span><span class="cx"> {
</span><span class="cx">     MESSAGE_CHECK(clientIdentifier);
</span><del>-    ASSERT(!m_pendingRequests.contains(clientIdentifier));
-    ASSERT(!m_ongoingRequests.contains(clientIdentifier));
</del><ins>+    ASSERT(!m_requests.contains(clientIdentifier));
</ins><span class="cx">     auto requestInfo = WebCore::SpeechRecognitionRequestInfo { clientIdentifier, WTFMove(lang), continuous, interimResults, maxAlternatives, WTFMove(origin) };
</span><del>-    auto& pendingRequest = m_pendingRequests.add(clientIdentifier, makeUnique<WebCore::SpeechRecognitionRequest>(WTFMove(requestInfo))).iterator->value;
</del><ins>+    auto& newRequest = m_requests.add(clientIdentifier, makeUnique<WebCore::SpeechRecognitionRequest>(WTFMove(requestInfo))).iterator->value;
</ins><span class="cx"> 
</span><del>-    requestPermissionForRequest(*pendingRequest);
</del><ins>+    requestPermissionForRequest(*newRequest);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void SpeechRecognitionServer::requestPermissionForRequest(WebCore::SpeechRecognitionRequest& request)
</span><span class="lines">@@ -64,68 +63,72 @@
</span><span class="cx">             return;
</span><span class="cx"> 
</span><span class="cx">         auto identifier = weakRequest->clientIdentifier();
</span><del>-        auto takenRequest = m_pendingRequests.take(identifier);
</del><span class="cx">         if (decision == SpeechRecognitionPermissionDecision::Deny) {
</span><ins>+            m_requests.remove(identifier);
</ins><span class="cx">             auto error = WebCore::SpeechRecognitionError { WebCore::SpeechRecognitionErrorType::NotAllowed, "Permission check failed"_s };
</span><span class="cx">             sendUpdate(identifier, WebCore::SpeechRecognitionUpdateType::Error, error);
</span><span class="cx">             return;
</span><span class="cx">         }
</span><span class="cx"> 
</span><del>-        m_ongoingRequests.add(identifier, WTFMove(takenRequest));
-        handleRequest(*m_ongoingRequests.get(identifier));
</del><ins>+        handleRequest(identifier);
</ins><span class="cx">     });
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void SpeechRecognitionServer::handleRequest(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier)
+{
+    if (!m_recognizer) {
+        m_recognizer = makeUnique<SpeechRecognizer>([this, weakThis = makeWeakPtr(this)](auto& update) {
+            if (!weakThis)
+                return;
+
+            auto clientIdentifier = update.clientIdentifier();
+            if (!m_requests.contains(clientIdentifier))
+                return;
+
+            auto type = update.type();
+            if (type == SpeechRecognitionUpdateType::Error || type == SpeechRecognitionUpdateType::End)
+                m_requests.remove(clientIdentifier);
+
+            sendUpdate(update);
+        });
+    }
+
+    m_recognizer->start(clientIdentifier);
+}
+
</ins><span class="cx"> void SpeechRecognitionServer::stop(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier)
</span><span class="cx"> {
</span><span class="cx">     MESSAGE_CHECK(clientIdentifier);
</span><del>-    if (m_pendingRequests.remove(clientIdentifier)) {
-        sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::End);
</del><ins>+    if (m_recognizer && m_recognizer->currentClientIdentifier() == clientIdentifier) {
+        m_recognizer->stop();
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    ASSERT(m_ongoingRequests.contains(clientIdentifier));
-    stopRequest(*m_ongoingRequests.get(clientIdentifier));
</del><ins>+    if (m_requests.remove(clientIdentifier))
+        sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::End);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void SpeechRecognitionServer::abort(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier)
</span><span class="cx"> {
</span><span class="cx">     MESSAGE_CHECK(clientIdentifier);
</span><del>-    if (m_pendingRequests.remove(clientIdentifier)) {
-        sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::End);
</del><ins>+    if (m_recognizer && m_recognizer->currentClientIdentifier() == clientIdentifier) {
+        m_recognizer->stop(WebCore::SpeechRecognizer::ShouldGenerateFinalResult::No);
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    ASSERT(m_ongoingRequests.contains(clientIdentifier));
-    auto request = m_ongoingRequests.take(clientIdentifier);
-    abortRequest(*request);
-    auto update = WebCore::SpeechRecognitionUpdate::create(clientIdentifier, WebCore::SpeechRecognitionUpdateType::End);
-    send(Messages::WebSpeechRecognitionConnection::DidReceiveUpdate(update), m_identifier);
</del><ins>+    if (m_requests.remove(clientIdentifier))
+        sendUpdate(clientIdentifier, WebCore::SpeechRecognitionUpdateType::End);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void SpeechRecognitionServer::invalidate(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier)
</span><span class="cx"> {
</span><span class="cx">     MESSAGE_CHECK(clientIdentifier);
</span><del>-    auto request = m_ongoingRequests.take(clientIdentifier);
-    if (request)
-        abortRequest(*request);
</del><ins>+    if (m_requests.remove(clientIdentifier)) {
+        if (m_recognizer && m_recognizer->currentClientIdentifier() == clientIdentifier)
+            m_recognizer->stop();
+    }
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-void SpeechRecognitionServer::handleRequest(WebCore::SpeechRecognitionRequest& request)
-{
-    // TODO: start capturing audio and recognition.
-}
-
-void SpeechRecognitionServer::stopRequest(WebCore::SpeechRecognitionRequest& request)
-{
-    // TODO: stop capturing audio and finalizing results by recognizing captured audio.
-}
-
-void SpeechRecognitionServer::abortRequest(WebCore::SpeechRecognitionRequest& request)
-{
-    // TODO: stop capturing audio and recognition immediately without generating results.
-}
-
</del><span class="cx"> void SpeechRecognitionServer::sendUpdate(WebCore::SpeechRecognitionConnectionClientIdentifier clientIdentifier, WebCore::SpeechRecognitionUpdateType type, Optional<WebCore::SpeechRecognitionError> error, Optional<Vector<WebCore::SpeechRecognitionResultData>> result)
</span><span class="cx"> {
</span><span class="cx">     auto update = WebCore::SpeechRecognitionUpdate::create(clientIdentifier, type);
</span><span class="lines">@@ -133,9 +136,14 @@
</span><span class="cx">         update = WebCore::SpeechRecognitionUpdate::createError(clientIdentifier, *error);
</span><span class="cx">     if (type == WebCore::SpeechRecognitionUpdateType::Result)
</span><span class="cx">         update = WebCore::SpeechRecognitionUpdate::createResult(clientIdentifier, *result);
</span><del>-    send(Messages::WebSpeechRecognitionConnection::DidReceiveUpdate(update), m_identifier);
</del><ins>+    sendUpdate(update);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void SpeechRecognitionServer::sendUpdate(const WebCore::SpeechRecognitionUpdate& update)
+{
+    send(Messages::WebSpeechRecognitionConnection::DidReceiveUpdate(update));
+}
+
</ins><span class="cx"> IPC::Connection* SpeechRecognitionServer::messageSenderConnection() const
</span><span class="cx"> {
</span><span class="cx">     return m_connection.ptr();
</span></span></pre></div>
<a id="trunkSourceWebKitUIProcessSpeechRecognitionServerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.h (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.h  2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/UIProcess/SpeechRecognitionServer.h     2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -31,6 +31,7 @@
</span><span class="cx"> #include <WebCore/SpeechRecognitionError.h>
</span><span class="cx"> #include <WebCore/SpeechRecognitionRequest.h>
</span><span class="cx"> #include <WebCore/SpeechRecognitionResultData.h>
</span><ins>+#include <WebCore/SpeechRecognizer.h>
</ins><span class="cx"> #include <wtf/Deque.h>
</span><span class="cx"> 
</span><span class="cx"> namespace WebCore {
</span><span class="lines">@@ -58,10 +59,9 @@
</span><span class="cx"> 
</span><span class="cx"> private:
</span><span class="cx">     void requestPermissionForRequest(WebCore::SpeechRecognitionRequest&);
</span><del>-    void handleRequest(WebCore::SpeechRecognitionRequest&);
-    void stopRequest(WebCore::SpeechRecognitionRequest&);
-    void abortRequest(WebCore::SpeechRecognitionRequest&);
</del><ins>+    void handleRequest(WebCore::SpeechRecognitionConnectionClientIdentifier);
</ins><span class="cx">     void sendUpdate(WebCore::SpeechRecognitionConnectionClientIdentifier, WebCore::SpeechRecognitionUpdateType, Optional<WebCore::SpeechRecognitionError> = WTF::nullopt, Optional<Vector<WebCore::SpeechRecognitionResultData>> = WTF::nullopt);
</span><ins>+    void sendUpdate(const WebCore::SpeechRecognitionUpdate&);
</ins><span class="cx"> 
</span><span class="cx">     // IPC::MessageReceiver.
</span><span class="cx">     void didReceiveMessage(IPC::Connection&, IPC::Decoder&) override;
</span><span class="lines">@@ -72,9 +72,9 @@
</span><span class="cx"> 
</span><span class="cx">     Ref<IPC::Connection> m_connection;
</span><span class="cx">     SpeechRecognitionServerIdentifier m_identifier;
</span><del>-    HashMap<WebCore::SpeechRecognitionConnectionClientIdentifier, std::unique_ptr<WebCore::SpeechRecognitionRequest>> m_pendingRequests;
-    HashMap<WebCore::SpeechRecognitionConnectionClientIdentifier, std::unique_ptr<WebCore::SpeechRecognitionRequest>> m_ongoingRequests;
</del><ins>+    HashMap<WebCore::SpeechRecognitionConnectionClientIdentifier, std::unique_ptr<WebCore::SpeechRecognitionRequest>> m_requests;
</ins><span class="cx">     SpeechRecognitionPermissionChecker m_permissionChecker;
</span><ins>+    std::unique_ptr<WebCore::SpeechRecognizer> m_recognizer;
</ins><span class="cx"> };
</span><span class="cx"> 
</span><span class="cx"> } // namespace WebKit
</span></span></pre></div>
<a id="trunkSourceWebKitUIProcessWebPageProxycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.cpp (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/UIProcess/WebPageProxy.cpp   2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.cpp      2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -8174,6 +8174,13 @@
</span><span class="cx"> #endif
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void WebPageProxy::syncIfMockDevicesEnabledChanged()
+{
+#if ENABLE(MEDIA_STREAM)
+    userMediaPermissionRequestManager().syncWithWebCorePrefs();
+#endif
+}
+
</ins><span class="cx"> void WebPageProxy::beginMonitoringCaptureDevices()
</span><span class="cx"> {
</span><span class="cx"> #if ENABLE(MEDIA_STREAM)
</span></span></pre></div>
<a id="trunkSourceWebKitUIProcessWebPageProxyh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit/UIProcess/WebPageProxy.h (270157 => 270158)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit/UIProcess/WebPageProxy.h     2020-11-22 04:51:13 UTC (rev 270157)
+++ trunk/Source/WebKit/UIProcess/WebPageProxy.h        2020-11-22 05:51:10 UTC (rev 270158)
</span><span class="lines">@@ -1825,6 +1825,8 @@
</span><span class="cx">     void requestSpeechRecognitionPermission(const WebCore::ClientOrigin&, CompletionHandler<void(SpeechRecognitionPermissionDecision)>&&);
</span><span class="cx">     void requestSpeechRecognitionPermissionByDefaultAction(const WebCore::SecurityOrigin&, CompletionHandler<void(bool)>&&);
</span><span class="cx"> 
</span><ins>+    void syncIfMockDevicesEnabledChanged();
+
</ins><span class="cx"> private:
</span><span class="cx">     WebPageProxy(PageClient&, WebProcessProxy&, Ref<API::PageConfiguration>&&);
</span><span class="cx">     void platformInitialize();
</span></span></pre>
</div>
</div>

</body>
</html>