<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[282196] trunk</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/282196">282196</a></dd>
<dt>Author</dt> <dd>jya@apple.com</dd>
<dt>Date</dt> <dd>2021-09-09 00:02:21 -0700 (Thu, 09 Sep 2021)</dd>
</dl>

<h3>Log Message</h3>
<pre>[WebAudio/WebM] Incorrect number of frames returned if decoding frame rate doesn't match original
https://bugs.webkit.org/show_bug.cgi?id=229251
rdar://problem/82095650

Source/WebCore:

Reviewed by Eric Carlson..

We can't rely on CoreMedia to performed the trimming correctly when resampling is also to be done.
It gives unexpected results. Let's do it ourselves instead.
Test: webaudio/decode-audio-data-webm-opus-resample.html

* platform/audio/cocoa/AudioFileReaderCocoa.cpp:
(WebCore::AudioFileReader::decodeWebMData const):

LayoutTests:

Reviewed by Eric Carlson.

* webaudio/decode-audio-data-webm-opus-resample-expected.txt: Added.
* webaudio/decode-audio-data-webm-opus-resample.html: Added.</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkLayoutTestsChangeLog">trunk/LayoutTests/ChangeLog</a></li>
<li><a href="#trunkLayoutTestsplatformmacTestExpectations">trunk/LayoutTests/platform/mac/TestExpectations</a></li>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCoreplatformaudiococoaAudioFileReaderCocoacpp">trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp</a></li>
</ul>

<h3>Added Paths</h3>
<ul>
<li><a href="#trunkLayoutTestswebaudiodecodeaudiodatawebmopusresampleexpectedtxt">trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample-expected.txt</a></li>
<li><a href="#trunkLayoutTestswebaudiodecodeaudiodatawebmopusresamplehtml">trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample.html</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkLayoutTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/ChangeLog (282195 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/ChangeLog      2021-09-09 07:01:12 UTC (rev 282195)
+++ trunk/LayoutTests/ChangeLog 2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -1,3 +1,14 @@
</span><ins>+2021-09-09  Jean-Yves Avenard  <jya@apple.com>
+
+        [WebAudio/WebM] Incorrect number of frames returned if decoding frame rate doesn't match original
+        https://bugs.webkit.org/show_bug.cgi?id=229251
+        rdar://problem/82095650
+
+        Reviewed by Eric Carlson.
+
+        * webaudio/decode-audio-data-webm-opus-resample-expected.txt: Added.
+        * webaudio/decode-audio-data-webm-opus-resample.html: Added.
+
</ins><span class="cx"> 2021-09-07  Tim Nguyen  <ntim@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Re-import css/css-pseudo WPT
</span></span></pre></div>
<a id="trunkLayoutTestsplatformmacTestExpectations"></a>
<div class="modfile"><h4>Modified: trunk/LayoutTests/platform/mac/TestExpectations (282195 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/platform/mac/TestExpectations  2021-09-09 07:01:12 UTC (rev 282195)
+++ trunk/LayoutTests/platform/mac/TestExpectations     2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -1712,6 +1712,7 @@
</span><span class="cx"> [ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial.html [ Skip ]
</span><span class="cx"> [ Catalina Mojave BigSur ] media/media-source/media-webm-opus-partial-abort.html [ Skip ]
</span><span class="cx"> [ Catalina Mojave BigSur ] webaudio/decode-audio-data-webm-opus.html [ Skip ]
</span><ins>+[ Catalina Mojave BigSur ] webaudio/decode-audio-data-webm-opus-resample.html [ Skip ]
</ins><span class="cx"> [ Catalina Mojave BigSur ] webaudio/decode-audio-data-webm-vorbis.html [ Skip ]
</span><span class="cx"> 
</span><span class="cx"> webkit.org/b/214422 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-audiocontext-interface/suspend-after-construct.html [ Pass Failure ]
</span></span></pre></div>
<a id="trunkLayoutTestswebaudiodecodeaudiodatawebmopusresampleexpectedtxt"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample-expected.txt (0 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample-expected.txt                             (rev 0)
+++ trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample-expected.txt        2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -0,0 +1,11 @@
</span><ins>+Test that decoding an opus webm file with resampling succeeds
+
+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
+
+
+PASS Successfully decoded content
+PASS Decoding returned the right number of frames.
+PASS successfullyParsed is true
+
+TEST COMPLETE
+
</ins></span></pre></div>
<a id="trunkLayoutTestswebaudiodecodeaudiodatawebmopusresamplehtml"></a>
<div class="addfile"><h4>Added: trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample.html (0 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample.html                             (rev 0)
+++ trunk/LayoutTests/webaudio/decode-audio-data-webm-opus-resample.html        2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -0,0 +1,36 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<head>
+<script src="../resources/js-test.js"></script>
+<script type="text/javascript" src="resources/audio-testing.js"></script>
+</head>
+<body>
+<script>
+description("Test that decoding an opus webm file with resampling succeeds");
+
+window.jsTestIsAsync = true;
+
+var context = new window.AudioContext({ sampleRate: 44100 });
+var request = new XMLHttpRequest();
+request.open("GET", 'resources/media/opus.webm', true);
+request.responseType = "arraybuffer";
+
+request.onload = function() {
+    context.decodeAudioData(request.response, (buffer) => {
+        testPassed("Successfully decoded content");
+        // File is exactly 1-0.0065s long @ 48000Hz, so 47688 frames, after resampling it should be 47688/48000*44100 = 43813.
+        if (buffer.length === 43813)
+          testPassed("Decoding returned the right number of frames.");
+        else
+          testFailed("Decoding returned the wrong number of frames: " + buffer.length);
+        finishJSTest();
+      }, () => {
+        testFailed("Failed to decode file");
+        finishJSTest();
+      });
+}
+request.send();
+
+</script>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (282195 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog   2021-09-09 07:01:12 UTC (rev 282195)
+++ trunk/Source/WebCore/ChangeLog      2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -1,3 +1,18 @@
</span><ins>+2021-09-09  Jean-Yves Avenard  <jya@apple.com>
+
+        [WebAudio/WebM] Incorrect number of frames returned if decoding frame rate doesn't match original
+        https://bugs.webkit.org/show_bug.cgi?id=229251
+        rdar://problem/82095650
+
+        Reviewed by Eric Carlson..
+
+        We can't rely on CoreMedia to performed the trimming correctly when resampling is also to be done.
+        It gives unexpected results. Let's do it ourselves instead.
+        Test: webaudio/decode-audio-data-webm-opus-resample.html
+
+        * platform/audio/cocoa/AudioFileReaderCocoa.cpp:
+        (WebCore::AudioFileReader::decodeWebMData const):
+
</ins><span class="cx"> 2021-09-08  Youenn Fablet  <youenn@apple.com>
</span><span class="cx"> 
</span><span class="cx">         RTCPeerConnection.addIceCandidate takes an optional argument
</span></span></pre></div>
<a id="trunkSourceWebCoreplatformaudiococoaAudioFileReaderCocoacpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp (282195 => 282196)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp       2021-09-09 07:01:12 UTC (rev 282195)
+++ trunk/Source/WebCore/platform/audio/cocoa/AudioFileReaderCocoa.cpp  2021-09-09 07:02:21 UTC (rev 282196)
</span><span class="lines">@@ -318,12 +318,6 @@
</span><span class="cx">     if (magicCookie && magicCookieSize)
</span><span class="cx">         PAL::AudioConverterSetProperty(converter, kAudioConverterDecompressionMagicCookie, magicCookieSize, magicCookie);
</span><span class="cx"> 
</span><del>-    AudioConverterPrimeInfo primeInfo = { UInt32(m_webmData->m_track->codecDelay().value_or(MediaTime()).toDouble() * outFormat.mSampleRate), 0 };
-    INFO_LOG(LOGIDENTIFIER, "Will drop %u leading frames out of %llu", primeInfo.leadingFrames, numberOfFrames);
-    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeInfo, sizeof(primeInfo), &primeInfo);
-    UInt32 primeMethod = kConverterPrimeMethod_None;
-    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeMethod, sizeof(primeMethod), &primeMethod);
-
</del><span class="cx">     AudioBufferListHolder decodedBufferList(inFormat.mChannelsPerFrame);
</span><span class="cx">     if (!decodedBufferList) {
</span><span class="cx">         RELEASE_LOG_FAULT(WebAudio, "Unable to create decoder");
</span><span class="lines">@@ -330,7 +324,20 @@
</span><span class="cx">         return { };
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    // Instruct the decoder to not drop any frames
+    // (by default the Opus decoder assumes that SampleRate / 400 frames are to be dropped.
+    AudioConverterPrimeInfo primeInfo = { 0, 0 };
+    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeInfo, sizeof(primeInfo), &primeInfo);
+    UInt32 primeMethod = kConverterPrimeMethod_None;
+    PAL::AudioConverterSetProperty(converter, kAudioConverterPrimeMethod, sizeof(primeMethod), &primeMethod);
+
+    uint32_t leadingTrim = m_webmData->m_track->codecDelay().value_or(MediaTime::zeroTime()).toDouble() * outFormat.mSampleRate;
+    // Calculate the number of trailing frames to be trimmed by rounding to nearest integer while minimizing cummulative rounding errors.
+    uint32_t trailingTrim = (m_webmData->m_track->codecDelay().value_or(MediaTime::zeroTime()) + m_webmData->m_track->discardPadding().value_or(MediaTime::zeroTime())).toDouble() * outFormat.mSampleRate - leadingTrim + 0.5;
+    INFO_LOG(LOGIDENTIFIER, "Will drop ", leadingTrim, " leading and ", trailingTrim, " trailing frames out of ", numberOfFrames);
+
</ins><span class="cx">     size_t decodedFrames = 0;
</span><ins>+    size_t totalDecodedFrames = 0;
</ins><span class="cx">     OSStatus status;
</span><span class="cx">     for (size_t i = 0; i < m_webmData->m_samples.size(); i++) {
</span><span class="cx">         auto& sample = m_webmData->m_samples[i];
</span><span class="lines">@@ -357,12 +364,14 @@
</span><span class="cx"> 
</span><span class="cx">         do {
</span><span class="cx">             if (numberOfFrames < decodedFrames) {
</span><del>-                RELEASE_LOG_FAULT(WebAudio, "Decoded more frames than first calculated");
</del><ins>+                RELEASE_LOG_FAULT(WebAudio, "Decoded more frames than first calculated, no available space left");
</ins><span class="cx">                 return { };
</span><span class="cx">             }
</span><span class="cx">             // in: the max number of packets we can handle from the decoder.
</span><span class="cx">             // out: the number of packets the decoder is actually returning.
</span><del>-            UInt32 numFrames = std::min<uint32_t>(std::numeric_limits<int32_t>::max() / sizeof(float), numberOfFrames - decodedFrames);
</del><ins>+            // The AudioConverter will sometimes pad with trailing silence if we set the free space to what it actually is (numberOfFrames - decodedFrames).
+            // So we set it to what there is left to decode instead.
+            UInt32 numFrames = std::min<uint32_t>(std::numeric_limits<int32_t>::max() / sizeof(float), numberOfFrames - totalDecodedFrames);
</ins><span class="cx"> 
</span><span class="cx">             for (UInt32 i = 0; i < inFormat.mChannelsPerFrame; i++) {
</span><span class="cx">                 decodedBufferList->mBuffers[i].mNumberChannels = 1;
</span><span class="lines">@@ -374,12 +383,19 @@
</span><span class="cx">                 RELEASE_LOG_FAULT(WebAudio, "Error decoding data");
</span><span class="cx">                 return { };
</span><span class="cx">             }
</span><ins>+            totalDecodedFrames += numFrames;
+            if (leadingTrim > 0) {
+                UInt32 toTrim = std::min(leadingTrim, numFrames);
+                for (UInt32 i = 0; i < outFormat.mChannelsPerFrame; i++)
+                    memcpy(decodedBufferList->mBuffers[i].mData, static_cast<float*>(decodedBufferList->mBuffers[i].mData) + toTrim, (numFrames - toTrim) * sizeof(float));
+                leadingTrim -= toTrim;
+                numFrames -= toTrim;
+            }
</ins><span class="cx">             decodedFrames += numFrames;
</span><span class="cx">         } while (status != kNoMoreDataErr && status != noErr);
</span><span class="cx">     }
</span><del>-    size_t paddingFrames = m_webmData->m_track->discardPadding().value_or(MediaTime()).toDouble() * outFormat.mSampleRate;
-    if (decodedFrames > paddingFrames)
-        return decodedFrames - paddingFrames;
</del><ins>+    if (decodedFrames > trailingTrim)
+        return decodedFrames - trailingTrim;
</ins><span class="cx">     return 0;
</span><span class="cx"> }
</span><span class="cx"> #endif
</span></span></pre>
</div>
</div>

</body>
</html>