[Webkit-unassigned] [Bug 218815] New: speechSynthesis.speak pauses video elements, mutes getUserMedia streams

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=218815

            Bug ID: 218815
           Summary: speechSynthesis.speak pauses video elements, mutes
                    getUserMedia streams
           Product: WebKit
           Version: Safari 14
          Hardware: iPhone / iPad
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: WebKit Misc.
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: andrew.vermie at gmail.com

Created attachment 413846

  --> https://bugs.webkit.org/attachment.cgi?id=413846&action=review

repro page

On Safari, speechSynthesis.speak invocations interrupt other active media.
To reproduce:
 1) get a MediaStream from getUserMedia, add it to a video element, play the video element
 2) call speechSynthesis.speak(new SpeechSynthesisUtterance('hello')

This interrupts both the video element and media stream indepedently.

 1) (mobile safari) video elements are paused
Calling videoElement.play() resumes playback.
If video playback is resumed before the SpeechSynthesisUtterance ends, the speech is sometimes interrupted (this part is really inconsistent).
It seems like, after resuming playback from a user-gesture handler, future calls to speak no longer pause this video.
If, on the other hand, the video is resumed from a non-user-gesture (e.g. in response to utterance.end event), future calls to speak will re-pause the video.

 2) (mobile safari) audio tracks from getUserMedia are muted (MediaStreamTrack - mute event is raised, muted is true)
The track is never unmuted, and there is no API for user-code to unmute a track. It seems the only option would be to stop the track, and get a new one by getUserMedia.

 3) (desktop safari) audio tracks from getUserMedia are silenced (all samples are 0) while the utterance is playing (mute event is not raised, volume is restored after utterance ends)


This may not be a complete list - I haven't checked how Web Speech interacts with other APIs, e.g. Web Audio.

For Mobile Safari, I tested on iOS 14.1
For Desktop Safari, I tested on Safari 13.1.3 (OSX 10.15.7)
Other browsers (Chrome, Firefox) do not exhibit these issues.

It would be great if Safari behaved the same as Chrome or Firefox.
At the bare minimum, I ask that media streams could be unmuted after speech finishes.

I have attached a page which demonstrates the behavior.
The page includes WebRTC, which serves to demonstrate that only the getUserMedia-created stream is muted.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20201111/8127d11c/attachment-0001.htm>