[webkit-changes] [WebKit/WebKit] c2f759: AX: Support VTT-based extended audio descriptions

[Eric Carlson]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: c2f7594742ab1169aa5e66097a2a4de8f928a746
      https://github.com/WebKit/WebKit/commit/c2f7594742ab1169aa5e66097a2a4de8f928a746
  Author: Eric Carlson <eric.carlson at apple.com>
  Date:   2022-09-14 (Wed, 14 Sep 2022)

  Changed paths:
    M LayoutTests/TestExpectations
    A LayoutTests/media/track/captions-webvtt/captions-extended-descriptions.vtt
    M LayoutTests/media/track/track-description-cue.html
    A LayoutTests/media/track/track-extended-descriptions-expected.txt
    A LayoutTests/media/track/track-extended-descriptions.html
    M LayoutTests/platform/ios/TestExpectations
    M LayoutTests/platform/mac/TestExpectations
    M Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml
    M Source/WebCore/Modules/speech/SpeechSynthesis.cpp
    M Source/WebCore/Modules/speech/SpeechSynthesis.h
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/html/HTMLMediaElement.h
    M Source/WebCore/html/track/TextTrack.cpp
    M Source/WebCore/html/track/TextTrack.h
    M Source/WebCore/html/track/TextTrackCue.h
    M Source/WebCore/html/track/VTTCue.cpp
    M Source/WebCore/html/track/VTTCue.h
    M Source/WebCore/page/CaptionUserPreferences.cpp
    M Source/WebCore/platform/PlatformSpeechSynthesizer.h
    M Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm
    M Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.cpp
    M Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.h
    M Source/WebCore/testing/Internals.cpp
    M Source/WebCore/testing/Internals.h
    M Source/WebCore/testing/Internals.idl
    M Source/WebKit/UIProcess/WebPageProxy.cpp
    M Source/WebKit/UIProcess/WebPageProxy.h

  Log Message:
  -----------
  AX: Support VTT-based extended audio descriptions
https://bugs.webkit.org/show_bug.cgi?id=244931
<rdar://99697422>

Reviewed by Jer Noble.

* LayoutTests/media/track/captions-webvtt/captions-extended-descriptions.vtt: Added.
* LayoutTests/media/track/track-description-cue.html:
* LayoutTests/media/track/track-extended-descriptions-expected.txt: Added.
* LayoutTests/media/track/track-extended-descriptions.html: Added.
* LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally.
* LayoutTests/platform/ios/TestExpectations: Enable test on iOS.
* LayoutTests/platform/mac/TestExpectations: Enable test on macOS.

* Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'ExtendedAudioDescriptionsEnabled'
setting.

* Source/WebCore/Modules/speech/SpeechSynthesis.cpp:
(WebCore::SpeechSynthesis::setPlatformSynthesizer): Take a Ref<> synthesizer instead of a
unique_ptr.
(WebCore::SpeechSynthesis::ensurePlatformSpeechSynthesizer): Use PlatformSpeechSynthesizer::create.
* Source/WebCore/Modules/speech/SpeechSynthesis.h:

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::convertEnumerationToString): Added.
(WebCore::HTMLMediaElement::unregisterWithDocument): Cancel speech, clear synthesize.
(WebCore::HTMLMediaElement::updateActiveTextTrackCues): Cleanup for readability.
(WebCore::HTMLMediaElement::setSpeechSynthesisState): Update for extended descriptions.
(WebCore::HTMLMediaElement::speakCueText): Call cue.prepareToSpeak as we don't always want
a cue to begin speaking immediately.
(WebCore::HTMLMediaElement::pauseSpeakingCueText): Support completing an extended description.
(WebCore::HTMLMediaElement::resumeSpeakingCueText): Ditto.
(WebCore::HTMLMediaElement::pausePlaybackForExtendedTextDescription): New, pause playback
to allow an extended description to complete.
(WebCore::HTMLMediaElement::speechSynthesis): Have the media element own the synthesizer
since the logic for managing it is here.
(WebCore::HTMLMediaElement::executeCueEnterOrExitActionForTime): Support extended descriptions.
(WebCore::HTMLMediaElement::addTextTrack): Set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::configureTextTrackGroup): Use m_userPrefersTextDescriptions.
(WebCore::HTMLMediaElement::updatePlayState): Pause and resume speaking here, not in `playPlayer`
and `pausePlayer`.
(WebCore::HTMLMediaElement::playPlayer): Ditto.
(WebCore::HTMLMediaElement::pausePlayer): Ditto.
(WebCore::HTMLMediaElement::captionPreferencesChanged): set m_userPrefersExtendedDescriptions.
(WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
(WTF::LogArgument<WebCore::HTMLMediaElement::SpeechSynthesisState>::toString):

* Source/WebCore/html/track/TextTrack.cpp:
(WebCore::TextTrack::speechSynthesis): Deleted.
* Source/WebCore/html/track/TextTrack.h:

* Source/WebCore/html/track/TextTrackCue.h:
(WebCore::TextTrackCue::prepareToSpeak): Renamed from `speak`.
(WebCore::TextTrackCue::beginSpeaking): Added.
(WebCore::TextTrackCue::pauseSpeaking): Added.
(WebCore::TextTrackCue::cancelSpeaking): Added.
(WebCore::TextTrackCue::speak): Deleted.

* Source/WebCore/html/track/VTTCue.cpp:
(WebCore::mapVideoRateToSpeechRate): Convert playback rate to web speech rate.
(WebCore::VTTCue::prepareToSpeak): Call completion handler when there is nothing to speak
or when the track is null. Stash `speechSynthesis` for future use.
(WebCore::VTTCue::beginSpeaking): Begin or resume speaking.
(WebCore::VTTCue::pauseSpeaking):
(WebCore::VTTCue::cancelSpeaking):
(WebCore::VTTCue::speak): Deleted.
* Source/WebCore/html/track/VTTCue.h:

* Source/WebCore/page/CaptionUserPreferences.cpp:
(WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Consider setting for
extended descriptions.
* Source/WebCore/platform/PlatformSpeechSynthesizer.h: Make refcounted.

* Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm:
(-[WebSpeechSynthesisWrapper speakUtterance:]): `client()` is a ref, not a pointer.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didStartSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didFinishSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didPauseSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didContinueSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:didCancelSpeechUtterance:]): Ditto.
(-[WebSpeechSynthesisWrapper speechSynthesizer:willSpeakRangeOfSpeechString:utterance:]): Ditto.
(WebCore::PlatformSpeechSynthesizer::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizer::PlatformSpeechSynthesizer):

* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.cpp:
(WebCore::PlatformSpeechSynthesizerMock::create): Client can never be null so take ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::PlatformSpeechSynthesizerMock):
(WebCore::PlatformSpeechSynthesizerMock::speakingFinished): `client()` is a ref, not a pointer.
(WebCore::PlatformSpeechSynthesizerMock::speak): Ditto. Use configurable utterance duration.
(WebCore::PlatformSpeechSynthesizerMock::cancel): Ditto.
(WebCore::PlatformSpeechSynthesizerMock::pause): Ditto. Null-check `m_utterance`.
(WebCore::PlatformSpeechSynthesizerMock::resume): Ditto.
* Source/WebCore/platform/mock/PlatformSpeechSynthesizerMock.h:
(WebCore::PlatformSpeechSynthesizerMock::setUtteranceDuration):

* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::enableMockSpeechSynthesizer):
(WebCore::Internals::enableMockSpeechSynthesizerForMediaElement):
(WebCore::Internals::setSpeechUtteranceDuration):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::resetSpeechSynthesizer): PlatformSpeechSynthesizer is a Ref, not
a unique_ptr.
(WebKit::WebPageProxy::speechSynthesisData): Ditto.
* Source/WebKit/UIProcess/WebPageProxy.h:

Canonical link: https://commits.webkit.org/254502@main