[webkit-changes] [WebKit/WebKit] 332cd8: AX: Consider VTT-based audio descriptions with tex...

[Eric Carlson]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 332cd88dafc8f0e5dbd5547cc6d6b91cb992f0c8
      https://github.com/WebKit/WebKit/commit/332cd88dafc8f0e5dbd5547cc6d6b91cb992f0c8
  Author: Eric Carlson <eric.carlson at apple.com>
  Date:   2022-09-08 (Thu, 08 Sep 2022)

  Changed paths:
    M LayoutTests/TestExpectations
    A LayoutTests/media/track/captions-webvtt/captions-descriptions.vtt
    A LayoutTests/media/track/track-description-cue-expected.txt
    A LayoutTests/media/track/track-description-cue.html
    M LayoutTests/platform/ios/TestExpectations
    M LayoutTests/platform/mac/TestExpectations
    M Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml
    M Source/WebCore/Modules/speech/SpeechSynthesis.cpp
    M Source/WebCore/Modules/speech/SpeechSynthesis.h
    M Source/WebCore/Modules/speech/SpeechSynthesisUtterance.cpp
    M Source/WebCore/Modules/speech/SpeechSynthesisUtterance.h
    M Source/WebCore/Modules/speech/SpeechSynthesisUtterance.idl
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/html/HTMLMediaElement.h
    M Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp
    M Source/WebCore/html/track/InbandGenericTextTrack.cpp
    M Source/WebCore/html/track/TextTrack.cpp
    M Source/WebCore/html/track/TextTrack.h
    M Source/WebCore/html/track/TextTrackCue.cpp
    M Source/WebCore/html/track/TextTrackCue.h
    M Source/WebCore/html/track/VTTCue.cpp
    M Source/WebCore/html/track/VTTCue.h
    M Source/WebCore/html/track/VTTCue.idl
    M Source/WebCore/page/CaptionUserPreferences.cpp
    M Source/WebCore/page/CaptionUserPreferencesMediaAF.cpp
    M Source/WebCore/page/CaptionUserPreferencesMediaAF.h
    M Source/WebCore/platform/cf/MediaAccessibilitySoftLink.cpp
    M Source/WebCore/platform/cf/MediaAccessibilitySoftLink.h
    M Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm
    M Source/WebCore/platform/graphics/InbandGenericCue.cpp
    M Source/WebCore/platform/graphics/iso/ISOVTTCue.cpp
    M Source/WebCore/testing/Internals.cpp
    M Source/WebCore/testing/Internals.h
    M Source/WebCore/testing/Internals.idl
    M Source/WebKit/UIProcess/WebPageProxy.cpp

  Log Message:
  -----------
  AX: Consider VTT-based audio descriptions with text-to-speech.
https://bugs.webkit.org/show_bug.cgi?id=243600
rdar://98206665

Reviewed by Jer Noble.

* LayoutTests/media/track/captions-webvtt/captions-descriptions.vtt: Added.
* LayoutTests/media/track/track-description-cue-expected.txt: Added.
* LayoutTests/media/track/track-description-cue.html: Added.
* LayoutTests/TestExpectations: Feature is Cocoa-specific so far, skip test globally.
* LayoutTests/platform/ios/TestExpectations: Enable test on iOS.
* LayoutTests/platform/mac/TestExpectations: Enable test on macOS.

* Source/WTF/Scripts/Preferences/WebPreferencesExperimental.yaml: Add 'AudioDescriptionsEnabled'
setting.

* Source/WebCore/Modules/speech/SpeechSynthesis.cpp:
(WebCore::SpeechSynthesis::cancel): Don't assert if `m_currentSpeechUtterance` isn't
null immediately after calling cancel() because the the callback from AVSpeechSynthesis
is asynchronous. Apparently it was synchronous when the code was originally written, but
it has changed and there is no way to modify it.
(WebCore::SpeechSynthesis::handleSpeakingCompleted): Call utterance to fire event.
(WebCore::SpeechSynthesis::boundaryEventOccurred): Ditto.
(WebCore::SpeechSynthesis::didStartSpeaking): Ditto.
(WebCore::SpeechSynthesis::didPauseSpeaking): Ditto.
(WebCore::SpeechSynthesis::didResumeSpeaking): Ditto.
(WebCore::SpeechSynthesis::fireEvent const): Deleted.
(WebCore::SpeechSynthesis::fireErrorEvent const): Deleted.
* Source/WebCore/Modules/speech/SpeechSynthesis.h:
(WebCore::SpeechSynthesis::userGestureRequiredForSpeechStart const):
(WebCore::SpeechSynthesis::removeBehaviorRestriction): Make public.

* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.cpp:
(WebCore::SpeechSynthesisUtterance::create): Add version that takes a completion handler.
(WebCore::SpeechSynthesisUtterance::SpeechSynthesisUtterance): Add completion handler
parameter.
(WebCore::SpeechSynthesisUtterance::eventOccurred): New. Call completion handler or
dispatch event.
(WebCore::SpeechSynthesisUtterance::errorEventOccurred): Ditto.
* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.h:
* Source/WebCore/Modules/speech/SpeechSynthesisUtterance.idl: JSGenerateToJSObject.

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::updateActiveTextTrackCues): Return early if a seek
is pending. Call new executeCueEnterOrLeaveAction method instead of dispatching
events directly.
(WebCore::HTMLMediaElement::setSpeechSynthesisState): Maintain synthesis state.
(WebCore::HTMLMediaElement::speakCueText): Speak a cue.
(WebCore::HTMLMediaElement::pauseSpeakingCueText):
(WebCore::HTMLMediaElement::resumeSpeakingCueText):
(WebCore::HTMLMediaElement::cancelSpeakingCueText):
(WebCore::HTMLMediaElement::shouldSpeakCueTextForTime):
(WebCore::HTMLMediaElement::executeCueEnterOrLeaveAction): Trigger cue speech if
the track contains descriptions and we are entering a cue range, schedule an enter
or exit event.
(WebCore::HTMLMediaElement::seekWithTolerance): INFO_LOG -> ALWAYS_LOG
(WebCore::HTMLMediaElement::seekTask): Cancel speaking if necessary.
(WebCore::HTMLMediaElement::finishSeek): Update logging. If there isn't a pending
seek, queue a task to update text track cues.
(WebCore::HTMLMediaElement::configureTextTrackGroup): When processing descriptions
and the user wants text descriptions, set `fallbackTrack` to the first track seen
in case none of the tracks matches the audio track language.
(WebCore::HTMLMediaElement::playPlayer): Call resumeSpeakingCueText.
(WebCore::HTMLMediaElement::pausePlayer): Call pauseSpeakingCueText.
(WebCore::HTMLMediaElement::effectiveVolume const): Use the speech volume multiplier
when calculating the effective volume.
(WebCore::m_categoryAtMostRecentPlayback): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
(WebCore::HTMLMediaElement::cueBeingSpoken const):

* Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp:
(WebCore::MediaControlTextTrackContainerElement::updateDisplay): Skip spoken tracks.

* Source/WebCore/html/track/InbandGenericTextTrack.cpp: Remove unneeded include.

* Source/WebCore/html/track/TextTrack.cpp:
(WebCore::TextTrack::trackIndex): Use textTrackList() instead of m_textTrackList.
(WebCore::TextTrack::isRendered): Consider descriptions.
(WebCore::TextTrack::isSpoken):
(WebCore::TextTrack::trackIndexRelativeToRenderedTracks):  Use textTrackList()
instead of m_textTrackList.
(WebCore::TextTrack::speechSynthesis):
* Source/WebCore/html/track/TextTrack.h:

* Source/WebCore/html/track/TextTrackCue.cpp:
(WebCore::operator<<): All cues have a `text()` method, just use it.
* Source/WebCore/html/track/TextTrackCue.h:
(WebCore::TextTrackCue::text const):
(WebCore::TextTrackCue::speak):

* Source/WebCore/html/track/VTTCue.cpp:
(WebCore::VTTCue::updateDisplayTree): `track()` can return null, check it.
(WebCore::VTTCue::getDisplayTree): Ditto.
(WebCore::VTTCue::toJSON const): Drive-by: address a Darin FIXME.
(WebCore::VTTCue::speak):
* Source/WebCore/html/track/VTTCue.h:
(WebCore::VTTCue::speechUtterance const):
(WebCore::VTTCue::text const): Deleted.
* Source/WebCore/html/track/VTTCue.idl:

* Source/WebCore/page/CaptionUserPreferences.cpp:
(WebCore::CaptionUserPreferences::userPrefersTextDescriptions const): Check audioDescriptionsEnabled
setting.
(WebCore::CaptionUserPreferences::textTrackSelectionScore const): Consider description
tracks if the user preference is set. Clean up logic.

* Source/WebCore/page/CaptionUserPreferencesMediaAF.cpp:
(WebCore::CaptionUserPreferencesMediaAF::userPrefersCaptions const):
(WebCore::CaptionUserPreferencesMediaAF::userPrefersTextDescriptions const): check
MediaAccessibility framework preference.
* Source/WebCore/page/CaptionUserPreferencesMediaAF.h:
* Source/WebCore/platform/cf/MediaAccessibilitySoftLink.cpp:
* Source/WebCore/platform/cf/MediaAccessibilitySoftLink.h:

* Source/WebCore/platform/cocoa/PlatformSpeechSynthesizerCocoa.mm:
(-[WebSpeechSynthesisWrapper speakUtterance:]): If `utteranceVoice` is non-NULL
and the URI is empty, it is invalid so check the language.

* Source/WebCore/platform/graphics/InbandGenericCue.cpp:
(WebCore::InbandGenericCue::toJSONString const): Log cue text like VTTCue now does.

* Source/WebCore/platform/graphics/iso/ISOVTTCue.cpp:
(WebCore::ISOWebVTTCue::toJSONString const): Ditto.

* Source/WebCore/testing/Internals.cpp:
(WebCore::Internals::speechSynthesisUtteranceForCue):
(WebCore::Internals::mediaElementCurrentlySpokenCue):
* Source/WebCore/testing/Internals.h:
* Source/WebCore/testing/Internals.idl:

* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::speechSynthesisSpeak): Remove the `startTime` parameter name,
it isn't used.

Canonical link: https://commits.webkit.org/254266@main