[webkit-changes] [WebKit/WebKit] 4918b7: Implement MediaSession voiceactivity action

[youennf]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 4918b7c46048ecbb3de1f268c7f424fc7bb5dca3
      https://github.com/WebKit/WebKit/commit/4918b7c46048ecbb3de1f268c7f424fc7bb5dca3
  Author: Youenn Fablet <youenn at apple.com>
  Date:   2024-09-06 (Fri, 06 Sep 2024)

  Changed paths:
    M LayoutTests/http/wpt/mediasession/resources/toggleCapture-iframe.html
    A LayoutTests/http/wpt/mediasession/voiceActivityDetection-expected.txt
    A LayoutTests/http/wpt/mediasession/voiceActivityDetection.html
    M LayoutTests/platform/glib/TestExpectations
    M LayoutTests/platform/mac-ventura/TestExpectations
    M Source/WTF/wtf/PlatformHave.h
    M Source/WebCore/Modules/mediasession/MediaSession.cpp
    M Source/WebCore/Modules/mediasession/MediaSessionAction.h
    M Source/WebCore/Modules/mediasession/MediaSessionAction.idl
    M Source/WebCore/Modules/mediasession/MediaSessionActionDetails.h
    M Source/WebCore/Modules/mediastream/UserMediaClient.h
    M Source/WebCore/Modules/mediastream/UserMediaController.cpp
    M Source/WebCore/Modules/mediastream/UserMediaController.h
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/dom/Document.cpp
    M Source/WebCore/dom/Document.h
    M Source/WebCore/page/Page.cpp
    M Source/WebCore/page/Page.h
    M Source/WebCore/platform/mediastream/RealtimeMediaSourceFactory.h
    M Source/WebCore/platform/mediastream/mac/BaseAudioSharedUnit.cpp
    M Source/WebCore/platform/mediastream/mac/BaseAudioSharedUnit.h
    M Source/WebCore/platform/mediastream/mac/CoreAudioCaptureSource.cpp
    M Source/WebCore/platform/mediastream/mac/CoreAudioCaptureSource.h
    A Source/WebCore/platform/mediastream/mac/CoreAudioSharedInternalUnit.h
    M Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.cpp
    M Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.h
    A Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.mm
    M Source/WebCore/platform/mediastream/mac/MockAudioSharedUnit.mm
    M Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp
    M Source/WebKit/GPUProcess/GPUProcess.cpp
    M Source/WebKit/GPUProcess/GPUProcess.h
    M Source/WebKit/GPUProcess/GPUProcess.messages.in
    M Source/WebKit/UIProcess/GPU/GPUProcessProxy.cpp
    M Source/WebKit/UIProcess/GPU/GPUProcessProxy.h
    M Source/WebKit/UIProcess/GPU/GPUProcessProxy.messages.in
    M Source/WebKit/UIProcess/WebPageProxy.cpp
    M Source/WebKit/UIProcess/WebPageProxy.h
    M Source/WebKit/UIProcess/WebPageProxy.messages.in
    M Source/WebKit/WebProcess/WebCoreSupport/WebUserMediaClient.cpp
    M Source/WebKit/WebProcess/WebCoreSupport/WebUserMediaClient.h
    M Source/WebKit/WebProcess/WebPage/WebPage.cpp
    M Source/WebKit/WebProcess/WebPage/WebPage.h
    M Source/WebKit/WebProcess/WebPage/WebPage.messages.in

  Log Message:
  -----------
  Implement MediaSession voiceactivity action
rdar://133054919
https://bugs.webkit.org/show_bug.cgi?id=277545

Reviewed by Eric Carlson.

We implement voiceactivity detection on macOS and iOS based on OS APIs.
This is done in GPU process and triggers voiceactivity action for documents having muted microphone tracks.

This detection works even if microphone is not in use, as long as the audio unit is running.
When being asked to stop the audio unit, we check whether voiceactivity detection is on.
If so, we mute microphone capture and let the audio unit running.

Added a mock based test.
Manually tested.

* LayoutTests/http/wpt/mediasession/resources/toggleCapture-iframe.html:
* LayoutTests/http/wpt/mediasession/voiceActivityDetection-expected.txt: Added.
* LayoutTests/http/wpt/mediasession/voiceActivityDetection.html: Added.
* LayoutTests/platform/glib/TestExpectations:
* LayoutTests/platform/mac-ventura/TestExpectations:
* Source/WTF/wtf/PlatformHave.h:
* Source/WebCore/Modules/mediasession/MediaSession.cpp:
(WebCore::platformCommandForMediaSessionAction):
(WebCore::MediaSession::setActionHandler):
* Source/WebCore/Modules/mediasession/MediaSessionAction.h:
* Source/WebCore/Modules/mediasession/MediaSessionAction.idl:
* Source/WebCore/Modules/mediastream/UserMediaClient.h:
* Source/WebCore/Modules/mediastream/UserMediaController.cpp:
(WebCore::UserMediaController::setShouldListenToVoiceActivity):
(WebCore::UserMediaController::checkDocumentForVoiceActivity):
(WebCore::UserMediaController::voiceActivityDetected):
* Source/WebCore/Modules/mediastream/UserMediaController.h:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/dom/Document.cpp:
(WebCore::Document::updateIsPlayingMedia):
(WebCore::Document::hasMutedAudioCaptureDevice const):
(WebCore::Document::setShouldListenToVoiceActivity):
(WebCore::Document::voiceActivityDetected):
* Source/WebCore/dom/Document.h:
* Source/WebCore/page/Page.cpp:
(WebCore::Page::voiceActivityDetected):
* Source/WebCore/page/Page.h:
* Source/WebCore/platform/mediastream/RealtimeMediaSourceFactory.h:
(WebCore::AudioCaptureFactory::enableMutedSpeechActivityEventListener):
(WebCore::AudioCaptureFactory::disableMutedSpeechActivityEventListener):
* Source/WebCore/platform/mediastream/mac/BaseAudioSharedUnit.cpp:
(WebCore::BaseAudioSharedUnit::stopProducingData):
* Source/WebCore/platform/mediastream/mac/BaseAudioSharedUnit.h:
(WebCore::BaseAudioSharedUnit::captureDeviceID const):
(WebCore::BaseAudioSharedUnit::persistentID const):
(WebCore::BaseAudioSharedUnit::setVoiceActivityListenerCallback):
(WebCore::BaseAudioSharedUnit::voiceActivityDetected):
(WebCore::BaseAudioSharedUnit::isListeningToVoiceActivity const):
* Source/WebCore/platform/mediastream/mac/CoreAudioCaptureSource.cpp:
(WebCore::CoreAudioCaptureSourceFactory::enableMutedSpeechActivityEventListener):
(WebCore::CoreAudioCaptureSourceFactory::disableMutedSpeechActivityEventListener):
* Source/WebCore/platform/mediastream/mac/CoreAudioCaptureSource.h:
* Source/WebCore/platform/mediastream/mac/CoreAudioSharedInternalUnit.h: Added.
* Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.cpp:
(WebCore::CoreAudioSharedUnit::startInternal):
(WebCore::CoreAudioSharedUnit::isProducingMicrophoneSamplesChanged):
(WebCore::CoreAudioSharedUnit::prewarmAudioUnitCreation):
(): Deleted.
* Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.h:
* Source/WebCore/platform/mediastream/mac/CoreAudioSharedUnit.mm: Added.
(WebCore::speechActivityListenerCallback):
(WebCore::manageSpeechActivityListener):
(WebCore::CoreAudioSharedUnit::processVoiceActivityEvent):
(WebCore::CoreAudioSharedInternalUnit::setVoiceActivityDetection):
(WebCore::CoreAudioSharedUnit::enableMutedSpeechActivityEventListener):
(WebCore::CoreAudioSharedUnit::disableMutedSpeechActivityEventListener):
* Source/WebCore/platform/mediastream/mac/MockAudioSharedUnit.mm:
(WebCore::m_voiceDetectionTimer):
(WebCore::MockAudioSharedInternalUnit::setVoiceActivityDetection):
(WebCore::MockAudioSharedInternalUnit::voiceDetected):
(WebCore::MockAudioSharedInternalUnit::set):
* Source/WebCore/platform/mock/MockRealtimeMediaSourceCenter.cpp:
* Source/WebKit/GPUProcess/GPUProcess.cpp:
(WebKit::GPUProcess::setShouldListenToVoiceActivity):
* Source/WebKit/GPUProcess/GPUProcess.h:
* Source/WebKit/GPUProcess/GPUProcess.messages.in:
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.cpp:
(WebKit::GPUProcessProxy::setShouldListenToVoiceActivity):
(WebKit::GPUProcessProxy::voiceActivityDetected):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.h:
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.messages.in:
* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::resetState):
(WebKit::WebPageProxy::setShouldListenToVoiceActivity):
(WebKit::WebPageProxy::voiceActivityDetected):
(WebKit::WebPageProxy::gpuProcessExited):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/UIProcess/WebPageProxy.messages.in:
* Source/WebKit/WebProcess/WebCoreSupport/WebUserMediaClient.cpp:
(WebKit::WebUserMediaClient::setShouldListenToVoiceActivity):
* Source/WebKit/WebProcess/WebCoreSupport/WebUserMediaClient.h:
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::voiceActivityDetected):
* Source/WebKit/WebProcess/WebPage/WebPage.h:
* Source/WebKit/WebProcess/WebPage/WebPage.messages.in:

Canonical link: https://commits.webkit.org/283278@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications