[webkit-changes] [WebKit/WebKit] 89ffa3: [visionOS] Audio coming from wrong location in ful...

Thu May 23 10:21:21 PDT 2024

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 89ffa3deb5e76c98593f533a16ea7c5fa211597f
      https://github.com/WebKit/WebKit/commit/89ffa3deb5e76c98593f533a16ea7c5fa211597f
  Author: Jer Noble <jer.noble at apple.com>
  Date:   2024-05-23 (Thu, 23 May 2024)

  Changed paths:
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/platform/audio/AudioSession.cpp
    M Source/WebCore/platform/audio/AudioSession.h
    M Source/WebCore/platform/audio/ios/AudioSessionIOS.h
    M Source/WebCore/platform/audio/ios/AudioSessionIOS.mm
    M Source/WebCore/platform/cocoa/PlaybackSessionModel.h
    M Source/WebCore/platform/cocoa/PlaybackSessionModelMediaElement.h
    M Source/WebCore/platform/cocoa/VideoFullscreenCaptions.h
    M Source/WebCore/platform/graphics/MediaPlayer.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.h
    M Source/WebCore/platform/graphics/MediaPlayerPrivate.h
    M Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp
    M Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm
    M Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h
    M Source/WebCore/platform/graphics/holepunch/MediaPlayerPrivateHolePunch.h
    M Source/WebCore/platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp
    M Source/WebCore/platform/graphics/win/MediaPlayerPrivateMediaFoundation.h
    M Source/WebCore/platform/ios/WebVideoFullscreenControllerAVKit.mm
    M Source/WebCore/platform/mock/mediasource/MockMediaPlayerMediaSource.cpp
    M Source/WebCore/platform/mock/mediasource/MockMediaPlayerMediaSource.h
    M Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.cpp
    M Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.h
    M Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.messages.in
    M Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.cpp
    M Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.h
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.messages.in
    M Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.h
    M Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.mm
    M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
    M Source/WebKit/UIProcess/Cocoa/PlaybackSessionManagerProxy.h
    M Source/WebKit/UIProcess/Cocoa/PlaybackSessionManagerProxy.mm
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h
    M Source/WebKit/WebProcess/GPU/media/RemoteAudioSession.cpp
    M Source/WebKit/WebProcess/GPU/media/RemoteAudioSession.h
    M Source/WebKit/WebProcess/GPU/media/RemoteAudioSessionConfiguration.h
    M Source/WebKit/WebProcess/GPU/media/RemoteAudioSessionConfiguration.serialization.in
    M Source/WebKit/WebProcess/WebPage/WebPage.cpp
    M Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.h
    M Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.messages.in
    M Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.mm

  Log Message:
  -----------
  [visionOS] Audio coming from wrong location in fullscreen mode
https://bugs.webkit.org/show_bug.cgi?id=274529
rdar://127009932

Reviewed by Andy Estes.

When video playback goes into native fullscreen, the rendering path switches from being CALayer-backed
to being Entity/VideoTarget-backed. This switch means the system loses the connection between the layer's
location in space, and the audio associated with that layer. In order to restore that connection, add
a separated layer containing a spatial tracking label as a sublayer of the captions layer we provide to
the native fullscreen presentation, and that tracking label will be passed all the down to the GPU process.

Now that video playback is correctly spatialized in fullscreen, we also need to increase the soundstage
size to "Large" when in fullscreen as well. Previously this was only managed by individual players, and
only MSE-backed players. Instead, pull responsibility for setting the soundstage size up into the UI
process, and propogate that setting down to the GPU process by calculating a "maximum" soundstage size
at both the WebContent and GPU process level.

Since setting the soundstage size also requires a UIScene identifier, that will be pulled out from
MediaPlayer and passed alongside the soundstage size.

* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::visibilityStateChanged):
* Source/WebCore/platform/audio/AudioSession.cpp:
(WebCore::convertEnumerationToString):
* Source/WebCore/platform/audio/AudioSession.h:
(WTF::LogArgument<WebCore::AudioSession::SoundStageSize>::toString):
* Source/WebCore/platform/audio/ios/AudioSessionIOS.h:
* Source/WebCore/platform/audio/ios/AudioSessionIOS.mm:
(WebCore::AudioSessionIOS::updateSpatialExperience):
(WebCore::AudioSessionIOS::setSceneIdentifier):
(WebCore::AudioSessionIOS::setSoundStageSize):
* Source/WebCore/platform/cocoa/PlaybackSessionModel.h:
* Source/WebCore/platform/cocoa/PlaybackSessionModelMediaElement.h:
* Source/WebCore/platform/cocoa/VideoFullscreenCaptions.h:
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::setPageIsVisible):
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerPrivate.h:
* Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp:
(WebCore::MediaPlayerPrivateAVFoundation::setPageIsVisible):
* Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::setPageIsVisible):
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::setPageIsVisible):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::setPageIsVisible):
* Source/WebCore/platform/ios/WebVideoFullscreenControllerAVKit.mm:
* Source/WebCore/platform/mock/mediasource/MockMediaPlayerMediaSource.cpp:
(WebCore::MockMediaPlayerMediaSource::setPageIsVisible):
* Source/WebCore/platform/mock/mediasource/MockMediaPlayerMediaSource.h:
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.cpp:
(WebKit::RemoteAudioSessionProxy::configuration):
(WebKit::RemoteAudioSessionProxy::tryToSetActive):
(WebKit::RemoteAudioSessionProxy::setSceneIdentifier):
(WebKit::RemoteAudioSessionProxy::setSoundStageSize):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.h:
(WebKit::RemoteAudioSessionProxy::sceneIdentifier const):
(WebKit::RemoteAudioSessionProxy::soundStageSize const):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxy.messages.in:
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.cpp:
(WebKit::RemoteAudioSessionProxyManager::updateSpatialExperience):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.h:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::setPageIsVisible):
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h:
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
* Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.h:
* Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.mm:
(-[WKLinearMediaKitCaptionsLayer layoutSublayers]):
(WebKit::VideoPresentationInterfaceLMK::presentFullscreen):
(WebKit::VideoPresentationInterfaceLMK::dismissFullscreen):
(WebKit::VideoPresentationInterfaceLMK::captionsLayer):
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Source/WebKit/UIProcess/Cocoa/PlaybackSessionManagerProxy.h:
* Source/WebKit/UIProcess/Cocoa/PlaybackSessionManagerProxy.mm:
(WebKit::PlaybackSessionModelContext::setSoundStageSize):
(WebKit::PlaybackSessionManagerProxy::setSoundStageSize):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
(WebKit::MediaPlayerPrivateRemote::setPageIsVisible):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
* Source/WebKit/WebProcess/GPU/media/RemoteAudioSession.cpp:
(WebKit::RemoteAudioSession::setSceneIdentifier):
(WebKit::RemoteAudioSession::setSoundStageSize):
* Source/WebKit/WebProcess/GPU/media/RemoteAudioSession.h:
* Source/WebKit/WebProcess/GPU/media/RemoteAudioSessionConfiguration.h:
* Source/WebKit/WebProcess/GPU/media/RemoteAudioSessionConfiguration.serialization.in:
* Source/WebKit/WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::setSceneIdentifier):
* Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.h:
* Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.messages.in:
* Source/WebKit/WebProcess/cocoa/PlaybackSessionManager.mm:
(WebKit::PlaybackSessionManager::setSoundStageSize):
(WebKit::PlaybackSessionManager::forEachModel):

Canonical link: https://commits.webkit.org/279214@main

To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications