[webkit-changes] [WebKit/WebKit] a8256b: [visionOS] Captions are missing in fullscreen mode

Jer Noble noreply at github.com
Tue May 28 14:34:55 PDT 2024 

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: a8256bf61cd70cb214e683a81a617354b57b3a42
      https://github.com/WebKit/WebKit/commit/a8256bf61cd70cb214e683a81a617354b57b3a42
  Author: Jer Noble <jer.noble at apple.com>
  Date:   2024-05-28 (Tue, 28 May 2024)

  Changed paths:
    M Source/WebCore/Modules/mediacontrols/MediaControlsHost.cpp
    M Source/WebCore/Modules/mediacontrols/MediaControlsHost.h
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/html/HTMLMediaElement.h
    M Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp
    M Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.h
    M Source/WebCore/platform/cocoa/VideoFullscreenCaptions.mm
    M Source/WebCore/platform/cocoa/VideoPresentationModel.h
    M Source/WebCore/platform/cocoa/VideoPresentationModelVideoElement.h
    M Source/WebCore/platform/cocoa/VideoPresentationModelVideoElement.mm
    M Source/WebCore/platform/graphics/MediaPlayer.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.h
    M Source/WebCore/platform/graphics/MediaPlayerPrivate.h
    M Source/WebCore/platform/graphics/TextTrackRepresentation.cpp
    M Source/WebCore/platform/graphics/TextTrackRepresentation.h
    M Source/WebCore/platform/graphics/VideoLayerManager.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm
    M Source/WebCore/platform/graphics/avfoundation/objc/VideoLayerManagerObjC.h
    M Source/WebCore/platform/graphics/avfoundation/objc/VideoLayerManagerObjC.mm
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h
    M Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm
    M Source/WebCore/platform/graphics/cocoa/TextTrackRepresentationCocoa.h
    M Source/WebCore/platform/graphics/cocoa/TextTrackRepresentationCocoa.mm
    M Source/WebCore/platform/ios/VideoPresentationInterfaceIOS.mm
    M Source/WebCore/platform/ios/WebVideoFullscreenControllerAVKit.mm
    M Source/WebCore/platform/mac/VideoPresentationInterfaceMac.mm
    M Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.h
    M Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.mm
    M Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.h
    M Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.mm
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp
    M Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h
    M Source/WebKit/WebProcess/cocoa/TextTrackRepresentationCocoa.h
    M Source/WebKit/WebProcess/cocoa/TextTrackRepresentationCocoa.mm
    M Source/WebKit/WebProcess/cocoa/VideoPresentationManager.h
    M Source/WebKit/WebProcess/cocoa/VideoPresentationManager.messages.in
    M Source/WebKit/WebProcess/cocoa/VideoPresentationManager.mm

  Log Message:
  -----------
  [visionOS] Captions are missing in fullscreen mode
https://bugs.webkit.org/show_bug.cgi?id=274592
rdar://128502343

Reviewed by Andy Estes.

When enabling the entity-backed rendering path on visionOS, we inadventantly broke
caption display in fullscreen. The reasons are multifaceted, but stem from the lack
of a WebAVPlayerLayer containing the video content in that path.

Untangling the requirement to have a fullscreen video layer requires changes at every
process level:

- Rather than have individual MediaPlayers report whether they need a TextTrackRepresentation,
  move responibility for telling HTMLMediaPlayer so up to the UI layer (and VideoPlaybackInterface).
- Notify the HTMLMediaElement of TextTrackRepresentation size changes directly, rather than
  implicitly via fullscreen video layer size changes.
- Allow the bounds of the TextTrackRepresentation to be set directly, rather than implictly
  via the representation's layer.

This removes large swaths of text track responsibilities from MediaPlayer/Private, but requires
VideoPresentationInterface subclasess to directly notify the element when they need a
representation for text tracks to be created.

* Source/WebCore/Modules/mediacontrols/MediaControlsHost.cpp:
(WebCore::MediaControlsHost::textTrackRepresentation const):
(WebCore::MediaControlsHost::requiresTextTrackRepresentationChanged):
* Source/WebCore/Modules/mediacontrols/MediaControlsHost.h:
* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::setTextTrackRepresentataionBounds):
(WebCore::HTMLMediaElement::setRequiresTextTrackRepresentation):
(WebCore::HTMLMediaElement::requiresTextTrackRepresentation const):
(WebCore::HTMLMediaElement::setTextTrackRepresentation):
* Source/WebCore/html/HTMLMediaElement.h:
* Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.cpp:
(WebCore::MediaControlTextTrackContainerElement::requiresTextTrackRepresentationChanged):
* Source/WebCore/html/shadow/MediaControlTextTrackContainerElement.h:
* Source/WebCore/platform/cocoa/VideoFullscreenCaptions.h:
* Source/WebCore/platform/cocoa/VideoFullscreenCaptions.mm:
(WebCore::VideoFullscreenCaptions::setTrackRepresentationImage):
* Source/WebCore/platform/cocoa/VideoPresentationModel.h:
(WebCore::VideoPresentationModel::setRequiresTextTrackRepresentation):
(WebCore::VideoPresentationModel::setTextTrackRepresentationBounds):
* Source/WebCore/platform/cocoa/VideoPresentationModelVideoElement.h:
* Source/WebCore/platform/cocoa/VideoPresentationModelVideoElement.mm:
(WebCore::VideoPresentationModelVideoElement::setVideoFullscreenFrame):
(WebCore::VideoPresentationModelVideoElement::setRequiresTextTrackRepresentation):
(WebCore::VideoPresentationModelVideoElement::setTextTrackRepresentationBounds):
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerPrivate.h:
(WebCore::MediaPlayerPrivateInterface::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/TextTrackRepresentation.cpp:
* Source/WebCore/platform/graphics/TextTrackRepresentation.h:
* Source/WebCore/platform/graphics/VideoLayerManager.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
(WebCore::MediaPlayerPrivateAVFoundationObjC::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/avfoundation/objc/VideoLayerManagerObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/VideoLayerManagerObjC.mm:
(WebCore::VideoLayerManagerObjC::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::requiresTextTrackRepresentation const): Deleted.
* Source/WebCore/platform/graphics/cocoa/TextTrackRepresentationCocoa.h:
* Source/WebCore/platform/graphics/cocoa/TextTrackRepresentationCocoa.mm:
(WebCore::TextTrackRepresentationCocoa::setBounds):
* Source/WebCore/platform/ios/VideoPresentationInterfaceIOS.mm:
(WebCore::VideoPresentationInterfaceIOS::willStartPictureInPicture):
(WebCore::VideoPresentationInterfaceIOS::finalizeSetup):
(WebCore::VideoPresentationInterfaceIOS::setMode):
(WebCore::VideoPresentationInterfaceIOS::clearMode):
* Source/WebCore/platform/ios/WebVideoFullscreenControllerAVKit.mm:
* Source/WebCore/platform/mac/VideoPresentationInterfaceMac.mm:
(-[WebVideoPresentationInterfaceMacObjC pipDidClose:]):
(WebCore::VideoPresentationInterfaceMac::setMode):
(WebCore::VideoPresentationInterfaceMac::clearMode):
(WebCore::VideoPresentationInterfaceMac::setupFullscreen):
* Source/WebKit/Platform/ios/PlaybackSessionInterfaceLMK.h:
* Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.h:
* Source/WebKit/Platform/ios/VideoPresentationInterfaceLMK.mm:
(-[WKLinearMediaKitCaptionsLayer initWithParent:]):
(-[WKLinearMediaKitCaptionsLayer layoutSublayers]):
(WebKit::VideoPresentationInterfaceLMK::captionsLayer):
(WebKit::VideoPresentationInterfaceLMK::captionsLayerBoundsChanged):
* Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.h:
* Source/WebKit/UIProcess/Cocoa/VideoPresentationManagerProxy.mm:
(WebKit::VideoPresentationModelContext::setVideoFullscreenFrame):
(WebKit::VideoPresentationModelContext::setRequiresTextTrackRepresentation):
(WebKit::VideoPresentationModelContext::setTextTrackRepresentationBounds):
(WebKit::VideoPresentationManagerProxy::setRequiresTextTrackRepresentation):
(WebKit::VideoPresentationManagerProxy::setTextTrackRepresentationBounds):
(WebKit::VideoPresentationManagerProxy::setVideoFullscreenFrame):
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
(WebKit::MediaPlayerPrivateRemote::requiresTextTrackRepresentation const): Deleted.
* Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
* Source/WebKit/WebProcess/cocoa/TextTrackRepresentationCocoa.h:
* Source/WebKit/WebProcess/cocoa/TextTrackRepresentationCocoa.mm:
(WebKit::WebTextTrackRepresentationCocoa::setBounds):
* Source/WebKit/WebProcess/cocoa/VideoPresentationManager.h:
* Source/WebKit/WebProcess/cocoa/VideoPresentationManager.messages.in:
* Source/WebKit/WebProcess/cocoa/VideoPresentationManager.mm:
(WebKit::VideoPresentationManager::setVideoFullscreenFrame):
(WebKit::VideoPresentationManager::setRequiresTextTrackRepresentation):
(WebKit::VideoPresentationManager::setTextTrackRepresentationBounds):

Canonical link: https://commits.webkit.org/279405@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications

More information about the webkit-changes mailing list