[webkit-changes] [WebKit/WebKit] 93f630: Add ability to sniff media content should we faile...

Jean-Yves Avenard noreply at github.com
Mon Mar 18 02:21:41 PDT 2024 

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 93f6303ff83e5c20dbd79f5e6ae7c8a801ee519d
      https://github.com/WebKit/WebKit/commit/93f6303ff83e5c20dbd79f5e6ae7c8a801ee519d
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-03-18 (Mon, 18 Mar 2024)

  Changed paths:
    M LayoutTests/http/tests/media/video-throttled-load-metadata-expected.txt
    M LayoutTests/http/tests/media/video-throttled-load-metadata.html
    A LayoutTests/media/video-src-mp4-blob-expected.txt
    A LayoutTests/media/video-src-mp4-blob.html
    A LayoutTests/media/video-src-webm-blob-expected.txt
    A LayoutTests/media/video-src-webm-blob.html
    A LayoutTests/media/video-srcobject-mp4-blob-expected.txt
    A LayoutTests/media/video-srcobject-mp4-blob.html
    M LayoutTests/platform/mac-wk1/TestExpectations
    M LayoutTests/platform/mac/TestExpectations
    M Source/WTF/wtf/NativePromise.h
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Sources.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj
    M Source/WebCore/html/HTMLMediaElement.cpp
    M Source/WebCore/html/HTMLMediaElement.h
    M Source/WebCore/platform/CommonAtomStrings.h
    M Source/WebCore/platform/ContentType.cpp
    M Source/WebCore/platform/ContentType.h
    M Source/WebCore/platform/PlatformMediaError.cpp
    M Source/WebCore/platform/PlatformMediaError.h
    A Source/WebCore/platform/graphics/MIMESniffer.cpp
    A Source/WebCore/platform/graphics/MIMESniffer.h
    M Source/WebCore/platform/graphics/MediaPlayer.cpp
    M Source/WebCore/platform/graphics/MediaPlayer.h
    A Source/WebCore/platform/graphics/MediaResourceSniffer.cpp
    A Source/WebCore/platform/graphics/MediaResourceSniffer.h
    M Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp
    M Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h
    M Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp
    M Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h
    M Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in
    M Tools/TestWebKitAPI/CMakeLists.txt
    M Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj
    A Tools/TestWebKitAPI/Tests/WebCore/MIMESniffer.cpp

  Log Message:
  -----------
  Add ability to sniff media content should we failed to play a file or if the provided content type was invalid.
https://bugs.webkit.org/show_bug.cgi?id=270975
rdar://problem/124614908

Reviewed by Jer Noble.

We have always relied on the server to provide a valid content-type and if that failed
we always used the URL's file name extension instead.
While the HTML5 specs clearly states that determining the type of a resource
should be done through sniffing, this is however a too significant change
to enable right away. There are also performance advantages in using the provided
content-type: it's typically immediately available.
Should the provided content-type be invalid or non-existent (such as with some blobs)
playback would have failed.

We add a MIMEtype sniffer for media content as per HTML5 specs.
Should we fail to play a media using the older content type detection rather
than immediately fail, we will also sniff the type as a last attempt before
retrying.

In order to not unnecessarily retry following a sniffing, we needed to
distinguish a format error from a network error which the MediaPlayerPrivateAVFoundationObjC
didn't do. As a fly-by fix, we add a way to distinguish the two errors by
recording in the WebCoreNSURLSession any network failures.

For now will limit sniffing to media element where the src attribute is set
(and so exclude element where alternative sources are defined)

Added API tests for the mimetype sniffer and tests for webm and mp4 in a blob.

* LayoutTests/media/video-src-mp4-blob-expected.txt: Added.
* LayoutTests/media/video-src-mp4-blob.html: Added.
* LayoutTests/media/video-src-webm-blob-expected.txt: Added.
* LayoutTests/media/video-src-webm-blob.html: Added.
* LayoutTests/media/video-srcobject-mp4-blob-expected.txt: Added.
* LayoutTests/media/video-srcobject-mp4-blob.html: Added.
* LayoutTests/http/tests/media/video-throttled-load-metadata-expected.txt:
* LayoutTests/http/tests/media/video-throttled-load-metadata.html: The test was racy and could
caused the loadedmetadata message to have been received before the worker's message.
* LayoutTests/platform/mac-wk1/TestExpectations:
* LayoutTests/platform/mac/TestExpectations: Remove failure expectations. Tests were failing as an incorrect
content-type was provided. Now that we sniff if the provided content-type was incorrect
files can properly play again.
* Source/WTF/wtf/NativePromise.h:
* Source/WebCore/Headers.cmake:
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/html/HTMLMediaElement.cpp: Move all the logic from MediaPlayer related
to guessing the ContentType if missing into the HTMLMediaElement.
If all have failed, we will sniff the content.
(WebCore::HTMLMediaElement::~HTMLMediaElement):
(WebCore::HTMLMediaElement::loadResource): If the content is blocked, signal it as a network error rather than content error.
Otherwise we will attempt to load the content twice (once for sniffing and once for playing) causing test failures.
(WebCore::HTMLMediaElement::needsContentTypeToPlay const): Add methods to determine if sniffing is ultimately required.
(WebCore::HTMLMediaElement::sniffForContentType):
(WebCore::HTMLMediaElement::mediaLoadingFailed): Add one ultimate attempt if we failed to decode and
didn't encounter a network error and didn't attempt to previously sniff content.
(WebCore::HTMLMediaElement::cancelPendingTasks):
(WebCore::HTMLMediaElement::cancelSniffer):
(WebCore::HTMLMediaElement::mediaPlayerEngineFailedToLoad): Save if a network error was encountered.
If so, we won't attempt to sniff later on. This is required as multiple tests are
relying on the network being accessed only once, doing one extra access for sniffing
made the tests fail.
(WebCore::HTMLMediaElement::mediaPlayerEngineFailedToLoad const): Deleted.
* Source/WebCore/html/HTMLMediaElement.h:
* Source/WebCore/platform/CommonAtomStrings.h:
* Source/WebCore/platform/ContentType.cpp:
(WebCore::ContentType::ContentType):
(WebCore::ContentType::fromURL):
* Source/WebCore/platform/ContentType.h:
(WebCore::ContentType::typeWasInferredFromExtension const):
* Source/WebCore/platform/PlatformMediaError.cpp:
(WebCore::convertEnumerationToString):
* Source/WebCore/platform/PlatformMediaError.h:
* Source/WebCore/platform/graphics/MIMESniffer.cpp: Added.
(WebCore::MIMESniffer::span8):
(WebCore::MIMESniffer::hasSignatureForMP4):
(WebCore::MIMESniffer::parseWebMVint):
(WebCore::MIMESniffer::hasSignatureForWebM):
(WebCore::MIMESniffer::matchMP3Header):
(WebCore::MIMESniffer::mp3FrameSize):
(WebCore::MIMESniffer::parseMP3Frame):
(WebCore::MIMESniffer::hasSignatureForMP3WithoutID3):
(WebCore::MIMESniffer::mimeTypeFromSnifferEntries):
(WebCore::MIMESniffer::getMIMETypeFromContent):
* Source/WebCore/platform/graphics/MIMESniffer.h: Added.
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::applicationOctetStream):
(WebCore::MediaPlayer::load):
(WebCore::MediaPlayer::loadWithNextMediaEngine):
* Source/WebCore/platform/graphics/MediaPlayer.h:
(WebCore::MediaPlayerClient::mediaPlayerEngineFailedToLoad):
(WebCore::MediaPlayerClient::mediaPlayerEngineFailedToLoad const): Deleted.
* Source/WebCore/platform/graphics/MediaResourceSniffer.cpp: Added.
(WebCore::MediaResourceSniffer::create):
(WebCore::MediaResourceSniffer::MediaResourceSniffer):
(WebCore::MediaResourceSniffer::~MediaResourceSniffer):
(WebCore::MediaResourceSniffer::cancel):
(WebCore::MediaResourceSniffer::promise const):
(WebCore::MediaResourceSniffer::dataReceived):
(WebCore::MediaResourceSniffer::loadFailed):
(WebCore::MediaResourceSniffer::loadFinished):
* Source/WebCore/platform/graphics/MediaResourceSniffer.h: Added.
* Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp:
(WebCore::MediaPlayerPrivateAVFoundation::updateStates): We needed to be able
to distinguish FormatError vs NetworkError so that we don't unnecessarily retry
playback following sniffing.
* Source/WebCore/platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
(WebCore::MediaPlayerPrivateAVFoundationObjC::assetStatus const): Distinguish
load failure due to decoding error vs networking error.
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.cpp:
(WebKit::RemoteMediaPlayerProxy::mediaPlayerEngineFailedToLoad):
(WebKit::RemoteMediaPlayerProxy::mediaPlayerEngineFailedToLoad const): Deleted.
* Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h:
* Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in:
* Tools/TestWebKitAPI/CMakeLists.txt:
* Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* Tools/TestWebKitAPI/Tests/WebCore/MIMESniffer.cpp: Added.
(TestWebKitAPI::TEST):

Canonical link: https://commits.webkit.org/276258@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications

More information about the webkit-changes mailing list