[webkit-changes] [WebKit/WebKit] 53a806: [WebCodecs] Limit the number of codec operations w...

[Jean-Yves Avenard]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 53a8065088fbe559932c62608a687cf6000cb798
      https://github.com/WebKit/WebKit/commit/53a8065088fbe559932c62608a687cf6000cb798
  Author: Jean-Yves Avenard <jya at apple.com>
  Date:   2024-12-13 (Fri, 13 Dec 2024)

  Changed paths:
    M Source/WebCore/Headers.cmake
    M Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp
    M Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h
    M Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp
    M Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h
    A Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp
    A Source/WebCore/Modules/webcodecs/WebCodecsBase.h
    M Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h
    M Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp
    M Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h
    M Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp
    M Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h
    M Source/WebCore/Sources.txt
    M Source/WebCore/WebCore.xcodeproj/project.pbxproj

  Log Message:
  -----------
  [WebCodecs] Limit the number of codec operations we can enqueue
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by Youenn Fablet.

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (https://github.com/w3c/webcodecs/issues/864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueCodecControlMessageAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementCodecQueueSize):
(WebCore::WebCodecsBase::decrementCodecQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decrementCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::codecQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::incrementCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:

Canonical link: https://commits.webkit.org/287798@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications