[webkit-dev] Position on emerging standard: video.requestAnimationFrame()

[Thomas Guilbert]

I am moving the execution of the video.rAF callbacks to be part of the
rendering steps. This should guarantee that "changes made inside the
callback appear at the same time as other changes in the same event loop
cycle".
I don't know if there is a place within the rendering steps where it makes
most sense for them to run. I'm planning on having them run before the
window.rAF callbacks, but I don't know after which of the other steps for
now.

That being said, if you paint a 24fps <video> into a <canvas> on every
video.rAF call, the canvas might still be 1/60th of a second behind the
video. This happens when we miss the rendering steps after jumping back on
the main thread, and have to wait for the next rendering steps (waiting
doesn't really change anything though, if the callbacks were fired earlier,
their changes would still appear later).

Simon, does having video.rAF callbacks run as part of the rendering steps
address some of your concerns?

On Mon, Feb 3, 2020 at 2:38 PM Ken Russell <kbr at google.com> wrote:

> It seems to have similar behavior to me: on a window or worker, the
> requestAnimationFrame callback needs to re-schedule itself before exiting,
> in order to be called again in the future. The proposed
> HTMLVideoElement.requestAnimationFrame works similarly. A single
> onFrameAvailable callback defined on the HTMLVideoElement would behave
> fairly differently. Domenic Denicola's feedback on using
> requestAnimationFrame vs. a promise
> <https://github.com/w3ctag/design-reviews/issues/429#issuecomment-558850179> resonates
> with me.
>
> -Ken
>
>
> On Mon, Feb 3, 2020 at 11:52 AM Simon Fraser <simon.fraser at apple.com>
> wrote:
>
>>
>>
>> On Feb 3, 2020, at 11:25 AM, Ken Russell <kbr at google.com> wrote:
>>
>> The name "requestAnimationFrame" was chosen mainly to achieve parity with
>> the AnimationFrameProvider mixin, which now provides the same animation
>> facility to the main thread and dedicated workers:
>>
>> https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animation-frames
>> https://discourse.wicg.io/t/proposal-video-requestanimationframe/3691
>>
>> It offers a nice symmetry with other JavaScript-driven animations.
>>
>>
>> But the video.requestAnimationFrame behavior seems fundamentally
>> different to window.requestAnimationFrame. It feels like it's conflating
>> two different things.
>>
>> Simon
>>
>>
>> -Ken
>>
>>
>>
>> On Mon, Feb 3, 2020 at 6:58 AM Philip Jägenstedt <foolip at chromium.org>
>> wrote:
>>
>>> Hi Simon,
>>>
>>> Naming is hard as usual and was discussed on
>>> https://github.com/w3ctag/design-reviews/issues/429, where you've
>>> already commented.
>>>
>>> Is there a pair of names that you think would work better
>>> here? onFrameAvailable() would IMHO be quite unidiomatic, the web platform
>>> doesn't have any other onFoo() methods, and what would the "cancel" variant
>>> be called?
>>>
>>> Can you file an issue in https://github.com/WICG/video-raf/issues if
>>> you see a good alternative?
>>>
>>> Also curious if +Eric Carlson <eric.carlson at apple.com> has any feedback
>>> on this?
>>>
>>> On Wed, Jan 22, 2020 at 10:49 PM Simon Fraser <simon.fraser at apple.com>
>>> wrote:
>>>
>>>>
>>>> On Jan 21, 2020, at 5:27 PM, Thomas Guilbert <tguilbert at google.com>
>>>> wrote:
>>>>
>>>> The idea was to reuse an API name that developers are already
>>>> familiar with, in a similar context. The name is also being used in
>>>> XRSession (
>>>> https://developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame),
>>>> and in OffscreenCanvas (or technically DedicatedWorkerGlobalScope). The AnimationFrameProvider
>>>> mixin
>>>> <https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider> could
>>>> also be updated so HTMLVideoElement can extend it.
>>>>
>>>> Yes, this isn't formally spec'ed out, but it will be. For now, they are
>>>> added to the task queue and run like any other task. So, going off the spec
>>>> you linked, I think this would be "5) Perform oldestTask's step"  and not
>>>> "10) Rendering: [...] 11. Foreach document run animation frame callbacks
>>>> for that Document".
>>>>
>>>>
>>>> I would expect something that's called "requestAnimationFrame" to only
>>>> fire in the "update the rendering" steps; requestAnimationFrame is a
>>>> "before rendering" callback. So firing a callback with the same name at
>>>> other times seems like it will lead to author confusion.
>>>>
>>>> The author's expectation should be that any content/style changes they
>>>> make inside a requestAnimationFrame callback will appear on-screen in the
>>>> same frame as other changes in the same event loop cycle, and that
>>>> requestAnimationFrame won't be called more often than is necessary to
>>>> update the screen at the appropriate frame rate.
>>>>
>>>> Simon
>>>>
>>>>
>>>> On Tue, Jan 21, 2020 at 1:01 PM Simon Fraser <simon.fraser at apple.com>
>>>> wrote:
>>>>
>>>>> On Jan 21, 2020, at 12:37 PM, Thomas Guilbert <tguilbert at google.com>
>>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm reaching out to see if webkit would like to weigh in on the following proposal:https://discourse.wicg.io/t/proposal-video-requestanimationframe/3691
>>>>>
>>>>> The HTMLVideoElement.requestAnimationFrame() API allows web developers to be notified when a video frame has been presented for composition, and provides metadata for that frame.
>>>>>
>>>>> If you want to try it out, a prototype is available in Chromium Dev,
>>>>> behind the enable-experimental-web-platform-features flag.
>>>>>
>>>>>
>>>>> This is not official feedback, but I have some issues with the
>>>>> proposal.
>>>>>
>>>>> First, the name is confusing. It sounds like you're requesting a frame
>>>>> from the video, but it's really a "frame available" callback. Why not call
>>>>> it onFrameAvailable()?
>>>>>
>>>>> Second, its interaction with normal requestAnimationFrame() and the
>>>>> HTML event loop needs to be better defined. Where in in the
>>>>> https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model do
>>>>> these callbacks fire?
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org
>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20200205/57e23e18/attachment.htm>