[webkit-dev] Position on emerging standard: video.requestAnimationFrame()

[Thomas Guilbert]

Yes, I agree with your comments, and I have some clarifications (but I know
there are still some unanswered questions). Some of this might be
implementation details of the prototype rather than abstract spec...
I made a mistake, the callbacks are not actually queued in the event loop
task queue. They are fired on the main thread ASAP after frame+info is sent
from the compositor thread. The compositor thread submits that info only as
part of the rendering phase, so the callbacks are actually tied to the
"update the rendering" phase (but the precise timing is undefined at the
moment). This means that the callbacks should be fired no more than is
necessary (e.g. a 60fps video in a 30fps will receive video-rAF callbacks
at 30fps).
Due to the thread jump, guaranteeing that content/style changes in the
video-raf callback will appear in the same event loop-cycle is tricky (and
definitively not guaranteed in the current state of the Chromium
prototype). However, the video-raf callback guarantees that there is enough
time before the next window.rAF to do something useful with the video
frame. The following is currently always true:

video.requestAnimationFrame((x, metadata) => {
  window.requestAnimationFrame((now) => {
    console.log("Useful?: " + (now <= metadata.expectedPresentationTime));
  });
});

This means that a video.rAF callback can always be used to update a webgl
texture, and paint it during the window.rAF, in time for its expected frame.

On Wed, Jan 22, 2020 at 1:49 PM Simon Fraser <simon.fraser at apple.com> wrote:

>
> On Jan 21, 2020, at 5:27 PM, Thomas Guilbert <tguilbert at google.com> wrote:
>
> The idea was to reuse an API name that developers are already
> familiar with, in a similar context. The name is also being used in
> XRSession (
> https://developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame),
> and in OffscreenCanvas (or technically DedicatedWorkerGlobalScope). The AnimationFrameProvider
> mixin
> <https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider> could
> also be updated so HTMLVideoElement can extend it.
>
> Yes, this isn't formally spec'ed out, but it will be. For now, they are
> added to the task queue and run like any other task. So, going off the spec
> you linked, I think this would be "5) Perform oldestTask's step"  and not
> "10) Rendering: [...] 11. Foreach document run animation frame callbacks
> for that Document".
>
>
> I would expect something that's called "requestAnimationFrame" to only
> fire in the "update the rendering" steps; requestAnimationFrame is a
> "before rendering" callback. So firing a callback with the same name at
> other times seems like it will lead to author confusion.
>
> The author's expectation should be that any content/style changes they
> make inside a requestAnimationFrame callback will appear on-screen in the
> same frame as other changes in the same event loop cycle, and that
> requestAnimationFrame won't be called more often than is necessary to
> update the screen at the appropriate frame rate.
>
> Simon
>
>
> On Tue, Jan 21, 2020 at 1:01 PM Simon Fraser <simon.fraser at apple.com>
> wrote:
>
>> On Jan 21, 2020, at 12:37 PM, Thomas Guilbert <tguilbert at google.com>
>> wrote:
>>
>> Hello,
>>
>> I'm reaching out to see if webkit would like to weigh in on the following proposal:https://discourse.wicg.io/t/proposal-video-requestanimationframe/3691
>>
>> The HTMLVideoElement.requestAnimationFrame() API allows web developers to be notified when a video frame has been presented for composition, and provides metadata for that frame.
>>
>> If you want to try it out, a prototype is available in Chromium Dev,
>> behind the enable-experimental-web-platform-features flag.
>>
>>
>> This is not official feedback, but I have some issues with the proposal.
>>
>> First, the name is confusing. It sounds like you're requesting a frame
>> from the video, but it's really a "frame available" callback. Why not call
>> it onFrameAvailable()?
>>
>> Second, its interaction with normal requestAnimationFrame() and the HTML
>> event loop needs to be better defined. Where in in the
>> https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model do
>> these callbacks fire?
>>
>> Simon
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20200123/4b5dbaff/attachment.htm>