[webkit-dev] Position on emerging standard: WebCodecs

[Dan Sanders]

Thanks for the detailed feedback! Responses to some of the items are below.

There is a bi-weekly meeting to discuss WebCodecs development, if you
(or others working on WebKit) are interested in participating please
let me know and I can invite you.

> (let me know if there is a more detailed proposal).

The explainer is the primary source at this time. There is IDL in
Chromium that has yet to be proposed:
https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/modules/webcodecs/?q=webcodecs&ss=chromium

> - Any media pipeline should be off the JS main thread, by default.
> This does not seem guaranteed by the proposal.

I filed https://github.com/WICG/web-codecs/issues/51

> - Providing deep access to codecs (in terms of capabilities,
> observability of timing of operations...) requires careful thinking of
> how much fingerprinting this ends up creating and how the processing
> model will ensure to keep the whole API fingerprinting neutral.

We have avoided providing codec enumeration API for this reason. Since
a site can already run experiments on the <video> implementation it's
not clear that there is substantial new surface, but it may be easier
to implement those experiments accurately using WebCodecs.

> - A codec implementation used for RTC may differ significantly from a
> codec implementation used for recording/MSE.

Chrome's implementation of <video> does not really distinguish here.
There are some tuning parameters (eg. latencyhint) which would be nice
to expose to WebCodecs users.

Do you foresee cases where usage hints are not sufficient?

> - The complexity behind WebRTC, MSE or the media recorder API is not
> to be neglected. There might be drawbacks in solving these issues at
> the JS level instead of the browser level. I am for instance uncertain
> that the level of complexity of a WebRTC pipeline can best be solved
> by JS.

Agreed; WebCodecs is a low-level API and is unlikely to be a better
choice for use cases that are served by current APIs.


- Dan