[webkit-dev] Position on emerging standard: WebCodecs

[youenn fablet]

> > - Providing deep access to codecs (in terms of capabilities,
> > observability of timing of operations...) requires careful thinking of
> > how much fingerprinting this ends up creating and how the processing
> > model will ensure to keep the whole API fingerprinting neutral.
>
> We have avoided providing codec enumeration API for this reason. Since
> a site can already run experiments on the <video> implementation it's
> not clear that there is substantial new surface, but it may be easier
> to implement those experiments accurately using WebCodecs.

Capabilities is indeed an issue. The more you need to understand the
codec, the wider is the API and the potential fingerprinting surface.
For instance, is the encoder supporting a realtime mode or not? Some
of it might be already retrieved from using a video element but the
potential mitigations might be easier with a video element than with a
JS API.

WebCodec might also potentially expose precise timing information in
how long it takes to decode/encode frames. Is it a way to identify
which hardware is being used underneath?

> > - A codec implementation used for RTC may differ significantly from a
> > codec implementation used for recording/MSE.
>
> Chrome's implementation of <video> does not really distinguish here.
> There are some tuning parameters (eg. latencyhint) which would be nice
> to expose to WebCodecs users.
>
> Do you foresee cases where usage hints are not sufficient?

I did not look at the usage hints but here are two examples:
1. A RTC decoder might want to do real-time error concealment / A MSE
decoder might not need to do any error concealment.
2. A MediaRecorder encoder might buffer frames to compress more
aggressively / a RTC encoder should not buffer frames.

> > - The complexity behind WebRTC, MSE or the media recorder API is not
> > to be neglected. There might be drawbacks in solving these issues at
> > the JS level instead of the browser level. I am for instance uncertain
> > that the level of complexity of a WebRTC pipeline can best be solved
> > by JS.
>
> Agreed; WebCodecs is a low-level API and is unlikely to be a better
> choice for use cases that are served by current APIs.
>
>
> - Dan