[Webkit-unassigned] [Bug 264893] New: VideoEncoder outputs mostly key frames

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=264893

            Bug ID: 264893
           Summary: VideoEncoder outputs mostly key frames
           Product: WebKit
           Version: Safari Technology Preview
          Hardware: Mac (Apple Silicon)
                OS: macOS 14
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Media
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: jacob at bandes-stor.ch

VideoEncoder with `avc1.42001f` seems to output mostly `key` frames, and rarely any `delta` frames, using the following configuration:

  {
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    avc: { format: "annexb" },
  }

Steps to reproduce:
1. Visit https://codesandbox.io/s/crimson-http-nvddmr?file=/src/index.mjs
2. Open the console tab and observe the output is "{keyFrames: 20, deltaFrames: 0}".
3. Try changing the config's `framerate` and `bitrate`, but no changes seem to produce delta frames.

This occurs on macOS 14.0 (23A344). Same behavior in Safari 17.0 (19616.1.27.211.1) and also in Safari Technology Preview 182 (Safari 17.4, WebKit 19618.1.4.1).


Full code below:

```
async function main() {
  const width = 200;
  const height = 100;
  const img = new ImageData(width, height);

  for (let r=0;r<height;r++) {
    for (let c=0;c<width;c++) {
      img.data[(r*width+c)*4+0] = Math.floor(r/height*255);
      img.data[(r*width+c)*4+1] = Math.floor(c/width*255);
      img.data[(r*width+c)*4+2] = 127;
      img.data[(r*width+c)*4+3] = 255;
    }
  }

  const bitmap = await createImageBitmap(img);

  let keyFrames=0;
  let deltaFrames=0;
  const encoder = new VideoEncoder({
    output: (chunk, metadata) => {
      if (chunk.type==="key") {
        ++keyFrames;
      } else if (chunk.type === "delta") {
        ++deltaFrames;
      }
      console.log('output chunk',chunk,metadata);
    },
    error: (err) => {
      console.error('encode error', err)
    }
  })

  encoder.addEventListener("dequeue", (event) => {
    console.log('dequeued')
  })

  const fps = 30;

  encoder.configure({
    codec: "avc1.42001f", // Baseline profile (42 00) with level 3.1 (1f)
    width,
    height,
    // latencyMode: "realtime",
    // framerate: fps,
    avc: { format: "annexb" },
    // hardwareAcceleration:"prefer-software"
    // bitrate:1000000,
    // bitrate: 10000000,
  });

  for (let i=0; i < 20; i++) {
    const frame = new VideoFrame(bitmap, {
      timestamp:i * 1/fps * 1e6,
      duration:1/fps * 1e6,
    });
    encoder.encode(frame, {keyFrame: i === 0});
    frame.close();
  }

  bitmap.close();
  console.log("sent frames");

  await encoder.flush();
  console.log("flushed", {keyFrames,deltaFrames})
  encoder.close();
  console.log("closed", {keyFrames,deltaFrames})
}

main().catch((e) => console.error(e,e.message));
```

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20231115/99b1597a/attachment.htm>