[Webkit-unassigned] [Bug 277537] New: Video element stalls even after stalled video track removed

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=277537

            Bug ID: 277537
           Summary: Video element stalls even after stalled video track
                    removed
           Product: WebKit
           Version: Safari 17
          Hardware: Mac (Apple Silicon)
                OS: macOS 14
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Media
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: szymon.witamborski at bibo.com.ph

When video track stops during RTC negotiation (either by user action or by device failure), the video element used to play remote tracks will stall and calls to `video.play()` methods will never resolve. This happens even after the video track is eventually removed from the element and the audio track is active and streaming data.

Context: in RTC negotiations, the `track` events for remote tracks on `RTCPeerConnection` are dispatched when the remote offer is received and local answer is created. Importantly, this happens before the P2P connection is established. That means that when the track is created there's no guarantee that it will ever produce a frame. But at the same time, it wouldn't be unreasonable for a WebRTC application the track is added to a `HTMLVideoElement.srcObject` the moment the `track` event is received.

This creates the condition for the element to stall and not play the valid (streaming) audio track once remaining negotiation steps complete. Track stopping on the sending end will trigger renegotiation but even when that triggers removal of the video track on the receiving end, the element remains stalled.

The steps to reproduce use `canvas` to simulate the situation for brevity and seems to replicate the behavior with RTC closely.

When the video track does produce a frame, removing the track works correctly. (And of course the video plays even before the track is removed.) So if this is considered expected behavior then this bug report operates on the assumption that it would be reasonable to expect the same result when a stalled track is removed. Please let me know whether this assumption is correct.

Notes:
1. The stalled remote track is indistinguishable from an active one, it's not `muted`, it's `enabled` and `readyState` is `live`. (Note that this is where `canvas`-based example differs, the track is muted until the first frame.)
2. The example code doesn't call `play()` when the element is created initially for brevity but the results there would be that it's going to stall until the call to `load()` where it will reject with an error.
3. Chrome 127 and Firefox 128 have similar issue. Small difference in Firefox is that audio plays but the `play()` method still doesn't resolve. Neither needs the `setTimeout` for a repro.
4. Safari needs the `new Promise(resolve => setTimeout(resolve, 0))` line in order for a repro. Presumably, some processing needs to happen before the track stalls. It's not necessary when the track comes from RTC though.

------------------

Steps to reproduce:

Interactive version: https://codepen.io/brainshave/pen/abgJydJ?editors=1011 (make sure "Console" is open)

1. Open new tab and go to `about:blank`
2. Run below code via dev tools:

```
async function test() {
  // Create sine wave track
  const audioContext = new AudioContext();
  const audioNode = audioContext.createMediaStreamDestination();
  const audioTrack = audioNode.stream.getAudioTracks()[0];
  const oscillator = audioContext.createOscillator();
  oscillator.type = "sine";
  oscillator.frequency.value = 412;
  oscillator.connect(audioNode);
  oscillator.start();

  // Create stalled video track
  const canvas = document.createElement('canvas');
  const videoTrack = canvas.captureStream().getTracks()[0]; 

  // Necessary for repro in Safari, can be skipped in Chrome/Firefox
  await new Promise(resolve => setTimeout(resolve, 0));

  const video = document.createElement('video');
  video.srcObject = new MediaStream([videoTrack, audioTrack]);
  await new Promise(resolve => video.addEventListener('loadstart', resolve, { once: true }));

  // Control: non-stalled video track doesn't have this problem
  // canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
  // await video.play();

  console.log('removing track');
  video.srcObject.removeTrack(videoTrack);

  // video.load();

  await video.play();
  console.log('playing');
}

test()
```

This should print only `removing track` on the console and no sound will play

3. Uncomment the `video.load()` and run again, this time `removing track` will be followed by  `playing` and sound should be heard
4. Reload the page to stop the sound
5. Comment the `video.load()` and uncomment the two lines starting with `canvas.getContext` and the following `await video.play()` line
6. Run this and again, `removing track` will be followed by `playing` and audio should be audible
7. Reload the page to stop the sound

The expected result would be `await video.play()` to resolve and `playing` text to show without the `video.load()` workaround.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20240802/836e2fac/attachment-0001.htm>