[Webkit-unassigned] [Bug 206843] New: Can't draw stream->video->texture->canvas without also drawing video.

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=206843

            Bug ID: 206843
           Summary: Can't draw stream->video->texture->canvas without also
                    drawing video.
           Product: WebKit
           Version: Safari 13
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: WebRTC
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: nb at 8thwall.com
                CC: youennf at gmail.com

Created attachment 388902

  --> https://bugs.webkit.org/attachment.cgi?id=388902&action=review

source code for repro case

WebKit appears to stop streaming the camera feed to a <video> tag when the element is not visible, either because it is display none, or it is fully occluded by other DOM elements. Streaming to an off-screen <video> is common when drawing the camera feed to a texture for canvas display, for example to apply visual effects on top of the camera feed, or to synchronize display of the camera feed with the result of some visual processing (QR code detection, image target tracking, SLAM, etc.).

Our workaround solution is to make the camera feed visible but all-but-hidden, e.g. by displaying as a block element behind the drawing canvas, offset by one pixel:

  const setVideoForSafari = ({video, canvas}) => {
    if (!canvas.style['z-index']) {
      canvas.style['z-index'] = 1
    }
    canvas.style.top = '1px'
    canvas.style.left = '0px'
    canvas.style.position = 'absolute'
    video.setAttribute('playsinline', true)
    video.style.display = 'block'
    video.style['z-index'] = canvas.style['z-index'] - 1
    video.style.top = '0px'
    video.style.left = '0px'
    video.style.position = 'absolute'
    video.style.opacity = '0.01'
    video.parentNode.style.position = 'absolute'
  }

This approach adds inefficiency because camera pixels are drawn twice, and adds a subtle visual defect because the camera feed is one-pixel visible. This workaround was required for WebAR activations by Pink Floyd, BBC America's Dr Who, Bailey's, Jumanji, Johnnie Walker, Game of Thrones, Monopoly, Ally Bank, Adidas, Porsche, Eva Air, London's Waterloo Station, Spider-Man: Far From Home, The Phillies, Lego, General Mills, Red Bull, Toyota, and many more.

A basic repro case can be viewed on an iPhone here: 8th.io/k4c5d

The repro app does the following:
* getUserMedia -> stream -> video
* iOS only: set the video style to be display: block, z-index: canvas.style.zIndex - 1, 1 pixel offset from canvas
* video -> texture (there is a rotating pool of 10 textures)
* texture -> sepia shader -> canvas -> window
* on a button press, toggle video.style.display to block / none

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20200127/af744e6c/attachment.htm>