[Webkit-unassigned] [Bug 207128] New: Drawing canvas to texture or canvas2d sometimes fails, flashes white

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=207128

            Bug ID: 207128
           Summary: Drawing canvas to texture or canvas2d sometimes fails,
                    flashes white
           Product: WebKit
           Version: Safari 13
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: WebRTC
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: nb at 8thwall.com
                CC: youennf at gmail.com

Created attachment 389533

  --> https://bugs.webkit.org/attachment.cgi?id=389533&action=review

source code for repro case

WebKit regularly appears to fail to draw canvas data to another source, and when this happens the source canvas flashes white. We have observed this behavior when drawing from canvas to canvas:

   captureCanvas_.getContext('2d').drawImage(srcCanvas_, ...)

and we have also seen the same behavior with populating a different texture in the same WebGlContext with texImage2D, e.g.

   gl.bindTexture(gl.TEXTURE_2D, captureTex)
   gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, canvas)

When drawing the canvas fails, the destination image is black, including the alpha channel. One workaround solution is to detect this transparent black image and retry capture until we have image data, as in this example:

  const CAPTURE_HEIGHT = 1280
  const REPEAT_TRIES = 4
  let captureCanvas_ = document.createElement('canvas')
  let captureContext_ = captureCanvas_.getContext('2d')
  captureCanvas_.width = CAPTURE_HEIGHT * 3 / 4
  captureCanvas_.height = CAPTURE_HEIGHT

  // Returns base64 string of image data upon success, null upon failure
  const getCanvasData = (canvas) => {
    captureContext_.drawImage(canvas, 0, 0, CAPTURE_HEIGHT * 3 / 4, CAPTURE_HEIGHT)

    // drawImage can fail on iOS leaving the canvas transparent. We return null so we can retry.
    const firstPixel = captureContext_.getImageData(0, 0, 1, 1)
    if (firstPixel.data[3] === 0) {
      return null
    }
    return captureContext_.getImageData(0, 0, 1, 1)
      ? null
      : captureCanvas_.toDataURL('image/jpeg', JPG_COMPRESSION / 100)
          .substring('data:image/jpeg;base64,'.length)
  }

  const takeScreenshot = (canvas, repeatTries = 0) => new Promise((resolve, reject) => 
    window.requestAnimationFrame(() => {
      const imageData = getCanvasData(canvas)
      if (repeatTries < MAX_REPEAT_TRIES && !imageData) {
        setTimeout(() => takeScreenshot(canvas, repeatTries + 1).then(resolve, reject), 60)
      } else {
        if (imageData) {
          updateStatusText(`OK after ${repeatTries} tries.`)
          resolve(imageData)
        } else {
          updateStatusText(`Failed ${repeatTries} times.`)
          reject(new Error('Unable to read pixels from canvas.'))
        }
      }
    })
  )

This workaround is imperfect for a number of reasons:
 * It adds expense to read back pixel values to perform the test.
 * Whenever drawing fails, the canvas flashes white, leading to a strobing effect on one or more retries.  To work around this, in production, we always need to show a full screen white div that fades out after capture succeeds or ultimately fails so that at least the effect appears intentional.
 * Capturing multiple frames in sequence (for example to capture a gif) gives inconsistent frame delay, leading to a jumpy video with poor quality.
 * Sometimes capture fails even after 10 tries, and the user is ultimately unable to receive an image, which is a poor and confusing experience.

Capturing and sharing augmented image content is a key flow for user engagement. This workaround was required for WebAR activations by Pink Floyd, Bailey's, Nike, Johnnie Walker, Game of Thrones, Monopoly, Ally Bank, Adidas, Porsche,  Spider-Man: Far From Home, The Phillies, Lego, Red Bull, Toyota, Coach and many more.

A basic repro case can be viewed on an iPhone here: 8th.io/3usye

The repro app does the following:
* getUserMedia -> stream -> video
* video -> texture (there is a rotating pool of 10 textures)
* texture -> sepia shader -> canvas -> window
* on a button press, call takeScreenshot(canvas) as above
* print the number of retries before successful capture, or a failure message if capture failed.

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20200203/dc021978/attachment-0001.htm>