[Webkit-unassigned] [Bug 226830] New: texImage2D is slow with HTMLVideoObjects from getUserMedia

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=226830

            Bug ID: 226830
           Summary: texImage2D is slow with HTMLVideoObjects from
                    getUserMedia
           Product: WebKit
           Version: Safari 14
          Hardware: iPhone / iPad
                OS: All
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: WebGL
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: paris at 8thwall.com
                CC: dino at apple.com, kbr at google.com, kkinnunen at apple.com

Created attachment 430980

  --> https://bugs.webkit.org/attachment.cgi?id=430980&action=review

Results of the benchmark script on three different devices.

Getting HTMLVideoObject pixels from getUserMedia into a WebGL texture with texImage2D is slow. 

Repro app: https://8w.8thwall.app/performance-test
Repro code: https://codepen.io/jparismorgan/pen/vYxrapM

This sample code assigns the stream from getUserMedia() to a HTMLMediaElement.srcObject, then renders it to a texture with texImage2D. texImage2D is quite slow - it takes 6.904 ms on an iPhone 12 Pro running iOS 14.4.2 with WebGL 1. On an iPhone 12 Pro Max with iOS 15.0 with WebGL 1 and "WebGL via Metal"=true it takes 2.503 ms. 
On a Galaxy S10 it takes 0.422 ms. In the wild (on pages that do more GPU processing and rendering), we typically see this taking even longer. For example typical realistic in the wild values for an iPhone 12 Pro are 7.5 ms.

I also tested two other cases:
  1) Setting gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true) before texImage2D, which speeds up the call slightly, i.e. from 6.904 ms down to 4.454 ms.
  2) Instead of calling texImage2D with a HTMLVideoElement, call it with an ArrayBufferView. This makes texImage2D faster - from 6.904 ms down to 0.401 ms. But this is not possible in practice because going from getUserMedia to an array buffer is very slow.

The repro above shows off the main path as well as these two modifications. Also attached are screenshots, and here is the data:

                                              iPhone 12 Pro iOS 14.4.2 WebGL 1 | iPhone 12 Pro Max iOS 15.0 WebGL 1 & "WebGL via Metal"=true | Galaxy S10 WebGL 1
texImage2D                                    6.904 ms                         | 4.297 ms                                                    | 0.407 ms
texImage2D + UNPACK_PREMULTIPLY_ALPHA_WEBGL   4.454 ms                         | 2.963 ms                                                    | 0.546 ms
texImage2D (from ArrayBufferView)             0.401 ms                         | 0.840 ms                                                    | 0.518 ms

Please let me know if I can provide more info, thank you.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20210609/a853d5c2/attachment-0001.htm>