[Webkit-unassigned] [Bug 253186] New: REGRESSION iOS 16.4 beta selects ultra-wide for facingMode: environment

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=253186

            Bug ID: 253186
           Summary: REGRESSION iOS 16.4 beta selects ultra-wide for
                    facingMode: environment
           Product: WebKit
           Version: Safari 16
          Hardware: iPhone / iPad
                OS: iOS 16
            Status: NEW
          Severity: Major
          Priority: P2
         Component: WebRTC
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: simontaylor1 at ntlworld.com
                CC: youennf at gmail.com

Related: Bug 250142 and Bug 252560

Recently WebKit has added support for exposing the virtual cameras on iOS and logic to prefer these virtual cameras when choosing a camera based on a facingMode constraint.

The virtual cameras start with videoZoomFactor set to 1, which means the field of view is set to the widest-FOV constituent device in the set. In most recent iPhone devices (all of them since iPhone 11) that means the default camera stream that comes from a {facingMode: 'environment'} constraint is now an ultra-wide one. I've verified this on an iPhone 12 Pro using the latest 16.4 Beta 2 build.

After https://github.com/WebKit/WebKit/pull/10784 was applied, the current priority list in AVVideoCaptureSource::facingModeFitnessScoreAdjustment() looks like this:
 1. AVCaptureDeviceTypeBuiltInTripleCamera
 2. AVCaptureDeviceTypeBuiltInDualWideCamera
 3. AVCaptureDeviceTypeBuiltInUltraWideCamera
 4. AVCaptureDeviceTypeBuiltInDualCamera
 5. AVCaptureDeviceTypeBuiltInWideAngleCamera
 6. AVCaptureDeviceTypeBuiltInTelephotoCamera
 7. AVCaptureDeviceTypeDeskViewCamera

The previous default was to use the standard FOV AVCaptureDeviceTypeBuiltInWideAngleCamera (now only the 5th priority).

Changing this default has far-ranging consequences and will likely break many of our sites (due to assumptions on camera model used in computer vision algorithms, as the Web API doesn't expose any specific characteristics on this).

Sites that want a simple camera UI such as ours just use a toggle button that swaps between 'user' and 'environment' with a facingMode constraint. I think most users would expect that to swap between the standard 1x rear camera and the front-facing one, hence why I have called this a regression.

Right now WebKit doesn't expose the zoom settings, so the virtual cameras always use the smallest possible zoom and hence are effectively just identical to the ultra-wide cameras (also exposed in the enumeration).

>From the discussion on bug 250142 it's possible the intent was to match the default behaviour seen in the iOS camera app on an iPhone 13/14 Pro. The standard camera on those devices has quite a high minimum focus distance, so the iPhone camera app viewfinder in Photo mode switches to the ultrawide when getting close to objects as it provides better macro focus.

If the changes to add virtual camera support to WebKit were intended to match that behaviour then the videoZoomFactor needs to be set to 2x when using the virtual cameras to achieve that.

However the auto switching for macro is not the right default for every possible getUserMedia use-case. It would definitely help for scanning barcodes from closer distances on the iPhone 14 Pro, but for AR use cases (which is our primary focus) the switch of camera is visually jarring and we'd have a strong preference to lock to the standard camera and put up with a bit of blur in the background view if users move too close.

Note not all native camera use cases on iPhone 14 Pro use the auto-switching behaviour - when a video is being recorded the camera doesn't switch until you manually tap-to-focus. AR Quick Look (and presumably other ARKit experiences) don't switch to the ultrawide when getting close to a surface.

Therefore for the best backward compatibility, I'd suggest switching back to a preference for the standard cameras when selecting just based on the facingMode constraint. Note that's also what the native iOS API does for backwards compatibility - the original `[AVCaptureDevice devicesWithMediaType:] method only returns the standard one.

Of course I would also welcome a way for sites such as barcode scanners to opt-in to "auto macro" mode involving a lens switch - perhaps a new "Auto Macro" virtual camera that would be the Triple or DualWide with videoZoomFactor set to 2x. I still think the lens switch is pretty jarring for many getUserMedia use cases so I'd argue that should be opt-in rather than the default for the environment facing mode.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20230301/c2d43ef3/attachment-0001.htm>