[Webkit-unassigned] [Bug 161450] No reliable way to get a snapshot of WKWebView (macOS)

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=161450

--- Comment #10 from Dan <dasau at microsoft.com> ---
Created attachment 289304
  --> https://bugs.webkit.org/attachment.cgi?id=289304&action=review
Snapshot API test case outputs

I did some testing to make it easier for reviewers to see the behavior of this API for different cases, on iOS/Mac. Please refer to snapshot_testing_output.zip for the output images. There are some potential bugs with SnapshotOptionsExcludeDeviceScaleFactor described below, but I believe these are bug level fixes in WebKit and should not impact the design as long as we can agree on the input/outputs expected from this API.

Test application:
All snapshots were taken with a WKWebView that has a width of 700 and a height of 600. I pass the entire viewport size into the snapshotRect API (0, 0, 700, 600) in device independent points. I pass in different bitmap widths and make it clear in the image name the dimensions used. If not specified in the file name, the device had a Retina display.

Test devices:
MacBook Pro with Retina display (El Capitan) calling new API snapshotRect:intoImageOfWidth:completionHandler:
iPad Air 2 with Retina display (iOS 10) calling private API _snapshotRect:intoImageOfWidth:completionHandler:
iPhone 5s simulator (iOS 9.3) calling new API snapshotRect:intoImageOfWidth:completionHandler:
Mac Pro with non-Retina display (El Capitan) calling new API snapshotRect:intoImageOfWidth:completionHandler:

Bitmap size tests:
https://www.google.com
- It seems that through testing and analyzing the code, SnapshotOptionsExcludeDeviceScaleFactor does change the behavior of the snapshot but it is not intuitive. The logic in snapshotAtSize will call graphicsContext->applyDeviceScaleFactor(scaleFactor) and then immediately call the inverse graphicsContext->scale(1/scaleFactor). This means that that whatever is passed as the imageWidth parameter will be the width of the CGImage (in pixels) returned by this method regardless of device scale. The only difference is that CGContextSetBaseCTM will get called with device scale which affects some CoreGraphics features. Because scale is actually computed and not always equal to the device scale, this will cause CG features such as focus rectangle to appear at incorrect sizes unless imageWidth / viewportWidth == deviceScaleFactor. This will be the usual case for our intended purposes of the API, but I think this is a bug in WebPage::snapshotAtSize that could be looked into. I believe we want C
- Software snapshot will cause text to still look crisp as higher resolutions than the viewport, could be useful in some cases.
- 700_600_device_scale the focus rectangle looks too large because of the device scale factor issue above.
- iOS device private API starts looking blurry when imageWidth / viewportWidth != deviceScaleFactor because it is using a hardware snapshot.
- A common case will be to pass output imageWidth that factors in the device scale, in this case the snapshot looks good. It should be clearly documented that the imageWidth is in pixels (not scaled for device scale factor), while rectInViewCoordinates are in device independent points inside WKWebView bounds. This is the behavior of _snapshotRect private API, if a different design is better please let me know.

CSS 3D transform tests:
1400_1200_css_3d_transform.png - https://desandro.github.io/3dtransforms/examples/carousel-02-dynamic.html
- Software snapshot looks distorted for CSS 3d transform. Not ideal, but I am able to understand what the thumbnail or placeholder represents. Not a better option available now for Mac that allows snapshots of obscured views which will occur.
- iOS device snapshot looks good in this case.

HTML5 video
1400_1200_html_video.png - http://www.w3schools.com/html/html5_video.asp
- iOS 10 device hardware snapshot is not working as assumed. It is just showing a white rectangle as a placeholder for HTML5 video content when the snapshot is taken while video is playing. Could this be improved in the future, or are there technical limitations?
- Software snapshot is actually performing better for HTML5 video than hardware snapshot, it is able to capture an image for this sample video on my Mac that the iOS 10 device could not.
- Mac software snapshot is getting a blank rectangle for youtube video HTML5 case. Have not investigated the reason why it is able to capture one example, but not another (1400_1200_youtube.png - https://www.youtube.com/watch?v=CQY3KUR3VzM).

WebGL tests:
1400_1200_webgl.png - http://globe.chromeexperiments.com
- Neither iOS device or Mac software snapshot could capture this WebGL content.
- I waited until the WebGL content was fully loaded before I called the snapshot API. Regardless of this, I am only getting the HTML content that is not using hardware acceleration such as loading div, texts, and tabs.


Code relevant to SnapshotOptionsExcludeDeviceScaleFactor for this new API. I describe how it works above.
PassRefPtr<WebImage> WebPage::snapshotAtSize(const IntRect& rect, const IntSize& bitmapSize, SnapshotOptions options)
{
    Frame* coreFrame = m_mainFrame->coreFrame();
    if (!coreFrame)
        return nullptr;

    FrameView* frameView = coreFrame->view();
    if (!frameView)
        return nullptr;

    IntRect snapshotRect = rect;
    float horizontalScaleFactor = static_cast<float>(bitmapSize.width()) / rect.width();
    float verticalScaleFactor = static_cast<float>(bitmapSize.height()) / rect.height();
    float scaleFactor = std::max(horizontalScaleFactor, verticalScaleFactor);

    auto snapshot = WebImage::create(bitmapSize, snapshotOptionsToImageOptions(options));
    if (!snapshot->bitmap())
        return nullptr;

    auto graphicsContext = snapshot->bitmap()->createGraphicsContext();

    if (options & SnapshotOptionsPrinting) {
        PrintContext::spoolAllPagesWithBoundaries(*coreFrame, *graphicsContext, snapshotRect.size());
        return snapshot;
    }

    Color documentBackgroundColor = frameView->documentBackgroundColor();
    Color backgroundColor = (coreFrame->settings().backgroundShouldExtendBeyondPage() && documentBackgroundColor.isValid()) ? documentBackgroundColor : frameView->baseBackgroundColor();
    graphicsContext->fillRect(IntRect(IntPoint(), bitmapSize), backgroundColor);

    if (!(options & SnapshotOptionsExcludeDeviceScaleFactor)) {
        double deviceScaleFactor = corePage()->deviceScaleFactor();
        graphicsContext->applyDeviceScaleFactor(deviceScaleFactor);
        scaleFactor /= deviceScaleFactor;
    }

    graphicsContext->scale(FloatSize(scaleFactor, scaleFactor));
    graphicsContext->translate(-snapshotRect.x(), -snapshotRect.y());

    FrameView::SelectionInSnapshot shouldPaintSelection = FrameView::IncludeSelection;
    if (options & SnapshotOptionsExcludeSelectionHighlighting)
        shouldPaintSelection = FrameView::ExcludeSelection;

    FrameView::CoordinateSpaceForSnapshot coordinateSpace = FrameView::DocumentCoordinates;
    if (options & SnapshotOptionsInViewCoordinates)
        coordinateSpace = FrameView::ViewCoordinates;

    frameView->paintContentsForSnapshot(*graphicsContext, snapshotRect, shouldPaintSelection, coordinateSpace);

    if (options & SnapshotOptionsPaintSelectionRectangle) {
        FloatRect selectionRectangle = m_mainFrame->coreFrame()->selection().selectionBounds();
        graphicsContext->setStrokeColor(Color(0xFF, 0, 0));
        graphicsContext->strokeRect(selectionRectangle, 1);
    }

    return snapshot;
}

void GraphicsContext::applyDeviceScaleFactor(float deviceScaleFactor)
{
    scale(FloatSize(deviceScaleFactor, deviceScaleFactor));

    if (isRecording()) {
        m_displayListRecorder->applyDeviceScaleFactor(deviceScaleFactor);
        return;
    }

    platformApplyDeviceScaleFactor(deviceScaleFactor);
}

void GraphicsContext::platformApplyDeviceScaleFactor(float deviceScaleFactor)
{
    if (paintingDisabled())
        return;

    ASSERT(!isRecording());

    // CoreGraphics expects the base CTM of a HiDPI context to have the scale factor applied to it.
    // Failing to change the base level CTM will cause certain CG features, such as focus rings,
    // to draw with a scale factor of 1 rather than the actual scale factor.
    CGContextSetBaseCTM(platformContext(), CGAffineTransformScale(CGContextGetBaseCTM(platformContext()), deviceScaleFactor, deviceScaleFactor));
}

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-unassigned/attachments/20160920/6bf4ed5b/attachment.html>