[webkit-changes] [WebKit/WebKit] 1021d6: Crash under RenderLayer::calculateClipRects() when...

[Simon Fraser]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 1021d66fe7c33f8661132fbe8803e7bca4e91692
      https://github.com/WebKit/WebKit/commit/1021d66fe7c33f8661132fbe8803e7bca4e91692
  Author: Simon Fraser <simon.fraser at apple.com>
  Date:   2024-02-08 (Thu, 08 Feb 2024)

  Changed paths:
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-dialog-expected.txt
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-dialog.html
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-expected.txt
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-variant-expected.txt
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-variant.html
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen.html
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-popover-expected.txt
    A LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-popover.html
    M Source/WebCore/rendering/RenderLayer.cpp
    M Source/WebCore/rendering/RenderLayerCompositor.cpp
    M Source/WebCore/rendering/RenderLayerCompositor.h

  Log Message:
  -----------
  Crash under RenderLayer::calculateClipRects() when going into fullscreen
https://bugs.webkit.org/show_bug.cgi?id=268891
rdar://121960496

Reviewed by Alan Baradlay.

A combination of top layer and compositing backing sharing can cause a null de-ref when entering fullscreen,
or using modal dialogs or popovers.

The issue occurs when the renderer going into top layer participates in a backing sharing sequence, in the
`RenderLayer::paintsIntoProvidedBacking()` sense. What happens in that case is that after the top layer
configuration is changed we do a layout, after which `RenderLayerBacking::updateAfterLayout()` calls
`RenderLayerBacking::updateCompositedBounds()` (this seems like an odd thing to do, because we're going
to do a compositing update anyway, but a comment explains why we do it). This call requires that we compute
clip rects, which calls `RenderLayer::canUseOffsetFromAncestor()`, which gets confused because the ancestor
layer is no longer an ancestor.

The fix is to clear any relevant backing sharing sequences when going into top layer, where "relevant" means
backing sharing sequences in the stacking context of the layer that's going into top layer. We do that
by calling into RenderLayerCompositor from `RenderLayer::establishesTopLayerWillChange()`. Normally traversing
layers in a stacking context would walk the z-order lists, and this works for popover and dialog, but fullscreen
triggers a style update before this code runs, which clears the z-order lists. So this stacking context
traversal is written in terms of the RenderLayer tree (like `collectLayers()`).

* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-dialog-expected.txt: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-dialog.html: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-expected.txt: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-variant-expected.txt: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen-variant.html: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-fullscreen.html: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-popover-expected.txt: Added.
* LayoutTests/compositing/shared-backing/top-layer/backing-sharing-split-by-popover.html: Added.
* Source/WebCore/rendering/RenderLayer.cpp:
(WebCore::RenderLayer::establishesTopLayerWillChange):
(WebCore::RenderLayer::calculateClipRects const):
(WebCore::outputPaintOrderTreeLegend):
(WebCore::outputPaintOrderTreeRecursive):
* Source/WebCore/rendering/RenderLayerCompositor.cpp:
(WebCore::RenderLayerCompositor::establishesTopLayerWillChangeForLayer):
(WebCore::clearBackingSharingWithinStackingContext):
(WebCore::RenderLayerCompositor::clearBackingProviderSequencesInStackingContextOfLayer):
* Source/WebCore/rendering/RenderLayerCompositor.h:

Canonical link: https://commits.webkit.org/274290@main