[webkit-changes] [WebKit/WebKit] c688b3: AX: Allow strict replacements via AXTextOperation

[Samar Sunkaria]

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: c688b31fdd986a6fdbe6643d1d5983042b601af7
      https://github.com/WebKit/WebKit/commit/c688b31fdd986a6fdbe6643d1d5983042b601af7
  Author: Samar Sunkaria <sameesunkaria at me.com>
  Date:   2024-09-16 (Mon, 16 Sep 2024)

  Changed paths:
    A LayoutTests/accessibility/mac/text-operation/text-operation-capitalize-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-capitalize.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-lowercase-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-lowercase.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-replace-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-replace-preserve-case-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-replace-preserve-case.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-replace.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-select-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-select.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-smart-replace-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-smart-replace.html
    A LayoutTests/accessibility/mac/text-operation/text-operation-uppercase-expected.txt
    A LayoutTests/accessibility/mac/text-operation/text-operation-uppercase.html
    M LayoutTests/platform/mac-wk1/TestExpectations
    M LayoutTests/platform/mac-wk2/TestExpectations
    M LayoutTests/resources/accessibility-helper.js
    M Source/WebCore/accessibility/AXCoreObject.h
    M Source/WebCore/accessibility/AccessibilityObject.cpp
    M Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
    M Tools/WebKitTestRunner/InjectedBundle/AccessibilityTextMarkerRange.cpp
    M Tools/WebKitTestRunner/InjectedBundle/AccessibilityTextMarkerRange.h
    M Tools/WebKitTestRunner/InjectedBundle/AccessibilityUIElement.cpp
    M Tools/WebKitTestRunner/InjectedBundle/AccessibilityUIElement.h
    M Tools/WebKitTestRunner/InjectedBundle/Bindings/AccessibilityUIElement.idl
    M Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm

  Log Message:
  -----------
  AX: Allow strict replacements via AXTextOperation
https://bugs.webkit.org/show_bug.cgi?id=278458
rdar://134409037

Reviewed by Tyler Wilcock.

The AXTextOperation accessibility API for macOS allows text replacement within a specified
list of text marker ranges, among other operations. This AX API is particularly beneficial
for clients like Grammarly for macOS, that utilize the accessibility subsystem to perform
text replacements. However, in its current form, AXTextOperation arbitrarily changes the
case of the replacement string and performs a “smart replacement” that adds spaces around
the replaced text, severely limiting the usability of the API. This commit introduces the
following modifications to the API to allow better control over the replacement. While the
same outcome can be achieved by chaining multiple calls to AX APIs/emitting CGEvents,
AXTextOperation allows for a more efficient implementation.

Preserving the Case of the Replacement String:

`TextOperationReplace` modifies the case of the replacement string supplied to the API,
changing it to either capitalized or lowercase, presumably to match the case of the text
being replaced. As a result, clients of AXTextOperation cannot enforce the case of the
replacement string they provide.

A real example of the existing behavior that makes incorrect/unexpected changes shows up
when the case of the replacement string must be preserved, e.g. when replacing “Mac OS X”
with “macOS”. The current implementation would infer that “Mac OS X” is capitalized and
would attempt to capitalize the replacement string “macOS” -> “MacOS” to preserve the
case.

To address this, an additional `AXTextOperationType` called
"TextOperationReplacePreserveCase" has been introduced, which allows for text replacements
that preserve the case of the replacement string for clients that know/want the
replacement string to maintain its case. When supplying the new type, the replacement
string will be used as is, without attempting to match its case to the string being
replaced. Since this is an additive change, the existing clients of AXTextOperation will
remain unaffected.

Allow Disabling Smart Replacements:

All AXTextOperation types that replace text (i.e. all operations types except “select”),
always perform a “smart replace”, which assumes that the replacement string is not
replacing partial words and attempts to add spaces around it. While this is an excellent
feature for certain replacements, it can severely limit the usability of AXTextOperation
for partial word replacements.

For example, when replacing “MacOS” with “macOS”, we could attempt to perform a minimal
change, where the “M” is replaced with “m” (given we already have
`TextOperationReplacePreserveCase` from above). The current implementation would perform
a smart replacement and, therefore, add a space after the replaced “m”,
“MacOS” -> “m acOS”.

To address this, a new key called "AXTextOperationSmartReplace" has been added to the
AXTextOperation dictionary to support _non-smart_ replacements. The value of this key is
a boolean, where NO/false disables smart replacements. Since this is an additive change,
the existing clients of AXTextOperation will remain unaffected.

Usage:

Client pseudo-code below:
```
// Replace “Mac OS X” -> “macOS”
CFTypeRef result;
AXUIElementCopyParameterizedAttributeValue(element, "AXTextOperation", @[
  NSAccessibilityTextOperationType: NSAccessibilityTextOperationReplacePreserveCase,
  NSAccessibilityTextOperationReplacementString: @"macOS",
  NSAccessibilityTextOperationMarkerRanges: @[ rangeOfMacOSX ]
], &result)

// Replace "MacOS" -> "macOS", by only replacing the first character
CFTypeRef result;
AXUIElementCopyParameterizedAttributeValue(element, "AXTextOperation", @[
  NSAccessibilityTextOperationType: NSAccessibilityTextOperationReplacePreserveCase,
  NSAccessibilityTextOperationReplacementString: @"m",
  NSAccessibilityTextOperationSmartReplace: kCFBooleanFalse,
  NSAccessibilityTextOperationMarkerRanges: @[ rangeOfM ]
], &result)
```

Layout tests have been introduced for the various AXTextOperation types. At the time of
writing, AXTextOperation has a bug with replacements/transformations over multiple
ranges, which is why some of the tests cases are expected to fail until the bug is
resolved. See https://bugs.webkit.org/show_bug.cgi?id=278928

* LayoutTests/accessibility/mac/text-operation/text-operation-capitalize-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-capitalize.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-lowercase-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-lowercase.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-replace-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-replace-preserve-case-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-replace-preserve-case.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-replace.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-select-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-select.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-smart-replace-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-smart-replace.html: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-uppercase-expected.txt: Added.
* LayoutTests/accessibility/mac/text-operation/text-operation-uppercase.html: Added.
* LayoutTests/platform/mac-wk1/TestExpectations:
* LayoutTests/platform/mac-wk2/TestExpectations:
* Source/WebCore/accessibility/AXCoreObject.h:
(WebCore::AccessibilityTextOperation::AccessibilityTextOperation): Deleted.
* Source/WebCore/accessibility/AccessibilityObject.cpp:
(WebCore::AccessibilityObject::performTextOperation):
* Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:
(accessibilityTextOperationForParameterizedAttribute):
* Tools/WebKitTestRunner/InjectedBundle/AccessibilityTextMarkerRange.cpp:
(WTR::toTextMarkerRange):
* Tools/WebKitTestRunner/InjectedBundle/AccessibilityTextMarkerRange.h:
* Tools/WebKitTestRunner/InjectedBundle/AccessibilityUIElement.cpp:
(WTR::AccessibilityUIElement::performTextOperation):
* Tools/WebKitTestRunner/InjectedBundle/AccessibilityUIElement.h:
* Tools/WebKitTestRunner/InjectedBundle/Bindings/AccessibilityUIElement.idl:
* Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm:
(WTR::textOperationParameterizedAttribute):
(WTR::AccessibilityUIElement::performTextOperation):

Canonical link: https://commits.webkit.org/283740@main



To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications