[webkit-dev] Richer text change notifications for accessibility RFC

[Doug Russell]

-- Goals --

Text selection change notifications allow assistive technology to inform a user about change contents and state. The notification sent by WebKit are insufficient because multiple actions result in the same notification, which makes it impossible in some cases to provide the user with appropriate feedback. For example, with the notifications it is impossible for VoiceOver to distinguish between a user typing text, and a user pasting text. Another example is javascript that has swapped one form value for another (for example, formatting a phone number).

-- Implementation --

Richer text change notifications are implemented by adding infrastructure to EditCommands and adding logic to Editor, FrameSelection and FocusController to thread a selection intent through to the logic that posts notifications.

--- New Edit Commands ---

Each of these EditCommands is a subclass of an existing subclass that is used in the appropriate logic for applying that command:

CutDeleteSelectionCommand
CutDeleteSelectionCommand
DictationInsertIntoTextNodeCommand
DictationInsertTextCommand
PasteInsertNodeBeforeCommand
PasteReplaceInsertIntoTextNodeCommand
ReplaceDeleteFromTextNodeCommand
ReplaceInsertIntoTextNodeCommand
TypingInsertIntoTextNodeCommand
TypingInsertNodeBeforeCommand
TypingInsertTextCommand
TypingReplaceInsertIntoTextNodeCommand

--- Edit Types ---

Value changes now include edit types. This is largely a subset of the existing EditAction enum, but needs to be it's own type since it's value are exposed via notifications and need to be stable. It also has a notion of generic Insert that EditAction doesn't which is used for cases where there isn't a strong need for context (currently).

enum AXTextEditType {
    AXTextEditTypeUnknown,
    AXTextEditTypeDelete, // Generic text delete
    AXTextEditTypeInsert, // Generic text insert
    AXTextEditTypeTyping, // Insert via typing
    AXTextEditTypeDictation, // Insert via dictation
    AXTextEditTypeCut, // Delete via Cut
    AXTextEditTypePaste // Insert via Paste
};

Each EditCommand now exposes an edit type for when it is applied and when it is unapplied (undo), which is used when posting value change notifications.

virtual AXObjectCache::AXTextEditType WebCore::SimpleEditCommand::applyEditType();
virtual AXObjectCache::AXTextEditType WebCore::SimpleEditCommand::unapplyEditType();

virtual void WebCore::SimpleEditCommand::notifyAccessibilityForTextChange(Node*, AXObjectCache::AXTextEditType, const String&, const VisiblePosition&);

--- Intents ---

Selection changes now include the values contained in a new struct AXTextStateChangeIntent. Intent allows a selection change to be annotated as the result of an edit event or the result of a selection change (in the form of navigation or a range selection). This follows closely with the existing selection model of Safari and like edit type is it's own type because it is exposed via notifications and needs to be stable and also adds some accessibility specific values. Intent includes a sync member to flag selection changes that are the result of an assistive app setting selection.

enum AXTextStateChangeType {
    AXTextStateChangeTypeUnknown,
    AXTextStateChangeTypeEdit,
    AXTextStateChangeTypeSelectionMove,
    AXTextStateChangeTypeSelectionExtend
};

enum AXTextSelectionDirection {
    AXTextSelectionDirectionUnknown,
    AXTextSelectionDirectionBeginning,
    AXTextSelectionDirectionEnd,
    AXTextSelectionDirectionPrevious,
    AXTextSelectionDirectionNext,
    AXTextSelectionDirectionDiscontiguous
};

enum AXTextSelectionGranularity {
    AXTextSelectionGranularityUnknown,
    AXTextSelectionGranularityCharacter,
    AXTextSelectionGranularityWord,
    AXTextSelectionGranularityLine,
    AXTextSelectionGranularitySentence,
    AXTextSelectionGranularityParagraph,
    AXTextSelectionGranularityPage,
    AXTextSelectionGranularityDocument,
    AXTextSelectionGranularityAll // All granularity represents the action of selecting the whole document as a single action. Extending selection by some other granularity until it encompasses the whole document will not result in a all granularity notification.
};

struct AXTextSelection {
    AXTextSelectionDirection direction;
    AXTextSelectionGranularity granularity;
};

struct AXTextStateChangeIntent {
    AXTextStateChangeType type;
    union {
        AXTextSelection selection;
        AXTextEditType change;
    };
    bool sync;
};

EditCommand and EditCommandComposition now expose intents for selection changes as a result of applying editing and unapplying (undoing) editing. In the case of a composite command, intent is taken from the last applied command.

virtual AXObjectCache::AXTextStateChangeIntent WebCore::EditCommand::applyIntent();
virtual AXObjectCache::AXTextStateChangeIntent WebCore::EditCommandComposition::unapplyIntent();

--- Changes to Editor, FrameSelection, FocusController ---

FrameSelection setSelection() now includes a new argument with a default value for intent. This allows cases that don't have an explicit intent to omit the argument so that unrelated code paths can remain unchanged.

void WebCore::FrameSelection::setSelection(const VisibleSelection&, SetSelectionOptions = defaultSetSelectionOptions(), AXObjectCache::AXTextStateChangeIntent = defaultSetSelectionIntent(), CursorAlignOnScroll = AlignCursorOnScrollIfNeeded, TextGranularity = CharacterGranularity);

FrameSelection setSelectionByMouseIfDifferent() now calls set selection with a selection intent for Move if it's a caret or Extend if it's a range and direction of Discontiguous.

Editor uses the edit type and intent data added to edit commands when calling changeSelectionAfterCommand() which now accepts an optional argument for intent.

void WebCore::Editor::changeSelectionAfterCommand(const VisibleSelection& newSelection, FrameSelection::SetSelectionOptions, AXObjectCache::AXTextStateChangeIntent = AXObjectCache::AXTextStateChangeIntent());

FocusController advanceFocusInDocumentOrder() (called when tabbing through a document) now calls selection with an selection intent for Move Discontiguous.

--- Notifications ---

Simple changes, like typing, pasting, deleting, etc, can now be posted with the type, the value of the change and the position of that change. The position can help inform if the change is within the currently focused element of an assistive app.

void WebCore::AXObjectCache::postTextStateChangeNotification(Node*, AXTextEditType, const String&, const VisiblePosition&);

Compound changes, for example replacing the value for an input form field for formatting, can now be posted as a single atomic notification.

void WebCore::AXObjectCache::postTextReplacementNotification(Node*, AXTextEditType deletionType, const String& deletedText, AXTextEditType insertionType, const String& insertedText, const VisiblePosition&);

Selection changes can now be posted with intent and the selection resulting from that intent.

void WebCore::AXObjectCache::postTextStateChangeNotification(Node*, AXTextStateChangeIntent, const VisibleSelection&);

--- Security ---

This change does not expose any additional user content than what was previously available to an assistive-technology user. This change only expands on informing the assistive-technology product about the user intent. 

In password fields no assistive technology product can get to user-typed text. This change also include better obfuscation of typing cadence in password fields using the approach implemented in AppKit which was provided and approved by Product Security.

--- Compatibility ---

Because this work overlaps with the previous work done by GTK, the implementations of postTextChangeNotification use PLATFORM macros to translate new data types into the data types that GTK is expecting when not building for Cocoa.

--- Links ---

https://bugs.webkit.org/show_bug.cgi?id=142719 <https://bugs.webkit.org/show_bug.cgi?id=142719>
https://bugs.webkit.org/show_bug.cgi?id=25898 <https://bugs.webkit.org/show_bug.cgi?id=25898>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20150414/cf12b2f4/attachment.html>