<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[181917] trunk</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/181917">181917</a></dd>
<dt>Author</dt> <dd>benjamin@webkit.org</dd>
<dt>Date</dt> <dd>2015-03-24 16:24:36 -0700 (Tue, 24 Mar 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>Make URL filter patterns matching consistent and add a simple canonicalization step
https://bugs.webkit.org/show_bug.cgi?id=142998
Patch by Benjamin Poulain <bpoulain@apple.com> on 2015-03-24
Reviewed by Alex Christensen.
Source/WebCore:
This patch makes two changes to the url filter input:
-Make the matching "Search" by default, the pattern can now appear anywhere
in the URL by default.
-Make the input a little less fragile: do not explode on valid input
that is not formatted in a certain way.
To implement the search behavior, I simply add an implict ".*" in front of the patterns
when that make sense.
To make the input more solid, we do some little modification on the input:
-Remove duplicated ".*".
-Remove matching suffixes that do not bring new information.
-Unify all the ".*" in the same format.
Why do that here? That should be done through a graph analysis on the machine.
The reason is this is incredibly cheap compared to the graph analysis. Any state
removed upfront will save the handling of several hundred nodes in the deterministic
graph.
* contentextensions/URLFilterParser.cpp:
(WebCore::ContentExtensions::Term::isKnownToMatchAnyString):
(WebCore::ContentExtensions::Term::isUniversalTransition):
(WebCore::ContentExtensions::GraphBuilder::finalize):
(WebCore::ContentExtensions::GraphBuilder::assertionBOL):
(WebCore::ContentExtensions::GraphBuilder::fail):
(WebCore::ContentExtensions::GraphBuilder::simplifySunkTerms):
Tools:
* TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp:</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCorecontentextensionsURLFilterParsercpp">trunk/Source/WebCore/contentextensions/URLFilterParser.cpp</a></li>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsTestWebKitAPITestsWebCoreContentExtensionscpp">trunk/Tools/TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (181916 => 181917)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog 2015-03-24 23:10:30 UTC (rev 181916)
+++ trunk/Source/WebCore/ChangeLog 2015-03-24 23:24:36 UTC (rev 181917)
</span><span class="lines">@@ -1,3 +1,38 @@
</span><ins>+2015-03-24 Benjamin Poulain <bpoulain@apple.com>
+
+ Make URL filter patterns matching consistent and add a simple canonicalization step
+ https://bugs.webkit.org/show_bug.cgi?id=142998
+
+ Reviewed by Alex Christensen.
+
+ This patch makes two changes to the url filter input:
+ -Make the matching "Search" by default, the pattern can now appear anywhere
+ in the URL by default.
+ -Make the input a little less fragile: do not explode on valid input
+ that is not formatted in a certain way.
+
+ To implement the search behavior, I simply add an implict ".*" in front of the patterns
+ when that make sense.
+
+ To make the input more solid, we do some little modification on the input:
+ -Remove duplicated ".*".
+ -Remove matching suffixes that do not bring new information.
+ -Unify all the ".*" in the same format.
+
+ Why do that here? That should be done through a graph analysis on the machine.
+
+ The reason is this is incredibly cheap compared to the graph analysis. Any state
+ removed upfront will save the handling of several hundred nodes in the deterministic
+ graph.
+
+ * contentextensions/URLFilterParser.cpp:
+ (WebCore::ContentExtensions::Term::isKnownToMatchAnyString):
+ (WebCore::ContentExtensions::Term::isUniversalTransition):
+ (WebCore::ContentExtensions::GraphBuilder::finalize):
+ (WebCore::ContentExtensions::GraphBuilder::assertionBOL):
+ (WebCore::ContentExtensions::GraphBuilder::fail):
+ (WebCore::ContentExtensions::GraphBuilder::simplifySunkTerms):
+
</ins><span class="cx"> 2015-03-24 Chris Dumez <cdumez@apple.com>
</span><span class="cx">
</span><span class="cx"> Regression(r181671): Caused Timer-related crashes on iOS / WK1
</span></span></pre></div>
<a id="trunkSourceWebCorecontentextensionsURLFilterParsercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/contentextensions/URLFilterParser.cpp (181916 => 181917)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/contentextensions/URLFilterParser.cpp 2015-03-24 23:10:30 UTC (rev 181916)
+++ trunk/Source/WebCore/contentextensions/URLFilterParser.cpp 2015-03-24 23:24:36 UTC (rev 181917)
</span><span class="lines">@@ -244,6 +244,50 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ // Matches any string, the empty string included.
+ // This is very conservative. Patterns matching any string can return false here.
+ bool isKnownToMatchAnyString() const
+ {
+ ASSERT(isValid());
+
+ switch (m_termType) {
+ case TermType::Empty:
+ case TermType::Deleted:
+ ASSERT_NOT_REACHED();
+ break;
+ case TermType::CharacterSet:
+ // ".*" is the only simple term matching any string.
+ return isUniversalTransition() && m_quantifier == AtomQuantifier::ZeroOrMore;
+ break;
+ case TermType::Group: {
+ // There are infinitely many ways to match anything with groups, we just handle simple cases
+ if (m_atomData.group.terms.size() != 1)
+ return false;
+
+ const Term& firstTermInGroup = m_atomData.group.terms.first();
+ // -(.*) with any quantifier.
+ if (firstTermInGroup.isKnownToMatchAnyString())
+ return true;
+
+ if (firstTermInGroup.isUniversalTransition()) {
+ // -(.)*, (.+)*, (.?)* etc.
+ if (m_quantifier == AtomQuantifier::ZeroOrMore)
+ return true;
+
+ // -(.+)?.
+ if (m_quantifier == AtomQuantifier::ZeroOrOne && firstTermInGroup.m_quantifier == AtomQuantifier::OneOrMore)
+ return true;
+
+ // -(.?)+.
+ if (m_quantifier == AtomQuantifier::OneOrMore && firstTermInGroup.m_quantifier == AtomQuantifier::ZeroOrOne)
+ return true;
+ }
+ break;
+ }
+ }
+ return false;
+ }
+
</ins><span class="cx"> Term& operator=(const Term& other)
</span><span class="cx"> {
</span><span class="cx"> destroy();
</span><span class="lines">@@ -308,11 +352,24 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private:
</span><ins>+ // This is exact for character sets but conservative for groups.
+ // The return value can be false for a group equivalent to a universal transition.
</ins><span class="cx"> bool isUniversalTransition() const
</span><span class="cx"> {
</span><del>- return m_termType == TermType::CharacterSet
- && ((m_atomData.characterSet.inverted && !m_atomData.characterSet.characters.bitCount())
- || (!m_atomData.characterSet.inverted && m_atomData.characterSet.characters.bitCount() == 128));
</del><ins>+ ASSERT(isValid());
+
+ switch (m_termType) {
+ case TermType::Empty:
+ case TermType::Deleted:
+ ASSERT_NOT_REACHED();
+ break;
+ case TermType::CharacterSet:
+ return (m_atomData.characterSet.inverted && !m_atomData.characterSet.characters.bitCount())
+ || (!m_atomData.characterSet.inverted && m_atomData.characterSet.characters.bitCount() == 128);
+ case TermType::Group:
+ return m_atomData.group.terms.size() == 1 && m_atomData.group.terms.first().isUniversalTransition();
+ }
+ return false;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> unsigned generateSubgraphForAtom(NFA& nfa, uint64_t patternId, unsigned source) const
</span><span class="lines">@@ -459,6 +516,8 @@
</span><span class="cx">
</span><span class="cx"> sinkFloatingTermIfNecessary();
</span><span class="cx">
</span><ins>+ simplifySunkTerms();
+
</ins><span class="cx"> // Check to see if there are any terms without ? or *.
</span><span class="cx"> bool matchesEverything = true;
</span><span class="cx"> for (const auto& term : m_sunkTerms) {
</span><span class="lines">@@ -467,8 +526,10 @@
</span><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> }
</span><del>- if (matchesEverything)
</del><ins>+ if (matchesEverything) {
</ins><span class="cx"> fail(URLFilterParser::MatchesEverything);
</span><ins>+ return;
+ }
</ins><span class="cx">
</span><span class="cx"> for (const auto& term : m_sunkTerms) {
</span><span class="cx"> ASSERT(m_lastPrefixTreeEntry);
</span><span class="lines">@@ -484,12 +545,7 @@
</span><span class="cx">
</span><span class="cx"> auto addResult = m_lastPrefixTreeEntry->nextPattern.set(term, WTF::move(nextPrefixTreeEntry));
</span><span class="cx"> ASSERT(addResult.isNewEntry);
</span><del>-
- if (!m_newPrefixSubtreeRoot) {
- m_newPrefixSubtreeRoot = m_lastPrefixTreeEntry;
- m_newPrefixStaringPoint = term;
- }
-
</del><ins>+
</ins><span class="cx"> m_lastPrefixTreeEntry = addResult.iterator->value.get();
</span><span class="cx"> }
</span><span class="cx"> m_subtreeEnd = m_lastPrefixTreeEntry->nfaNode;
</span><span class="lines">@@ -564,8 +620,12 @@
</span><span class="cx"> if (hasError())
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- if (m_subtreeStart != m_subtreeEnd || m_floatingTerm.isValid() || !m_openGroups.isEmpty())
</del><ins>+ if (m_subtreeStart != m_subtreeEnd || m_floatingTerm.isValid() || !m_openGroups.isEmpty()) {
</ins><span class="cx"> fail(URLFilterParser::MisplacedStartOfLine);
</span><ins>+ return;
+ }
+
+ m_hasBeginningOfLineAssertion = true;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void assertionEOL()
</span><span class="lines">@@ -671,9 +731,6 @@
</span><span class="cx"> if (hasError())
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- if (m_newPrefixSubtreeRoot)
- m_newPrefixSubtreeRoot->nextPattern.remove(m_newPrefixStaringPoint);
-
</del><span class="cx"> m_parseStatus = reason;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -701,6 +758,50 @@
</span><span class="cx"> m_floatingTerm = Term();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ void simplifySunkTerms()
+ {
+ ASSERT(!m_floatingTerm.isValid());
+
+ if (m_sunkTerms.isEmpty())
+ return;
+
+ Term canonicalDotStar(Term::UniversalTransition);
+ canonicalDotStar.quantify(AtomQuantifier::ZeroOrMore);
+
+ // Replace every ".*"-like terms by our canonical version. Remove any duplicate ".*".
+ {
+ unsigned termIndex = 0;
+ bool isAfterDotStar = false;
+ while (termIndex < m_sunkTerms.size()) {
+ if (isAfterDotStar && m_sunkTerms[termIndex].isKnownToMatchAnyString()) {
+ m_sunkTerms.remove(termIndex);
+ continue;
+ }
+ isAfterDotStar = false;
+
+ if (m_sunkTerms[termIndex].isKnownToMatchAnyString()) {
+ m_sunkTerms[termIndex] = canonicalDotStar;
+ isAfterDotStar = true;
+ }
+ ++termIndex;
+ }
+ }
+
+ // Add our ".*" in front if needed.
+ if (!m_hasBeginningOfLineAssertion && !m_sunkTerms.first().isKnownToMatchAnyString())
+ m_sunkTerms.insert(0, canonicalDotStar);
+
+ // Remove trailing ".*$".
+ if (m_sunkTerms.size() > 2 && m_sunkTerms.last().isEndOfLineAssertion() && m_sunkTerms[m_sunkTerms.size() - 2].isKnownToMatchAnyString())
+ m_sunkTerms.shrink(m_sunkTerms.size() - 2);
+
+ // Remove irrelevant terms that can match empty. For example in "foob?", matching "b" is irrelevant.
+ if (m_sunkTerms.last().isEndOfLineAssertion())
+ return;
+ while (!m_sunkTerms.isEmpty() && !m_sunkTerms.last().matchesAtLeastOneCharacter())
+ m_sunkTerms.removeLast();
+ }
+
</ins><span class="cx"> NFA& m_nfa;
</span><span class="cx"> bool m_patternIsCaseSensitive;
</span><span class="cx"> const uint64_t m_patternId;
</span><span class="lines">@@ -712,9 +813,9 @@
</span><span class="cx"> Deque<Term> m_openGroups;
</span><span class="cx"> Vector<Term> m_sunkTerms;
</span><span class="cx"> Term m_floatingTerm;
</span><ins>+ bool m_hasBeginningOfLineAssertion { false };
</ins><span class="cx"> bool m_hasProcessedEndOfLineAssertion { false };
</span><span class="cx">
</span><del>- PrefixTreeEntry* m_newPrefixSubtreeRoot = nullptr;
</del><span class="cx"> Term m_newPrefixStaringPoint;
</span><span class="cx">
</span><span class="cx"> URLFilterParser::ParseStatus m_parseStatus;
</span></span></pre></div>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (181916 => 181917)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2015-03-24 23:10:30 UTC (rev 181916)
+++ trunk/Tools/ChangeLog 2015-03-24 23:24:36 UTC (rev 181917)
</span><span class="lines">@@ -1,3 +1,12 @@
</span><ins>+2015-03-24 Benjamin Poulain <bpoulain@apple.com>
+
+ Make URL filter patterns matching consistent and add a simple canonicalization step
+ https://bugs.webkit.org/show_bug.cgi?id=142998
+
+ Reviewed by Alex Christensen.
+
+ * TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp:
+
</ins><span class="cx"> 2015-03-24 Csaba Osztrogonác <ossy@webkit.org>
</span><span class="cx">
</span><span class="cx"> [EFL] Add OpenWebRTC in jhbuild
</span></span></pre></div>
<a id="trunkToolsTestWebKitAPITestsWebCoreContentExtensionscpp"></a>
<div class="modfile"><h4>Modified: trunk/Tools/TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp (181916 => 181917)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp 2015-03-24 23:10:30 UTC (rev 181916)
+++ trunk/Tools/TestWebKitAPI/Tests/WebCore/ContentExtensions.cpp 2015-03-24 23:24:36 UTC (rev 181917)
</span><span class="lines">@@ -115,7 +115,7 @@
</span><span class="cx"> return { URL(URL(), url), URL(URL(), url), resourceType };
</span><span class="cx"> }
</span><span class="cx">
</span><del>-const char* basicFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*webkit.org\"}}]";
</del><ins>+const char* basicFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"webkit.org\"}}]";
</ins><span class="cx">
</span><span class="cx"> TEST_F(ContentExtensionTest, Basic)
</span><span class="cx"> {
</span><span class="lines">@@ -130,7 +130,8 @@
</span><span class="cx">
</span><span class="cx"> TEST_F(ContentExtensionTest, RangeBasic)
</span><span class="cx"> {
</span><del>- const char* rangeBasicFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*w[0-9]c\", \"url-filter-is-case-sensitive\":true}},{\"action\":{\"type\":\"block-cookies\"},\"trigger\":{\"url-filter\":\".*[A-H][a-z]cko\", \"url-filter-is-case-sensitive\":true}}]";
</del><ins>+ const char* rangeBasicFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"w[0-9]c\", \"url-filter-is-case-sensitive\":true}},"
+ "{\"action\":{\"type\":\"block-cookies\"},\"trigger\":{\"url-filter\":\"[A-H][a-z]cko\", \"url-filter-is-case-sensitive\":true}}]";
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(rangeBasicFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -157,7 +158,7 @@
</span><span class="cx"> TEST_F(ContentExtensionTest, RangeExclusionGeneratingUniversalTransition)
</span><span class="cx"> {
</span><span class="cx"> // Transition of the type ([^X]X) effictively transition on every input.
</span><del>- const char* rangeExclusionGeneratingUniversalTransitionFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*[^a]+afoobar\"}}]";
</del><ins>+ const char* rangeExclusionGeneratingUniversalTransitionFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"[^a]+afoobar\"}}]";
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(rangeExclusionGeneratingUniversalTransitionFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -178,10 +179,9 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://w3c.org/AAfoobar"), { });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-const char* patternsStartingWithGroupFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"(http://whatwg\\\\.org/)?webkit\134\134.org\"}}]";
-
</del><span class="cx"> TEST_F(ContentExtensionTest, PatternStartingWithGroup)
</span><span class="cx"> {
</span><ins>+ const char* patternsStartingWithGroupFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^(http://whatwg\\\\.org/)?webkit\134\134.org\"}}]";
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(patternsStartingWithGroupFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -195,10 +195,10 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://whatwg.org"), { });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-const char* patternNestedGroupsFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"http://webkit\\\\.org/(foo(bar)*)+\"}}]";
-
</del><span class="cx"> TEST_F(ContentExtensionTest, PatternNestedGroups)
</span><span class="cx"> {
</span><ins>+ const char* patternNestedGroupsFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^http://webkit\\\\.org/(foo(bar)*)+\"}}]";
+
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(patternNestedGroupsFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -218,10 +218,10 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://webkit.org/fobar"), { });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-const char* matchPastEndOfStringFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".+\"}}]";
-
</del><span class="cx"> TEST_F(ContentExtensionTest, MatchPastEndOfString)
</span><span class="cx"> {
</span><ins>+ const char* matchPastEndOfStringFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".+\"}}]";
+
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(matchPastEndOfStringFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -258,10 +258,9 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://foobar.org/"), { });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-const char* endOfLineAssertionFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*foobar$\"}}]";
-
</del><span class="cx"> TEST_F(ContentExtensionTest, EndOfLineAssertion)
</span><span class="cx"> {
</span><ins>+ const char* endOfLineAssertionFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"foobar$\"}}]";
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(endOfLineAssertionFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -278,7 +277,7 @@
</span><span class="cx">
</span><span class="cx"> TEST_F(ContentExtensionTest, EndOfLineAssertionWithInvertedCharacterSet)
</span><span class="cx"> {
</span><del>- const char* endOfLineAssertionWithInvertedCharacterSetFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*[^y]$\"}}]";
</del><ins>+ const char* endOfLineAssertionWithInvertedCharacterSetFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"[^y]$\"}}]";
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(endOfLineAssertionWithInvertedCharacterSetFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -295,13 +294,183 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://webkit.org/foobary"), { });
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://webkit.org/foobarY"), { });
</span><span class="cx"> }
</span><del>-
-const char* loadTypeFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*webkit.org\",\"load-type\":[\"third-party\"]}},"
- "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*whatwg.org\",\"load-type\":[\"first-party\"]}},"
- "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*alwaysblock.pdf\"}}]";
</del><span class="cx">
</span><ins>+TEST_F(ContentExtensionTest, PrefixInfixSuffixExactMatch)
+{
+ const char* prefixInfixSuffixExactMatchFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"infix\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^prefix\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"suffix$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^http://exact\\\\.org/$\"}}]";
+ auto extensionData = ContentExtensions::compileRuleList(prefixInfixSuffixExactMatchFilter);
+ auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
+
+ ContentExtensions::ContentExtensionsBackend backend;
+ backend.addContentExtension("PrefixInfixSuffixExactMatch", extension);
+
+ testRequest(backend, mainDocumentRequest("infix://webkit.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("http://infix.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/infix"), { ContentExtensions::ActionType::BlockLoad });
+
+ testRequest(backend, mainDocumentRequest("prefix://webkit.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://prefix.org/"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/prefix"), { });
+
+ testRequest(backend, mainDocumentRequest("https://webkit.org/suffix"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://suffix.org/"), { });
+ testRequest(backend, mainDocumentRequest("suffix://webkit.org/"), { });
+
+ testRequest(backend, mainDocumentRequest("http://exact.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("http://exact.org/oops"), { });
+}
+
+TEST_F(ContentExtensionTest, DuplicatedMatchAllTermsInVariousFormat)
+{
+ const char* duplicatedMatchAllTermsInVariousFormatFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*.*(.)*(.*)(.+)*(.?)*infix\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"pre(.?)+(.+)?post\"}}]";
+ auto extensionData = ContentExtensions::compileRuleList(duplicatedMatchAllTermsInVariousFormatFilter);
+ auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
+
+ ContentExtensions::ContentExtensionsBackend backend;
+ backend.addContentExtension("DuplicatedMatchAllTermsInVariousFormat", extension);
+
+ testRequest(backend, mainDocumentRequest("infix://webkit.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("http://infix.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/infix"), { ContentExtensions::ActionType::BlockLoad });
+
+ testRequest(backend, mainDocumentRequest("pre://webkit.org/post"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("http://prepost.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://pre.org/posttail"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://pre.pre/posttail"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://pre.org/posttailpost"), { ContentExtensions::ActionType::BlockLoad });
+
+ testRequest(backend, mainDocumentRequest("https://post.org/pre"), { });
+ testRequest(backend, mainDocumentRequest("https://pre.org/pre"), { });
+ testRequest(backend, mainDocumentRequest("https://post.org/post"), { });
+}
+
+TEST_F(ContentExtensionTest, TermsKnownToMatchAnything)
+{
+ const char* termsKnownToMatchAnythingFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre1.*post1$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre2(.*)post2$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre3(.*)?post3$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre4(.*)+post4$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre5(.*)*post5$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre6(.)*post6$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre7(.+)*post7$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre8(.?)*post8$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre9(.+)?post9$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"^pre0(.?)+post0$\"}}]";
+ auto extensionData = ContentExtensions::compileRuleList(termsKnownToMatchAnythingFilter);
+ auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
+
+ ContentExtensions::ContentExtensionsBackend backend;
+ backend.addContentExtension("TermsKnownToMatchAnything", extension);
+
+ testRequest(backend, mainDocumentRequest("pre1://webkit.org/post1"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre2://webkit.org/post2"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre3://webkit.org/post3"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre4://webkit.org/post4"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre5://webkit.org/post5"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre6://webkit.org/post6"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre7://webkit.org/post7"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre8://webkit.org/post8"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre9://webkit.org/post9"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("pre0://webkit.org/post0"), { ContentExtensions::ActionType::BlockLoad });
+
+ testRequest(backend, mainDocumentRequest("pre1://webkit.org/post2"), { });
+ testRequest(backend, mainDocumentRequest("pre2://webkit.org/post3"), { });
+ testRequest(backend, mainDocumentRequest("pre3://webkit.org/post4"), { });
+ testRequest(backend, mainDocumentRequest("pre4://webkit.org/post5"), { });
+ testRequest(backend, mainDocumentRequest("pre5://webkit.org/post6"), { });
+ testRequest(backend, mainDocumentRequest("pre6://webkit.org/post7"), { });
+ testRequest(backend, mainDocumentRequest("pre7://webkit.org/post8"), { });
+ testRequest(backend, mainDocumentRequest("pre8://webkit.org/post9"), { });
+ testRequest(backend, mainDocumentRequest("pre9://webkit.org/post0"), { });
+ testRequest(backend, mainDocumentRequest("pre0://webkit.org/post1"), { });
+
+ testRequest(backend, mainDocumentRequest("pre0://webkit.org/post1"), { });
+ testRequest(backend, mainDocumentRequest("pre1://webkit.org/post2"), { });
+ testRequest(backend, mainDocumentRequest("pre2://webkit.org/post3"), { });
+ testRequest(backend, mainDocumentRequest("pre3://webkit.org/post4"), { });
+ testRequest(backend, mainDocumentRequest("pre4://webkit.org/post5"), { });
+ testRequest(backend, mainDocumentRequest("pre5://webkit.org/post6"), { });
+ testRequest(backend, mainDocumentRequest("pre6://webkit.org/post7"), { });
+ testRequest(backend, mainDocumentRequest("pre7://webkit.org/post8"), { });
+ testRequest(backend, mainDocumentRequest("pre8://webkit.org/post9"), { });
+ testRequest(backend, mainDocumentRequest("pre9://webkit.org/post0"), { });
+}
+
+TEST_F(ContentExtensionTest, TrailingDotStar)
+{
+ const char* trailingDotStarFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"foo.*$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"bar(.*)$\"}}]";
+ auto extensionData = ContentExtensions::compileRuleList(trailingDotStarFilter);
+ auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
+
+ ContentExtensions::ContentExtensionsBackend backend;
+ backend.addContentExtension("TrailingDotStar", extension);
+
+ testRequest(backend, mainDocumentRequest("https://webkit.org/"), { });
+
+ testRequest(backend, mainDocumentRequest("foo://webkit.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://foo.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.foo/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/foo"), { ContentExtensions::ActionType::BlockLoad });
+
+ testRequest(backend, mainDocumentRequest("bar://webkit.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://bar.org/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.bar/"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bar"), { ContentExtensions::ActionType::BlockLoad });
+}
+
+TEST_F(ContentExtensionTest, TrailingTermsCarryingNoData)
+{
+ const char* trailingTermsCarryingNoDataFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"foob?a?r?\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"bazo(ok)?a?$\"}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"cats*$\"}}]";
+ auto extensionData = ContentExtensions::compileRuleList(trailingTermsCarryingNoDataFilter);
+ auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
+
+ ContentExtensions::ContentExtensionsBackend backend;
+ backend.addContentExtension("TrailingTermsCarryingNoData", extension);
+
+ testRequest(backend, mainDocumentRequest("https://webkit.org/"), { });
+
+ // Anything is fine after foo.
+ testRequest(backend, mainDocumentRequest("https://webkit.org/foo"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/foob"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/fooc"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/fooba"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/foobar"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/foobar-stuff"), { ContentExtensions::ActionType::BlockLoad });
+
+ // Bazooka has to be at the tail without any character not defined by the filter.
+ testRequest(backend, mainDocumentRequest("https://webkit.org/baz"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazo"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazoa"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazob"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazoo"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazook"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazookb"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazooka"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/bazookaa"), { });
+
+ // The pattern must finish with cat, with any number of 's' following it, but no other character.
+ testRequest(backend, mainDocumentRequest("https://cat.org/"), { });
+ testRequest(backend, mainDocumentRequest("https://cats.org/"), { });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/cat"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/cats"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/catss"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/catsss"), { ContentExtensions::ActionType::BlockLoad });
+ testRequest(backend, mainDocumentRequest("https://webkit.org/catso"), { });
+}
+
</ins><span class="cx"> TEST_F(ContentExtensionTest, LoadType)
</span><span class="cx"> {
</span><ins>+ const char* loadTypeFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"webkit.org\",\"load-type\":[\"third-party\"]}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"whatwg.org\",\"load-type\":[\"first-party\"]}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"alwaysblock.pdf\"}}]";
+
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(loadTypeFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -317,12 +486,12 @@
</span><span class="cx"> testRequest(backend, mainDocumentRequest("http://foobar.org/alwaysblock.pdf"), { ContentExtensions::ActionType::BlockLoad });
</span><span class="cx"> testRequest(backend, {URL(URL(), "http://foobar.org/alwaysblock.pdf"), URL(URL(), "http://not_foobar.org/alwaysblock.pdf"), ResourceType::Document}, { ContentExtensions::ActionType::BlockLoad });
</span><span class="cx"> }
</span><del>-
-const char* resourceTypeFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*block_all_types.org\",\"resource-type\":[\"document\",\"image\",\"style-sheet\",\"script\",\"font\",\"raw\",\"svg-document\",\"media\"]}},"
- "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\".*block_only_images\",\"resource-type\":[\"image\"]}}]";
-
</del><ins>+
</ins><span class="cx"> TEST_F(ContentExtensionTest, ResourceType)
</span><span class="cx"> {
</span><ins>+ const char* resourceTypeFilter = "[{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"block_all_types.org\",\"resource-type\":[\"document\",\"image\",\"style-sheet\",\"script\",\"font\",\"raw\",\"svg-document\",\"media\"]}},"
+ "{\"action\":{\"type\":\"block\"},\"trigger\":{\"url-filter\":\"block_only_images\",\"resource-type\":[\"image\"]}}]";
+
</ins><span class="cx"> auto extensionData = ContentExtensions::compileRuleList(resourceTypeFilter);
</span><span class="cx"> auto extension = InMemoryCompiledContentExtension::create(WTF::move(extensionData));
</span><span class="cx">
</span><span class="lines">@@ -426,6 +595,13 @@
</span><span class="cx"> testPatternStatus("([a-z]*)", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
</span><span class="cx"> testPatternStatus("([a-z]?)", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
</span><span class="cx">
</span><ins>+ testPatternStatus("(.)*", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+ testPatternStatus("(.+)*", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+ testPatternStatus("(.?)*", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+ testPatternStatus("(.*)*", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+ testPatternStatus("(.+)?", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+ testPatternStatus("(.?)+", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
+
</ins><span class="cx"> // Nested groups.
</span><span class="cx"> testPatternStatus("((foo)?((.)*)(bar)*)", ContentExtensions::URLFilterParser::ParseStatus::MatchesEverything);
</span><span class="cx"> }
</span></span></pre>
</div>
</div>
</body>
</html>