<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[182935] trunk/Source/WebCore</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/182935">182935</a></dd>
<dt>Author</dt> <dd>achristensen@apple.com</dd>
<dt>Date</dt> <dd>2015-04-16 18:12:51 -0700 (Thu, 16 Apr 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>Use less memory when compiling content extensions
https://bugs.webkit.org/show_bug.cgi?id=143857
Reviewed by Benjamin Poulain.
When compiling a content extension, we convert the rule list into several intermediate forms:
1) A String.
2) A JSValue from JSONParse in loadEncodedRules.
3) A Vector of ContentExtensionRules.
4) A CombinedURLFilters object representing the pieces of the regular expressions from the triggers.
5) A Vector of NFAs.
6) A DFA for each NFA.
7) A Vector of DFABytecode.
Each one of these contains all the information contained in the content extension,
so we do not need to keep them all in memory at the same time like we are doing now.
When we are done with one, we can free that memory to greatly reduce the maximum memory usage while compiling.
The next step will be to reduce the copies of the original JSON String and to generate NFAs one at a time.
* contentextensions/CombinedURLFilters.cpp:
(WebCore::ContentExtensions::CombinedURLFilters::clear):
* contentextensions/CombinedURLFilters.h:
* contentextensions/ContentExtensionCompiler.cpp:
(WebCore::ContentExtensions::compileRuleList):
Clear structures when finished using them.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceWebCoreChangeLog">trunk/Source/WebCore/ChangeLog</a></li>
<li><a href="#trunkSourceWebCorecontentextensionsCombinedURLFilterscpp">trunk/Source/WebCore/contentextensions/CombinedURLFilters.cpp</a></li>
<li><a href="#trunkSourceWebCorecontentextensionsCombinedURLFiltersh">trunk/Source/WebCore/contentextensions/CombinedURLFilters.h</a></li>
<li><a href="#trunkSourceWebCorecontentextensionsContentExtensionCompilercpp">trunk/Source/WebCore/contentextensions/ContentExtensionCompiler.cpp</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceWebCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/ChangeLog (182934 => 182935)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/ChangeLog 2015-04-17 01:03:55 UTC (rev 182934)
+++ trunk/Source/WebCore/ChangeLog 2015-04-17 01:12:51 UTC (rev 182935)
</span><span class="lines">@@ -1,3 +1,32 @@
</span><ins>+2015-04-16 Alex Christensen <achristensen@webkit.org>
+
+ Use less memory when compiling content extensions
+ https://bugs.webkit.org/show_bug.cgi?id=143857
+
+ Reviewed by Benjamin Poulain.
+
+ When compiling a content extension, we convert the rule list into several intermediate forms:
+
+ 1) A String.
+ 2) A JSValue from JSONParse in loadEncodedRules.
+ 3) A Vector of ContentExtensionRules.
+ 4) A CombinedURLFilters object representing the pieces of the regular expressions from the triggers.
+ 5) A Vector of NFAs.
+ 6) A DFA for each NFA.
+ 7) A Vector of DFABytecode.
+
+ Each one of these contains all the information contained in the content extension,
+ so we do not need to keep them all in memory at the same time like we are doing now.
+ When we are done with one, we can free that memory to greatly reduce the maximum memory usage while compiling.
+ The next step will be to reduce the copies of the original JSON String and to generate NFAs one at a time.
+
+ * contentextensions/CombinedURLFilters.cpp:
+ (WebCore::ContentExtensions::CombinedURLFilters::clear):
+ * contentextensions/CombinedURLFilters.h:
+ * contentextensions/ContentExtensionCompiler.cpp:
+ (WebCore::ContentExtensions::compileRuleList):
+ Clear structures when finished using them.
+
</ins><span class="cx"> 2015-04-16 Brady Eidson <beidson@apple.com>
</span><span class="cx">
</span><span class="cx"> Compiling a content extension fails when user's home directory is on a different volume from /var/tmp.
</span></span></pre></div>
<a id="trunkSourceWebCorecontentextensionsCombinedURLFilterscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/contentextensions/CombinedURLFilters.cpp (182934 => 182935)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/contentextensions/CombinedURLFilters.cpp 2015-04-17 01:03:55 UTC (rev 182934)
+++ trunk/Source/WebCore/contentextensions/CombinedURLFilters.cpp 2015-04-17 01:12:51 UTC (rev 182935)
</span><span class="lines">@@ -52,6 +52,11 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void CombinedURLFilters::clear()
+{
+ m_prefixTreeRoot = std::make_unique<PrefixTreeVertex>();
+}
+
</ins><span class="cx"> void CombinedURLFilters::addPattern(uint64_t actionId, const Vector<Term>& pattern)
</span><span class="cx"> {
</span><span class="cx"> ASSERT_WITH_MESSAGE(!pattern.isEmpty(), "The parser should have excluded empty patterns before reaching CombinedURLFilters.");
</span></span></pre></div>
<a id="trunkSourceWebCorecontentextensionsCombinedURLFiltersh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/contentextensions/CombinedURLFilters.h (182934 => 182935)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/contentextensions/CombinedURLFilters.h 2015-04-17 01:03:55 UTC (rev 182934)
+++ trunk/Source/WebCore/contentextensions/CombinedURLFilters.h 2015-04-17 01:12:51 UTC (rev 182935)
</span><span class="lines">@@ -45,6 +45,7 @@
</span><span class="cx"> void addPattern(uint64_t patternId, const Vector<Term>& pattern);
</span><span class="cx">
</span><span class="cx"> Vector<NFA> createNFAs() const;
</span><ins>+ void clear();
</ins><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> std::unique_ptr<PrefixTreeVertex> m_prefixTreeRoot;
</span></span></pre></div>
<a id="trunkSourceWebCorecontentextensionsContentExtensionCompilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/contentextensions/ContentExtensionCompiler.cpp (182934 => 182935)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/contentextensions/ContentExtensionCompiler.cpp 2015-04-17 01:03:55 UTC (rev 182934)
+++ trunk/Source/WebCore/contentextensions/ContentExtensionCompiler.cpp 2015-04-17 01:12:51 UTC (rev 182935)
</span><span class="lines">@@ -133,6 +133,9 @@
</span><span class="cx">
</span><span class="cx"> Vector<SerializedActionByte> actions;
</span><span class="cx"> Vector<unsigned> actionLocations = serializeActions(parsedRuleList, actions);
</span><ins>+ client.writeActions(WTF::move(actions));
+ actions.clear();
+
</ins><span class="cx"> HashSet<uint64_t, DefaultHash<uint64_t>::Hash, WTF::UnsignedWithZeroKeyHashTraits<uint64_t>> universalActionLocations;
</span><span class="cx">
</span><span class="cx"> CombinedURLFilters combinedURLFilters;
</span><span class="lines">@@ -161,6 +164,8 @@
</span><span class="cx"> if (contentExtensionRule.action().type() == ActionType::IgnorePreviousRules)
</span><span class="cx"> ignorePreviousRulesSeen = true;
</span><span class="cx"> }
</span><ins>+ parsedRuleList.clear();
+ actionLocations.clear();
</ins><span class="cx">
</span><span class="cx"> #if CONTENT_EXTENSIONS_PERFORMANCE_REPORTING
</span><span class="cx"> double patternPartitioningEnd = monotonicallyIncreasingTime();
</span><span class="lines">@@ -172,6 +177,7 @@
</span><span class="cx"> #endif
</span><span class="cx">
</span><span class="cx"> Vector<NFA> nfas = combinedURLFilters.createNFAs();
</span><ins>+ combinedURLFilters.clear();
</ins><span class="cx"> if (!nfas.size() && universalActionLocations.size())
</span><span class="cx"> nfas.append(NFA());
</span><span class="cx">
</span><span class="lines">@@ -226,6 +232,7 @@
</span><span class="cx"> DFABytecodeCompiler compiler(dfa, bytecode);
</span><span class="cx"> compiler.compile();
</span><span class="cx"> }
</span><ins>+ universalActionLocations.clear();
</ins><span class="cx">
</span><span class="cx"> #if CONTENT_EXTENSIONS_PERFORMANCE_REPORTING
</span><span class="cx"> double totalNFAToByteCodeBuildTimeEnd = monotonicallyIncreasingTime();
</span><span class="lines">@@ -233,9 +240,9 @@
</span><span class="cx"> dataLogF(" Bytecode size %zu\n", bytecode.size());
</span><span class="cx"> dataLogF(" DFA count %zu\n", nfas.size());
</span><span class="cx"> #endif
</span><ins>+ nfas.clear();
</ins><span class="cx">
</span><span class="cx"> client.writeBytecode(WTF::move(bytecode));
</span><del>- client.writeActions(WTF::move(actions));
</del><span class="cx">
</span><span class="cx"> return { };
</span><span class="cx"> }
</span></span></pre>
</div>
</div>
</body>
</html>