<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[207179] trunk/Source</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/207179">207179</a></dd>
<dt>Author</dt> <dd>fpizlo@apple.com</dd>
<dt>Date</dt> <dd>2016-10-11 16:52:02 -0700 (Tue, 11 Oct 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>MarkedBlock should know what objects are live during marking
https://bugs.webkit.org/show_bug.cgi?id=162309
Reviewed by Geoffrey Garen.
Source/JavaScriptCore:
It used to be that we would forget which objects are live the moment we started collection.
That's because the flip at the beginning clears all mark bits.
But we already have a facility for tracking objects that are live-but-not-marked. It's called
newlyAllocated. So, instead of clearing mark bits, we want to just transfer them to
newlyAllocated. Then we want to clear all newlyAllocated after GC.
This implements such an approach, along with a versioning optimization for newlyAllocated.
Instead of walking the whole heap to clear newlyAllocated bits at the end of the GC, we bump
the newlyAllocatedVersion, which causes MarkedBlock to treat newlyAllocated as if it was
clear.
We could have even avoided allocating newlyAllocated in most cases, since empirically most
blocks are either completely empty or completely full. An earlier version of this patch did
this, but it was not better than this patch. In fact, it seemed to actually be worse for PLT
and membuster.
To validate this change, we now run the conservative scan after the beginMarking flip. And it
totally works!
This is a huge step towards concurrent GC. It means that we ought to be able to run the
allocator while marking. Since we already separately made it possible to run the barrier
while marking, this means that we're pretty much ready for some serious concurrency action.
This appears to be perf-neutral and space-neutral.
* JavaScriptCore.xcodeproj/project.pbxproj:
* bytecode/CodeBlock.cpp:
* bytecode/CodeBlock.h:
(JSC::CodeBlockSet::mark): Deleted.
* heap/CodeBlockSet.cpp:
(JSC::CodeBlockSet::writeBarrierCurrentlyExecuting):
(JSC::CodeBlockSet::clearCurrentlyExecuting):
(JSC::CodeBlockSet::writeBarrierCurrentlyExecutingCodeBlocks): Deleted.
* heap/CodeBlockSet.h:
* heap/CodeBlockSetInlines.h: Added.
(JSC::CodeBlockSet::mark):
* heap/ConservativeRoots.cpp:
* heap/Heap.cpp:
(JSC::Heap::markRoots):
(JSC::Heap::beginMarking):
(JSC::Heap::collectImpl):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::clearCurrentlyExecutingCodeBlocks):
* heap/Heap.h:
* heap/HeapUtil.h:
(JSC::HeapUtil::findGCObjectPointersForMarking):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::isPagedOut):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::aboutToMarkSlow):
(JSC::MarkedBlock::Handle::resetAllocated):
(JSC::MarkedBlock::resetMarks):
(JSC::MarkedBlock::setNeedsDestruction):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::clearMarks): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::newlyAllocatedVersion):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): Deleted.
(JSC::MarkedBlock::Handle::clearNewlyAllocated): Deleted.
* heap/MarkedBlockInlines.h:
(JSC::MarkedBlock::Handle::cellsPerBlock):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::isNewlyAllocatedStale):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocatedWithSweep):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::heap):
(JSC::MarkedBlock::space):
(JSC::MarkedBlock::Handle::space):
(JSC::MarkedBlock::resetMarkingVersion): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::beginMarking):
(JSC::MarkedSpace::endMarking):
(JSC::MarkedSpace::clearNewlyAllocated): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::nextVersion):
(JSC::MarkedSpace::newlyAllocatedVersion):
(JSC::MarkedSpace::markingVersion): Deleted.
* runtime/SamplingProfiler.cpp:
Source/WTF:
This removes the atomicity mode, because it's not really used: it only affects the
concurrentBlah methods, but their only users turn on atomicity. This was useful because
previously, some binary Bitmap methods (like merge(const Bitmap&)) couldn't be used
effectively in the GC because some of the GC's bitmaps set the atomic mode and some didn't.
Removing this useless mode is the best solution.
Also added some new binary Bitmap methods: mergeAndClear(Bitmap& other) and
setAndClear(Bitmap& other). They perform their action on 'this' (either merge or set,
respectively) while also clearing the contents of 'other'. This is great for one of the GC
hot paths.
* wtf/Bitmap.h:
(WTF::WordType>::Bitmap):
(WTF::WordType>::get):
(WTF::WordType>::set):
(WTF::WordType>::testAndSet):
(WTF::WordType>::testAndClear):
(WTF::WordType>::concurrentTestAndSet):
(WTF::WordType>::concurrentTestAndClear):
(WTF::WordType>::clear):
(WTF::WordType>::clearAll):
(WTF::WordType>::nextPossiblyUnset):
(WTF::WordType>::findRunOfZeros):
(WTF::WordType>::count):
(WTF::WordType>::isEmpty):
(WTF::WordType>::isFull):
(WTF::WordType>::merge):
(WTF::WordType>::filter):
(WTF::WordType>::exclude):
(WTF::WordType>::forEachSetBit):
(WTF::WordType>::mergeAndClear):
(WTF::WordType>::setAndClear):
(WTF::=):
(WTF::WordType>::hash):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreJavaScriptCorexcodeprojprojectpbxproj">trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCodeBlockcpp">trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCodeBlockh">trunk/Source/JavaScriptCore/bytecode/CodeBlock.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapCodeBlockSetcpp">trunk/Source/JavaScriptCore/heap/CodeBlockSet.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapCodeBlockSeth">trunk/Source/JavaScriptCore/heap/CodeBlockSet.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapConservativeRootscpp">trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapcpp">trunk/Source/JavaScriptCore/heap/Heap.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeaph">trunk/Source/JavaScriptCore/heap/Heap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapUtilh">trunk/Source/JavaScriptCore/heap/HeapUtil.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedAllocatorcpp">trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockcpp">trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockh">trunk/Source/JavaScriptCore/heap/MarkedBlock.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockInlinesh">trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpacecpp">trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpaceh">trunk/Source/JavaScriptCore/heap/MarkedSpace.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorcpp">trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeSamplingProfilercpp">trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp</a></li>
<li><a href="#trunkSourceWTFChangeLog">trunk/Source/WTF/ChangeLog</a></li>
<li><a href="#trunkSourceWTFwtfBitmaph">trunk/Source/WTF/wtf/Bitmap.h</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkSourceJavaScriptCoreheapCodeBlockSetInlinesh">trunk/Source/JavaScriptCore/heap/CodeBlockSetInlines.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/ChangeLog 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -1,3 +1,98 @@
</span><ins>+2016-10-06 Filip Pizlo <fpizlo@apple.com>
+
+ MarkedBlock should know what objects are live during marking
+ https://bugs.webkit.org/show_bug.cgi?id=162309
+
+ Reviewed by Geoffrey Garen.
+
+ It used to be that we would forget which objects are live the moment we started collection.
+ That's because the flip at the beginning clears all mark bits.
+
+ But we already have a facility for tracking objects that are live-but-not-marked. It's called
+ newlyAllocated. So, instead of clearing mark bits, we want to just transfer them to
+ newlyAllocated. Then we want to clear all newlyAllocated after GC.
+
+ This implements such an approach, along with a versioning optimization for newlyAllocated.
+ Instead of walking the whole heap to clear newlyAllocated bits at the end of the GC, we bump
+ the newlyAllocatedVersion, which causes MarkedBlock to treat newlyAllocated as if it was
+ clear.
+
+ We could have even avoided allocating newlyAllocated in most cases, since empirically most
+ blocks are either completely empty or completely full. An earlier version of this patch did
+ this, but it was not better than this patch. In fact, it seemed to actually be worse for PLT
+ and membuster.
+
+ To validate this change, we now run the conservative scan after the beginMarking flip. And it
+ totally works!
+
+ This is a huge step towards concurrent GC. It means that we ought to be able to run the
+ allocator while marking. Since we already separately made it possible to run the barrier
+ while marking, this means that we're pretty much ready for some serious concurrency action.
+
+ This appears to be perf-neutral and space-neutral.
+
+ * JavaScriptCore.xcodeproj/project.pbxproj:
+ * bytecode/CodeBlock.cpp:
+ * bytecode/CodeBlock.h:
+ (JSC::CodeBlockSet::mark): Deleted.
+ * heap/CodeBlockSet.cpp:
+ (JSC::CodeBlockSet::writeBarrierCurrentlyExecuting):
+ (JSC::CodeBlockSet::clearCurrentlyExecuting):
+ (JSC::CodeBlockSet::writeBarrierCurrentlyExecutingCodeBlocks): Deleted.
+ * heap/CodeBlockSet.h:
+ * heap/CodeBlockSetInlines.h: Added.
+ (JSC::CodeBlockSet::mark):
+ * heap/ConservativeRoots.cpp:
+ * heap/Heap.cpp:
+ (JSC::Heap::markRoots):
+ (JSC::Heap::beginMarking):
+ (JSC::Heap::collectImpl):
+ (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
+ (JSC::Heap::clearCurrentlyExecutingCodeBlocks):
+ * heap/Heap.h:
+ * heap/HeapUtil.h:
+ (JSC::HeapUtil::findGCObjectPointersForMarking):
+ * heap/MarkedAllocator.cpp:
+ (JSC::MarkedAllocator::isPagedOut):
+ * heap/MarkedBlock.cpp:
+ (JSC::MarkedBlock::Handle::Handle):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated):
+ (JSC::MarkedBlock::Handle::stopAllocating):
+ (JSC::MarkedBlock::Handle::lastChanceToFinalize):
+ (JSC::MarkedBlock::Handle::resumeAllocating):
+ (JSC::MarkedBlock::aboutToMarkSlow):
+ (JSC::MarkedBlock::Handle::resetAllocated):
+ (JSC::MarkedBlock::resetMarks):
+ (JSC::MarkedBlock::setNeedsDestruction):
+ (JSC::MarkedBlock::Handle::didAddToAllocator):
+ (JSC::MarkedBlock::Handle::isLive):
+ (JSC::MarkedBlock::Handle::isLiveCell):
+ (JSC::MarkedBlock::clearMarks): Deleted.
+ * heap/MarkedBlock.h:
+ (JSC::MarkedBlock::Handle::newlyAllocatedVersion):
+ (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): Deleted.
+ (JSC::MarkedBlock::Handle::clearNewlyAllocated): Deleted.
+ * heap/MarkedBlockInlines.h:
+ (JSC::MarkedBlock::Handle::cellsPerBlock):
+ (JSC::MarkedBlock::Handle::isLive):
+ (JSC::MarkedBlock::Handle::isLiveCell):
+ (JSC::MarkedBlock::Handle::isNewlyAllocatedStale):
+ (JSC::MarkedBlock::Handle::hasAnyNewlyAllocatedWithSweep):
+ (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
+ (JSC::MarkedBlock::heap):
+ (JSC::MarkedBlock::space):
+ (JSC::MarkedBlock::Handle::space):
+ (JSC::MarkedBlock::resetMarkingVersion): Deleted.
+ * heap/MarkedSpace.cpp:
+ (JSC::MarkedSpace::beginMarking):
+ (JSC::MarkedSpace::endMarking):
+ (JSC::MarkedSpace::clearNewlyAllocated): Deleted.
+ * heap/MarkedSpace.h:
+ (JSC::MarkedSpace::nextVersion):
+ (JSC::MarkedSpace::newlyAllocatedVersion):
+ (JSC::MarkedSpace::markingVersion): Deleted.
+ * runtime/SamplingProfiler.cpp:
+
</ins><span class="cx"> 2016-10-11 Mark Lam <mark.lam@apple.com>
</span><span class="cx">
</span><span class="cx"> Array.prototype.concat should not modify frozen objects.
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreJavaScriptCorexcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -445,6 +445,7 @@
</span><span class="cx"> 0F64B2791A7957B2006E4E66 /* CallEdge.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F64B2771A7957B2006E4E66 /* CallEdge.cpp */; };
</span><span class="cx"> 0F64B27A1A7957B2006E4E66 /* CallEdge.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F64B2781A7957B2006E4E66 /* CallEdge.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0F64EAF31C4ECD0600621E9B /* AirArgInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F64EAF21C4ECD0600621E9B /* AirArgInlines.h */; };
</span><ins>+ 0F664CE81DA304EF00B00A11 /* CodeBlockSetInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F664CE71DA304ED00B00A11 /* CodeBlockSetInlines.h */; };
</ins><span class="cx"> 0F666EC0183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F666EBE183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0F666EC1183566F900D017F1 /* FullBytecodeLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F666EBF183566F900D017F1 /* FullBytecodeLiveness.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0F666EC61835672B00D017F1 /* DFGAvailability.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F666EC21835672B00D017F1 /* DFGAvailability.cpp */; };
</span><span class="lines">@@ -2709,6 +2710,7 @@
</span><span class="cx"> 0F64B2771A7957B2006E4E66 /* CallEdge.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallEdge.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0F64B2781A7957B2006E4E66 /* CallEdge.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallEdge.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F64EAF21C4ECD0600621E9B /* AirArgInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = AirArgInlines.h; path = b3/air/AirArgInlines.h; sourceTree = "<group>"; };
</span><ins>+ 0F664CE71DA304ED00B00A11 /* CodeBlockSetInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CodeBlockSetInlines.h; sourceTree = "<group>"; };
</ins><span class="cx"> 0F666EBE183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BytecodeLivenessAnalysisInlines.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F666EBF183566F900D017F1 /* FullBytecodeLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FullBytecodeLiveness.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F666EC21835672B00D017F1 /* DFGAvailability.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGAvailability.cpp; path = dfg/DFGAvailability.cpp; sourceTree = "<group>"; };
</span><span class="lines">@@ -5325,6 +5327,7 @@
</span><span class="cx"> 0F1C3DD91BBCE09E00E523E4 /* CellState.h */,
</span><span class="cx"> 0FD8A31117D4326C00CA2C40 /* CodeBlockSet.cpp */,
</span><span class="cx"> 0FD8A31217D4326C00CA2C40 /* CodeBlockSet.h */,
</span><ins>+ 0F664CE71DA304ED00B00A11 /* CodeBlockSetInlines.h */,
</ins><span class="cx"> 146B14DB12EB5B12001BEC1B /* ConservativeRoots.cpp */,
</span><span class="cx"> 149DAAF212EB559D0083B12B /* ConservativeRoots.h */,
</span><span class="cx"> 2A7A58EE1808A4C40020BDF7 /* DeferGC.cpp */,
</span><span class="lines">@@ -7960,6 +7963,7 @@
</span><span class="cx"> BC18C41C0E16F5CD00B34460 /* JSCallbackObjectFunctions.h in Headers */,
</span><span class="cx"> 657CF45919BF6662004ACBF2 /* JSCallee.h in Headers */,
</span><span class="cx"> A7D801A91880D6A80026C39B /* JSCBuiltins.h in Headers */,
</span><ins>+ 0F664CE81DA304EF00B00A11 /* CodeBlockSetInlines.h in Headers */,
</ins><span class="cx"> BC1167DA0E19BCC9008066DD /* JSCell.h in Headers */,
</span><span class="cx"> 0F9749711687ADE400A4FF6A /* JSCellInlines.h in Headers */,
</span><span class="cx"> 0F1DD84A18A945BE0026F3FA /* JSCInlines.h in Headers */,
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCodeBlockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -36,6 +36,7 @@
</span><span class="cx"> #include "BytecodeLivenessAnalysis.h"
</span><span class="cx"> #include "BytecodeUseDef.h"
</span><span class="cx"> #include "CallLinkStatus.h"
</span><ins>+#include "CodeBlockSet.h"
</ins><span class="cx"> #include "DFGCapabilities.h"
</span><span class="cx"> #include "DFGCommon.h"
</span><span class="cx"> #include "DFGDriver.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCodeBlockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CodeBlock.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CodeBlock.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/bytecode/CodeBlock.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -35,7 +35,6 @@
</span><span class="cx"> #include "CallLinkInfo.h"
</span><span class="cx"> #include "CallReturnOffsetToBytecodeOffset.h"
</span><span class="cx"> #include "CodeBlockHash.h"
</span><del>-#include "CodeBlockSet.h"
</del><span class="cx"> #include "CodeOrigin.h"
</span><span class="cx"> #include "CodeType.h"
</span><span class="cx"> #include "CompactJITCodeMap.h"
</span><span class="lines">@@ -76,6 +75,7 @@
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="cx"> class BytecodeLivenessAnalysis;
</span><ins>+class CodeBlockSet;
</ins><span class="cx"> class ExecState;
</span><span class="cx"> class JSModuleEnvironment;
</span><span class="cx"> class LLIntOffsetsExtractor;
</span><span class="lines">@@ -1311,38 +1311,6 @@
</span><span class="cx"> m_visitWeaklyHasBeenCalled.store(false, std::memory_order_relaxed);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void CodeBlockSet::mark(const LockHolder& locker, void* candidateCodeBlock)
-{
- ASSERT(m_lock.isLocked());
- // We have to check for 0 and -1 because those are used by the HashMap as markers.
- uintptr_t value = reinterpret_cast<uintptr_t>(candidateCodeBlock);
-
- // This checks for both of those nasty cases in one go.
- // 0 + 1 = 1
- // -1 + 1 = 0
- if (value + 1 <= 1)
- return;
-
- CodeBlock* codeBlock = static_cast<CodeBlock*>(candidateCodeBlock);
- if (!m_oldCodeBlocks.contains(codeBlock) && !m_newCodeBlocks.contains(codeBlock))
- return;
-
- mark(locker, codeBlock);
-}
-
-inline void CodeBlockSet::mark(const LockHolder&, CodeBlock* codeBlock)
-{
- if (!codeBlock)
- return;
-
- // Try to recover gracefully if we forget to execute a barrier for a
- // CodeBlock that does value profiling. This is probably overkill, but we
- // have always done it.
- Heap::heap(codeBlock)->writeBarrier(codeBlock);
-
- m_currentlyExecuting.add(codeBlock);
-}
-
</del><span class="cx"> template <typename Functor> inline void ScriptExecutable::forEachCodeBlock(Functor&& functor)
</span><span class="cx"> {
</span><span class="cx"> switch (type()) {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCodeBlockSetcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/CodeBlockSet.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CodeBlockSet.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/CodeBlockSet.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -107,7 +107,7 @@
</span><span class="cx"> return m_oldCodeBlocks.contains(codeBlock) || m_newCodeBlocks.contains(codeBlock) || m_currentlyExecuting.contains(codeBlock);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void CodeBlockSet::writeBarrierCurrentlyExecutingCodeBlocks(Heap* heap)
</del><ins>+void CodeBlockSet::writeBarrierCurrentlyExecuting(Heap* heap)
</ins><span class="cx"> {
</span><span class="cx"> LockHolder locker(&m_lock);
</span><span class="cx"> if (verbose)
</span><span class="lines">@@ -114,9 +114,10 @@
</span><span class="cx"> dataLog("Remembering ", m_currentlyExecuting.size(), " code blocks.\n");
</span><span class="cx"> for (CodeBlock* codeBlock : m_currentlyExecuting)
</span><span class="cx"> heap->writeBarrier(codeBlock);
</span><ins>+}
</ins><span class="cx">
</span><del>- // It's safe to clear this set because we won't delete the CodeBlocks
- // in it until the next GC, and we'll recompute it at that time.
</del><ins>+void CodeBlockSet::clearCurrentlyExecuting()
+{
</ins><span class="cx"> m_currentlyExecuting.clear();
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCodeBlockSeth"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/CodeBlockSet.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CodeBlockSet.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/CodeBlockSet.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -71,8 +71,10 @@
</span><span class="cx">
</span><span class="cx"> // Add all currently executing CodeBlocks to the remembered set to be
</span><span class="cx"> // re-scanned during the next collection.
</span><del>- void writeBarrierCurrentlyExecutingCodeBlocks(Heap*);
</del><ins>+ void writeBarrierCurrentlyExecuting(Heap*);
</ins><span class="cx">
</span><ins>+ void clearCurrentlyExecuting();
+
</ins><span class="cx"> bool contains(const LockHolder&, void* candidateCodeBlock);
</span><span class="cx"> Lock& getLock() { return m_lock; }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCodeBlockSetInlinesh"></a>
<div class="addfile"><h4>Added: trunk/Source/JavaScriptCore/heap/CodeBlockSetInlines.h (0 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CodeBlockSetInlines.h (rev 0)
+++ trunk/Source/JavaScriptCore/heap/CodeBlockSetInlines.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -0,0 +1,64 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of Apple Inc. ("Apple") nor the names of
+ * its contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE AND ITS CONTRIBUTORS "AS IS" AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL APPLE OR ITS CONTRIBUTORS BE LIABLE FOR ANY
+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+ * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "CodeBlock.h"
+#include "CodeBlockSet.h"
+
+namespace JSC {
+
+inline void CodeBlockSet::mark(const LockHolder& locker, void* candidateCodeBlock)
+{
+ ASSERT(m_lock.isLocked());
+ // We have to check for 0 and -1 because those are used by the HashMap as markers.
+ uintptr_t value = reinterpret_cast<uintptr_t>(candidateCodeBlock);
+
+ // This checks for both of those nasty cases in one go.
+ // 0 + 1 = 1
+ // -1 + 1 = 0
+ if (value + 1 <= 1)
+ return;
+
+ CodeBlock* codeBlock = static_cast<CodeBlock*>(candidateCodeBlock);
+ if (!m_oldCodeBlocks.contains(codeBlock) && !m_newCodeBlocks.contains(codeBlock))
+ return;
+
+ mark(locker, codeBlock);
+}
+
+inline void CodeBlockSet::mark(const LockHolder&, CodeBlock* codeBlock)
+{
+ if (!codeBlock)
+ return;
+
+ m_currentlyExecuting.add(codeBlock);
+}
+
+} // namespace JSC
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapConservativeRootscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -27,7 +27,7 @@
</span><span class="cx"> #include "ConservativeRoots.h"
</span><span class="cx">
</span><span class="cx"> #include "CodeBlock.h"
</span><del>-#include "CodeBlockSet.h"
</del><ins>+#include "CodeBlockSetInlines.h"
</ins><span class="cx"> #include "HeapInlines.h"
</span><span class="cx"> #include "HeapUtil.h"
</span><span class="cx"> #include "JITStubRoutineSet.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -22,6 +22,7 @@
</span><span class="cx"> #include "Heap.h"
</span><span class="cx">
</span><span class="cx"> #include "CodeBlock.h"
</span><ins>+#include "CodeBlockSet.h"
</ins><span class="cx"> #include "ConservativeRoots.h"
</span><span class="cx"> #include "DFGWorklist.h"
</span><span class="cx"> #include "EdenGCActivityCallback.h"
</span><span class="lines">@@ -388,18 +389,8 @@
</span><span class="cx">
</span><span class="cx"> HeapRootVisitor heapRootVisitor(m_slotVisitor);
</span><span class="cx">
</span><del>- ConservativeRoots conservativeRoots(*this);
</del><span class="cx"> {
</span><span class="cx"> TimingScope preConvergenceTimingScope(*this, "Heap::markRoots before convergence");
</span><del>- // We gather conservative roots before clearing mark bits because conservative
- // gathering uses the mark bits to determine whether a reference is valid.
- {
- TimingScope preConvergenceTimingScope(*this, "Heap::markRoots conservative scan");
- SuperSamplerScope superSamplerScope(false);
- gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
- gatherJSStackRoots(conservativeRoots);
- gatherScratchBufferRoots(conservativeRoots);
- }
</del><span class="cx">
</span><span class="cx"> #if ENABLE(DFG_JIT)
</span><span class="cx"> DFG::rememberCodeBlocks(*m_vm);
</span><span class="lines">@@ -463,9 +454,24 @@
</span><span class="cx"> ParallelModeEnabler enabler(m_slotVisitor);
</span><span class="cx">
</span><span class="cx"> m_slotVisitor.donateAndDrain();
</span><ins>+
+ {
+ TimingScope preConvergenceTimingScope(*this, "Heap::markRoots conservative scan");
+ ConservativeRoots conservativeRoots(*this);
+ SuperSamplerScope superSamplerScope(false);
+ gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
+ gatherJSStackRoots(conservativeRoots);
+ gatherScratchBufferRoots(conservativeRoots);
+ visitConservativeRoots(conservativeRoots);
+
+ // We want to do this to conservatively ensure that we rescan any code blocks that are
+ // running right now. However, we need to be sure to do it *after* we mark the code block
+ // so that we know for sure if it really needs a barrier.
+ m_codeBlocks->writeBarrierCurrentlyExecuting(this);
+ }
+
</ins><span class="cx"> visitExternalRememberedSet();
</span><span class="cx"> visitSmallStrings();
</span><del>- visitConservativeRoots(conservativeRoots);
</del><span class="cx"> visitProtectedObjects(heapRootVisitor);
</span><span class="cx"> visitArgumentBuffers(heapRootVisitor);
</span><span class="cx"> visitException(heapRootVisitor);
</span><span class="lines">@@ -524,11 +530,6 @@
</span><span class="cx"> m_codeBlocks->clearMarksForFullCollection();
</span><span class="cx">
</span><span class="cx"> {
</span><del>- TimingScope clearNewlyAllocatedTimingScope(*this, "m_objectSpace.clearNewlyAllocated");
- m_objectSpace.clearNewlyAllocated();
- }
-
- {
</del><span class="cx"> TimingScope clearMarksTimingScope(*this, "m_objectSpace.beginMarking");
</span><span class="cx"> m_objectSpace.beginMarking();
</span><span class="cx"> }
</span><span class="lines">@@ -1059,7 +1060,8 @@
</span><span class="cx"> deleteSourceProviderCaches();
</span><span class="cx">
</span><span class="cx"> notifyIncrementalSweeper();
</span><del>- writeBarrierCurrentlyExecutingCodeBlocks();
</del><ins>+ m_codeBlocks->writeBarrierCurrentlyExecuting(this);
+ m_codeBlocks->clearCurrentlyExecuting();
</ins><span class="cx">
</span><span class="cx"> prepareForAllocation();
</span><span class="cx"> updateAllocationLimits();
</span><span class="lines">@@ -1201,11 +1203,6 @@
</span><span class="cx"> m_sweeper->startSweeping();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::writeBarrierCurrentlyExecutingCodeBlocks()
-{
- m_codeBlocks->writeBarrierCurrentlyExecutingCodeBlocks(this);
-}
-
</del><span class="cx"> void Heap::prepareForAllocation()
</span><span class="cx"> {
</span><span class="cx"> m_objectSpace.prepareForAllocation();
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/Heap.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -349,7 +349,6 @@
</span><span class="cx"> void snapshotUnswept();
</span><span class="cx"> void deleteSourceProviderCaches();
</span><span class="cx"> void notifyIncrementalSweeper();
</span><del>- void writeBarrierCurrentlyExecutingCodeBlocks();
</del><span class="cx"> void prepareForAllocation();
</span><span class="cx"> void harvestWeakReferences();
</span><span class="cx"> void finalizeUnconditionalFinalizers();
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapUtilh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/HeapUtil.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/HeapUtil.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/HeapUtil.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -51,6 +51,9 @@
</span><span class="cx"> {
</span><span class="cx"> const HashSet<MarkedBlock*>& set = heap.objectSpace().blocks().set();
</span><span class="cx">
</span><ins>+ ASSERT(heap.objectSpace().isMarking());
+ static const bool isMarking = true;
+
</ins><span class="cx"> char* pointer = static_cast<char*>(passedPointer);
</span><span class="cx">
</span><span class="cx"> // It could point to a large allocation.
</span><span class="lines">@@ -85,7 +88,7 @@
</span><span class="cx"> && set.contains(previousCandidate)
</span><span class="cx"> && previousCandidate->handle().cellKind() == HeapCell::Auxiliary) {
</span><span class="cx"> previousPointer = static_cast<char*>(previousCandidate->handle().cellAlign(previousPointer));
</span><del>- if (previousCandidate->handle().isLiveCell(markingVersion, previousPointer))
</del><ins>+ if (previousCandidate->handle().isLiveCell(markingVersion, isMarking, previousPointer))
</ins><span class="cx"> func(previousPointer);
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="lines">@@ -99,7 +102,7 @@
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> auto tryPointer = [&] (void* pointer) {
</span><del>- if (candidate->handle().isLiveCell(markingVersion, pointer))
</del><ins>+ if (candidate->handle().isLiveCell(markingVersion, isMarking, pointer))
</ins><span class="cx"> func(pointer);
</span><span class="cx"> };
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedAllocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -53,11 +53,8 @@
</span><span class="cx"> unsigned itersSinceLastTimeCheck = 0;
</span><span class="cx"> for (size_t index = 0; index < m_blocks.size(); ++index) {
</span><span class="cx"> MarkedBlock::Handle* block = m_blocks[index];
</span><del>- if (block) {
- // Forces us to touch the memory of the block, but has no semantic effect.
- if (block->areMarksStale())
- block->block().resetMarkingVersion();
- }
</del><ins>+ if (block)
+ block->block().updateNeedsDestruction();
</ins><span class="cx"> ++itersSinceLastTimeCheck;
</span><span class="cx"> if (itersSinceLastTimeCheck >= Heap::s_timeCheckResolution) {
</span><span class="cx"> double currentTime = WTF::monotonicallyIncreasingTime();
</span><span class="lines">@@ -436,7 +433,8 @@
</span><span class="cx"> {
</span><span class="cx"> m_unswept.forEachSetBit(
</span><span class="cx"> [&] (size_t index) {
</span><del>- m_blocks[index]->sweep();
</del><ins>+ MarkedBlock::Handle* block = m_blocks[index];
+ block->sweep();
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -54,6 +54,7 @@
</span><span class="cx">
</span><span class="cx"> MarkedBlock::Handle::Handle(Heap& heap, void* blockSpace)
</span><span class="cx"> : m_weakSet(heap.vm(), CellContainer())
</span><ins>+ , m_newlyAllocatedVersion(MarkedSpace::nullVersion)
</ins><span class="cx"> {
</span><span class="cx"> m_block = new (NotNull, blockSpace) MarkedBlock(*heap.vm(), *this);
</span><span class="cx">
</span><span class="lines">@@ -125,7 +126,7 @@
</span><span class="cx"> for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
</span><span class="cx"> if (emptyMode == NotEmpty
</span><span class="cx"> && ((marksMode == MarksNotStale && block.m_marks.get(i))
</span><del>- || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated->get(i)))) {
</del><ins>+ || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated.get(i)))) {
</ins><span class="cx"> isEmpty = false;
</span><span class="cx"> continue;
</span><span class="cx"> }
</span><span class="lines">@@ -148,7 +149,7 @@
</span><span class="cx"> // We only want to discard the newlyAllocated bits if we're creating a FreeList,
</span><span class="cx"> // otherwise we would lose information on what's currently alive.
</span><span class="cx"> if (sweepMode == SweepToFreeList && newlyAllocatedMode == HasNewlyAllocated)
</span><del>- m_newlyAllocated = nullptr;
</del><ins>+ m_newlyAllocatedVersion = MarkedSpace::nullVersion;
</ins><span class="cx">
</span><span class="cx"> FreeList result = FreeList::list(head, count * cellSize());
</span><span class="cx"> if (sweepMode == SweepToFreeList)
</span><span class="lines">@@ -207,7 +208,7 @@
</span><span class="cx"> template<MarkedBlock::Handle::EmptyMode emptyMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</span><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated(SweepMode sweepMode)
</span><span class="cx"> {
</span><del>- if (m_newlyAllocated)
</del><ins>+ if (hasAnyNewlyAllocated())
</ins><span class="cx"> return sweepHelperSelectSweepMode<emptyMode, destructionMode, scribbleMode, HasNewlyAllocated>(sweepMode);
</span><span class="cx"> return sweepHelperSelectSweepMode<emptyMode, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>(sweepMode);
</span><span class="cx"> }
</span><span class="lines">@@ -275,8 +276,8 @@
</span><span class="cx"> // allocated from our free list are not currently marked, so we need another
</span><span class="cx"> // way to tell what's live vs dead.
</span><span class="cx">
</span><del>- ASSERT(!m_newlyAllocated);
- m_newlyAllocated = std::make_unique<WTF::Bitmap<atomsPerBlock>>();
</del><ins>+ m_newlyAllocated.clearAll();
+ m_newlyAllocatedVersion = heap()->objectSpace().newlyAllocatedVersion();
</ins><span class="cx">
</span><span class="cx"> SetNewlyAllocatedFunctor functor(this);
</span><span class="cx"> forEachCell(functor);
</span><span class="lines">@@ -295,10 +296,12 @@
</span><span class="cx"> void MarkedBlock::Handle::lastChanceToFinalize()
</span><span class="cx"> {
</span><span class="cx"> allocator()->setIsAllocated(this, false);
</span><del>- m_block->clearMarks();
</del><ins>+ m_block->m_marks.clearAll();
+ m_block->clearHasAnyMarked();
+ m_block->m_markingVersion = heap()->objectSpace().markingVersion();
</ins><span class="cx"> m_weakSet.lastChanceToFinalize();
</span><del>-
- clearNewlyAllocated();
</del><ins>+ m_newlyAllocated.clearAll();
+ m_newlyAllocatedVersion = heap()->objectSpace().newlyAllocatedVersion();
</ins><span class="cx"> sweep();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -307,7 +310,7 @@
</span><span class="cx"> ASSERT(!allocator()->isAllocated(this));
</span><span class="cx"> ASSERT(!isFreeListed());
</span><span class="cx">
</span><del>- if (!m_newlyAllocated) {
</del><ins>+ if (!hasAnyNewlyAllocated()) {
</ins><span class="cx"> // This means we had already exhausted the block when we stopped allocation.
</span><span class="cx"> return FreeList();
</span><span class="cx"> }
</span><span class="lines">@@ -346,24 +349,56 @@
</span><span class="cx"> {
</span><span class="cx"> ASSERT(vm()->heap.objectSpace().isMarking());
</span><span class="cx"> LockHolder locker(m_lock);
</span><del>- if (areMarksStale(markingVersion)) {
- clearMarks(markingVersion);
- // This means we're the first ones to mark any object in this block.
- handle().allocator()->atomicSetAndCheckIsMarkingNotEmpty(&handle(), true);
</del><ins>+ if (!areMarksStale(markingVersion))
+ return;
+
+ if (handle().allocator()->isAllocated(&handle())
+ || !marksConveyLivenessDuringMarking(markingVersion)) {
+ // We already know that the block is full and is already recognized as such, or that the
+ // block did not survive the previous GC. So, we can clear mark bits the old fashioned
+ // way. Note that it's possible for such a block to have newlyAllocated with an up-to-
+ // date version! If it does, then we want to leave the newlyAllocated alone, since that
+ // means that we had allocated in this previously empty block but did not fill it up, so
+ // we created a newlyAllocated.
+ m_marks.clearAll();
+ } else {
+ HeapVersion newlyAllocatedVersion = space()->newlyAllocatedVersion();
+ if (handle().m_newlyAllocatedVersion == newlyAllocatedVersion) {
+ // Merge the contents of marked into newlyAllocated. If we get the full set of bits
+ // then invalidate newlyAllocated and set allocated.
+ handle().m_newlyAllocated.mergeAndClear(m_marks);
+ } else {
+ // Replace the contents of newlyAllocated with marked. If we get the full set of
+ // bits then invalidate newlyAllocated and set allocated.
+ handle().m_newlyAllocated.setAndClear(m_marks);
+ }
+ handle().m_newlyAllocatedVersion = newlyAllocatedVersion;
</ins><span class="cx"> }
</span><ins>+ clearHasAnyMarked();
+ WTF::storeStoreFence();
+ m_markingVersion = markingVersion;
+
+ // This means we're the first ones to mark any object in this block.
+ handle().allocator()->atomicSetAndCheckIsMarkingNotEmpty(&handle(), true);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::clearMarks()
</del><ins>+void MarkedBlock::Handle::resetAllocated()
</ins><span class="cx"> {
</span><del>- clearMarks(vm()->heap.objectSpace().markingVersion());
</del><ins>+ m_newlyAllocated.clearAll();
+ m_newlyAllocatedVersion = MarkedSpace::nullVersion;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::clearMarks(HeapVersion markingVersion)
</del><ins>+void MarkedBlock::resetMarks()
</ins><span class="cx"> {
</span><del>- m_marks.clearAll();
- clearHasAnyMarked();
- WTF::storeStoreFence();
- m_markingVersion = markingVersion;
</del><ins>+ // We want aboutToMarkSlow() to see what the mark bits were after the last collection. It uses
+ // the version number to distinguish between the marks having already been stale before
+ // beginMarking(), or just stale now that beginMarking() bumped the version. If we have a version
+ // wraparound, then we will call this method before resetting the version to null. When the
+ // version is null, aboutToMarkSlow() will assume that the marks were not stale as of before
+ // beginMarking(). Hence the need to whip the marks into shape.
+ if (areMarksStale())
+ m_marks.clearAll();
+ m_markingVersion = MarkedSpace::nullVersion;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #if !ASSERT_DISABLED
</span><span class="lines">@@ -425,6 +460,11 @@
</span><span class="cx"> m_allocator->removeBlock(this);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedBlock::updateNeedsDestruction()
+{
+ m_needsDestruction = handle().needsDestruction();
+}
+
</ins><span class="cx"> void MarkedBlock::Handle::didAddToAllocator(MarkedAllocator* allocator, size_t index)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(m_index == std::numeric_limits<size_t>::max());
</span><span class="lines">@@ -442,10 +482,9 @@
</span><span class="cx"> if (m_attributes.cellKind != HeapCell::JSCell)
</span><span class="cx"> RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
</span><span class="cx">
</span><del>- block().m_needsDestruction = needsDestruction();
</del><ins>+ block().updateNeedsDestruction();
</ins><span class="cx">
</span><del>- unsigned cellsPerBlock = MarkedSpace::blockPayload / cellSize;
- double markCountBias = -(Options::minMarkedBlockUtilization() * cellsPerBlock);
</del><ins>+ double markCountBias = -(Options::minMarkedBlockUtilization() * cellsPerBlock());
</ins><span class="cx">
</span><span class="cx"> // The mark count bias should be comfortably within this range.
</span><span class="cx"> RELEASE_ASSERT(markCountBias > static_cast<double>(std::numeric_limits<int16_t>::min()));
</span><span class="lines">@@ -466,12 +505,12 @@
</span><span class="cx">
</span><span class="cx"> bool MarkedBlock::Handle::isLive(const HeapCell* cell)
</span><span class="cx"> {
</span><del>- return isLive(vm()->heap.objectSpace().markingVersion(), cell);
</del><ins>+ return isLive(space()->markingVersion(), space()->isMarking(), cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool MarkedBlock::Handle::isLiveCell(const void* p)
</span><span class="cx"> {
</span><del>- return isLiveCell(vm()->heap.objectSpace().markingVersion(), p);
</del><ins>+ return isLiveCell(space()->markingVersion(), space()->isMarking(), p);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -40,6 +40,7 @@
</span><span class="cx"> class Heap;
</span><span class="cx"> class JSCell;
</span><span class="cx"> class MarkedAllocator;
</span><ins>+class MarkedSpace;
</ins><span class="cx">
</span><span class="cx"> typedef uintptr_t Bits;
</span><span class="cx"> typedef uint32_t HeapVersion;
</span><span class="lines">@@ -110,6 +111,7 @@
</span><span class="cx">
</span><span class="cx"> MarkedAllocator* allocator() const;
</span><span class="cx"> Heap* heap() const;
</span><ins>+ inline MarkedSpace* space() const;
</ins><span class="cx"> VM* vm() const;
</span><span class="cx"> WeakSet& weakSet();
</span><span class="cx">
</span><span class="lines">@@ -141,11 +143,9 @@
</span><span class="cx"> void stopAllocating(const FreeList&);
</span><span class="cx"> FreeList resumeAllocating(); // Call this if you canonicalized a block for some non-collection related purpose.
</span><span class="cx">
</span><del>- // Returns true if the "newly allocated" bitmap was non-null
- // and was successfully cleared and false otherwise.
- bool clearNewlyAllocated();
-
</del><span class="cx"> size_t cellSize();
</span><ins>+ inline unsigned cellsPerBlock();
+
</ins><span class="cx"> const AllocatorAttributes& attributes() const;
</span><span class="cx"> DestructionMode destruction() const;
</span><span class="cx"> bool needsDestruction() const;
</span><span class="lines">@@ -153,9 +153,9 @@
</span><span class="cx">
</span><span class="cx"> size_t markCount();
</span><span class="cx"> size_t size();
</span><del>-
- inline bool isLive(HeapVersion markingVersion, const HeapCell*);
- inline bool isLiveCell(HeapVersion markingVersion, const void*);
</del><ins>+
+ inline bool isLive(HeapVersion markingVersion, bool isMarking, const HeapCell*);
+ inline bool isLiveCell(HeapVersion markingVersion, bool isMarking, const void*);
</ins><span class="cx">
</span><span class="cx"> bool isLive(const HeapCell*);
</span><span class="cx"> bool isLiveCell(const void*);
</span><span class="lines">@@ -164,8 +164,13 @@
</span><span class="cx"> void setNewlyAllocated(const void*);
</span><span class="cx"> void clearNewlyAllocated(const void*);
</span><span class="cx">
</span><del>- bool hasAnyNewlyAllocated() const { return !!m_newlyAllocated; }
-
</del><ins>+ HeapVersion newlyAllocatedVersion() const { return m_newlyAllocatedVersion; }
+
+ inline bool isNewlyAllocatedStale() const;
+
+ inline bool hasAnyNewlyAllocated();
+ void resetAllocated();
+
</ins><span class="cx"> template <typename Functor> IterationStatus forEachCell(const Functor&);
</span><span class="cx"> template <typename Functor> inline IterationStatus forEachLiveCell(const Functor&);
</span><span class="cx"> template <typename Functor> inline IterationStatus forEachDeadCell(const Functor&);
</span><span class="lines">@@ -185,7 +190,7 @@
</span><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> Handle(Heap&, void*);
</span><del>-
</del><ins>+
</ins><span class="cx"> template<DestructionMode>
</span><span class="cx"> FreeList sweepHelperSelectScribbleMode(SweepMode = SweepOnly);
</span><span class="cx">
</span><span class="lines">@@ -223,7 +228,7 @@
</span><span class="cx"> size_t m_atomsPerCell { std::numeric_limits<size_t>::max() };
</span><span class="cx"> size_t m_endAtom { std::numeric_limits<size_t>::max() }; // This is a fuzzy end. Always test for < m_endAtom.
</span><span class="cx">
</span><del>- std::unique_ptr<WTF::Bitmap<atomsPerBlock>> m_newlyAllocated;
</del><ins>+ WTF::Bitmap<atomsPerBlock> m_newlyAllocated;
</ins><span class="cx">
</span><span class="cx"> AllocatorAttributes m_attributes;
</span><span class="cx"> bool m_isFreeListed { false };
</span><span class="lines">@@ -231,6 +236,8 @@
</span><span class="cx"> MarkedAllocator* m_allocator { nullptr };
</span><span class="cx"> size_t m_index { std::numeric_limits<size_t>::max() };
</span><span class="cx"> WeakSet m_weakSet;
</span><ins>+
+ HeapVersion m_newlyAllocatedVersion;
</ins><span class="cx">
</span><span class="cx"> MarkedBlock* m_block { nullptr };
</span><span class="cx"> };
</span><span class="lines">@@ -240,6 +247,8 @@
</span><span class="cx"> Handle& handle();
</span><span class="cx">
</span><span class="cx"> VM* vm() const;
</span><ins>+ inline Heap* heap() const;
+ inline MarkedSpace* space() const;
</ins><span class="cx">
</span><span class="cx"> static bool isAtomAligned(const void*);
</span><span class="cx"> static MarkedBlock* blockFor(const void*);
</span><span class="lines">@@ -278,8 +287,11 @@
</span><span class="cx">
</span><span class="cx"> bool needsDestruction() const { return m_needsDestruction; }
</span><span class="cx">
</span><del>- inline void resetMarkingVersion();
</del><ins>+ // This is usually a no-op, and we use it as a no-op that touches the page in isPagedOut().
+ void updateNeedsDestruction();
</ins><span class="cx">
</span><ins>+ void resetMarks();
+
</ins><span class="cx"> private:
</span><span class="cx"> static const size_t atomAlignmentMask = atomSize - 1;
</span><span class="cx">
</span><span class="lines">@@ -289,13 +301,13 @@
</span><span class="cx"> Atom* atoms();
</span><span class="cx">
</span><span class="cx"> void aboutToMarkSlow(HeapVersion markingVersion);
</span><del>- void clearMarks();
- void clearMarks(HeapVersion markingVersion);
</del><span class="cx"> void clearHasAnyMarked();
</span><span class="cx">
</span><span class="cx"> void noteMarkedSlow();
</span><ins>+
+ inline bool marksConveyLivenessDuringMarking(HeapVersion markingVersion);
</ins><span class="cx">
</span><del>- WTF::Bitmap<atomsPerBlock, WTF::BitmapAtomic, uint8_t> m_marks;
</del><ins>+ WTF::Bitmap<atomsPerBlock> m_marks;
</ins><span class="cx">
</span><span class="cx"> bool m_needsDestruction;
</span><span class="cx"> Lock m_lock;
</span><span class="lines">@@ -521,28 +533,19 @@
</span><span class="cx">
</span><span class="cx"> inline bool MarkedBlock::Handle::isNewlyAllocated(const void* p)
</span><span class="cx"> {
</span><del>- return m_newlyAllocated->get(m_block->atomNumber(p));
</del><ins>+ return m_newlyAllocated.get(m_block->atomNumber(p));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void MarkedBlock::Handle::setNewlyAllocated(const void* p)
</span><span class="cx"> {
</span><del>- m_newlyAllocated->set(m_block->atomNumber(p));
</del><ins>+ m_newlyAllocated.set(m_block->atomNumber(p));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void MarkedBlock::Handle::clearNewlyAllocated(const void* p)
</span><span class="cx"> {
</span><del>- m_newlyAllocated->clear(m_block->atomNumber(p));
</del><ins>+ m_newlyAllocated.clear(m_block->atomNumber(p));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::Handle::clearNewlyAllocated()
-{
- if (m_newlyAllocated) {
- m_newlyAllocated = nullptr;
- return true;
- }
- return false;
-}
-
</del><span class="cx"> inline bool MarkedBlock::isAtom(const void* p)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(MarkedBlock::isAtomAligned(p));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -27,11 +27,36 @@
</span><span class="cx">
</span><span class="cx"> #include "MarkedAllocator.h"
</span><span class="cx"> #include "MarkedBlock.h"
</span><ins>+#include "MarkedSpace.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><del>-inline bool MarkedBlock::Handle::isLive(HeapVersion markingVersion, const HeapCell* cell)
</del><ins>+inline unsigned MarkedBlock::Handle::cellsPerBlock()
</ins><span class="cx"> {
</span><ins>+ return MarkedSpace::blockPayload / cellSize();
+}
+
+inline bool MarkedBlock::marksConveyLivenessDuringMarking(HeapVersion markingVersion)
+{
+ // This returns true if any of these is true:
+ // - We just created the block and so the bits are clear already.
+ // - This block has objects marked during the last GC, and so its version was up-to-date just
+ // before the current collection did beginMarking(). This means that any objects that have
+ // their mark bit set are valid objects that were never deleted, and so are candidates for
+ // marking in any conservative scan. Using our jargon, they are "live".
+ // - We did ~2^32 collections and rotated the version back to null, so we needed to hard-reset
+ // everything. If the marks had been stale, we would have cleared them. So, we can be sure that
+ // any set mark bit reflects objects marked during last GC, i.e. "live" objects.
+ // It would be absurd to use this method when not collecting, since this special "one version
+ // back" state only makes sense when we're in a concurrent collection and have to be
+ // conservative.
+ ASSERT(space()->isMarking());
+ return m_markingVersion == MarkedSpace::nullVersion
+ || MarkedSpace::nextVersion(m_markingVersion) == markingVersion;
+}
+
+inline bool MarkedBlock::Handle::isLive(HeapVersion markingVersion, bool isMarking, const HeapCell* cell)
+{
</ins><span class="cx"> ASSERT(!isFreeListed());
</span><span class="cx">
</span><span class="cx"> if (UNLIKELY(hasAnyNewlyAllocated())) {
</span><span class="lines">@@ -39,24 +64,38 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- MarkedBlock& block = this->block();
-
</del><span class="cx"> if (allocator()->isAllocated(this))
</span><span class="cx"> return true;
</span><span class="cx">
</span><del>- if (block.areMarksStale(markingVersion))
- return false;
</del><ins>+ MarkedBlock& block = this->block();
+
+ if (block.areMarksStale()) {
+ if (!isMarking)
+ return false;
+ if (!block.marksConveyLivenessDuringMarking(markingVersion))
+ return false;
+ }
</ins><span class="cx">
</span><del>- return block.isMarked(cell);
</del><ins>+ return block.m_marks.get(block.atomNumber(cell));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::Handle::isLiveCell(HeapVersion markingVersion, const void* p)
</del><ins>+inline bool MarkedBlock::Handle::isLiveCell(HeapVersion markingVersion, bool isMarking, const void* p)
</ins><span class="cx"> {
</span><span class="cx"> if (!m_block->isAtom(p))
</span><span class="cx"> return false;
</span><del>- return isLive(markingVersion, static_cast<const HeapCell*>(p));
</del><ins>+ return isLive(markingVersion, isMarking, static_cast<const HeapCell*>(p));
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+inline bool MarkedBlock::Handle::isNewlyAllocatedStale() const
+{
+ return m_newlyAllocatedVersion != space()->newlyAllocatedVersion();
+}
+
+inline bool MarkedBlock::Handle::hasAnyNewlyAllocated()
+{
+ return !isNewlyAllocatedStale();
+}
+
</ins><span class="cx"> template <typename Functor>
</span><span class="cx"> inline IterationStatus MarkedBlock::Handle::forEachLiveCell(const Functor& functor)
</span><span class="cx"> {
</span><span class="lines">@@ -87,10 +126,20 @@
</span><span class="cx"> return IterationStatus::Continue;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void MarkedBlock::resetMarkingVersion()
</del><ins>+inline Heap* MarkedBlock::heap() const
</ins><span class="cx"> {
</span><del>- m_markingVersion = MarkedSpace::nullVersion;
</del><ins>+ return &vm()->heap;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+inline MarkedSpace* MarkedBlock::space() const
+{
+ return &heap()->objectSpace();
+}
+
+inline MarkedSpace* MarkedBlock::Handle::space() const
+{
+ return &heap()->objectSpace();
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpacecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -451,29 +451,6 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedSpace::clearNewlyAllocated()
-{
- forEachAllocator(
- [&] (MarkedAllocator& allocator) -> IterationStatus {
- if (MarkedBlock::Handle* block = allocator.takeLastActiveBlock())
- block->clearNewlyAllocated();
- return IterationStatus::Continue;
- });
-
- for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
- m_largeAllocations[i]->clearNewlyAllocated();
-
- if (!ASSERT_DISABLED) {
- forEachBlock(
- [&] (MarkedBlock::Handle* block) {
- ASSERT_UNUSED(block, !block->clearNewlyAllocated());
- });
-
- for (LargeAllocation* allocation : m_largeAllocations)
- ASSERT_UNUSED(allocation, !allocation->isNewlyAllocated());
- }
-}
-
</del><span class="cx"> void MarkedSpace::beginMarking()
</span><span class="cx"> {
</span><span class="cx"> if (m_heap->operationInProgress() == FullCollection) {
</span><span class="lines">@@ -483,16 +460,15 @@
</span><span class="cx"> return IterationStatus::Continue;
</span><span class="cx"> });
</span><span class="cx">
</span><del>- m_markingVersion = nextVersion(m_markingVersion);
-
- if (UNLIKELY(m_markingVersion == initialVersion)) {
- // Oh no! Version wrap-around! We handle this by setting all block versions to null.
</del><ins>+ if (UNLIKELY(nextVersion(m_markingVersion) == initialVersion)) {
</ins><span class="cx"> forEachBlock(
</span><span class="cx"> [&] (MarkedBlock::Handle* handle) {
</span><del>- handle->block().resetMarkingVersion();
</del><ins>+ handle->block().resetMarks();
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ m_markingVersion = nextVersion(m_markingVersion);
+
</ins><span class="cx"> for (LargeAllocation* allocation : m_largeAllocations)
</span><span class="cx"> allocation->flip();
</span><span class="cx"> }
</span><span class="lines">@@ -511,6 +487,23 @@
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::endMarking()
</span><span class="cx"> {
</span><ins>+ if (UNLIKELY(nextVersion(m_newlyAllocatedVersion) == initialVersion)) {
+ forEachBlock(
+ [&] (MarkedBlock::Handle* handle) {
+ handle->resetAllocated();
+ });
+ }
+
+ m_newlyAllocatedVersion = nextVersion(m_newlyAllocatedVersion);
+
+ for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
+ m_largeAllocations[i]->clearNewlyAllocated();
+
+ if (!ASSERT_DISABLED) {
+ for (LargeAllocation* allocation : m_largeAllocations)
+ ASSERT_UNUSED(allocation, !allocation->isNewlyAllocated());
+ }
+
</ins><span class="cx"> forEachAllocator(
</span><span class="cx"> [&] (MarkedAllocator& allocator) -> IterationStatus {
</span><span class="cx"> allocator.endMarking();
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpaceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedSpace.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpace.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpace.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -65,13 +65,13 @@
</span><span class="cx"> static const size_t numSizeClasses = largeCutoff / sizeStep;
</span><span class="cx">
</span><span class="cx"> static const HeapVersion nullVersion = 0; // The version of freshly allocated blocks.
</span><del>- static const HeapVersion initialVersion = 1; // The version that the heap starts out with.
</del><ins>+ static const HeapVersion initialVersion = 2; // The version that the heap starts out with. Set to make sure that nextVersion(nullVersion) != initialVersion.
</ins><span class="cx">
</span><del>- HeapVersion nextVersion(HeapVersion version)
</del><ins>+ static HeapVersion nextVersion(HeapVersion version)
</ins><span class="cx"> {
</span><span class="cx"> version++;
</span><span class="cx"> if (version == nullVersion)
</span><del>- version++;
</del><ins>+ version = initialVersion;
</ins><span class="cx"> return version;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -171,6 +171,7 @@
</span><span class="cx"> bool isPagedOut(double deadline);
</span><span class="cx">
</span><span class="cx"> HeapVersion markingVersion() const { return m_markingVersion; }
</span><ins>+ HeapVersion newlyAllocatedVersion() const { return m_newlyAllocatedVersion; }
</ins><span class="cx">
</span><span class="cx"> const Vector<LargeAllocation*>& largeAllocations() const { return m_largeAllocations; }
</span><span class="cx"> unsigned largeAllocationsNurseryOffset() const { return m_largeAllocationsNurseryOffset; }
</span><span class="lines">@@ -217,6 +218,7 @@
</span><span class="cx">
</span><span class="cx"> Heap* m_heap;
</span><span class="cx"> HeapVersion m_markingVersion { initialVersion };
</span><ins>+ HeapVersion m_newlyAllocatedVersion { initialVersion };
</ins><span class="cx"> size_t m_capacity;
</span><span class="cx"> bool m_isIterating;
</span><span class="cx"> bool m_isMarking { false };
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -191,8 +191,6 @@
</span><span class="cx"> validate(cell);
</span><span class="cx"> #endif
</span><span class="cx">
</span><del>- //dataLog(" Marking ", RawPointer(cell), "\n");
-
</del><span class="cx"> if (cell->isLargeAllocation())
</span><span class="cx"> setMarkedAndAppendToMarkStack(cell->largeAllocation(), cell);
</span><span class="cx"> else
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeSamplingProfilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx">
</span><span class="cx"> #include "CallFrame.h"
</span><span class="cx"> #include "CodeBlock.h"
</span><ins>+#include "CodeBlockSet.h"
</ins><span class="cx"> #include "Executable.h"
</span><span class="cx"> #include "HeapInlines.h"
</span><span class="cx"> #include "HeapIterationScope.h"
</span></span></pre></div>
<a id="trunkSourceWTFChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/ChangeLog (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/ChangeLog 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/WTF/ChangeLog 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -1,3 +1,45 @@
</span><ins>+2016-10-08 Filip Pizlo <fpizlo@apple.com>
+
+ MarkedBlock should know what objects are live during marking
+ https://bugs.webkit.org/show_bug.cgi?id=162309
+
+ Reviewed by Geoffrey Garen.
+
+ This removes the atomicity mode, because it's not really used: it only affects the
+ concurrentBlah methods, but their only users turn on atomicity. This was useful because
+ previously, some binary Bitmap methods (like merge(const Bitmap&)) couldn't be used
+ effectively in the GC because some of the GC's bitmaps set the atomic mode and some didn't.
+ Removing this useless mode is the best solution.
+
+ Also added some new binary Bitmap methods: mergeAndClear(Bitmap& other) and
+ setAndClear(Bitmap& other). They perform their action on 'this' (either merge or set,
+ respectively) while also clearing the contents of 'other'. This is great for one of the GC
+ hot paths.
+
+ * wtf/Bitmap.h:
+ (WTF::WordType>::Bitmap):
+ (WTF::WordType>::get):
+ (WTF::WordType>::set):
+ (WTF::WordType>::testAndSet):
+ (WTF::WordType>::testAndClear):
+ (WTF::WordType>::concurrentTestAndSet):
+ (WTF::WordType>::concurrentTestAndClear):
+ (WTF::WordType>::clear):
+ (WTF::WordType>::clearAll):
+ (WTF::WordType>::nextPossiblyUnset):
+ (WTF::WordType>::findRunOfZeros):
+ (WTF::WordType>::count):
+ (WTF::WordType>::isEmpty):
+ (WTF::WordType>::isFull):
+ (WTF::WordType>::merge):
+ (WTF::WordType>::filter):
+ (WTF::WordType>::exclude):
+ (WTF::WordType>::forEachSetBit):
+ (WTF::WordType>::mergeAndClear):
+ (WTF::WordType>::setAndClear):
+ (WTF::=):
+ (WTF::WordType>::hash):
+
</ins><span class="cx"> 2016-10-11 Said Abou-Hallawa <sabouhallawa@apple.com>
</span><span class="cx">
</span><span class="cx"> Add SynchronizedFixedQueue class
</span></span></pre></div>
<a id="trunkSourceWTFwtfBitmaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/Bitmap.h (207178 => 207179)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/Bitmap.h 2016-10-11 23:25:38 UTC (rev 207178)
+++ trunk/Source/WTF/wtf/Bitmap.h 2016-10-11 23:52:02 UTC (rev 207179)
</span><span class="lines">@@ -27,16 +27,7 @@
</span><span class="cx">
</span><span class="cx"> namespace WTF {
</span><span class="cx">
</span><del>-enum BitmapAtomicMode {
- // This makes concurrentTestAndSet behave just like testAndSet.
- BitmapNotAtomic,
-
- // This makes concurrentTestAndSet use compareAndSwap, so that it's
- // atomic even when used concurrently.
- BitmapAtomic
-};
-
-template<size_t bitmapSize, BitmapAtomicMode atomicMode = BitmapNotAtomic, typename WordType = uint32_t>
</del><ins>+template<size_t bitmapSize, typename WordType = uint32_t>
</ins><span class="cx"> class Bitmap {
</span><span class="cx"> WTF_MAKE_FAST_ALLOCATED;
</span><span class="cx">
</span><span class="lines">@@ -59,8 +50,8 @@
</span><span class="cx"> size_t nextPossiblyUnset(size_t) const;
</span><span class="cx"> void clear(size_t);
</span><span class="cx"> void clearAll();
</span><del>- int64_t findRunOfZeros(size_t) const;
- size_t count(size_t = 0) const;
</del><ins>+ int64_t findRunOfZeros(size_t runLength) const;
+ size_t count(size_t start = 0) const;
</ins><span class="cx"> size_t isEmpty() const;
</span><span class="cx"> size_t isFull() const;
</span><span class="cx">
</span><span class="lines">@@ -71,6 +62,9 @@
</span><span class="cx"> template<typename Func>
</span><span class="cx"> void forEachSetBit(const Func&) const;
</span><span class="cx">
</span><ins>+ void mergeAndClear(Bitmap&);
+ void setAndClear(Bitmap&);
+
</ins><span class="cx"> bool operator==(const Bitmap&) const;
</span><span class="cx"> bool operator!=(const Bitmap&) const;
</span><span class="cx">
</span><span class="lines">@@ -90,26 +84,26 @@
</span><span class="cx"> std::array<WordType, words> bits;
</span><span class="cx"> };
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline Bitmap<bitmapSize, atomicMode, WordType>::Bitmap()
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline Bitmap<bitmapSize, WordType>::Bitmap()
</ins><span class="cx"> {
</span><span class="cx"> clearAll();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::get(size_t n) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::get(size_t n) const
</ins><span class="cx"> {
</span><span class="cx"> return !!(bits[n / wordSize] & (one << (n % wordSize)));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::set(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::set(size_t n)
</ins><span class="cx"> {
</span><span class="cx"> bits[n / wordSize] |= (one << (n % wordSize));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::set(size_t n, bool value)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::set(size_t n, bool value)
</ins><span class="cx"> {
</span><span class="cx"> if (value)
</span><span class="cx"> set(n);
</span><span class="lines">@@ -117,8 +111,8 @@
</span><span class="cx"> clear(n);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::testAndSet(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::testAndSet(size_t n)
</ins><span class="cx"> {
</span><span class="cx"> WordType mask = one << (n % wordSize);
</span><span class="cx"> size_t index = n / wordSize;
</span><span class="lines">@@ -127,8 +121,8 @@
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::testAndClear(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::testAndClear(size_t n)
</ins><span class="cx"> {
</span><span class="cx"> WordType mask = one << (n % wordSize);
</span><span class="cx"> size_t index = n / wordSize;
</span><span class="lines">@@ -137,14 +131,9 @@
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::concurrentTestAndSet(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::concurrentTestAndSet(size_t n)
</ins><span class="cx"> {
</span><del>- if (atomicMode == BitmapNotAtomic)
- return testAndSet(n);
-
- ASSERT(atomicMode == BitmapAtomic);
-
</del><span class="cx"> WordType mask = one << (n % wordSize);
</span><span class="cx"> size_t index = n / wordSize;
</span><span class="cx"> WordType* wordPtr = bits.data() + index;
</span><span class="lines">@@ -157,14 +146,9 @@
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::concurrentTestAndClear(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::concurrentTestAndClear(size_t n)
</ins><span class="cx"> {
</span><del>- if (atomicMode == BitmapNotAtomic)
- return testAndClear(n);
-
- ASSERT(atomicMode == BitmapAtomic);
-
</del><span class="cx"> WordType mask = one << (n % wordSize);
</span><span class="cx"> size_t index = n / wordSize;
</span><span class="cx"> WordType* wordPtr = bits.data() + index;
</span><span class="lines">@@ -177,20 +161,20 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::clear(size_t n)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::clear(size_t n)
</ins><span class="cx"> {
</span><span class="cx"> bits[n / wordSize] &= ~(one << (n % wordSize));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::clearAll()
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::clearAll()
</ins><span class="cx"> {
</span><span class="cx"> memset(bits.data(), 0, sizeof(bits));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline size_t Bitmap<bitmapSize, atomicMode, WordType>::nextPossiblyUnset(size_t start) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline size_t Bitmap<bitmapSize, WordType>::nextPossiblyUnset(size_t start) const
</ins><span class="cx"> {
</span><span class="cx"> if (!~bits[start / wordSize])
</span><span class="cx"> return ((start / wordSize) + 1) * wordSize;
</span><span class="lines">@@ -197,8 +181,8 @@
</span><span class="cx"> return start + 1;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline int64_t Bitmap<bitmapSize, atomicMode, WordType>::findRunOfZeros(size_t runLength) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline int64_t Bitmap<bitmapSize, WordType>::findRunOfZeros(size_t runLength) const
</ins><span class="cx"> {
</span><span class="cx"> if (!runLength)
</span><span class="cx"> runLength = 1;
</span><span class="lines">@@ -217,8 +201,8 @@
</span><span class="cx"> return -1;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline size_t Bitmap<bitmapSize, atomicMode, WordType>::count(size_t start) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline size_t Bitmap<bitmapSize, WordType>::count(size_t start) const
</ins><span class="cx"> {
</span><span class="cx"> size_t result = 0;
</span><span class="cx"> for ( ; (start % wordSize); ++start) {
</span><span class="lines">@@ -230,8 +214,8 @@
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline size_t Bitmap<bitmapSize, atomicMode, WordType>::isEmpty() const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline size_t Bitmap<bitmapSize, WordType>::isEmpty() const
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span><span class="cx"> if (bits[i])
</span><span class="lines">@@ -239,8 +223,8 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline size_t Bitmap<bitmapSize, atomicMode, WordType>::isFull() const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline size_t Bitmap<bitmapSize, WordType>::isFull() const
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span><span class="cx"> if (~bits[i])
</span><span class="lines">@@ -248,30 +232,30 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::merge(const Bitmap& other)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::merge(const Bitmap& other)
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span><span class="cx"> bits[i] |= other.bits[i];
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::filter(const Bitmap& other)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::filter(const Bitmap& other)
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span><span class="cx"> bits[i] &= other.bits[i];
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline void Bitmap<bitmapSize, atomicMode, WordType>::exclude(const Bitmap& other)
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::exclude(const Bitmap& other)
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span><span class="cx"> bits[i] &= ~other.bits[i];
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
</del><ins>+template<size_t bitmapSize, typename WordType>
</ins><span class="cx"> template<typename Func>
</span><del>-inline void Bitmap<bitmapSize, atomicMode, WordType>::forEachSetBit(const Func& func) const
</del><ins>+inline void Bitmap<bitmapSize, WordType>::forEachSetBit(const Func& func) const
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i) {
</span><span class="cx"> WordType word = bits[i];
</span><span class="lines">@@ -286,10 +270,28 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::operator==(const Bitmap& other) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::mergeAndClear(Bitmap& other)
</ins><span class="cx"> {
</span><span class="cx"> for (size_t i = 0; i < words; ++i) {
</span><ins>+ bits[i] |= other.bits[i];
+ other.bits[i] = 0;
+ }
+}
+
+template<size_t bitmapSize, typename WordType>
+inline void Bitmap<bitmapSize, WordType>::setAndClear(Bitmap& other)
+{
+ for (size_t i = 0; i < words; ++i) {
+ bits[i] = other.bits[i];
+ other.bits[i] = 0;
+ }
+}
+
+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::operator==(const Bitmap& other) const
+{
+ for (size_t i = 0; i < words; ++i) {
</ins><span class="cx"> if (bits[i] != other.bits[i])
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="lines">@@ -296,14 +298,14 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline bool Bitmap<bitmapSize, atomicMode, WordType>::operator!=(const Bitmap& other) const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline bool Bitmap<bitmapSize, WordType>::operator!=(const Bitmap& other) const
</ins><span class="cx"> {
</span><span class="cx"> return !(*this == other);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<size_t bitmapSize, BitmapAtomicMode atomicMode, typename WordType>
-inline unsigned Bitmap<bitmapSize, atomicMode, WordType>::hash() const
</del><ins>+template<size_t bitmapSize, typename WordType>
+inline unsigned Bitmap<bitmapSize, WordType>::hash() const
</ins><span class="cx"> {
</span><span class="cx"> unsigned result = 0;
</span><span class="cx"> for (size_t i = 0; i < words; ++i)
</span></span></pre>
</div>
</div>
</body>
</html>