<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[206154] trunk</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/206154">206154</a></dd>
<dt>Author</dt> <dd>fpizlo@apple.com</dd>
<dt>Date</dt> <dd>2016-09-20 11:12:18 -0700 (Tue, 20 Sep 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>Make MarkedBlock state tracking support overlapped allocation and marking state
https://bugs.webkit.org/show_bug.cgi?id=161581
Reviewed by Geoffrey Garen.
JSTests:
Add a microbenchmark for why we want to reclaim empty blocks from other allocators.
* microbenchmarks/switching-size-classes.js: Added.
Source/JavaScriptCore:
Concurrent GCs must allow for mutation and allocation during collection. We already know
how to mutate during collection. We have a write barrier for that. Allocation during
collection is more involved: the collector modifies the the mark bits, as well as other
kinds of MarkedBlock state, in-place during a collection. The allocator uses that same
MarkedBlock state to decide which regions of memory are free. This works if the allocator
never runs while the collector is running, but if we want to allow them to run at the same
time, then we need to have two versions of the state: one version built up by the
collector and another consumed by the allocator. We clear the collector state at the
beginning of collection, and splat the collector state onto the allocator state after
collection.
This could be super expensive, but we can make it cheap with some cleverness. The biggest
observation is just that most of the state is a handful of bits per block: is the block
free-listed? is it completely full? completely empty? in the incremental sweeper's
snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark
bits, but I have a solid plan there and I'll save it for another patch. Once we view the
state of blocks as bits, we can put that state into bitvectors, so that if the collector
needs to transform the state of some blocks, it can do it with a single operation over
bitvectors. I like to think of this as 32-way parallelizing block operations, since
doing one operation on a 32-bit word in one of those bitvectors instantly affects 32
blocks.
This change converts all previous collections of MarkedBlocks, along with the MarkedBlock
state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept,
markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty,
allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired).
As a nice side-effect of switching to bitvectors, we get size class rebalancing for free.
It used to be that if a MarkedAllocator had an empty block, we would only allow that
memory to be reused by a different MarkedAllocator if we did an incremental sweep or a
full eager sweep. Now we hunt down all destructorless empty blocks before allocating new
MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but
the theory is that those might be expensive to sweep, so it might still be better to leave
those to the incremental sweeper.
This change is perf-neutral all around. I did some tests with two different kinds of
allocation strategies - something that is somewhat easier to do now that you can look for
blocks that are candidates for allocation by just scanning some bitvectors. I tried two
variants:
- Allocate out of non-empty blocks first, leaving empty blocks for last in case a
different allocator needed them. This is sort of a best-fit strategy. I tried this
first, and it can be expressed as:
m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true)
- Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty
blocks equally. This is sort of a first-fit strategy. This is what I ended up settling
on, and it can be expressed as:
m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true)
The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11%
regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew
towards first-fit because it's empirically better, so this result is not surprising.
Overall, the performance of this patch on my machine is as follows, where "neutral" means
less than 1% and not statistically significant.
run-jsc-benchmarks:
SunSpider: neutral
LongSpider: 0.6% slower
V8Spider: neutral
Octane: neutral
Kraken: neutral
Microbenchmarks: 0.37% slower
AsmBench: neutral
CompressionBench: maybe 1% faster
For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test
from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we
normally consider anything less than 95% confidence to be inconclusive.
Browser benchmarks:
PLT3: 0.3% faster with 67% confidence
membuster:
Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence
Snap3EndPost: 2.4% more memory with 61% confidence
JetStream: 0.2% slower with 32% confidence
Speedometer: 0.7% faster with 82% confidence
Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10%
progression. This is due to the allocator rebalancing feature.
Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I
filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal
tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace.
* JavaScriptCore.xcodeproj/project.pbxproj:
* debugger/Debugger.cpp:
* heap/CellContainer.h:
* heap/CellContainerInlines.h:
(JSC::CellContainer::vm):
(JSC::CellContainer::heap):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::aboutToMark):
(JSC::CellContainer::isMarked): Deleted.
(JSC::CellContainer::flipIfNecessary): Deleted.
* heap/ConservativeRoots.cpp:
* heap/Heap.cpp:
(JSC::Heap::beginMarking):
(JSC::Heap::endMarking):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collectImpl):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::prepareForAllocation):
(JSC::Heap::zombifyDeadObjects):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted.
(JSC::MarkedBlockSnapshotFunctor::operator()): Deleted.
(JSC::Heap::resetAllocators): Deleted.
* heap/Heap.h:
* heap/HeapInlines.h:
(JSC::Heap::isMarked):
(JSC::Heap::isMarkedConcurrently):
(JSC::Heap::testAndSetMarked):
* heap/HeapStatistics.cpp:
* heap/HeapUtil.h:
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
* heap/HeapVerifier.cpp:
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::IncrementalSweeper):
(JSC::IncrementalSweeper::doSweep):
(JSC::IncrementalSweeper::sweepNextBlock):
(JSC::IncrementalSweeper::startSweeping):
(JSC::IncrementalSweeper::willFinishSweeping):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.h:
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedConcurrently):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::aboutToMark):
(JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted.
(JSC::LargeAllocation::flipIfNecessary): Deleted.
(JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted.
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::findEmptyBlock):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::allocateIn):
(JSC::MarkedAllocator::tryAllocateIn):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::prepareForAllocation):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::resumeAllocating):
(JSC::MarkedAllocator::beginMarkingForFullCollection):
(JSC::MarkedAllocator::endMarking):
(JSC::MarkedAllocator::snapshotForEdenCollection):
(JSC::MarkedAllocator::snapshotForFullCollection):
(JSC::MarkedAllocator::findBlockToSweep):
(JSC::MarkedAllocator::sweep):
(JSC::MarkedAllocator::shrink):
(JSC::MarkedAllocator::assertSnapshotEmpty):
(JSC::MarkedAllocator::dump):
(JSC::MarkedAllocator::dumpBits):
(JSC::MarkedAllocator::retire): Deleted.
(JSC::MarkedAllocator::filterNextBlock): Deleted.
(JSC::MarkedAllocator::setNextBlockToSweep): Deleted.
(JSC::MarkedAllocator::reset): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::forEachBitVector):
(JSC::MarkedAllocator::forEachBitVectorWithName):
(JSC::MarkedAllocator::nextAllocator):
(JSC::MarkedAllocator::setNextAllocator):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::resumeAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated):
(JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::MarkedBlock::Handle::setIsFreeListed):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::aboutToMarkSlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::noteMarkedSlow):
(JSC::MarkedBlock::Handle::removeFromAllocator):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::didRemoveFromAllocator):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted.
(JSC::MarkedBlock::flipIfNecessary): Deleted.
(JSC::MarkedBlock::Handle::flipIfNecessary): Deleted.
(JSC::MarkedBlock::flipIfNecessarySlow): Deleted.
(JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted.
(JSC::MarkedBlock::Handle::willRemoveBlock): Deleted.
(WTF::printInternal): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::Handle::index):
(JSC::MarkedBlock::aboutToMark):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::isMarkedConcurrently):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted.
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted.
(JSC::MarkedBlock::Handle::state): Deleted.
(JSC::MarkedBlock::flipIfNecessary): Deleted.
(JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted.
(JSC::MarkedBlock::Handle::flipIfNecessary): Deleted.
(JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted.
(JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted.
(JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted.
(JSC::MarkedBlock::Handle::isLive): Deleted.
(JSC::MarkedBlock::Handle::isLiveCell): Deleted.
(JSC::MarkedBlock::Handle::forEachLiveCell): Deleted.
(JSC::MarkedBlock::Handle::forEachDeadCell): Deleted.
(JSC::MarkedBlock::Handle::needsSweeping): Deleted.
(JSC::MarkedBlock::Handle::isAllocated): Deleted.
(JSC::MarkedBlock::Handle::isMarked): Deleted.
* heap/MarkedBlockInlines.h: Added.
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::resetVersion):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::MarkedSpace::beginMarking):
(JSC::MarkedSpace::endMarking):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::findEmptyBlock):
(JSC::MarkedSpace::snapshot):
(JSC::MarkedSpace::assertSnapshotEmpty):
(JSC::MarkedSpace::dumpBits):
(JSC::MarkedSpace::zombifySweep): Deleted.
(JSC::MarkedSpace::resetAllocators): Deleted.
(JSC::VerifyMarked::operator()): Deleted.
(JSC::MarkedSpace::flip): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::nextVersion):
(JSC::MarkedSpace::firstAllocator):
(JSC::MarkedSpace::allocatorForEmptyAllocation):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::blocksWithNewObjects): Deleted.
(JSC::MarkedSpace::setIsMarking): Deleted.
(JSC::MarkedSpace::forEachLiveCell): Deleted.
(JSC::MarkedSpace::forEachDeadCell): Deleted.
* heap/MarkedSpaceInlines.h: Added.
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::visitChildren):
* heap/Weak.h:
(WTF::HashTraits<JSC::Weak<T>>::emptyValue):
(WTF::HashTraits<JSC::Weak<T>>::peek):
* heap/WeakBlock.cpp:
(JSC::WeakBlock::specializedVisit):
(JSC::WeakBlock::reap):
* heap/WeakInlines.h:
(WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted.
(WTF::HashTraits<JSC::Weak<T>>::peek): Deleted.
* jit/JITThunks.h:
* runtime/JSGlobalObject.cpp:
* runtime/PrototypeMap.h:
* runtime/SamplingProfiler.cpp:
* runtime/WeakGCMap.h:
* tools/JSDollarVMPrototype.cpp:
Source/WTF:
The main change here is to bring back FastBitVector.cpp, so that I could outline some
large slow path functions. This also adds some utilities, like atomicSetAndCheck() and
isEmpty(). The GC uses these.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastBitVector.cpp: Added.
(WTF::FastBitVectorWordOwner::setEqualsSlow):
(WTF::FastBitVectorWordOwner::resizeSlow):
* wtf/FastBitVector.h:
(WTF::FastBitVectorWordOwner::operator=):
(WTF::FastBitVectorWordOwner::resize):
(WTF::FastBitVectorImpl::isEmpty):
(WTF::FastBitVector::atomicSetAndCheck):
(WTF::FastBitVector::operator[]): Deleted.
Tools:
Remove the always-trigger-copy-phase configuration.
* Scripts/run-jsc-stress-tests:</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkJSTestsChangeLog">trunk/JSTests/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreJavaScriptCorexcodeprojprojectpbxproj">trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj</a></li>
<li><a href="#trunkSourceJavaScriptCoreScriptsbuiltinsbuiltins_generate_internals_wrapper_headerpy">trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_header.py</a></li>
<li><a href="#trunkSourceJavaScriptCoredebuggerDebuggercpp">trunk/Source/JavaScriptCore/debugger/Debugger.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapCellContainerh">trunk/Source/JavaScriptCore/heap/CellContainer.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapCellContainerInlinesh">trunk/Source/JavaScriptCore/heap/CellContainerInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapConservativeRootscpp">trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapcpp">trunk/Source/JavaScriptCore/heap/Heap.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeaph">trunk/Source/JavaScriptCore/heap/Heap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapInlinesh">trunk/Source/JavaScriptCore/heap/HeapInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapStatisticscpp">trunk/Source/JavaScriptCore/heap/HeapStatistics.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapUtilh">trunk/Source/JavaScriptCore/heap/HeapUtil.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapVerifiercpp">trunk/Source/JavaScriptCore/heap/HeapVerifier.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapIncrementalSweepercpp">trunk/Source/JavaScriptCore/heap/IncrementalSweeper.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapIncrementalSweeperh">trunk/Source/JavaScriptCore/heap/IncrementalSweeper.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapLargeAllocationh">trunk/Source/JavaScriptCore/heap/LargeAllocation.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedAllocatorcpp">trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedAllocatorh">trunk/Source/JavaScriptCore/heap/MarkedAllocator.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockcpp">trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockh">trunk/Source/JavaScriptCore/heap/MarkedBlock.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpacecpp">trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpaceh">trunk/Source/JavaScriptCore/heap/MarkedSpace.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorcpp">trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapWeakh">trunk/Source/JavaScriptCore/heap/Weak.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapWeakBlockcpp">trunk/Source/JavaScriptCore/heap/WeakBlock.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapWeakInlinesh">trunk/Source/JavaScriptCore/heap/WeakInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITThunksh">trunk/Source/JavaScriptCore/jit/JITThunks.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSGlobalObjectcpp">trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSObjectcpp">trunk/Source/JavaScriptCore/runtime/JSObject.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSProxycpp">trunk/Source/JavaScriptCore/runtime/JSProxy.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeOptionsh">trunk/Source/JavaScriptCore/runtime/Options.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimePrototypeMapcpp">trunk/Source/JavaScriptCore/runtime/PrototypeMap.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimePrototypeMaph">trunk/Source/JavaScriptCore/runtime/PrototypeMap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeRegExpCacheh">trunk/Source/JavaScriptCore/runtime/RegExpCache.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeSamplingProfilercpp">trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeWeakGCMaph">trunk/Source/JavaScriptCore/runtime/WeakGCMap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeWeakGCMapInlinesh">trunk/Source/JavaScriptCore/runtime/WeakGCMapInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoretoolsJSDollarVMPrototypecpp">trunk/Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp</a></li>
<li><a href="#trunkSourceWTFChangeLog">trunk/Source/WTF/ChangeLog</a></li>
<li><a href="#trunkSourceWTFWTFxcodeprojprojectpbxproj">trunk/Source/WTF/WTF.xcodeproj/project.pbxproj</a></li>
<li><a href="#trunkSourceWTFwtfCMakeListstxt">trunk/Source/WTF/wtf/CMakeLists.txt</a></li>
<li><a href="#trunkSourceWTFwtfFastBitVectorh">trunk/Source/WTF/wtf/FastBitVector.h</a></li>
<li><a href="#trunkSourceWebCoretestingGCObservationcpp">trunk/Source/WebCore/testing/GCObservation.cpp</a></li>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptsrunjscstresstests">trunk/Tools/Scripts/run-jsc-stress-tests</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkJSTestsmicrobenchmarksswitchingsizeclassesjs">trunk/JSTests/microbenchmarks/switching-size-classes.js</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapAllocationScopeh">trunk/Source/JavaScriptCore/heap/AllocationScope.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockInlinesh">trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpaceInlinesh">trunk/Source/JavaScriptCore/heap/MarkedSpaceInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimePrototypeMapInlinesh">trunk/Source/JavaScriptCore/runtime/PrototypeMapInlines.h</a></li>
<li><a href="#trunkSourceWTFwtfFastBitVectorcpp">trunk/Source/WTF/wtf/FastBitVector.cpp</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkJSTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/JSTests/ChangeLog (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/JSTests/ChangeLog 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/JSTests/ChangeLog 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,3 +1,14 @@
</span><ins>+2016-09-20 Filip Pizlo <fpizlo@apple.com>
+
+ Make MarkedBlock state tracking support overlapped allocation and marking state
+ https://bugs.webkit.org/show_bug.cgi?id=161581
+
+ Reviewed by Geoffrey Garen.
+
+ Add a microbenchmark for why we want to reclaim empty blocks from other allocators.
+
+ * microbenchmarks/switching-size-classes.js: Added.
+
</ins><span class="cx"> 2016-09-20 Saam Barati <sbarati@apple.com>
</span><span class="cx">
</span><span class="cx"> Unreviewed, added test for x86 32-bit failure for HasOwnProperty node in DFG.
</span></span></pre></div>
<a id="trunkJSTestsmicrobenchmarksswitchingsizeclassesjs"></a>
<div class="addfile"><h4>Added: trunk/JSTests/microbenchmarks/switching-size-classes.js (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/JSTests/microbenchmarks/switching-size-classes.js (rev 0)
+++ trunk/JSTests/microbenchmarks/switching-size-classes.js 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,4 @@
</span><ins>+for (var i = 0; i < 10; ++i) {
+ for (var j = 0; j < 100000; ++j)
+ new Array(10 * i);
+}
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/ChangeLog 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,3 +1,300 @@
</span><ins>+2016-09-20 Filip Pizlo <fpizlo@apple.com>
+
+ Make MarkedBlock state tracking support overlapped allocation and marking state
+ https://bugs.webkit.org/show_bug.cgi?id=161581
+
+ Reviewed by Geoffrey Garen.
+
+ Concurrent GCs must allow for mutation and allocation during collection. We already know
+ how to mutate during collection. We have a write barrier for that. Allocation during
+ collection is more involved: the collector modifies the the mark bits, as well as other
+ kinds of MarkedBlock state, in-place during a collection. The allocator uses that same
+ MarkedBlock state to decide which regions of memory are free. This works if the allocator
+ never runs while the collector is running, but if we want to allow them to run at the same
+ time, then we need to have two versions of the state: one version built up by the
+ collector and another consumed by the allocator. We clear the collector state at the
+ beginning of collection, and splat the collector state onto the allocator state after
+ collection.
+
+ This could be super expensive, but we can make it cheap with some cleverness. The biggest
+ observation is just that most of the state is a handful of bits per block: is the block
+ free-listed? is it completely full? completely empty? in the incremental sweeper's
+ snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark
+ bits, but I have a solid plan there and I'll save it for another patch. Once we view the
+ state of blocks as bits, we can put that state into bitvectors, so that if the collector
+ needs to transform the state of some blocks, it can do it with a single operation over
+ bitvectors. I like to think of this as 32-way parallelizing block operations, since
+ doing one operation on a 32-bit word in one of those bitvectors instantly affects 32
+ blocks.
+
+ This change converts all previous collections of MarkedBlocks, along with the MarkedBlock
+ state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept,
+ markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty,
+ allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired).
+
+ As a nice side-effect of switching to bitvectors, we get size class rebalancing for free.
+ It used to be that if a MarkedAllocator had an empty block, we would only allow that
+ memory to be reused by a different MarkedAllocator if we did an incremental sweep or a
+ full eager sweep. Now we hunt down all destructorless empty blocks before allocating new
+ MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but
+ the theory is that those might be expensive to sweep, so it might still be better to leave
+ those to the incremental sweeper.
+
+ This change is perf-neutral all around. I did some tests with two different kinds of
+ allocation strategies - something that is somewhat easier to do now that you can look for
+ blocks that are candidates for allocation by just scanning some bitvectors. I tried two
+ variants:
+
+ - Allocate out of non-empty blocks first, leaving empty blocks for last in case a
+ different allocator needed them. This is sort of a best-fit strategy. I tried this
+ first, and it can be expressed as:
+
+ m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true)
+
+ - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty
+ blocks equally. This is sort of a first-fit strategy. This is what I ended up settling
+ on, and it can be expressed as:
+
+ m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true)
+
+ The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11%
+ regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew
+ towards first-fit because it's empirically better, so this result is not surprising.
+
+ Overall, the performance of this patch on my machine is as follows, where "neutral" means
+ less than 1% and not statistically significant.
+
+ run-jsc-benchmarks:
+ SunSpider: neutral
+ LongSpider: 0.6% slower
+ V8Spider: neutral
+ Octane: neutral
+ Kraken: neutral
+ Microbenchmarks: 0.37% slower
+ AsmBench: neutral
+ CompressionBench: maybe 1% faster
+
+ For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test
+ from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we
+ normally consider anything less than 95% confidence to be inconclusive.
+
+ Browser benchmarks:
+ PLT3: 0.3% faster with 67% confidence
+ membuster:
+ Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence
+ Snap3EndPost: 2.4% more memory with 61% confidence
+ JetStream: 0.2% slower with 32% confidence
+ Speedometer: 0.7% faster with 82% confidence
+
+ Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10%
+ progression. This is due to the allocator rebalancing feature.
+
+ Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I
+ filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal
+ tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace.
+
+ * JavaScriptCore.xcodeproj/project.pbxproj:
+ * debugger/Debugger.cpp:
+ * heap/CellContainer.h:
+ * heap/CellContainerInlines.h:
+ (JSC::CellContainer::vm):
+ (JSC::CellContainer::heap):
+ (JSC::CellContainer::isMarkedOrNewlyAllocated):
+ (JSC::CellContainer::aboutToMark):
+ (JSC::CellContainer::isMarked): Deleted.
+ (JSC::CellContainer::flipIfNecessary): Deleted.
+ * heap/ConservativeRoots.cpp:
+ * heap/Heap.cpp:
+ (JSC::Heap::beginMarking):
+ (JSC::Heap::endMarking):
+ (JSC::Heap::collectAllGarbage):
+ (JSC::Heap::collectImpl):
+ (JSC::Heap::snapshotMarkedSpace):
+ (JSC::Heap::prepareForAllocation):
+ (JSC::Heap::zombifyDeadObjects):
+ (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted.
+ (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted.
+ (JSC::Heap::resetAllocators): Deleted.
+ * heap/Heap.h:
+ * heap/HeapInlines.h:
+ (JSC::Heap::isMarked):
+ (JSC::Heap::isMarkedConcurrently):
+ (JSC::Heap::testAndSetMarked):
+ * heap/HeapStatistics.cpp:
+ * heap/HeapUtil.h:
+ (JSC::HeapUtil::findGCObjectPointersForMarking):
+ (JSC::HeapUtil::isPointerGCObjectJSCell):
+ * heap/HeapVerifier.cpp:
+ * heap/IncrementalSweeper.cpp:
+ (JSC::IncrementalSweeper::IncrementalSweeper):
+ (JSC::IncrementalSweeper::doSweep):
+ (JSC::IncrementalSweeper::sweepNextBlock):
+ (JSC::IncrementalSweeper::startSweeping):
+ (JSC::IncrementalSweeper::willFinishSweeping):
+ * heap/IncrementalSweeper.h:
+ * heap/LargeAllocation.h:
+ (JSC::LargeAllocation::isMarked):
+ (JSC::LargeAllocation::isMarkedConcurrently):
+ (JSC::LargeAllocation::isMarkedOrNewlyAllocated):
+ (JSC::LargeAllocation::aboutToMark):
+ (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted.
+ (JSC::LargeAllocation::flipIfNecessary): Deleted.
+ (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted.
+ * heap/MarkedAllocator.cpp:
+ (JSC::MarkedAllocator::MarkedAllocator):
+ (JSC::MarkedAllocator::isPagedOut):
+ (JSC::MarkedAllocator::findEmptyBlock):
+ (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
+ (JSC::MarkedAllocator::allocateIn):
+ (JSC::MarkedAllocator::tryAllocateIn):
+ (JSC::MarkedAllocator::allocateSlowCaseImpl):
+ (JSC::MarkedAllocator::tryAllocateBlock):
+ (JSC::MarkedAllocator::addBlock):
+ (JSC::MarkedAllocator::removeBlock):
+ (JSC::MarkedAllocator::stopAllocating):
+ (JSC::MarkedAllocator::prepareForAllocation):
+ (JSC::MarkedAllocator::lastChanceToFinalize):
+ (JSC::MarkedAllocator::resumeAllocating):
+ (JSC::MarkedAllocator::beginMarkingForFullCollection):
+ (JSC::MarkedAllocator::endMarking):
+ (JSC::MarkedAllocator::snapshotForEdenCollection):
+ (JSC::MarkedAllocator::snapshotForFullCollection):
+ (JSC::MarkedAllocator::findBlockToSweep):
+ (JSC::MarkedAllocator::sweep):
+ (JSC::MarkedAllocator::shrink):
+ (JSC::MarkedAllocator::assertSnapshotEmpty):
+ (JSC::MarkedAllocator::dump):
+ (JSC::MarkedAllocator::dumpBits):
+ (JSC::MarkedAllocator::retire): Deleted.
+ (JSC::MarkedAllocator::filterNextBlock): Deleted.
+ (JSC::MarkedAllocator::setNextBlockToSweep): Deleted.
+ (JSC::MarkedAllocator::reset): Deleted.
+ * heap/MarkedAllocator.h:
+ (JSC::MarkedAllocator::forEachBitVector):
+ (JSC::MarkedAllocator::forEachBitVectorWithName):
+ (JSC::MarkedAllocator::nextAllocator):
+ (JSC::MarkedAllocator::setNextAllocator):
+ (JSC::MarkedAllocator::forEachBlock):
+ (JSC::MarkedAllocator::resumeAllocating): Deleted.
+ * heap/MarkedBlock.cpp:
+ (JSC::MarkedBlock::tryCreate):
+ (JSC::MarkedBlock::Handle::Handle):
+ (JSC::MarkedBlock::Handle::~Handle):
+ (JSC::MarkedBlock::MarkedBlock):
+ (JSC::MarkedBlock::Handle::specializedSweep):
+ (JSC::MarkedBlock::Handle::sweep):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode):
+ (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
+ (JSC::MarkedBlock::Handle::setIsFreeListed):
+ (JSC::MarkedBlock::Handle::stopAllocating):
+ (JSC::MarkedBlock::Handle::lastChanceToFinalize):
+ (JSC::MarkedBlock::Handle::resumeAllocating):
+ (JSC::MarkedBlock::aboutToMarkSlow):
+ (JSC::MarkedBlock::clearMarks):
+ (JSC::MarkedBlock::isMarked):
+ (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
+ (JSC::MarkedBlock::isMarkedOrNewlyAllocated):
+ (JSC::MarkedBlock::Handle::didConsumeFreeList):
+ (JSC::MarkedBlock::markCount):
+ (JSC::MarkedBlock::Handle::isEmpty):
+ (JSC::MarkedBlock::noteMarkedSlow):
+ (JSC::MarkedBlock::Handle::removeFromAllocator):
+ (JSC::MarkedBlock::Handle::didAddToAllocator):
+ (JSC::MarkedBlock::Handle::didRemoveFromAllocator):
+ (JSC::MarkedBlock::Handle::isLive):
+ (JSC::MarkedBlock::Handle::isLiveCell):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted.
+ (JSC::MarkedBlock::flipIfNecessary): Deleted.
+ (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted.
+ (JSC::MarkedBlock::flipIfNecessarySlow): Deleted.
+ (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted.
+ (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted.
+ (WTF::printInternal): Deleted.
+ * heap/MarkedBlock.h:
+ (JSC::MarkedBlock::Handle::isFreeListed):
+ (JSC::MarkedBlock::Handle::index):
+ (JSC::MarkedBlock::aboutToMark):
+ (JSC::MarkedBlock::isMarked):
+ (JSC::MarkedBlock::isMarkedConcurrently):
+ (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
+ (JSC::MarkedBlock::isMarkedOrNewlyAllocated):
+ (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted.
+ (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted.
+ (JSC::MarkedBlock::Handle::state): Deleted.
+ (JSC::MarkedBlock::flipIfNecessary): Deleted.
+ (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted.
+ (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted.
+ (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted.
+ (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted.
+ (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted.
+ (JSC::MarkedBlock::Handle::isLive): Deleted.
+ (JSC::MarkedBlock::Handle::isLiveCell): Deleted.
+ (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted.
+ (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted.
+ (JSC::MarkedBlock::Handle::needsSweeping): Deleted.
+ (JSC::MarkedBlock::Handle::isAllocated): Deleted.
+ (JSC::MarkedBlock::Handle::isMarked): Deleted.
+ * heap/MarkedBlockInlines.h: Added.
+ (JSC::MarkedBlock::Handle::isLive):
+ (JSC::MarkedBlock::Handle::isLiveCell):
+ (JSC::MarkedBlock::Handle::forEachLiveCell):
+ (JSC::MarkedBlock::Handle::forEachDeadCell):
+ (JSC::MarkedBlock::resetVersion):
+ * heap/MarkedSpace.cpp:
+ (JSC::MarkedSpace::MarkedSpace):
+ (JSC::MarkedSpace::allocate):
+ (JSC::MarkedSpace::tryAllocate):
+ (JSC::MarkedSpace::sweep):
+ (JSC::MarkedSpace::prepareForAllocation):
+ (JSC::MarkedSpace::shrink):
+ (JSC::MarkedSpace::clearNewlyAllocated):
+ (JSC::MarkedSpace::beginMarking):
+ (JSC::MarkedSpace::endMarking):
+ (JSC::MarkedSpace::didAllocateInBlock):
+ (JSC::MarkedSpace::findEmptyBlock):
+ (JSC::MarkedSpace::snapshot):
+ (JSC::MarkedSpace::assertSnapshotEmpty):
+ (JSC::MarkedSpace::dumpBits):
+ (JSC::MarkedSpace::zombifySweep): Deleted.
+ (JSC::MarkedSpace::resetAllocators): Deleted.
+ (JSC::VerifyMarked::operator()): Deleted.
+ (JSC::MarkedSpace::flip): Deleted.
+ * heap/MarkedSpace.h:
+ (JSC::MarkedSpace::nextVersion):
+ (JSC::MarkedSpace::firstAllocator):
+ (JSC::MarkedSpace::allocatorForEmptyAllocation):
+ (JSC::MarkedSpace::forEachAllocator):
+ (JSC::MarkedSpace::blocksWithNewObjects): Deleted.
+ (JSC::MarkedSpace::setIsMarking): Deleted.
+ (JSC::MarkedSpace::forEachLiveCell): Deleted.
+ (JSC::MarkedSpace::forEachDeadCell): Deleted.
+ * heap/MarkedSpaceInlines.h: Added.
+ (JSC::MarkedSpace::forEachLiveCell):
+ (JSC::MarkedSpace::forEachDeadCell):
+ * heap/SlotVisitor.cpp:
+ (JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
+ (JSC::SlotVisitor::markAuxiliary):
+ (JSC::SlotVisitor::visitChildren):
+ * heap/Weak.h:
+ (WTF::HashTraits<JSC::Weak<T>>::emptyValue):
+ (WTF::HashTraits<JSC::Weak<T>>::peek):
+ * heap/WeakBlock.cpp:
+ (JSC::WeakBlock::specializedVisit):
+ (JSC::WeakBlock::reap):
+ * heap/WeakInlines.h:
+ (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted.
+ (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted.
+ * jit/JITThunks.h:
+ * runtime/JSGlobalObject.cpp:
+ * runtime/PrototypeMap.h:
+ * runtime/SamplingProfiler.cpp:
+ * runtime/WeakGCMap.h:
+ * tools/JSDollarVMPrototype.cpp:
+
</ins><span class="cx"> 2016-09-20 Jonathan Bedard <jbedard@apple.com>
</span><span class="cx">
</span><span class="cx"> Undefined behavior: Left shift negative number
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreJavaScriptCorexcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -498,6 +498,8 @@
</span><span class="cx"> 0F7C39FB1C8F629300480151 /* RegExpInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C39FA1C8F629300480151 /* RegExpInlines.h */; };
</span><span class="cx"> 0F7C39FD1C8F659500480151 /* RegExpObjectInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C39FC1C8F659500480151 /* RegExpObjectInlines.h */; };
</span><span class="cx"> 0F7C39FF1C90C55B00480151 /* DFGOpInfo.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C39FE1C90C55B00480151 /* DFGOpInfo.h */; };
</span><ins>+ 0F7C5FB81D888A0C0044F5E2 /* MarkedBlockInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C5FB71D888A010044F5E2 /* MarkedBlockInlines.h */; };
+ 0F7C5FBA1D8895070044F5E2 /* MarkedSpaceInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F7C5FB91D8895050044F5E2 /* MarkedSpaceInlines.h */; };
</ins><span class="cx"> 0F8023EA1613832B00A0BA45 /* ByValInfo.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8023E91613832300A0BA45 /* ByValInfo.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0F8335B71639C1E6001443B5 /* ArrayAllocationProfile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */; };
</span><span class="cx"> 0F8335B81639C1EA001443B5 /* ArrayAllocationProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="lines">@@ -576,6 +578,7 @@
</span><span class="cx"> 0F9FB4F417FCB91700CB67F8 /* DFGStackLayoutPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F9FB4F217FCB91700CB67F8 /* DFGStackLayoutPhase.cpp */; };
</span><span class="cx"> 0F9FB4F517FCB91700CB67F8 /* DFGStackLayoutPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F9FB4F317FCB91700CB67F8 /* DFGStackLayoutPhase.h */; };
</span><span class="cx"> 0F9FC8C514E1B60400D52AE0 /* PutKind.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F9FC8C114E1B5FB00D52AE0 /* PutKind.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><ins>+ 0FA131711D8DD72B00EC130A /* PrototypeMapInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FA131701D8DD72900EC130A /* PrototypeMapInlines.h */; };
</ins><span class="cx"> 0FA2C17B17D7CF84009D015F /* TestRunnerUtils.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA2C17917D7CF84009D015F /* TestRunnerUtils.cpp */; };
</span><span class="cx"> 0FA2C17C17D7CF84009D015F /* TestRunnerUtils.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FA2C17A17D7CF84009D015F /* TestRunnerUtils.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0FA581BA150E952C00B9A2D9 /* DFGNodeFlags.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA581B7150E952A00B9A2D9 /* DFGNodeFlags.cpp */; };
</span><span class="lines">@@ -610,6 +613,7 @@
</span><span class="cx"> 0FB4FB731BC843140025CA5A /* FTLLazySlowPath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB4FB701BC843140025CA5A /* FTLLazySlowPath.cpp */; };
</span><span class="cx"> 0FB4FB741BC843140025CA5A /* FTLLazySlowPath.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB4FB711BC843140025CA5A /* FTLLazySlowPath.h */; };
</span><span class="cx"> 0FB4FB751BC843140025CA5A /* FTLLazySlowPathCall.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB4FB721BC843140025CA5A /* FTLLazySlowPathCall.h */; };
</span><ins>+ 0FB530391D90404C00F28AE3 /* AllocationScope.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB530381D90404600F28AE3 /* AllocationScope.h */; };
</ins><span class="cx"> 0FB5467714F59B5C002C2989 /* LazyOperandValueProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB5467614F59AD1002C2989 /* LazyOperandValueProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 0FB5467914F5C46B002C2989 /* LazyOperandValueProfile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB5467814F5C468002C2989 /* LazyOperandValueProfile.cpp */; };
</span><span class="cx"> 0FB5467B14F5C7E1002C2989 /* MethodOfGettingAValueProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FB5467A14F5C7D4002C2989 /* MethodOfGettingAValueProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="lines">@@ -2731,6 +2735,8 @@
</span><span class="cx"> 0F7C39FA1C8F629300480151 /* RegExpInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RegExpInlines.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F7C39FC1C8F659500480151 /* RegExpObjectInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RegExpObjectInlines.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F7C39FE1C90C55B00480151 /* DFGOpInfo.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGOpInfo.h; path = dfg/DFGOpInfo.h; sourceTree = "<group>"; };
</span><ins>+ 0F7C5FB71D888A010044F5E2 /* MarkedBlockInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MarkedBlockInlines.h; sourceTree = "<group>"; };
+ 0F7C5FB91D8895050044F5E2 /* MarkedSpaceInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MarkedSpaceInlines.h; sourceTree = "<group>"; };
</ins><span class="cx"> 0F8023E91613832300A0BA45 /* ByValInfo.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ByValInfo.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F8335B41639C1E3001443B5 /* ArrayAllocationProfile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ArrayAllocationProfile.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0F8335B51639C1E3001443B5 /* ArrayAllocationProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ArrayAllocationProfile.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -2808,6 +2814,7 @@
</span><span class="cx"> 0F9FB4F217FCB91700CB67F8 /* DFGStackLayoutPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStackLayoutPhase.cpp; path = dfg/DFGStackLayoutPhase.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0F9FB4F317FCB91700CB67F8 /* DFGStackLayoutPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStackLayoutPhase.h; path = dfg/DFGStackLayoutPhase.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F9FC8C114E1B5FB00D52AE0 /* PutKind.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PutKind.h; sourceTree = "<group>"; };
</span><ins>+ 0FA131701D8DD72900EC130A /* PrototypeMapInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PrototypeMapInlines.h; sourceTree = "<group>"; };
</ins><span class="cx"> 0FA2C17917D7CF84009D015F /* TestRunnerUtils.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = TestRunnerUtils.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0FA2C17A17D7CF84009D015F /* TestRunnerUtils.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = TestRunnerUtils.h; sourceTree = "<group>"; };
</span><span class="cx"> 0FA581B7150E952A00B9A2D9 /* DFGNodeFlags.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGNodeFlags.cpp; path = dfg/DFGNodeFlags.cpp; sourceTree = "<group>"; };
</span><span class="lines">@@ -2852,6 +2859,7 @@
</span><span class="cx"> 0FB4FB701BC843140025CA5A /* FTLLazySlowPath.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = FTLLazySlowPath.cpp; path = ftl/FTLLazySlowPath.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0FB4FB711BC843140025CA5A /* FTLLazySlowPath.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = FTLLazySlowPath.h; path = ftl/FTLLazySlowPath.h; sourceTree = "<group>"; };
</span><span class="cx"> 0FB4FB721BC843140025CA5A /* FTLLazySlowPathCall.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = FTLLazySlowPathCall.h; path = ftl/FTLLazySlowPathCall.h; sourceTree = "<group>"; };
</span><ins>+ 0FB530381D90404600F28AE3 /* AllocationScope.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AllocationScope.h; sourceTree = "<group>"; };
</ins><span class="cx"> 0FB5467614F59AD1002C2989 /* LazyOperandValueProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LazyOperandValueProfile.h; sourceTree = "<group>"; };
</span><span class="cx"> 0FB5467814F5C468002C2989 /* LazyOperandValueProfile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LazyOperandValueProfile.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0FB5467A14F5C7D4002C2989 /* MethodOfGettingAValueProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MethodOfGettingAValueProfile.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -5255,6 +5263,7 @@
</span><span class="cx"> 142E312A134FF0A600AFADB5 /* heap */ = {
</span><span class="cx"> isa = PBXGroup;
</span><span class="cx"> children = (
</span><ins>+ 0FB530381D90404600F28AE3 /* AllocationScope.h */,
</ins><span class="cx"> 0F9630351D4192C3005609D9 /* AllocatorAttributes.cpp */,
</span><span class="cx"> 0F9630361D4192C3005609D9 /* AllocatorAttributes.h */,
</span><span class="cx"> 0F070A421D543A89006E7232 /* CellContainer.h */,
</span><span class="lines">@@ -5338,9 +5347,11 @@
</span><span class="cx"> C2B916C114DA014E00CBAC86 /* MarkedAllocator.h */,
</span><span class="cx"> 142D6F0613539A2800B02E86 /* MarkedBlock.cpp */,
</span><span class="cx"> 142D6F0713539A2800B02E86 /* MarkedBlock.h */,
</span><ins>+ 0F7C5FB71D888A010044F5E2 /* MarkedBlockInlines.h */,
</ins><span class="cx"> 141448CA13A176EC00F5BA1A /* MarkedBlockSet.h */,
</span><span class="cx"> 14D2F3D8139F4BE200491031 /* MarkedSpace.cpp */,
</span><span class="cx"> 14D2F3D9139F4BE200491031 /* MarkedSpace.h */,
</span><ins>+ 0F7C5FB91D8895050044F5E2 /* MarkedSpaceInlines.h */,
</ins><span class="cx"> 142D6F0E13539A4100B02E86 /* MarkStack.cpp */,
</span><span class="cx"> 142D6F0F13539A4100B02E86 /* MarkStack.h */,
</span><span class="cx"> ADDB1F6218D77DB7009B58A8 /* OpaqueRootSet.h */,
</span><span class="lines">@@ -6076,6 +6087,7 @@
</span><span class="cx"> 65C02FBB0637462A003E7EE6 /* Protect.h */,
</span><span class="cx"> 14D844A216AA2C7000A65AF0 /* PrototypeMap.cpp */,
</span><span class="cx"> 14D844A316AA2C7000A65AF0 /* PrototypeMap.h */,
</span><ins>+ 0FA131701D8DD72900EC130A /* PrototypeMapInlines.h */,
</ins><span class="cx"> 79B00CB81C6AB07E0088C65D /* ProxyConstructor.cpp */,
</span><span class="cx"> 79B00CB91C6AB07E0088C65D /* ProxyConstructor.h */,
</span><span class="cx"> 79B00CBA1C6AB07E0088C65D /* ProxyObject.cpp */,
</span><span class="lines">@@ -7312,6 +7324,7 @@
</span><span class="cx"> 62D755D71B84FB4A001801FA /* CallFrameShuffler.h in Headers */,
</span><span class="cx"> 0F0B83B114BCF71800885B4F /* CallLinkInfo.h in Headers */,
</span><span class="cx"> 0F93329E14CA7DC50085F3C6 /* CallLinkStatus.h in Headers */,
</span><ins>+ 0FB530391D90404C00F28AE3 /* AllocationScope.h in Headers */,
</ins><span class="cx"> 627673241B680C1E00FD9F2E /* CallMode.h in Headers */,
</span><span class="cx"> 0F0B83B914BCF95F00885B4F /* CallReturnOffsetToBytecodeOffset.h in Headers */,
</span><span class="cx"> 0F3B7E2B19A11B8000D9BC56 /* CallVariant.h in Headers */,
</span><span class="lines">@@ -7394,6 +7407,7 @@
</span><span class="cx"> 0F66E16B14DF3F1600B7B2E4 /* DFGAdjacencyList.h in Headers */,
</span><span class="cx"> 0FFB921816D02EB20055A5DB /* DFGAllocator.h in Headers */,
</span><span class="cx"> 0F1E3A461534CBAF000F9456 /* DFGArgumentPosition.h in Headers */,
</span><ins>+ 0F7C5FB81D888A0C0044F5E2 /* MarkedBlockInlines.h in Headers */,
</ins><span class="cx"> 0F2DD8121AB3D8BE00BBB8E8 /* DFGArgumentsEliminationPhase.h in Headers */,
</span><span class="cx"> 0F2DD8141AB3D8BE00BBB8E8 /* DFGArgumentsUtilities.h in Headers */,
</span><span class="cx"> 0F485322187750560083B687 /* DFGArithMode.h in Headers */,
</span><span class="lines">@@ -7887,6 +7901,7 @@
</span><span class="cx"> 86E3C614167BABD7006D760A /* JSExport.h in Headers */,
</span><span class="cx"> A7B4ACAF1484C9CE00B38A36 /* JSExportMacros.h in Headers */,
</span><span class="cx"> 0F2B66EF17B6B5AB00A7AE3F /* JSFloat32Array.h in Headers */,
</span><ins>+ 0FA131711D8DD72B00EC130A /* PrototypeMapInlines.h in Headers */,
</ins><span class="cx"> 0F2B66F017B6B5AB00A7AE3F /* JSFloat64Array.h in Headers */,
</span><span class="cx"> BC18C41F0E16F5CD00B34460 /* JSFunction.h in Headers */,
</span><span class="cx"> A72028BA1797603D0098028C /* JSFunctionInlines.h in Headers */,
</span><span class="lines">@@ -7911,6 +7926,7 @@
</span><span class="cx"> A503FA2A188F105900110F14 /* JSGlobalObjectScriptDebugServer.h in Headers */,
</span><span class="cx"> 79A228361D35D71F00D8E067 /* ArithProfile.h in Headers */,
</span><span class="cx"> A513E5C0185BFACC007E95AD /* JSInjectedScriptHost.h in Headers */,
</span><ins>+ 0F7C5FBA1D8895070044F5E2 /* MarkedSpaceInlines.h in Headers */,
</ins><span class="cx"> A513E5C2185BFACC007E95AD /* JSInjectedScriptHostPrototype.h in Headers */,
</span><span class="cx"> C442CB251A6CDB8C005D3D7C /* JSInputs.json in Headers */,
</span><span class="cx"> 0F2B66F817B6B5AB00A7AE3F /* JSInt16Array.h in Headers */,
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreScriptsbuiltinsbuiltins_generate_internals_wrapper_headerpy"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_header.py (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_header.py 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_header.py 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -61,9 +61,8 @@
</span><span class="cx">
</span><span class="cx"> def generate_secondary_header_includes(self):
</span><span class="cx"> header_includes = [
</span><del>- (["WebCore"],
- ("JavaScriptCore", "runtime/VM.h"),
- ),
</del><ins>+ (["WebCore"], ("JavaScriptCore", "heap/WeakInlines.h")),
+ (["WebCore"], ("JavaScriptCore", "runtime/VM.h"))
</ins><span class="cx"> ]
</span><span class="cx"> for object in self.internals:
</span><span class="cx"> header_includes.append((["WebCore"], ("WebCore", object.object_name + "Builtins.h")))
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredebuggerDebuggercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/debugger/Debugger.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/debugger/Debugger.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/debugger/Debugger.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -31,6 +31,7 @@
</span><span class="cx"> #include "JSFunction.h"
</span><span class="cx"> #include "JSGlobalObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "Parser.h"
</span><span class="cx"> #include "Protect.h"
</span><span class="cx"> #include "VMEntryScope.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapAllocationScopeh"></a>
<div class="addfile"><h4>Added: trunk/Source/JavaScriptCore/heap/AllocationScope.h (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/AllocationScope.h (rev 0)
+++ trunk/Source/JavaScriptCore/heap/AllocationScope.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,53 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "Heap.h"
+
+namespace JSC {
+
+class AllocationScope {
+public:
+ AllocationScope(Heap& heap)
+ : m_heap(heap)
+ , m_lastOperation(m_heap.m_operationInProgress)
+ {
+ ASSERT(m_lastOperation == NoOperation || m_lastOperation == Allocation);
+ m_heap.m_operationInProgress = Allocation;
+ }
+
+ ~AllocationScope()
+ {
+ ASSERT(m_heap.m_operationInProgress == Allocation);
+ m_heap.m_operationInProgress = m_lastOperation;
+ }
+private:
+ Heap& m_heap;
+ HeapOperation m_lastOperation;
+};
+
+} // namespace JSC
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCellContainerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/CellContainer.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CellContainer.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/CellContainer.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,10 +29,12 @@
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><ins>+class Heap;
</ins><span class="cx"> class HeapCell;
</span><span class="cx"> class LargeAllocation;
</span><span class="cx"> class MarkedBlock;
</span><span class="cx"> class WeakSet;
</span><ins>+class VM;
</ins><span class="cx">
</span><span class="cx"> typedef uint32_t HeapVersion;
</span><span class="cx">
</span><span class="lines">@@ -56,6 +58,9 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ VM* vm() const;
+ Heap* heap() const;
+
</ins><span class="cx"> explicit operator bool() const { return !!m_encodedPointer; }
</span><span class="cx">
</span><span class="cx"> bool isMarkedBlock() const { return m_encodedPointer && !(m_encodedPointer & isLargeAllocationBit); }
</span><span class="lines">@@ -73,13 +78,13 @@
</span><span class="cx"> return *bitwise_cast<LargeAllocation*>(m_encodedPointer - isLargeAllocationBit);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- void flipIfNecessary(HeapVersion);
- void flipIfNecessary();
</del><ins>+ void aboutToMark(HeapVersion);
</ins><span class="cx"> bool needsFlip() const;
</span><span class="cx">
</span><del>- bool isMarked() const;
</del><span class="cx"> bool isMarked(HeapCell*) const;
</span><ins>+ bool isMarked(HeapVersion, HeapCell*) const;
</ins><span class="cx"> bool isMarkedOrNewlyAllocated(HeapCell*) const;
</span><ins>+ bool isMarkedOrNewlyAllocated(HeapVersion, HeapCell*) const;
</ins><span class="cx">
</span><span class="cx"> void noteMarked();
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCellContainerInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/CellContainerInlines.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CellContainerInlines.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/CellContainerInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,16 +29,22 @@
</span><span class="cx"> #include "JSCell.h"
</span><span class="cx"> #include "LargeAllocation.h"
</span><span class="cx"> #include "MarkedBlock.h"
</span><ins>+#include "VM.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><del>-inline bool CellContainer::isMarked() const
</del><ins>+inline VM* CellContainer::vm() const
</ins><span class="cx"> {
</span><span class="cx"> if (isLargeAllocation())
</span><del>- return true;
- return markedBlock().handle().isMarked();
</del><ins>+ return largeAllocation().vm();
+ return markedBlock().vm();
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+inline Heap* CellContainer::heap() const
+{
+ return &vm()->heap;
+}
+
</ins><span class="cx"> inline bool CellContainer::isMarked(HeapCell* cell) const
</span><span class="cx"> {
</span><span class="cx"> if (isLargeAllocation())
</span><span class="lines">@@ -46,6 +52,13 @@
</span><span class="cx"> return markedBlock().isMarked(cell);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline bool CellContainer::isMarked(HeapVersion version, HeapCell* cell) const
+{
+ if (isLargeAllocation())
+ return largeAllocation().isMarked();
+ return markedBlock().isMarked(version, cell);
+}
+
</ins><span class="cx"> inline bool CellContainer::isMarkedOrNewlyAllocated(HeapCell* cell) const
</span><span class="cx"> {
</span><span class="cx"> if (isLargeAllocation())
</span><span class="lines">@@ -53,6 +66,13 @@
</span><span class="cx"> return markedBlock().isMarkedOrNewlyAllocated(cell);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline bool CellContainer::isMarkedOrNewlyAllocated(HeapVersion version, HeapCell* cell) const
+{
+ if (isLargeAllocation())
+ return largeAllocation().isMarkedOrNewlyAllocated();
+ return markedBlock().isMarkedOrNewlyAllocated(version, cell);
+}
+
</ins><span class="cx"> inline void CellContainer::noteMarked()
</span><span class="cx"> {
</span><span class="cx"> if (!isLargeAllocation())
</span><span class="lines">@@ -73,18 +93,12 @@
</span><span class="cx"> return markedBlock().weakSet();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void CellContainer::flipIfNecessary(HeapVersion heapVersion)
</del><ins>+inline void CellContainer::aboutToMark(HeapVersion heapVersion)
</ins><span class="cx"> {
</span><span class="cx"> if (!isLargeAllocation())
</span><del>- markedBlock().flipIfNecessary(heapVersion);
</del><ins>+ markedBlock().aboutToMark(heapVersion);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline void CellContainer::flipIfNecessary()
-{
- if (!isLargeAllocation())
- markedBlock().flipIfNecessary();
-}
-
</del><span class="cx"> inline bool CellContainer::needsFlip() const
</span><span class="cx"> {
</span><span class="cx"> if (isLargeAllocation())
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapConservativeRootscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/ConservativeRoots.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -34,6 +34,7 @@
</span><span class="cx"> #include "JSCell.h"
</span><span class="cx"> #include "JSObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "MarkedBlockInlines.h"
</ins><span class="cx"> #include "Structure.h"
</span><span class="cx"> #include <wtf/OSAllocator.h>
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -45,6 +45,7 @@
</span><span class="cx"> #include "JSGlobalObject.h"
</span><span class="cx"> #include "JSLock.h"
</span><span class="cx"> #include "JSVirtualMachineInternal.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "SamplingProfiler.h"
</span><span class="cx"> #include "ShadowChicken.h"
</span><span class="cx"> #include "SuperSampler.h"
</span><span class="lines">@@ -527,11 +528,9 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> {
</span><del>- TimingScope clearMarksTimingScope(*this, "m_objectSpace.clearMarks");
- m_objectSpace.flip();
</del><ins>+ TimingScope clearMarksTimingScope(*this, "m_objectSpace.beginMarking");
+ m_objectSpace.beginMarking();
</ins><span class="cx"> }
</span><del>-
- m_objectSpace.setIsMarking(true);
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::visitExternalRememberedSet()
</span><span class="lines">@@ -780,7 +779,7 @@
</span><span class="cx"> ASSERT(m_sharedMarkStack.isEmpty());
</span><span class="cx"> m_weakReferenceHarvesters.removeAll();
</span><span class="cx">
</span><del>- m_objectSpace.setIsMarking(false);
</del><ins>+ m_objectSpace.endMarking();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> size_t Heap::objectCount()
</span><span class="lines">@@ -937,10 +936,19 @@
</span><span class="cx"> if (UNLIKELY(Options::useImmortalObjects()))
</span><span class="cx"> sweeper()->willFinishSweeping();
</span><span class="cx"> else {
</span><ins>+ double before = 0;
+ if (Options::logGC()) {
+ dataLog("[Full sweep: ", capacity() / 1024, " kb ");
+ before = currentTimeMS();
+ }
</ins><span class="cx"> m_objectSpace.sweep();
</span><span class="cx"> m_objectSpace.shrink();
</span><ins>+ if (Options::logGC()) {
+ double after = currentTimeMS();
+ dataLog("=> ", capacity() / 1024, " kb, ", after - before, " ms]\n");
+ }
</ins><span class="cx"> }
</span><del>- ASSERT(m_blockSnapshot.isEmpty());
</del><ins>+ m_objectSpace.assertNoUnswept();
</ins><span class="cx">
</span><span class="cx"> sweepAllLogicallyEmptyWeakBlocks();
</span><span class="cx"> }
</span><span class="lines">@@ -1043,7 +1051,7 @@
</span><span class="cx"> reapWeakHandles();
</span><span class="cx"> pruneStaleEntriesFromWeakGCMaps();
</span><span class="cx"> sweepArrayBuffers();
</span><del>- snapshotMarkedSpace();
</del><ins>+ snapshotUnswept();
</ins><span class="cx"> finalizeUnconditionalFinalizers();
</span><span class="cx"> removeDeadCompilerWorklistEntries();
</span><span class="cx"> deleteUnmarkedCompiledCode();
</span><span class="lines">@@ -1052,7 +1060,7 @@
</span><span class="cx"> notifyIncrementalSweeper();
</span><span class="cx"> writeBarrierCurrentlyExecutingCodeBlocks();
</span><span class="cx">
</span><del>- resetAllocators();
</del><ins>+ prepareForAllocation();
</ins><span class="cx"> updateAllocationLimits();
</span><span class="cx"> didFinishCollection(gcStartTime);
</span><span class="cx"> resumeCompilerThreads();
</span><span class="lines">@@ -1067,6 +1075,11 @@
</span><span class="cx"> double after = currentTimeMS();
</span><span class="cx"> dataLog(after - before, " ms]\n");
</span><span class="cx"> }
</span><ins>+
+ if (false) {
+ dataLog("Heap state after GC:\n");
+ m_objectSpace.dumpBits();
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::sweepLargeAllocations()
</span><span class="lines">@@ -1166,44 +1179,10 @@
</span><span class="cx"> m_arrayBuffers.sweep();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-struct MarkedBlockSnapshotFunctor : public MarkedBlock::VoidFunctor {
- MarkedBlockSnapshotFunctor(Vector<MarkedBlock::Handle*>& blocks)
- : m_index(0)
- , m_blocks(blocks)
- {
- }
-
- void operator()(MarkedBlock::Handle* block) const
- {
- block->setIsOnBlocksToSweep(true);
- m_blocks[m_index++] = block;
- }
-
- // FIXME: This is a mutable field becaue this isn't a C++ lambda.
- // https://bugs.webkit.org/show_bug.cgi?id=159644
- mutable size_t m_index;
- Vector<MarkedBlock::Handle*>& m_blocks;
-};
-
-void Heap::snapshotMarkedSpace()
</del><ins>+void Heap::snapshotUnswept()
</ins><span class="cx"> {
</span><del>- TimingScope timingScope(*this, "Heap::snapshotMarkedSpace");
- // FIXME: This should probably be renamed. It's not actually snapshotting all of MarkedSpace.
- // This is used by IncrementalSweeper, so it only needs to snapshot blocks. However, if we ever
- // wanted to add other snapshotting login, we'd probably put it here.
-
- if (m_operationInProgress == EdenCollection) {
- for (MarkedBlock::Handle* handle : m_objectSpace.blocksWithNewObjects()) {
- if (handle->isOnBlocksToSweep())
- continue;
- m_blockSnapshot.append(handle);
- handle->setIsOnBlocksToSweep(true);
- }
- } else {
- m_blockSnapshot.resizeToFit(m_objectSpace.blocks().set().size());
- MarkedBlockSnapshotFunctor functor(m_blockSnapshot);
- m_objectSpace.forEachBlock(functor);
- }
</del><ins>+ TimingScope timingScope(*this, "Heap::snapshotUnswept");
+ m_objectSpace.snapshotUnswept();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::deleteSourceProviderCaches()
</span><span class="lines">@@ -1226,9 +1205,9 @@
</span><span class="cx"> m_codeBlocks->writeBarrierCurrentlyExecutingCodeBlocks(this);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::resetAllocators()
</del><ins>+void Heap::prepareForAllocation()
</ins><span class="cx"> {
</span><del>- m_objectSpace.resetAllocators();
</del><ins>+ m_objectSpace.prepareForAllocation();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::updateAllocationLimits()
</span><span class="lines">@@ -1477,7 +1456,7 @@
</span><span class="cx"> void Heap::zombifyDeadObjects()
</span><span class="cx"> {
</span><span class="cx"> // Sweep now because destructors will crash once we're zombified.
</span><del>- m_objectSpace.zombifySweep();
</del><ins>+ m_objectSpace.sweep();
</ins><span class="cx"> HeapIterationScope iterationScope(*this);
</span><span class="cx"> m_objectSpace.forEachDeadCell(iterationScope, Zombify());
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/Heap.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -49,6 +49,7 @@
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><ins>+class AllocationScope;
</ins><span class="cx"> class CodeBlock;
</span><span class="cx"> class CodeBlockSet;
</span><span class="cx"> class EdenGCActivityCallback;
</span><span class="lines">@@ -255,6 +256,7 @@
</span><span class="cx"> void didFreeBlock(size_t capacity);
</span><span class="cx">
</span><span class="cx"> private:
</span><ins>+ friend class AllocationScope;
</ins><span class="cx"> friend class CodeBlock;
</span><span class="cx"> friend class DeferGC;
</span><span class="cx"> friend class DeferGCForAWhile;
</span><span class="lines">@@ -327,11 +329,11 @@
</span><span class="cx"> void reapWeakHandles();
</span><span class="cx"> void pruneStaleEntriesFromWeakGCMaps();
</span><span class="cx"> void sweepArrayBuffers();
</span><del>- void snapshotMarkedSpace();
</del><ins>+ void snapshotUnswept();
</ins><span class="cx"> void deleteSourceProviderCaches();
</span><span class="cx"> void notifyIncrementalSweeper();
</span><span class="cx"> void writeBarrierCurrentlyExecutingCodeBlocks();
</span><del>- void resetAllocators();
</del><ins>+ void prepareForAllocation();
</ins><span class="cx"> void harvestWeakReferences();
</span><span class="cx"> void finalizeUnconditionalFinalizers();
</span><span class="cx"> void clearUnmarkedExecutables();
</span><span class="lines">@@ -422,7 +424,6 @@
</span><span class="cx"> RefPtr<FullGCActivityCallback> m_fullActivityCallback;
</span><span class="cx"> RefPtr<GCActivityCallback> m_edenActivityCallback;
</span><span class="cx"> std::unique_ptr<IncrementalSweeper> m_sweeper;
</span><del>- Vector<MarkedBlock::Handle*> m_blockSnapshot;
</del><span class="cx">
</span><span class="cx"> Vector<HeapObserver*> m_observers;
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/HeapInlines.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/HeapInlines.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/HeapInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -81,9 +81,8 @@
</span><span class="cx"> if (cell->isLargeAllocation())
</span><span class="cx"> return cell->largeAllocation().isMarked();
</span><span class="cx"> MarkedBlock& block = cell->markedBlock();
</span><del>- if (block.needsFlip(block.vm()->heap.objectSpace().version()))
- return false;
- return block.isMarked(cell);
</del><ins>+ return block.isMarked(
+ block.vm()->heap.objectSpace().version(), cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> ALWAYS_INLINE bool Heap::isMarkedConcurrently(const void* rawCell)
</span><span class="lines">@@ -92,10 +91,8 @@
</span><span class="cx"> if (cell->isLargeAllocation())
</span><span class="cx"> return cell->largeAllocation().isMarked();
</span><span class="cx"> MarkedBlock& block = cell->markedBlock();
</span><del>- if (block.needsFlip(block.vm()->heap.objectSpace().version()))
- return false;
- WTF::loadLoadFence();
- return block.isMarked(cell);
</del><ins>+ return block.isMarkedConcurrently(
+ block.vm()->heap.objectSpace().version(), cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> ALWAYS_INLINE bool Heap::testAndSetMarked(HeapVersion version, const void* rawCell)
</span><span class="lines">@@ -104,7 +101,7 @@
</span><span class="cx"> if (cell->isLargeAllocation())
</span><span class="cx"> return cell->largeAllocation().testAndSetMarked();
</span><span class="cx"> MarkedBlock& block = cell->markedBlock();
</span><del>- block.flipIfNecessaryDuringMarking(version);
</del><ins>+ block.aboutToMark(version);
</ins><span class="cx"> return block.testAndSetMarked(cell);
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapStatisticscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/HeapStatistics.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/HeapStatistics.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/HeapStatistics.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx"> #include "HeapIterationScope.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><span class="cx"> #include "JSObject.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "Options.h"
</span><span class="cx"> #include <stdlib.h>
</span><span class="cx"> #include <wtf/CurrentTime.h>
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapUtilh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/HeapUtil.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/HeapUtil.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/HeapUtil.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -84,9 +84,8 @@
</span><span class="cx"> if (!filter.ruleOut(bitwise_cast<Bits>(previousCandidate))
</span><span class="cx"> && set.contains(previousCandidate)
</span><span class="cx"> && previousCandidate->handle().cellKind() == HeapCell::Auxiliary) {
</span><del>- previousCandidate->flipIfNecessary(heapVersion);
</del><span class="cx"> previousPointer = static_cast<char*>(previousCandidate->handle().cellAlign(previousPointer));
</span><del>- if (previousCandidate->handle().isLiveCell(previousPointer))
</del><ins>+ if (previousCandidate->handle().isLiveCell(heapVersion, previousPointer))
</ins><span class="cx"> func(previousPointer);
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="lines">@@ -99,10 +98,8 @@
</span><span class="cx"> if (!set.contains(candidate))
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- candidate->flipIfNecessary(heapVersion);
-
</del><span class="cx"> auto tryPointer = [&] (void* pointer) {
</span><del>- if (candidate->handle().isLiveCell(pointer))
</del><ins>+ if (candidate->handle().isLiveCell(heapVersion, pointer))
</ins><span class="cx"> func(pointer);
</span><span class="cx"> };
</span><span class="cx">
</span><span class="lines">@@ -170,7 +167,6 @@
</span><span class="cx"> if (candidate->handle().cellKind() != HeapCell::JSCell)
</span><span class="cx"> return false;
</span><span class="cx">
</span><del>- candidate->flipIfNecessary();
</del><span class="cx"> if (!candidate->handle().isLiveCell(pointer))
</span><span class="cx"> return false;
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapVerifiercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/HeapVerifier.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/HeapVerifier.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/HeapVerifier.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2014 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx"> #include "HeapIterationScope.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><span class="cx"> #include "JSObject.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapIncrementalSweepercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/IncrementalSweeper.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/IncrementalSweeper.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/IncrementalSweeper.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -50,7 +50,7 @@
</span><span class="cx"> #if USE(CF)
</span><span class="cx"> IncrementalSweeper::IncrementalSweeper(Heap* heap, CFRunLoopRef runLoop)
</span><span class="cx"> : HeapTimer(heap->vm(), runLoop)
</span><del>- , m_blocksToSweep(heap->m_blockSnapshot)
</del><ins>+ , m_currentAllocator(nullptr)
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -66,7 +66,7 @@
</span><span class="cx"> #elif PLATFORM(EFL)
</span><span class="cx"> IncrementalSweeper::IncrementalSweeper(Heap* heap)
</span><span class="cx"> : HeapTimer(heap->vm())
</span><del>- , m_blocksToSweep(heap->m_blockSnapshot)
</del><ins>+ , m_currentAllocator(nullptr)
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -86,7 +86,7 @@
</span><span class="cx"> #elif USE(GLIB)
</span><span class="cx"> IncrementalSweeper::IncrementalSweeper(Heap* heap)
</span><span class="cx"> : HeapTimer(heap->vm())
</span><del>- , m_blocksToSweep(heap->m_blockSnapshot)
</del><ins>+ , m_currentAllocator(nullptr)
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -121,19 +121,20 @@
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_blocksToSweep.clear();
</del><span class="cx"> cancelTimer();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool IncrementalSweeper::sweepNextBlock()
</span><span class="cx"> {
</span><del>- while (!m_blocksToSweep.isEmpty()) {
- MarkedBlock::Handle* block = m_blocksToSweep.takeLast();
- block->setIsOnBlocksToSweep(false);
-
- if (!block->needsSweeping())
- continue;
-
</del><ins>+ MarkedBlock::Handle* block = nullptr;
+
+ for (; m_currentAllocator; m_currentAllocator = m_currentAllocator->nextAllocator()) {
+ block = m_currentAllocator->findBlockToSweep();
+ if (block)
+ break;
+ }
+
+ if (block) {
</ins><span class="cx"> DeferGCForAWhile deferGC(m_vm->heap);
</span><span class="cx"> block->sweep();
</span><span class="cx"> m_vm->heap.objectSpace().freeOrShrinkBlock(block);
</span><span class="lines">@@ -146,11 +147,12 @@
</span><span class="cx"> void IncrementalSweeper::startSweeping()
</span><span class="cx"> {
</span><span class="cx"> scheduleTimer();
</span><ins>+ m_currentAllocator = m_vm->heap.objectSpace().firstAllocator();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void IncrementalSweeper::willFinishSweeping()
</span><span class="cx"> {
</span><del>- m_blocksToSweep.clear();
</del><ins>+ m_currentAllocator = nullptr;
</ins><span class="cx"> if (m_vm)
</span><span class="cx"> cancelTimer();
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapIncrementalSweeperh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/IncrementalSweeper.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/IncrementalSweeper.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/IncrementalSweeper.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -27,13 +27,12 @@
</span><span class="cx"> #define IncrementalSweeper_h
</span><span class="cx">
</span><span class="cx"> #include "HeapTimer.h"
</span><del>-#include "MarkedBlock.h"
</del><span class="cx"> #include <wtf/Vector.h>
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="cx"> class Heap;
</span><del>-class MarkedBlock;
</del><ins>+class MarkedAllocator;
</ins><span class="cx">
</span><span class="cx"> class IncrementalSweeper : public HeapTimer {
</span><span class="cx"> WTF_MAKE_FAST_ALLOCATED;
</span><span class="lines">@@ -56,7 +55,7 @@
</span><span class="cx"> void scheduleTimer();
</span><span class="cx"> void cancelTimer();
</span><span class="cx">
</span><del>- Vector<MarkedBlock::Handle*>& m_blocksToSweep;
</del><ins>+ MarkedAllocator* m_currentAllocator;
</ins><span class="cx"> #endif
</span><span class="cx"> };
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapLargeAllocationh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/LargeAllocation.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/LargeAllocation.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/LargeAllocation.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -70,9 +70,10 @@
</span><span class="cx">
</span><span class="cx"> bool isNewlyAllocated() const { return m_isNewlyAllocated; }
</span><span class="cx"> ALWAYS_INLINE bool isMarked() { return m_isMarked.load(std::memory_order_relaxed); }
</span><ins>+ ALWAYS_INLINE bool isMarked(HeapCell*) { return m_isMarked.load(std::memory_order_relaxed); }
+ ALWAYS_INLINE bool isMarkedConcurrently(HeapVersion, HeapCell*) { return m_isMarked.load(std::memory_order_relaxed); }
</ins><span class="cx"> bool isMarkedOrNewlyAllocated() { return isMarked() || isNewlyAllocated(); }
</span><span class="cx"> bool isMarkedOrNewlyAllocated(HeapCell*) { return isMarkedOrNewlyAllocated(); }
</span><del>- bool isMarkedDuringWeakVisiting(HeapVersion, HeapCell*) { return isMarked(); }
</del><span class="cx"> bool isLive() { return isMarkedOrNewlyAllocated(); }
</span><span class="cx">
</span><span class="cx"> bool hasValidCell() const { return m_hasValidCell; }
</span><span class="lines">@@ -106,8 +107,7 @@
</span><span class="cx">
</span><span class="cx"> const AllocatorAttributes& attributes() const { return m_attributes; }
</span><span class="cx">
</span><del>- void flipIfNecessary(uint64_t) { }
- void flipIfNecessaryDuringMarking(uint64_t) { }
</del><ins>+ void aboutToMark(HeapVersion) { }
</ins><span class="cx">
</span><span class="cx"> ALWAYS_INLINE bool testAndSetMarked()
</span><span class="cx"> {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedAllocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -26,10 +26,12 @@
</span><span class="cx"> #include "config.h"
</span><span class="cx"> #include "MarkedAllocator.h"
</span><span class="cx">
</span><ins>+#include "AllocationScope.h"
</ins><span class="cx"> #include "GCActivityCallback.h"
</span><span class="cx"> #include "Heap.h"
</span><span class="cx"> #include "IncrementalSweeper.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "MarkedBlockInlines.h"
</ins><span class="cx"> #include "SuperSampler.h"
</span><span class="cx"> #include "VM.h"
</span><span class="cx"> #include <wtf/CurrentTime.h>
</span><span class="lines">@@ -39,7 +41,6 @@
</span><span class="cx"> MarkedAllocator::MarkedAllocator(Heap* heap, MarkedSpace* markedSpace, size_t cellSize, const AllocatorAttributes& attributes)
</span><span class="cx"> : m_currentBlock(0)
</span><span class="cx"> , m_lastActiveBlock(0)
</span><del>- , m_nextBlockToSweep(nullptr)
</del><span class="cx"> , m_cellSize(static_cast<unsigned>(cellSize))
</span><span class="cx"> , m_attributes(attributes)
</span><span class="cx"> , m_heap(heap)
</span><span class="lines">@@ -50,11 +51,13 @@
</span><span class="cx"> bool MarkedAllocator::isPagedOut(double deadline)
</span><span class="cx"> {
</span><span class="cx"> unsigned itersSinceLastTimeCheck = 0;
</span><del>- MarkedBlock::Handle* block = m_blockList.begin();
- while (block) {
- block = filterNextBlock(block->next());
- if (block)
- block->flipIfNecessary(); // Forces us to touch the memory of the block, but has no semantic effect.
</del><ins>+ for (size_t index = 0; index < m_blocks.size(); ++index) {
+ MarkedBlock::Handle* block = m_blocks[index];
+ if (block) {
+ // Forces us to touch the memory of the block, but has no semantic effect.
+ if (block->needsFlip())
+ block->block().resetVersion();
+ }
</ins><span class="cx"> ++itersSinceLastTimeCheck;
</span><span class="cx"> if (itersSinceLastTimeCheck >= Heap::s_timeCheckResolution) {
</span><span class="cx"> double currentTime = WTF::monotonicallyIncreasingTime();
</span><span class="lines">@@ -66,84 +69,96 @@
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedAllocator::retire(MarkedBlock::Handle* block)
</del><ins>+bool MarkedAllocator::shouldStealEmptyBlocksFromOtherAllocators() const
</ins><span class="cx"> {
</span><del>- LockHolder locker(m_lock); // This will be called in parallel during GC.
- if (block == m_currentBlock) {
- // This happens when the mutator is running. We finished a full GC and marked too few things
- // to retire. Then we started allocating in this block. Then a barrier ran, which marked an
- // object in this block, which put it over the retirement threshold. It's OK to simply do
- // nothing in that case.
- return;
- }
- if (block == m_lastActiveBlock) {
- // This can easily happen during marking. It would be easy to handle this case, but it's
- // just as easy to ignore it.
- return;
- }
- RELEASE_ASSERT(block->isOnList());
- if (block == m_nextBlockToSweep)
- m_nextBlockToSweep = filterNextBlock(block->next());
- block->remove();
- m_retiredBlocks.push(block);
</del><ins>+ return !needsDestruction();
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-MarkedBlock::Handle* MarkedAllocator::filterNextBlock(MarkedBlock::Handle* block)
</del><ins>+MarkedBlock::Handle* MarkedAllocator::findEmptyBlockToSteal()
</ins><span class="cx"> {
</span><del>- if (block == m_blockList.end())
</del><ins>+ // Don't allow others to steal from us, if we wouldn't steal from others.
+ if (!shouldStealEmptyBlocksFromOtherAllocators())
</ins><span class="cx"> return nullptr;
</span><del>- return block;
</del><ins>+
+ m_emptyCursor = m_empty.findBit(m_emptyCursor, true);
+ if (m_emptyCursor >= m_blocks.size())
+ return nullptr;
+ return m_blocks[m_emptyCursor];
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedAllocator::setNextBlockToSweep(MarkedBlock::Handle* block)
</del><ins>+void MarkedAllocator::didConsumeFreeList()
</ins><span class="cx"> {
</span><del>- m_nextBlockToSweep = filterNextBlock(block);
</del><ins>+ if (m_currentBlock)
+ m_currentBlock->didConsumeFreeList();
+
+ setFreeList(FreeList());
+ m_currentBlock = nullptr;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void* MarkedAllocator::tryAllocateWithoutCollectingImpl()
</del><ins>+void* MarkedAllocator::tryAllocateWithoutCollecting()
</ins><span class="cx"> {
</span><span class="cx"> SuperSamplerScope superSamplerScope(false);
</span><span class="cx">
</span><del>- if (m_currentBlock) {
- ASSERT(m_currentBlock == m_nextBlockToSweep);
- m_currentBlock->didConsumeFreeList();
- setNextBlockToSweep(m_currentBlock->next());
- }
</del><ins>+ ASSERT(!m_currentBlock);
+ ASSERT(!m_freeList);
</ins><span class="cx">
</span><del>- setFreeList(FreeList());
-
- RELEASE_ASSERT(m_nextBlockToSweep != m_blockList.end());
-
- MarkedBlock::Handle* next;
- for (MarkedBlock::Handle*& block = m_nextBlockToSweep; block; block = next) {
- next = filterNextBlock(block->next());
-
- // It would be super weird if the blocks we are sweeping have anything allocated during this
- // cycle.
- ASSERT(!block->hasAnyNewlyAllocated());
</del><ins>+ for (;;) {
+ m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true);
+ if (m_allocationCursor >= m_blocks.size())
+ break;
</ins><span class="cx">
</span><del>- FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
-
- // It's possible to stumble on a complete-full block. Marking tries to retire these, but
- // that algorithm is racy and may forget to do it sometimes.
- if (freeList.allocationWillFail()) {
- ASSERT(block->isFreeListed());
- block->unsweepWithNoNewlyAllocated();
- ASSERT(block->isMarked());
- retire(block);
- continue;
- }
</del><ins>+ setIsCanAllocateButNotEmpty(m_allocationCursor, false);
</ins><span class="cx">
</span><del>- m_currentBlock = block;
- setFreeList(freeList);
- break;
</del><ins>+ if (void* result = tryAllocateIn(m_blocks[m_allocationCursor]))
+ return result;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- if (!m_freeList) {
- m_currentBlock = 0;
- return 0;
</del><ins>+ if (Options::stealEmptyBlocksFromOtherAllocators()
+ && shouldStealEmptyBlocksFromOtherAllocators()) {
+ if (MarkedBlock::Handle* block = m_markedSpace->findEmptyBlockToSteal()) {
+ block->sweep();
+
+ // It's good that this clears canAllocateButNotEmpty as well as all other bits,
+ // because there is a remote chance that a block may have both canAllocateButNotEmpty
+ // and empty set at the same time.
+ block->removeFromAllocator();
+ addBlock(block);
+ return allocateIn(block);
+ }
</ins><span class="cx"> }
</span><ins>+
+ return nullptr;
+}
</ins><span class="cx">
</span><ins>+void* MarkedAllocator::allocateIn(MarkedBlock::Handle* block)
+{
+ void* result = tryAllocateIn(block);
+ RELEASE_ASSERT(result);
+ return result;
+}
+
+void* MarkedAllocator::tryAllocateIn(MarkedBlock::Handle* block)
+{
+ ASSERT(block);
+ ASSERT(!block->hasAnyNewlyAllocated());
+ ASSERT(!block->isFreeListed());
+
+ FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
+
+ // It's possible to stumble on a completely full block. Marking tries to retire these, but
+ // that algorithm is racy and may forget to do it sometimes.
+ if (freeList.allocationWillFail()) {
+ ASSERT(block->isFreeListed());
+ block->unsweepWithNoNewlyAllocated();
+ ASSERT(!block->isFreeListed());
+ ASSERT(!isEmpty(block));
+ ASSERT(!isCanAllocateButNotEmpty(block));
+ return nullptr;
+ }
+
+ m_currentBlock = block;
+ setFreeList(freeList);
+
</ins><span class="cx"> void* result;
</span><span class="cx"> if (m_freeList.remaining) {
</span><span class="cx"> unsigned cellSize = m_cellSize;
</span><span class="lines">@@ -155,21 +170,11 @@
</span><span class="cx"> result = head;
</span><span class="cx"> }
</span><span class="cx"> RELEASE_ASSERT(result);
</span><ins>+ setIsEden(m_currentBlock, true);
</ins><span class="cx"> m_markedSpace->didAllocateInBlock(m_currentBlock);
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void* MarkedAllocator::tryAllocateWithoutCollecting()
-{
- ASSERT(!m_heap->isBusy());
- m_heap->m_operationInProgress = Allocation;
- void* result = tryAllocateWithoutCollectingImpl();
-
- m_heap->m_operationInProgress = NoOperation;
- ASSERT(result || !m_currentBlock);
- return result;
-}
-
</del><span class="cx"> ALWAYS_INLINE void MarkedAllocator::doTestCollectionsIfNeeded()
</span><span class="cx"> {
</span><span class="cx"> if (!Options::slowPathAllocsBetweenGCs())
</span><span class="lines">@@ -206,19 +211,17 @@
</span><span class="cx"> ASSERT(!m_markedSpace->isIterating());
</span><span class="cx"> m_heap->didAllocate(m_freeList.originalSize);
</span><span class="cx">
</span><ins>+ didConsumeFreeList();
+
+ m_heap->collectIfNecessaryOrDefer();
+
+ AllocationScope allocationScope(*m_heap);
+
</ins><span class="cx"> void* result = tryAllocateWithoutCollecting();
</span><span class="cx">
</span><span class="cx"> if (LIKELY(result != 0))
</span><span class="cx"> return result;
</span><span class="cx">
</span><del>- if (m_heap->collectIfNecessaryOrDefer()) {
- result = tryAllocateWithoutCollecting();
- if (result)
- return result;
- }
-
- ASSERT(!m_heap->shouldCollect());
-
</del><span class="cx"> MarkedBlock::Handle* block = tryAllocateBlock();
</span><span class="cx"> if (!block) {
</span><span class="cx"> if (crashOnFailure)
</span><span class="lines">@@ -227,8 +230,7 @@
</span><span class="cx"> return nullptr;
</span><span class="cx"> }
</span><span class="cx"> addBlock(block);
</span><del>-
- result = tryAllocateWithoutCollecting();
</del><ins>+ result = allocateIn(block);
</ins><span class="cx"> ASSERT(result);
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="lines">@@ -249,37 +251,75 @@
</span><span class="cx"> MarkedBlock::Handle* MarkedAllocator::tryAllocateBlock()
</span><span class="cx"> {
</span><span class="cx"> SuperSamplerScope superSamplerScope(false);
</span><del>- return MarkedBlock::tryCreate(*m_heap, this, m_cellSize, m_attributes);
</del><ins>+
+ MarkedBlock::Handle* handle = MarkedBlock::tryCreate(*m_heap);
+ if (!handle)
+ return nullptr;
+
+ m_markedSpace->didAddBlock(handle);
+
+ return handle;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedAllocator::addBlock(MarkedBlock::Handle* block)
</span><span class="cx"> {
</span><del>- ASSERT(!m_currentBlock);
- ASSERT(!m_freeList);
</del><ins>+ size_t index;
+ if (m_freeBlockIndices.isEmpty()) {
+ index = m_blocks.size();
+
+ size_t oldCapacity = m_blocks.capacity();
+ m_blocks.append(block);
+ if (m_blocks.capacity() != oldCapacity) {
+ forEachBitVector(
+ [&] (FastBitVector& vector) {
+ ASSERT_UNUSED(vector, vector.numBits() == oldCapacity);
+ });
+
+ ASSERT(m_blocks.capacity() > oldCapacity);
+
+ forEachBitVector(
+ [&] (FastBitVector& vector) {
+ vector.resize(m_blocks.capacity());
+ });
+ }
+ } else {
+ index = m_freeBlockIndices.takeLast();
+ ASSERT(!m_blocks[index]);
+ m_blocks[index] = block;
+ }
</ins><span class="cx">
</span><del>- m_blockList.append(block);
- setNextBlockToSweep(block);
- m_markedSpace->didAddBlock(block);
</del><ins>+ forEachBitVector(
+ [&] (FastBitVector& vector) {
+ ASSERT_UNUSED(vector, !vector[index]);
+ });
+
+ // This is the point at which the block learns of its cellSize() and attributes().
+ block->didAddToAllocator(this, index);
+
+ setIsLive(index, true);
+ setIsEmpty(index, true);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedAllocator::removeBlock(MarkedBlock::Handle* block)
</span><span class="cx"> {
</span><del>- if (m_currentBlock == block) {
- m_currentBlock = filterNextBlock(m_currentBlock->next());
- setFreeList(FreeList());
- }
- if (m_nextBlockToSweep == block)
- setNextBlockToSweep(m_nextBlockToSweep->next());
</del><ins>+ ASSERT(block->allocator() == this);
+ ASSERT(m_blocks[block->index()] == block);
</ins><span class="cx">
</span><del>- block->willRemoveBlock();
- m_blockList.remove(block);
</del><ins>+ m_blocks[block->index()] = nullptr;
+ m_freeBlockIndices.append(block->index());
+
+ forEachBitVector(
+ [&] (FastBitVector& vector) {
+ vector[block->index()] = false;
+ });
+
+ block->didRemoveFromAllocator();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedAllocator::stopAllocating()
</span><span class="cx"> {
</span><del>- if (m_heap->operationInProgress() == FullCollection)
- m_blockList.takeFrom(m_retiredBlocks);
-
</del><ins>+ if (false)
+ dataLog(RawPointer(this), ": MarkedAllocator::stopAllocating!\n");
</ins><span class="cx"> ASSERT(!m_lastActiveBlock);
</span><span class="cx"> if (!m_currentBlock) {
</span><span class="cx"> ASSERT(!m_freeList);
</span><span class="lines">@@ -292,29 +332,27 @@
</span><span class="cx"> m_freeList = FreeList();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedAllocator::reset()
</del><ins>+void MarkedAllocator::prepareForAllocation()
</ins><span class="cx"> {
</span><del>- m_lastActiveBlock = 0;
- m_currentBlock = 0;
</del><ins>+ m_lastActiveBlock = nullptr;
+ m_currentBlock = nullptr;
</ins><span class="cx"> setFreeList(FreeList());
</span><span class="cx">
</span><del>- setNextBlockToSweep(m_blockList.begin());
</del><ins>+ m_allocationCursor = 0;
+ m_emptyCursor = 0;
+ m_unsweptCursor = 0;
+
+ m_eden.clearAll();
</ins><span class="cx">
</span><span class="cx"> if (UNLIKELY(Options::useImmortalObjects())) {
</span><del>- MarkedBlock::Handle* next;
- for (MarkedBlock::Handle*& block = m_nextBlockToSweep; block; block = next) {
- next = filterNextBlock(block->next());
-
- FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
- block->zap(freeList);
- retire(block);
- }
</del><ins>+ // FIXME: Make this work again.
+ // https://bugs.webkit.org/show_bug.cgi?id=162296
+ RELEASE_ASSERT_NOT_REACHED();
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedAllocator::lastChanceToFinalize()
</span><span class="cx"> {
</span><del>- m_blockList.takeFrom(m_retiredBlocks);
</del><span class="cx"> forEachBlock(
</span><span class="cx"> [&] (MarkedBlock::Handle* block) {
</span><span class="cx"> block->lastChanceToFinalize();
</span><span class="lines">@@ -326,4 +364,121 @@
</span><span class="cx"> m_freeList = freeList;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedAllocator::resumeAllocating()
+{
+ if (!m_lastActiveBlock)
+ return;
+
+ m_freeList = m_lastActiveBlock->resumeAllocating();
+ m_currentBlock = m_lastActiveBlock;
+ m_lastActiveBlock = nullptr;
+}
+
+void MarkedAllocator::beginMarkingForFullCollection()
+{
+ // Mark bits are sticky and so is our summary of mark bits. We only clear these during full
+ // collections, so if you survived the last collection you will survive the next one so long
+ // as the next one is eden.
+ m_markingNotEmpty.clearAll();
+ m_markingRetired.clearAll();
+}
+
+void MarkedAllocator::endMarking()
+{
+ m_allocated.clearAll();
+
+ // It's surprising and frustrating to comprehend, but the end-of-marking flip does not need to
+ // know what kind of collection it is. That knowledge is already encoded in the m_markingXYZ
+ // vectors.
+
+ if (needsDestruction()) {
+ // If blocks need destruction then nothing is empty! This is a correct assertion but may
+ // become wrong once we go full concurrent: when we create a new block, it will flicker
+ // into the empty set for a tiny moment. On the other hand, this code is likely to be run
+ // in stopTheWorld.
+ ASSERT(m_empty.isEmpty());
+ m_canAllocateButNotEmpty = m_live & ~m_markingRetired;
+ return;
+ }
+
+ m_empty = m_live & ~m_markingNotEmpty;
+ m_canAllocateButNotEmpty = m_live & m_markingNotEmpty & ~m_markingRetired;
+
+ if (false) {
+ dataLog("Bits for ", m_cellSize, ", ", m_attributes, " after endMarking:\n");
+ dumpBits(WTF::dataFile());
+ }
+}
+
+void MarkedAllocator::snapshotUnsweptForEdenCollection()
+{
+ m_unswept |= m_eden;
+}
+
+void MarkedAllocator::snapshotUnsweptForFullCollection()
+{
+ m_unswept = m_live;
+}
+
+MarkedBlock::Handle* MarkedAllocator::findBlockToSweep()
+{
+ m_unsweptCursor = m_unswept.findBit(m_unsweptCursor, true);
+ if (m_unsweptCursor >= m_blocks.size())
+ return nullptr;
+ return m_blocks[m_unsweptCursor];
+}
+
+void MarkedAllocator::sweep()
+{
+ m_unswept.forEachSetBit(
+ [&] (size_t index) {
+ m_blocks[index]->sweep();
+ });
+}
+
+void MarkedAllocator::shrink()
+{
+ m_empty.forEachSetBit(
+ [&] (size_t index) {
+ m_markedSpace->freeBlock(m_blocks[index]);
+ });
+}
+
+void MarkedAllocator::assertNoUnswept()
+{
+ if (ASSERT_DISABLED)
+ return;
+
+ if (m_unswept.isEmpty())
+ return;
+
+ dataLog("Assertion failed: unswept not empty in ", *this, ".\n");
+ dumpBits();
+ ASSERT_NOT_REACHED();
+}
+
+void MarkedAllocator::dump(PrintStream& out) const
+{
+ out.print(RawPointer(this), ":", m_cellSize, "/", m_attributes);
+}
+
+void MarkedAllocator::dumpBits(PrintStream& out)
+{
+ unsigned maxNameLength = 0;
+ forEachBitVectorWithName(
+ [&] (FastBitVector&, const char* name) {
+ unsigned length = strlen(name);
+ maxNameLength = std::max(maxNameLength, length);
+ });
+
+ forEachBitVectorWithName(
+ [&] (FastBitVector& vector, const char* name) {
+ out.print(" ", name, ": ");
+ for (unsigned i = maxNameLength - strlen(name); i--;)
+ out.print(" ");
+ out.print(vector, "\n");
+ });
+}
+
</ins><span class="cx"> } // namespace JSC
</span><ins>+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedAllocatorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedAllocator.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedAllocator.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedAllocator.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,7 +29,9 @@
</span><span class="cx"> #include "AllocatorAttributes.h"
</span><span class="cx"> #include "FreeList.h"
</span><span class="cx"> #include "MarkedBlock.h"
</span><ins>+#include <wtf/FastBitVector.h>
</ins><span class="cx"> #include <wtf/SentinelLinkedList.h>
</span><ins>+#include <wtf/Vector.h>
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="lines">@@ -37,6 +39,95 @@
</span><span class="cx"> class MarkedSpace;
</span><span class="cx"> class LLIntOffsetsExtractor;
</span><span class="cx">
</span><ins>+#define FOR_EACH_MARKED_ALLOCATOR_BIT(macro) \
+ macro(live, Live) /* The set of block indices that have actual blocks. */\
+ macro(empty, Empty) /* The set of all blocks that have no live objects and nothing to destroy. */ \
+ macro(allocated, Allocated) /* The set of allblocks that are full of live objects. */\
+ macro(canAllocateButNotEmpty, CanAllocateButNotEmpty) /* The set of all blocks are neither empty nor retired (i.e. are more than minMarkedBlockUtilization full). */ \
+ macro(eden, Eden) /* The set of all blocks that have new objects since the last GC. */\
+ macro(unswept, Unswept) /* The set of all blocks that could be swept by the incremental sweeper. */\
+ \
+ /* These are computed during marking. */\
+ macro(markingNotEmpty, MarkingNotEmpty) /* The set of all blocks that are not empty. */ \
+ macro(markingRetired, MarkingRetired) /* The set of all blocks that are retired. */
+
+// FIXME: We defined canAllocateButNotEmpty and empty to be exclusive:
+//
+// canAllocateButNotEmpty & empty == 0
+//
+// Instead of calling it canAllocate and making it inclusive:
+//
+// canAllocate & empty == empty
+//
+// The latter is probably better. I'll leave it to a future bug to fix that, since breathing on
+// this code leads to regressions for days, and it's not clear that making this change would
+// improve perf since it would not change the collector's behavior, and either way the allocator
+// has to look at both bitvectors.
+// https://bugs.webkit.org/show_bug.cgi?id=162121
+
+// Note that this collector supports overlapping allocator state with marking state, since in a
+// concurrent collector you allow allocation while marking is running. So it's best to visualize a
+// full mutable->eden collect->mutate->full collect cycle and see how the bits above get affected.
+// The example below tries to be exhaustive about what happens to the bits, but omits a lot of
+// things that happen to other state.
+//
+// Create allocator
+// - all bits are empty
+// Start allocating in some block
+// - allocate the block and set the live bit.
+// - the empty bit for the block flickers on and then gets immediately cleared by sweeping.
+// - set the eden bit.
+// Finish allocating in that block
+// - set the allocated bit.
+// Do that to a lot of blocks and then start an eden collection.
+// - beginMarking() has nothing to do.
+// - by default we have cleared markingNotEmpty/markingRetired bits.
+// - marking builds up markingNotEmpty/markingRetired bits.
+// We do endMarking()
+// - clear all allocated bits.
+// - for destructor blocks: fragmented = live & ~markingRetired
+// - for non-destructor blocks:
+// empty = live & ~markingNotEmpty
+// fragmented = live & markingNotEmpty & ~markingRetired
+// Snapshotting.
+// - unswept |= eden
+// Prepare for allocation.
+// - clear eden
+// Finish collection.
+// Allocate in some block that had some free and some live objects.
+// - clear the canAllocateButNotEmpty bit
+// - clear the unswept bit
+// - set the eden bit
+// Finish allocating (set the allocated bit).
+// Allocate in some block that was completely empty.
+// - clear the empty bit
+// - clear the unswept bit
+// - set the eden bit.
+// Finish allocating (set the allocated bit).
+// Allocate in some block that was completely empty in another allocator.
+// - clear the empty bit
+// - clear all bits in that allocator
+// - set the live bit in another allocator and the empty bit.
+// - clear the empty, unswept bits.
+// - set the eden bit.
+// Finish allocating (set the allocated bit).
+// Start a full collection.
+// - beginMarking() clears markingNotEmpty, markingRetired
+// - the heap version is incremented
+// - marking rebuilds markingNotEmpty/markingretired bits.
+// We do endMarking()
+// - clear all allocated bits.
+// - set canAllocateButNotEmpty/empty the same way as in eden collection.
+// Snapshotting.
+// - unswept = live
+// prepare for allocation.
+// - clear eden.
+// Finish collection.
+//
+// Notice how in this scheme, the empty/canAllocateButNotEmpty state stays separate from the
+// markingNotEmpty/markingRetired state. This is one step towards having separated allocation and
+// marking state.
+
</ins><span class="cx"> class MarkedAllocator {
</span><span class="cx"> friend class LLIntOffsetsExtractor;
</span><span class="cx">
</span><span class="lines">@@ -46,9 +137,16 @@
</span><span class="cx">
</span><span class="cx"> MarkedAllocator(Heap*, MarkedSpace*, size_t cellSize, const AllocatorAttributes&);
</span><span class="cx"> void lastChanceToFinalize();
</span><del>- void reset();
</del><ins>+ void prepareForAllocation();
</ins><span class="cx"> void stopAllocating();
</span><span class="cx"> void resumeAllocating();
</span><ins>+ void beginMarkingForFullCollection();
+ void endMarking();
+ void snapshotUnsweptForEdenCollection();
+ void snapshotUnsweptForFullCollection();
+ void sweep();
+ void shrink();
+ void assertNoUnswept();
</ins><span class="cx"> size_t cellSize() const { return m_cellSize; }
</span><span class="cx"> const AllocatorAttributes& attributes() const { return m_attributes; }
</span><span class="cx"> bool needsDestruction() const { return m_attributes.destruction == NeedsDestruction; }
</span><span class="lines">@@ -72,35 +170,89 @@
</span><span class="cx"> bool isPagedOut(double deadline);
</span><span class="cx">
</span><span class="cx"> static size_t blockSizeForBytes(size_t);
</span><ins>+
+#define MARKED_ALLOCATOR_BIT_ACCESSORS(lowerBitName, capitalBitName) \
+ bool is ## capitalBitName(size_t index) const { return m_ ## lowerBitName[index]; } \
+ bool is ## capitalBitName(MarkedBlock::Handle* block) const { return is ## capitalBitName(block->index()); } \
+ void setIs ## capitalBitName(size_t index, bool value) { m_ ## lowerBitName[index] = value; } \
+ void setIs ## capitalBitName(MarkedBlock::Handle* block, bool value) { setIs ## capitalBitName(block->index(), value); } \
+ bool atomicSetAndCheckIs ## capitalBitName(size_t index, bool value) { return m_ ## lowerBitName.atomicSetAndCheck(index, value); } \
+ bool atomicSetAndCheckIs ## capitalBitName(MarkedBlock::Handle* block, bool value) { return m_ ## lowerBitName.atomicSetAndCheck(block->index(), value); }
+ FOR_EACH_MARKED_ALLOCATOR_BIT(MARKED_ALLOCATOR_BIT_ACCESSORS)
+#undef MARKED_ALLOCATOR_BIT_ACCESSORS
+
+ template<typename Func>
+ void forEachBitVector(const Func& func)
+ {
+#define MARKED_ALLOCATOR_BIT_CALLBACK(lowerBitName, capitalBitName) \
+ func(m_ ## lowerBitName);
+ FOR_EACH_MARKED_ALLOCATOR_BIT(MARKED_ALLOCATOR_BIT_CALLBACK);
+#undef MARKED_ALLOCATOR_BIT_CALLBACK
+ }
+
+ template<typename Func>
+ void forEachBitVectorWithName(const Func& func)
+ {
+#define MARKED_ALLOCATOR_BIT_CALLBACK(lowerBitName, capitalBitName) \
+ func(m_ ## lowerBitName, #capitalBitName);
+ FOR_EACH_MARKED_ALLOCATOR_BIT(MARKED_ALLOCATOR_BIT_CALLBACK);
+#undef MARKED_ALLOCATOR_BIT_CALLBACK
+ }
+
+ MarkedAllocator* nextAllocator() const { return m_nextAllocator; }
+
+ // MarkedSpace calls this during init.
+ void setNextAllocator(MarkedAllocator* allocator) { m_nextAllocator = allocator; }
+
+ MarkedBlock::Handle* findEmptyBlockToSteal();
+
+ MarkedBlock::Handle* findBlockToSweep();
+
+ void dump(PrintStream&) const;
+ void dumpBits(PrintStream& = WTF::dataFile());
</ins><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> friend class MarkedBlock;
</span><span class="cx">
</span><ins>+ bool shouldStealEmptyBlocksFromOtherAllocators() const;
+
</ins><span class="cx"> JS_EXPORT_PRIVATE void* allocateSlowCase();
</span><span class="cx"> JS_EXPORT_PRIVATE void* tryAllocateSlowCase();
</span><span class="cx"> void* allocateSlowCaseImpl(bool crashOnFailure);
</span><ins>+ void didConsumeFreeList();
</ins><span class="cx"> void* tryAllocateWithoutCollecting();
</span><del>- void* tryAllocateWithoutCollectingImpl();
</del><span class="cx"> MarkedBlock::Handle* tryAllocateBlock();
</span><ins>+ void* tryAllocateIn(MarkedBlock::Handle*);
+ void* allocateIn(MarkedBlock::Handle*);
</ins><span class="cx"> ALWAYS_INLINE void doTestCollectionsIfNeeded();
</span><del>- void retire(MarkedBlock::Handle*);
</del><span class="cx">
</span><span class="cx"> void setFreeList(const FreeList&);
</span><span class="cx">
</span><del>- MarkedBlock::Handle* filterNextBlock(MarkedBlock::Handle*);
- void setNextBlockToSweep(MarkedBlock::Handle*);
</del><ins>+ FreeList m_freeList;
</ins><span class="cx">
</span><del>- FreeList m_freeList;
</del><ins>+ Vector<MarkedBlock::Handle*> m_blocks;
+ Vector<unsigned> m_freeBlockIndices;
+
+#define MARKED_ALLOCATOR_BIT_DECLARATION(lowerBitName, capitalBitName) \
+ FastBitVector m_ ## lowerBitName;
+ FOR_EACH_MARKED_ALLOCATOR_BIT(MARKED_ALLOCATOR_BIT_DECLARATION)
+#undef MARKED_ALLOCATOR_BIT_DECLARATION
+
+ // After you do something to a block based on one of these cursors, you clear the bit in the
+ // corresponding bitvector and leave the cursor where it was.
+ size_t m_allocationCursor { 0 }; // Points to the next block that is a candidate for allocation.
+ size_t m_emptyCursor { 0 }; // Points to the next block that is a candidate for empty allocation (allocating in empty blocks).
+ size_t m_unsweptCursor { 0 }; // Points to the next block that is a candidate for incremental sweeping.
+
</ins><span class="cx"> MarkedBlock::Handle* m_currentBlock;
</span><span class="cx"> MarkedBlock::Handle* m_lastActiveBlock;
</span><del>- MarkedBlock::Handle* m_nextBlockToSweep;
- SentinelLinkedList<MarkedBlock::Handle, BasicRawSentinelNode<MarkedBlock::Handle>> m_blockList;
- SentinelLinkedList<MarkedBlock::Handle, BasicRawSentinelNode<MarkedBlock::Handle>> m_retiredBlocks;
</del><ins>+
</ins><span class="cx"> Lock m_lock;
</span><span class="cx"> unsigned m_cellSize;
</span><span class="cx"> AllocatorAttributes m_attributes;
</span><span class="cx"> Heap* m_heap;
</span><span class="cx"> MarkedSpace* m_markedSpace;
</span><ins>+ MarkedAllocator* m_nextAllocator;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> inline ptrdiff_t MarkedAllocator::offsetOfFreeList()
</span><span class="lines">@@ -149,20 +301,12 @@
</span><span class="cx"> return head;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void MarkedAllocator::resumeAllocating()
-{
- if (!m_lastActiveBlock)
- return;
-
- m_freeList = m_lastActiveBlock->resumeAllocating();
- m_currentBlock = m_lastActiveBlock;
- m_lastActiveBlock = 0;
-}
-
</del><span class="cx"> template <typename Functor> inline void MarkedAllocator::forEachBlock(const Functor& functor)
</span><span class="cx"> {
</span><del>- m_blockList.forEach(functor);
- m_retiredBlocks.forEach(functor);
</del><ins>+ m_live.forEachSetBit(
+ [&] (size_t index) {
+ functor(m_blocks[index]);
+ });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,6 +29,7 @@
</span><span class="cx"> #include "JSCell.h"
</span><span class="cx"> #include "JSDestructibleObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "MarkedBlockInlines.h"
</ins><span class="cx"> #include "SuperSampler.h"
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -36,7 +37,7 @@
</span><span class="cx"> static const bool computeBalance = false;
</span><span class="cx"> static size_t balance;
</span><span class="cx">
</span><del>-MarkedBlock::Handle* MarkedBlock::tryCreate(Heap& heap, MarkedAllocator* allocator, size_t cellSize, const AllocatorAttributes& attributes)
</del><ins>+MarkedBlock::Handle* MarkedBlock::tryCreate(Heap& heap)
</ins><span class="cx"> {
</span><span class="cx"> if (computeBalance) {
</span><span class="cx"> balance++;
</span><span class="lines">@@ -48,16 +49,11 @@
</span><span class="cx"> return nullptr;
</span><span class="cx"> if (scribbleFreeCells())
</span><span class="cx"> scribble(blockSpace, blockSize);
</span><del>- return new Handle(heap, allocator, cellSize, attributes, blockSpace);
</del><ins>+ return new Handle(heap, blockSpace);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-MarkedBlock::Handle::Handle(Heap& heap, MarkedAllocator* allocator, size_t cellSize, const AllocatorAttributes& attributes, void* blockSpace)
- : m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
- , m_endAtom(atomsPerBlock - m_atomsPerCell + 1)
- , m_attributes(attributes)
- , m_state(New) // All cells start out unmarked.
- , m_allocator(allocator)
- , m_weakSet(allocator->heap()->vm(), CellContainer())
</del><ins>+MarkedBlock::Handle::Handle(Heap& heap, void* blockSpace)
+ : m_weakSet(heap.vm(), CellContainer())
</ins><span class="cx"> {
</span><span class="cx"> m_block = new (NotNull, blockSpace) MarkedBlock(*heap.vm(), *this);
</span><span class="cx">
</span><span class="lines">@@ -64,10 +60,6 @@
</span><span class="cx"> m_weakSet.setContainer(*m_block);
</span><span class="cx">
</span><span class="cx"> heap.didAllocateBlock(blockSize);
</span><del>- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
- ASSERT(allocator);
- if (m_attributes.cellKind != HeapCell::JSCell)
- RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> MarkedBlock::Handle::~Handle()
</span><span class="lines">@@ -78,6 +70,7 @@
</span><span class="cx"> if (!(balance % 10))
</span><span class="cx"> dataLog("MarkedBlock Balance: ", balance, "\n");
</span><span class="cx"> }
</span><ins>+ removeFromAllocator();
</ins><span class="cx"> m_block->~MarkedBlock();
</span><span class="cx"> fastAlignedFree(m_block);
</span><span class="cx"> heap.didFreeBlock(blockSize);
</span><span class="lines">@@ -84,39 +77,28 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> MarkedBlock::MarkedBlock(VM& vm, Handle& handle)
</span><del>- : m_needsDestruction(handle.needsDestruction())
- , m_version(vm.heap.objectSpace().version())
</del><ins>+ : m_version(MarkedSpace::nullVersion)
</ins><span class="cx"> , m_handle(handle)
</span><span class="cx"> , m_vm(&vm)
</span><span class="cx"> {
</span><del>- unsigned cellsPerBlock = MarkedSpace::blockPayload / handle.cellSize();
- double markCountBias = -(Options::minMarkedBlockUtilization() * cellsPerBlock);
-
- // The mark count bias should be comfortably within this range.
- RELEASE_ASSERT(markCountBias > static_cast<double>(std::numeric_limits<int16_t>::min()));
- RELEASE_ASSERT(markCountBias < 0);
-
- m_markCountBias = static_cast<int16_t>(markCountBias);
-
- m_biasedMarkCount = m_markCountBias; // This means we haven't marked anything yet.
</del><span class="cx"> }
</span><span class="cx">
</span><del>-template<MarkedBlock::BlockState blockState, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
</del><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode, MarkedBlock::Handle::FlipMode flipMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::specializedSweep()
</span><span class="cx"> {
</span><ins>+ RELEASE_ASSERT(!(destructionMode == DoesNotNeedDestruction && sweepMode == SweepOnly));
+
</ins><span class="cx"> SuperSamplerScope superSamplerScope(false);
</span><del>- ASSERT(blockState == New || blockState == Marked);
- ASSERT(!(destructionMode == DoesNotNeedDestruction && sweepMode == SweepOnly));
-
- assertFlipped();
</del><ins>+
</ins><span class="cx"> MarkedBlock& block = this->block();
</span><span class="cx">
</span><del>- bool isNewBlock = blockState == New;
- bool isEmptyBlock = !block.hasAnyMarked()
- && newlyAllocatedMode == DoesNotHaveNewlyAllocated
- && destructionMode == DoesNotNeedDestruction;
- if (Options::useBumpAllocator() && (isNewBlock || isEmptyBlock)) {
- ASSERT(block.m_marks.isEmpty());
</del><ins>+ if (false)
+ dataLog(RawPointer(this), ": MarkedBlock::Handle::specializedSweep!\n");
+
+ if (Options::useBumpAllocator()
+ && emptyMode == IsEmpty
+ && newlyAllocatedMode == DoesNotHaveNewlyAllocated) {
+ ASSERT(flipMode == NeedsFlip);
</ins><span class="cx">
</span><span class="cx"> char* startOfLastCell = static_cast<char*>(cellAlign(block.atoms() + m_endAtom - 1));
</span><span class="cx"> char* payloadEnd = startOfLastCell + cellSize();
</span><span class="lines">@@ -124,7 +106,10 @@
</span><span class="cx"> char* payloadBegin = bitwise_cast<char*>(block.atoms() + firstAtom());
</span><span class="cx"> if (scribbleMode == Scribble)
</span><span class="cx"> scribble(payloadBegin, payloadEnd - payloadBegin);
</span><del>- m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Marked);
</del><ins>+ if (sweepMode == SweepToFreeList)
+ setIsFreeListed();
+ else
+ m_allocator->setIsEmpty(this, true);
</ins><span class="cx"> FreeList result = FreeList::bump(payloadEnd, payloadEnd - payloadBegin);
</span><span class="cx"> if (false)
</span><span class="cx"> dataLog("Quickly swept block ", RawPointer(this), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
</span><span class="lines">@@ -136,15 +121,18 @@
</span><span class="cx"> // order of the free list.
</span><span class="cx"> FreeCell* head = 0;
</span><span class="cx"> size_t count = 0;
</span><ins>+ bool isEmpty = true;
</ins><span class="cx"> for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
</span><del>- if (blockState == Marked
- && (block.m_marks.get(i)
- || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated->get(i))))
</del><ins>+ if (emptyMode == NotEmpty
+ && ((flipMode == DoesNotNeedFlip && block.m_marks.get(i))
+ || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated->get(i)))) {
+ isEmpty = false;
</ins><span class="cx"> continue;
</span><del>-
</del><ins>+ }
+
</ins><span class="cx"> HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&block.atoms()[i]);
</span><span class="cx">
</span><del>- if (destructionMode == NeedsDestruction && blockState != New)
</del><ins>+ if (destructionMode == NeedsDestruction && emptyMode == NotEmpty)
</ins><span class="cx"> static_cast<JSCell*>(cell)->callDestructor(*vm());
</span><span class="cx">
</span><span class="cx"> if (sweepMode == SweepToFreeList) {
</span><span class="lines">@@ -163,7 +151,10 @@
</span><span class="cx"> m_newlyAllocated = nullptr;
</span><span class="cx">
</span><span class="cx"> FreeList result = FreeList::list(head, count * cellSize());
</span><del>- m_state = (sweepMode == SweepToFreeList ? FreeListed : Marked);
</del><ins>+ if (sweepMode == SweepToFreeList)
+ setIsFreeListed();
+ else if (isEmpty)
+ m_allocator->setIsEmpty(this, true);
</ins><span class="cx"> if (false)
</span><span class="cx"> dataLog("Slowly swept block ", RawPointer(&block), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
</span><span class="cx"> return result;
</span><span class="lines">@@ -171,15 +162,20 @@
</span><span class="cx">
</span><span class="cx"> FreeList MarkedBlock::Handle::sweep(SweepMode sweepMode)
</span><span class="cx"> {
</span><del>- flipIfNecessary();
</del><ins>+ m_allocator->setIsUnswept(this, false);
</ins><span class="cx">
</span><del>- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
-
</del><span class="cx"> m_weakSet.sweep();
</span><span class="cx">
</span><span class="cx"> if (sweepMode == SweepOnly && m_attributes.destruction == DoesNotNeedDestruction)
</span><span class="cx"> return FreeList();
</span><span class="cx">
</span><ins>+ if (UNLIKELY(m_isFreeListed)) {
+ RELEASE_ASSERT(sweepMode == SweepToFreeList);
+ return FreeList();
+ }
+
+ ASSERT(!m_allocator->isAllocated(this));
+
</ins><span class="cx"> if (m_attributes.destruction == NeedsDestruction)
</span><span class="cx"> return sweepHelperSelectScribbleMode<NeedsDestruction>(sweepMode);
</span><span class="cx"> return sweepHelperSelectScribbleMode<DoesNotNeedDestruction>(sweepMode);
</span><span class="lines">@@ -189,49 +185,61 @@
</span><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectScribbleMode(SweepMode sweepMode)
</span><span class="cx"> {
</span><span class="cx"> if (scribbleFreeCells())
</span><del>- return sweepHelperSelectStateAndSweepMode<destructionMode, Scribble>(sweepMode);
- return sweepHelperSelectStateAndSweepMode<destructionMode, DontScribble>(sweepMode);
</del><ins>+ return sweepHelperSelectEmptyMode<destructionMode, Scribble>(sweepMode);
+ return sweepHelperSelectEmptyMode<destructionMode, DontScribble>(sweepMode);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</span><del>-FreeList MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode(SweepMode sweepMode)
</del><ins>+FreeList MarkedBlock::Handle::sweepHelperSelectEmptyMode(SweepMode sweepMode)
</ins><span class="cx"> {
</span><del>- switch (m_state) {
- case New:
- ASSERT(sweepMode == SweepToFreeList);
- return specializedSweep<New, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
- case FreeListed:
- // Happens when a block transitions to fully allocated.
- ASSERT(sweepMode == SweepToFreeList);
- return FreeList();
- case Allocated:
- RELEASE_ASSERT_NOT_REACHED();
- return FreeList();
- case Marked:
- if (m_newlyAllocated) {
- return sweepMode == SweepToFreeList
- ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, HasNewlyAllocated>()
- : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, HasNewlyAllocated>();
- } else {
- return sweepMode == SweepToFreeList
- ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>()
- : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
- }
- }
- RELEASE_ASSERT_NOT_REACHED();
- return FreeList();
</del><ins>+ // It's not obvious, but this is the only way to know if the block is empty. It's the only
+ // bit that captures these caveats:
+ // - It's true when the block is freshly allocated.
+ // - It's true if the block had been swept in the past, all destructors were called, and that
+ // sweep proved that the block is empty.
+ // - It's false if there are any destructors that need to be called, even if the block has no
+ // live objects.
+ if (m_allocator->isEmpty(this))
+ return sweepHelperSelectHasNewlyAllocated<IsEmpty, destructionMode, scribbleMode>(sweepMode);
+ return sweepHelperSelectHasNewlyAllocated<NotEmpty, destructionMode, scribbleMode>(sweepMode);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
+FreeList MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated(SweepMode sweepMode)
+{
+ if (m_newlyAllocated)
+ return sweepHelperSelectSweepMode<emptyMode, destructionMode, scribbleMode, HasNewlyAllocated>(sweepMode);
+ return sweepHelperSelectSweepMode<emptyMode, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>(sweepMode);
+}
+
+template<MarkedBlock::Handle::EmptyMode emptyMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
+FreeList MarkedBlock::Handle::sweepHelperSelectSweepMode(SweepMode sweepMode)
+{
+ if (sweepMode == SweepToFreeList)
+ return sweepHelperSelectFlipMode<emptyMode, SweepToFreeList, destructionMode, scribbleMode, newlyAllocatedMode>();
+ return sweepHelperSelectFlipMode<emptyMode, SweepOnly, destructionMode, scribbleMode, newlyAllocatedMode>();
+}
+
+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
+FreeList MarkedBlock::Handle::sweepHelperSelectFlipMode()
+{
+ if (needsFlip())
+ return specializedSweep<emptyMode, sweepMode, destructionMode, scribbleMode, newlyAllocatedMode, NeedsFlip>();
+ return specializedSweep<emptyMode, sweepMode, destructionMode, scribbleMode, newlyAllocatedMode, DoesNotNeedFlip>();
+}
+
</ins><span class="cx"> void MarkedBlock::Handle::unsweepWithNoNewlyAllocated()
</span><span class="cx"> {
</span><del>- flipIfNecessary();
-
- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
-
- RELEASE_ASSERT(m_state == FreeListed);
- m_state = Marked;
</del><ins>+ RELEASE_ASSERT(m_isFreeListed);
+ m_isFreeListed = false;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedBlock::Handle::setIsFreeListed()
+{
+ m_allocator->setIsEmpty(this, false);
+ m_isFreeListed = true;
+}
+
</ins><span class="cx"> class SetNewlyAllocatedFunctor : public MarkedBlock::VoidFunctor {
</span><span class="cx"> public:
</span><span class="cx"> SetNewlyAllocatedFunctor(MarkedBlock::Handle* block)
</span><span class="lines">@@ -252,26 +260,17 @@
</span><span class="cx">
</span><span class="cx"> void MarkedBlock::Handle::stopAllocating(const FreeList& freeList)
</span><span class="cx"> {
</span><del>- flipIfNecessary();
- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
</del><ins>+ if (false)
+ dataLog(RawPointer(this), ": MarkedBlock::Handle::stopAllocating!\n");
+ ASSERT(!allocator()->isAllocated(this));
</ins><span class="cx">
</span><del>- if (m_state == Marked) {
- // If the block is in the Marked state then we know that one of these
- // conditions holds:
- //
- // - It was not used for allocation during the previous allocation cycle.
- // It may have dead objects, and we only know them to be dead by the
- // fact that their mark bits are unset.
- //
- // - Someone had already done stopAllocating(), for example because of
- // heap iteration, and they had already
- // Hence if the block is Marked we need to leave it Marked.
</del><ins>+ if (!isFreeListed()) {
+ // This means that we either didn't use this block at all for allocation since last GC,
+ // or someone had already done stopAllocating() before.
</ins><span class="cx"> ASSERT(freeList.allocationWillFail());
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- ASSERT(m_state == FreeListed);
-
</del><span class="cx"> // Roll back to a coherent state for Heap introspection. Cells newly
</span><span class="cx"> // allocated from our free list are not currently marked, so we need another
</span><span class="cx"> // way to tell what's live vs dead.
</span><span class="lines">@@ -290,11 +289,12 @@
</span><span class="cx"> clearNewlyAllocated(cell);
</span><span class="cx"> });
</span><span class="cx">
</span><del>- m_state = Marked;
</del><ins>+ m_isFreeListed = false;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedBlock::Handle::lastChanceToFinalize()
</span><span class="cx"> {
</span><ins>+ allocator()->setIsAllocated(this, false);
</ins><span class="cx"> m_block->clearMarks();
</span><span class="cx"> m_weakSet.lastChanceToFinalize();
</span><span class="cx">
</span><span class="lines">@@ -304,14 +304,11 @@
</span><span class="cx">
</span><span class="cx"> FreeList MarkedBlock::Handle::resumeAllocating()
</span><span class="cx"> {
</span><del>- flipIfNecessary();
- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
</del><ins>+ ASSERT(!allocator()->isAllocated(this));
+ ASSERT(!isFreeListed());
</ins><span class="cx">
</span><del>- ASSERT(m_state == Marked);
-
</del><span class="cx"> if (!m_newlyAllocated) {
</span><del>- // We didn't have to create a "newly allocated" bitmap. That means we were already Marked
- // when we last stopped allocation, so return an empty free list and stay in the Marked state.
</del><ins>+ // This means we had already exhausted the block when we stopped allocation.
</ins><span class="cx"> return FreeList();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -345,40 +342,28 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::flipIfNecessary()
</del><ins>+void MarkedBlock::aboutToMarkSlow(HeapVersion heapVersion)
</ins><span class="cx"> {
</span><del>- flipIfNecessary(vm()->heap.objectSpace().version());
</del><ins>+ ASSERT(vm()->heap.objectSpace().isMarking());
+ LockHolder locker(m_lock);
+ if (needsFlip(heapVersion)) {
+ clearMarks(heapVersion);
+ // This means we're the first ones to mark any object in this block.
+ handle().allocator()->atomicSetAndCheckIsMarkingNotEmpty(&handle(), true);
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::Handle::flipIfNecessary()
</del><ins>+void MarkedBlock::clearMarks()
</ins><span class="cx"> {
</span><del>- block().flipIfNecessary();
</del><ins>+ clearMarks(vm()->heap.objectSpace().version());
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::flipIfNecessarySlow()
</del><ins>+void MarkedBlock::clearMarks(HeapVersion heapVersion)
</ins><span class="cx"> {
</span><del>- ASSERT(needsFlip());
- ASSERT(!vm()->heap.objectSpace().isMarking());
- clearMarks();
-}
-
-void MarkedBlock::flipIfNecessaryDuringMarkingSlow()
-{
- ASSERT(vm()->heap.objectSpace().isMarking());
- LockHolder locker(m_lock);
- if (needsFlip())
- clearMarks();
-}
-
-void MarkedBlock::clearMarks()
-{
</del><span class="cx"> m_marks.clearAll();
</span><span class="cx"> clearHasAnyMarked();
</span><del>- // This will become true at the end of the mark phase. We set it now to
- // avoid an extra pass to do so later.
- handle().m_state = Marked;
</del><span class="cx"> WTF::storeStoreFence();
</span><del>- m_version = vm()->heap.objectSpace().version();
</del><ins>+ m_version = heapVersion;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #if !ASSERT_DISABLED
</span><span class="lines">@@ -398,31 +383,38 @@
</span><span class="cx"> return m_block->needsFlip();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::Handle::willRemoveBlock()
</del><ins>+bool MarkedBlock::isMarked(const void* p)
</ins><span class="cx"> {
</span><del>- flipIfNecessary();
</del><ins>+ return isMarked(vm()->heap.objectSpace().version(), p);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedBlock::Handle::didConsumeFreeList()
</del><ins>+bool MarkedBlock::Handle::isMarkedOrNewlyAllocated(const HeapCell* cell)
</ins><span class="cx"> {
</span><del>- flipIfNecessary();
- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
-
- ASSERT(m_state == FreeListed);
</del><ins>+ return isMarkedOrNewlyAllocated(vm()->heap.objectSpace().version(), cell);
+}
</ins><span class="cx">
</span><del>- m_state = Allocated;
</del><ins>+bool MarkedBlock::isMarkedOrNewlyAllocated(const HeapCell* cell)
+{
+ return isMarkedOrNewlyAllocated(vm()->heap.objectSpace().version(), cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedBlock::Handle::didConsumeFreeList()
+{
+ if (false)
+ dataLog(RawPointer(this), ": MarkedBlock::Handle::didConsumeFreeList!\n");
+ ASSERT(isFreeListed());
+ m_isFreeListed = false;
+ allocator()->setIsAllocated(this, true);
+}
+
</ins><span class="cx"> size_t MarkedBlock::markCount()
</span><span class="cx"> {
</span><del>- flipIfNecessary();
- return m_marks.count();
</del><ins>+ return needsFlip() ? 0 : m_marks.count();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool MarkedBlock::Handle::isEmpty()
</span><span class="cx"> {
</span><del>- flipIfNecessary();
- return m_state == Marked && !block().hasAnyMarked() && m_weakSet.isEmpty() && (!m_newlyAllocated || m_newlyAllocated->isEmpty());
</del><ins>+ return m_allocator->isEmpty(this);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedBlock::clearHasAnyMarked()
</span><span class="lines">@@ -432,30 +424,79 @@
</span><span class="cx">
</span><span class="cx"> void MarkedBlock::noteMarkedSlow()
</span><span class="cx"> {
</span><del>- handle().m_allocator->retire(&handle());
</del><ins>+ handle().allocator()->atomicSetAndCheckIsMarkingRetired(&handle(), true);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedBlock::Handle::removeFromAllocator()
+{
+ if (!m_allocator)
+ return;
+
+ m_allocator->removeBlock(this);
+}
+
+void MarkedBlock::Handle::didAddToAllocator(MarkedAllocator* allocator, size_t index)
+{
+ ASSERT(m_index == std::numeric_limits<size_t>::max());
+ ASSERT(!m_allocator);
+
+ m_index = index;
+ m_allocator = allocator;
+
+ size_t cellSize = allocator->cellSize();
+ m_atomsPerCell = (cellSize + atomSize - 1) / atomSize;
+ m_endAtom = atomsPerBlock - m_atomsPerCell + 1;
+
+ m_attributes = allocator->attributes();
+
+ if (m_attributes.cellKind != HeapCell::JSCell)
+ RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
+
+ block().m_needsDestruction = needsDestruction();
+
+ unsigned cellsPerBlock = MarkedSpace::blockPayload / cellSize;
+ double markCountBias = -(Options::minMarkedBlockUtilization() * cellsPerBlock);
+
+ // The mark count bias should be comfortably within this range.
+ RELEASE_ASSERT(markCountBias > static_cast<double>(std::numeric_limits<int16_t>::min()));
+ RELEASE_ASSERT(markCountBias < 0);
+
+ // This means we haven't marked anything yet.
+ block().m_biasedMarkCount = block().m_markCountBias = static_cast<int16_t>(markCountBias);
+}
+
+void MarkedBlock::Handle::didRemoveFromAllocator()
+{
+ ASSERT(m_index != std::numeric_limits<size_t>::max());
+ ASSERT(m_allocator);
+
+ m_index = std::numeric_limits<size_t>::max();
+ m_allocator = nullptr;
+}
+
+bool MarkedBlock::Handle::isLive(const HeapCell* cell)
+{
+ return isLive(vm()->heap.objectSpace().version(), cell);
+}
+
+bool MarkedBlock::Handle::isLiveCell(const void* p)
+{
+ return isLiveCell(vm()->heap.objectSpace().version(), p);
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx">
</span><span class="cx"> namespace WTF {
</span><span class="cx">
</span><del>-using namespace JSC;
-
-void printInternal(PrintStream& out, MarkedBlock::BlockState blockState)
</del><ins>+void printInternal(PrintStream& out, JSC::MarkedBlock::Handle::SweepMode mode)
</ins><span class="cx"> {
</span><del>- switch (blockState) {
- case MarkedBlock::New:
- out.print("New");
</del><ins>+ switch (mode) {
+ case JSC::MarkedBlock::Handle::SweepToFreeList:
+ out.print("SweepToFreeList");
</ins><span class="cx"> return;
</span><del>- case MarkedBlock::FreeListed:
- out.print("FreeListed");
</del><ins>+ case JSC::MarkedBlock::Handle::SweepOnly:
+ out.print("SweepOnly");
</ins><span class="cx"> return;
</span><del>- case MarkedBlock::Allocated:
- out.print("Allocated");
- return;
- case MarkedBlock::Marked:
- out.print("Marked");
- return;
</del><span class="cx"> }
</span><span class="cx"> RELEASE_ASSERT_NOT_REACHED();
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -44,20 +44,6 @@
</span><span class="cx"> typedef uintptr_t Bits;
</span><span class="cx"> typedef uint32_t HeapVersion;
</span><span class="cx">
</span><del>-// Set to log state transitions of blocks.
-#define HEAP_LOG_BLOCK_STATE_TRANSITIONS 0
-
-#if HEAP_LOG_BLOCK_STATE_TRANSITIONS
-#define HEAP_LOG_BLOCK_STATE_TRANSITION(handle) do { \
- dataLogF( \
- "%s:%d %s: block %s = %p, %d\n", \
- __FILE__, __LINE__, __FUNCTION__, \
- #handle, &(handle)->block(), (handle)->m_state); \
- } while (false)
-#else
-#define HEAP_LOG_BLOCK_STATE_TRANSITION(handle) ((void)0)
-#endif
-
</del><span class="cx"> // A marked block is a page-aligned container for heap-allocated objects.
</span><span class="cx"> // Objects are allocated within cells of the marked block. For a given
</span><span class="cx"> // marked block, all cells have the same size. Objects smaller than the
</span><span class="lines">@@ -76,8 +62,6 @@
</span><span class="cx"> private:
</span><span class="cx"> friend class Handle;
</span><span class="cx"> public:
</span><del>- enum BlockState : uint8_t { New, FreeListed, Allocated, Marked };
-
</del><span class="cx"> static const size_t atomSize = 16; // bytes
</span><span class="cx"> static const size_t blockSize = 16 * KB;
</span><span class="cx"> static const size_t blockMask = ~(blockSize - 1); // blockSize must be a power of two.
</span><span class="lines">@@ -86,12 +70,12 @@
</span><span class="cx">
</span><span class="cx"> static_assert(!(MarkedBlock::atomSize & (MarkedBlock::atomSize - 1)), "MarkedBlock::atomSize must be a power of two.");
</span><span class="cx"> static_assert(!(MarkedBlock::blockSize & (MarkedBlock::blockSize - 1)), "MarkedBlock::blockSize must be a power of two.");
</span><del>-
</del><ins>+
</ins><span class="cx"> struct VoidFunctor {
</span><span class="cx"> typedef void ReturnType;
</span><span class="cx"> void returnValue() { }
</span><span class="cx"> };
</span><del>-
</del><ins>+
</ins><span class="cx"> class CountFunctor {
</span><span class="cx"> public:
</span><span class="cx"> typedef size_t ReturnType;
</span><span class="lines">@@ -106,10 +90,9 @@
</span><span class="cx"> mutable ReturnType m_count;
</span><span class="cx"> };
</span><span class="cx">
</span><del>- class Handle : public BasicRawSentinelNode<Handle> {
</del><ins>+ class Handle {
</ins><span class="cx"> WTF_MAKE_NONCOPYABLE(Handle);
</span><span class="cx"> WTF_MAKE_FAST_ALLOCATED;
</span><del>- friend class DoublyLinkedListNode<Handle>;
</del><span class="cx"> friend class LLIntOffsetsExtractor;
</span><span class="cx"> friend class MarkedBlock;
</span><span class="cx"> friend struct VerifyMarked;
</span><span class="lines">@@ -130,6 +113,15 @@
</span><span class="cx"> VM* vm() const;
</span><span class="cx"> WeakSet& weakSet();
</span><span class="cx">
</span><ins>+ // Sweeping ensures that destructors get called and removes the block from the unswept
+ // set. Sweeping to free list also removes the block from the empty set, if it was in that
+ // set. Sweeping with SweepOnly may add this block to the empty set, if the block is found
+ // to be empty.
+ //
+ // Note that you need to make sure that the empty bit reflects reality. If it's not set
+ // and the block is freshly created, then we'll make the mistake of running destructors in
+ // the block. If it's not set and the block has nothing marked, then we'll make the
+ // mistake of making a pop freelist rather than a bump freelist.
</ins><span class="cx"> enum SweepMode { SweepOnly, SweepToFreeList };
</span><span class="cx"> FreeList sweep(SweepMode = SweepOnly);
</span><span class="cx">
</span><span class="lines">@@ -153,8 +145,6 @@
</span><span class="cx"> // and was successfully cleared and false otherwise.
</span><span class="cx"> bool clearNewlyAllocated();
</span><span class="cx">
</span><del>- void flipForEdenCollection();
-
</del><span class="cx"> size_t cellSize();
</span><span class="cx"> const AllocatorAttributes& attributes() const;
</span><span class="cx"> DestructionMode destruction() const;
</span><span class="lines">@@ -164,9 +154,14 @@
</span><span class="cx"> size_t markCount();
</span><span class="cx"> size_t size();
</span><span class="cx">
</span><ins>+ inline bool isLive(HeapVersion, const HeapCell*);
+ inline bool isLiveCell(HeapVersion, const void*);
+
</ins><span class="cx"> bool isLive(const HeapCell*);
</span><span class="cx"> bool isLiveCell(const void*);
</span><ins>+
</ins><span class="cx"> bool isMarkedOrNewlyAllocated(const HeapCell*);
</span><ins>+ bool isMarkedOrNewlyAllocated(HeapVersion, const HeapCell*);
</ins><span class="cx">
</span><span class="cx"> bool isNewlyAllocated(const void*);
</span><span class="cx"> void setNewlyAllocated(const void*);
</span><span class="lines">@@ -174,31 +169,25 @@
</span><span class="cx">
</span><span class="cx"> bool hasAnyNewlyAllocated() const { return !!m_newlyAllocated; }
</span><span class="cx">
</span><del>- bool isAllocated() const;
- bool isMarked() const;
- bool isFreeListed() const;
- bool needsSweeping() const;
- void willRemoveBlock();
-
</del><span class="cx"> template <typename Functor> IterationStatus forEachCell(const Functor&);
</span><del>- template <typename Functor> IterationStatus forEachLiveCell(const Functor&);
- template <typename Functor> IterationStatus forEachDeadCell(const Functor&);
</del><ins>+ template <typename Functor> inline IterationStatus forEachLiveCell(const Functor&);
+ template <typename Functor> inline IterationStatus forEachDeadCell(const Functor&);
</ins><span class="cx">
</span><span class="cx"> bool needsFlip();
</span><del>-
- void flipIfNecessaryDuringMarking(HeapVersion);
- void flipIfNecessary(HeapVersion);
- void flipIfNecessary();
-
</del><ins>+
</ins><span class="cx"> void assertFlipped();
</span><span class="cx">
</span><del>- bool isOnBlocksToSweep() const { return m_isOnBlocksToSweep; }
- void setIsOnBlocksToSweep(bool value) { m_isOnBlocksToSweep = value; }
</del><ins>+ bool isFreeListed() const { return m_isFreeListed; }
</ins><span class="cx">
</span><del>- BlockState state() const { return m_state; }
-
</del><ins>+ size_t index() const { return m_index; }
+
+ void removeFromAllocator();
+
+ void didAddToAllocator(MarkedAllocator*, size_t index);
+ void didRemoveFromAllocator();
+
</ins><span class="cx"> private:
</span><del>- Handle(Heap&, MarkedAllocator*, size_t cellSize, const AllocatorAttributes&, void*);
</del><ins>+ Handle(Heap&, void*);
</ins><span class="cx">
</span><span class="cx"> template<DestructionMode>
</span><span class="cx"> FreeList sweepHelperSelectScribbleMode(SweepMode = SweepOnly);
</span><span class="lines">@@ -206,35 +195,50 @@
</span><span class="cx"> enum ScribbleMode { DontScribble, Scribble };
</span><span class="cx">
</span><span class="cx"> template<DestructionMode, ScribbleMode>
</span><del>- FreeList sweepHelperSelectStateAndSweepMode(SweepMode = SweepOnly);
</del><ins>+ FreeList sweepHelperSelectEmptyMode(SweepMode = SweepOnly);
</ins><span class="cx">
</span><ins>+ enum EmptyMode { IsEmpty, NotEmpty };
+
+ template<EmptyMode, DestructionMode, ScribbleMode>
+ FreeList sweepHelperSelectHasNewlyAllocated(SweepMode = SweepOnly);
+
</ins><span class="cx"> enum NewlyAllocatedMode { HasNewlyAllocated, DoesNotHaveNewlyAllocated };
</span><del>-
- template<BlockState, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
</del><ins>+
+ template<EmptyMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
+ FreeList sweepHelperSelectSweepMode(SweepMode = SweepOnly);
+
+ template<EmptyMode, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
+ FreeList sweepHelperSelectFlipMode();
+
+ enum FlipMode { NeedsFlip, DoesNotNeedFlip };
+
+ template<EmptyMode, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode, FlipMode>
</ins><span class="cx"> FreeList specializedSweep();
</span><span class="cx">
</span><span class="cx"> template<typename Func>
</span><span class="cx"> void forEachFreeCell(const FreeList&, const Func&);
</span><del>-
</del><ins>+
+ void setIsFreeListed();
+
</ins><span class="cx"> MarkedBlock::Handle* m_prev;
</span><span class="cx"> MarkedBlock::Handle* m_next;
</span><span class="cx">
</span><del>- size_t m_atomsPerCell;
- size_t m_endAtom; // This is a fuzzy end. Always test for < m_endAtom.
</del><ins>+ size_t m_atomsPerCell { std::numeric_limits<size_t>::max() };
+ size_t m_endAtom { std::numeric_limits<size_t>::max() }; // This is a fuzzy end. Always test for < m_endAtom.
</ins><span class="cx">
</span><span class="cx"> std::unique_ptr<WTF::Bitmap<atomsPerBlock>> m_newlyAllocated;
</span><span class="cx">
</span><span class="cx"> AllocatorAttributes m_attributes;
</span><del>- BlockState m_state;
- bool m_isOnBlocksToSweep { false };
</del><ins>+ bool m_isFreeListed { false };
</ins><span class="cx">
</span><del>- MarkedAllocator* m_allocator;
</del><ins>+ MarkedAllocator* m_allocator { nullptr };
+ size_t m_index { std::numeric_limits<size_t>::max() };
</ins><span class="cx"> WeakSet m_weakSet;
</span><span class="cx">
</span><del>- MarkedBlock* m_block;
</del><ins>+ MarkedBlock* m_block { nullptr };
</ins><span class="cx"> };
</span><span class="cx">
</span><del>- static MarkedBlock::Handle* tryCreate(Heap&, MarkedAllocator*, size_t cellSize, const AllocatorAttributes&);
</del><ins>+ static MarkedBlock::Handle* tryCreate(Heap&);
</ins><span class="cx">
</span><span class="cx"> Handle& handle();
</span><span class="cx">
</span><span class="lines">@@ -248,34 +252,35 @@
</span><span class="cx"> size_t markCount();
</span><span class="cx">
</span><span class="cx"> bool isMarked(const void*);
</span><ins>+ bool isMarked(HeapVersion, const void*);
+ bool isMarkedConcurrently(HeapVersion, const void*);
</ins><span class="cx"> bool testAndSetMarked(const void*);
</span><span class="cx">
</span><span class="cx"> bool isMarkedOrNewlyAllocated(const HeapCell*);
</span><ins>+ bool isMarkedOrNewlyAllocated(HeapVersion, const HeapCell*);
</ins><span class="cx">
</span><del>- bool isMarkedDuringWeakVisiting(HeapVersion, const HeapCell*);
-
</del><span class="cx"> bool isAtom(const void*);
</span><span class="cx"> void clearMarked(const void*);
</span><del>-
</del><ins>+
</ins><span class="cx"> size_t cellSize();
</span><span class="cx"> const AllocatorAttributes& attributes() const;
</span><del>-
</del><ins>+
</ins><span class="cx"> bool hasAnyMarked() const;
</span><span class="cx"> void noteMarked();
</span><span class="cx">
</span><span class="cx"> WeakSet& weakSet();
</span><del>-
</del><ins>+
</ins><span class="cx"> bool needsFlip(HeapVersion);
</span><span class="cx"> bool needsFlip();
</span><ins>+
+ void aboutToMark(HeapVersion);
</ins><span class="cx">
</span><del>- void flipIfNecessaryDuringMarking(HeapVersion);
- void flipIfNecessary(HeapVersion);
- void flipIfNecessary();
-
</del><span class="cx"> void assertFlipped();
</span><span class="cx">
</span><span class="cx"> bool needsDestruction() const { return m_needsDestruction; }
</span><del>-
</del><ins>+
+ inline void resetVersion();
+
</ins><span class="cx"> private:
</span><span class="cx"> static const size_t atomAlignmentMask = atomSize - 1;
</span><span class="cx">
</span><span class="lines">@@ -284,9 +289,9 @@
</span><span class="cx"> MarkedBlock(VM&, Handle&);
</span><span class="cx"> Atom* atoms();
</span><span class="cx">
</span><del>- void flipIfNecessaryDuringMarkingSlow();
- void flipIfNecessarySlow();
</del><ins>+ void aboutToMarkSlow(HeapVersion);
</ins><span class="cx"> void clearMarks();
</span><ins>+ void clearMarks(HeapVersion);
</ins><span class="cx"> void clearHasAnyMarked();
</span><span class="cx">
</span><span class="cx"> void noteMarkedSlow();
</span><span class="lines">@@ -469,40 +474,13 @@
</span><span class="cx"> return heapVersion != m_version;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void MarkedBlock::flipIfNecessary(HeapVersion heapVersion)
</del><ins>+inline void MarkedBlock::aboutToMark(HeapVersion heapVersion)
</ins><span class="cx"> {
</span><span class="cx"> if (UNLIKELY(needsFlip(heapVersion)))
</span><del>- flipIfNecessarySlow();
-}
-
-inline void MarkedBlock::flipIfNecessaryDuringMarking(HeapVersion heapVersion)
-{
- if (UNLIKELY(needsFlip(heapVersion)))
- flipIfNecessaryDuringMarkingSlow();
</del><ins>+ aboutToMarkSlow(heapVersion);
</ins><span class="cx"> WTF::loadLoadFence();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void MarkedBlock::Handle::flipIfNecessary(HeapVersion heapVersion)
-{
- block().flipIfNecessary(heapVersion);
-}
-
-inline void MarkedBlock::Handle::flipIfNecessaryDuringMarking(HeapVersion heapVersion)
-{
- block().flipIfNecessaryDuringMarking(heapVersion);
-}
-
-inline void MarkedBlock::Handle::flipForEdenCollection()
-{
- assertFlipped();
-
- HEAP_LOG_BLOCK_STATE_TRANSITION(this);
-
- ASSERT(m_state != New && m_state != FreeListed);
-
- m_state = Marked;
-}
-
</del><span class="cx"> #if ASSERT_DISABLED
</span><span class="cx"> inline void MarkedBlock::assertFlipped()
</span><span class="cx"> {
</span><span class="lines">@@ -514,9 +492,16 @@
</span><span class="cx"> block().assertFlipped();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::isMarked(const void* p)
</del><ins>+inline bool MarkedBlock::isMarked(HeapVersion heapVersion, const void* p)
</ins><span class="cx"> {
</span><del>- assertFlipped();
</del><ins>+ return needsFlip(heapVersion) ? false : m_marks.get(atomNumber(p));
+}
+
+inline bool MarkedBlock::isMarkedConcurrently(HeapVersion heapVersion, const void* p)
+{
+ if (needsFlip(heapVersion))
+ return false;
+ WTF::loadLoadFence();
</ins><span class="cx"> return m_marks.get(atomNumber(p));
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -550,45 +535,16 @@
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::Handle::isMarkedOrNewlyAllocated(const HeapCell* cell)
</del><ins>+inline bool MarkedBlock::Handle::isMarkedOrNewlyAllocated(HeapVersion version, const HeapCell* cell)
</ins><span class="cx"> {
</span><del>- ASSERT(m_state == Marked);
- return m_block->isMarked(cell) || (m_newlyAllocated && isNewlyAllocated(cell));
</del><ins>+ return m_block->isMarked(version, cell) || (m_newlyAllocated && isNewlyAllocated(cell));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::isMarkedOrNewlyAllocated(const HeapCell* cell)
</del><ins>+inline bool MarkedBlock::isMarkedOrNewlyAllocated(HeapVersion version, const HeapCell* cell)
</ins><span class="cx"> {
</span><del>- ASSERT(m_handle.m_state == Marked);
- return isMarked(cell) || (m_handle.m_newlyAllocated && m_handle.isNewlyAllocated(cell));
</del><ins>+ return isMarked(version, cell) || (m_handle.m_newlyAllocated && m_handle.isNewlyAllocated(cell));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::isMarkedDuringWeakVisiting(HeapVersion heapVersion, const HeapCell* cell)
-{
- if (needsFlip(heapVersion))
- return false;
- return isMarked(cell);
-}
-
-inline bool MarkedBlock::Handle::isLive(const HeapCell* cell)
-{
- assertFlipped();
- switch (m_state) {
- case Allocated:
- return true;
-
- case Marked:
- return isMarkedOrNewlyAllocated(cell);
-
- case New:
- case FreeListed:
- RELEASE_ASSERT_NOT_REACHED();
- return false;
- }
-
- RELEASE_ASSERT_NOT_REACHED();
- return false;
-}
-
</del><span class="cx"> inline bool MarkedBlock::isAtom(const void* p)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(MarkedBlock::isAtomAligned(p));
</span><span class="lines">@@ -603,13 +559,6 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool MarkedBlock::Handle::isLiveCell(const void* p)
-{
- if (!m_block->isAtom(p))
- return false;
- return isLive(static_cast<const HeapCell*>(p));
-}
-
</del><span class="cx"> template <typename Functor>
</span><span class="cx"> inline IterationStatus MarkedBlock::Handle::forEachCell(const Functor& functor)
</span><span class="cx"> {
</span><span class="lines">@@ -622,62 +571,6 @@
</span><span class="cx"> return IterationStatus::Continue;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template <typename Functor>
-inline IterationStatus MarkedBlock::Handle::forEachLiveCell(const Functor& functor)
-{
- flipIfNecessary();
- HeapCell::Kind kind = m_attributes.cellKind;
- for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
- HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
- if (!isLive(cell))
- continue;
-
- if (functor(cell, kind) == IterationStatus::Done)
- return IterationStatus::Done;
- }
- return IterationStatus::Continue;
-}
-
-template <typename Functor>
-inline IterationStatus MarkedBlock::Handle::forEachDeadCell(const Functor& functor)
-{
- flipIfNecessary();
- HeapCell::Kind kind = m_attributes.cellKind;
- for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
- HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
- if (isLive(cell))
- continue;
-
- if (functor(cell, kind) == IterationStatus::Done)
- return IterationStatus::Done;
- }
- return IterationStatus::Continue;
-}
-
-inline bool MarkedBlock::Handle::needsSweeping() const
-{
- const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
- return m_state == Marked;
-}
-
-inline bool MarkedBlock::Handle::isAllocated() const
-{
- const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
- return m_state == Allocated;
-}
-
-inline bool MarkedBlock::Handle::isMarked() const
-{
- const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
- return m_state == Marked;
-}
-
-inline bool MarkedBlock::Handle::isFreeListed() const
-{
- const_cast<MarkedBlock::Handle*>(this)->flipIfNecessary();
- return m_state == FreeListed;
-}
-
</del><span class="cx"> inline bool MarkedBlock::hasAnyMarked() const
</span><span class="cx"> {
</span><span class="cx"> return m_biasedMarkCount != m_markCountBias;
</span><span class="lines">@@ -711,7 +604,7 @@
</span><span class="cx"> typedef MarkedBlockHash Hash;
</span><span class="cx"> };
</span><span class="cx">
</span><del>-void printInternal(PrintStream& out, JSC::MarkedBlock::BlockState);
</del><ins>+void printInternal(PrintStream& out, JSC::MarkedBlock::Handle::SweepMode);
</ins><span class="cx">
</span><span class="cx"> } // namespace WTF
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockInlinesh"></a>
<div class="addfile"><h4>Added: trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h (rev 0)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlockInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,96 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "MarkedAllocator.h"
+#include "MarkedBlock.h"
+
+namespace JSC {
+
+inline bool MarkedBlock::Handle::isLive(HeapVersion version, const HeapCell* cell)
+{
+ ASSERT(!isFreeListed());
+
+ if (UNLIKELY(hasAnyNewlyAllocated())) {
+ if (isNewlyAllocated(cell))
+ return true;
+ }
+
+ MarkedBlock& block = this->block();
+
+ if (allocator()->isAllocated(this))
+ return true;
+
+ if (block.needsFlip(version))
+ return false;
+
+ return block.isMarked(cell);
+}
+
+inline bool MarkedBlock::Handle::isLiveCell(HeapVersion version, const void* p)
+{
+ if (!m_block->isAtom(p))
+ return false;
+ return isLive(version, static_cast<const HeapCell*>(p));
+}
+
+template <typename Functor>
+inline IterationStatus MarkedBlock::Handle::forEachLiveCell(const Functor& functor)
+{
+ HeapCell::Kind kind = m_attributes.cellKind;
+ for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
+ HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
+ if (!isLive(cell))
+ continue;
+
+ if (functor(cell, kind) == IterationStatus::Done)
+ return IterationStatus::Done;
+ }
+ return IterationStatus::Continue;
+}
+
+template <typename Functor>
+inline IterationStatus MarkedBlock::Handle::forEachDeadCell(const Functor& functor)
+{
+ HeapCell::Kind kind = m_attributes.cellKind;
+ for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
+ HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&m_block->atoms()[i]);
+ if (isLive(cell))
+ continue;
+
+ if (functor(cell, kind) == IterationStatus::Done)
+ return IterationStatus::Done;
+ }
+ return IterationStatus::Continue;
+}
+
+inline void MarkedBlock::resetVersion()
+{
+ m_version = MarkedSpace::nullVersion;
+}
+
+} // namespace JSC
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpacecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -24,6 +24,7 @@
</span><span class="cx"> #include "IncrementalSweeper.h"
</span><span class="cx"> #include "JSObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "MarkedBlockInlines.h"
</ins><span class="cx"> #include "SuperSampler.h"
</span><span class="cx"> #include <wtf/ListDump.h>
</span><span class="cx">
</span><span class="lines">@@ -195,6 +196,19 @@
</span><span class="cx">
</span><span class="cx"> return IterationStatus::Continue;
</span><span class="cx"> });
</span><ins>+
+ MarkedAllocator* previous = nullptr;
+ forEachSubspace(
+ [&] (Subspace& subspace, AllocatorAttributes) -> IterationStatus {
+ for (MarkedAllocator* allocator : subspace.bagOfAllocators) {
+ allocator->setNextAllocator(previous);
+ previous = allocator;
+ }
+
+ return IterationStatus::Continue;
+ });
+ m_firstAllocator = previous;
+ m_allocatorForEmptyAllocation = previous;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> MarkedSpace::~MarkedSpace()
</span><span class="lines">@@ -222,15 +236,19 @@
</span><span class="cx">
</span><span class="cx"> void* MarkedSpace::allocate(Subspace& subspace, size_t bytes)
</span><span class="cx"> {
</span><del>- if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
- return allocator->allocate();
</del><ins>+ if (MarkedAllocator* allocator = allocatorFor(subspace, bytes)) {
+ void* result = allocator->allocate();
+ return result;
+ }
</ins><span class="cx"> return allocateLarge(subspace, bytes);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void* MarkedSpace::tryAllocate(Subspace& subspace, size_t bytes)
</span><span class="cx"> {
</span><del>- if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
- return allocator->tryAllocate();
</del><ins>+ if (MarkedAllocator* allocator = allocatorFor(subspace, bytes)) {
+ void* result = allocator->tryAllocate();
+ return result;
+ }
</ins><span class="cx"> return tryAllocateLarge(subspace, bytes);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -259,9 +277,10 @@
</span><span class="cx"> void MarkedSpace::sweep()
</span><span class="cx"> {
</span><span class="cx"> m_heap->sweeper()->willFinishSweeping();
</span><del>- forEachBlock(
- [&] (MarkedBlock::Handle* block) {
- block->sweep();
</del><ins>+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.sweep();
+ return IterationStatus::Continue;
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -284,33 +303,23 @@
</span><span class="cx"> m_largeAllocationsNurseryOffset = m_largeAllocations.size();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void MarkedSpace::zombifySweep()
</del><ins>+void MarkedSpace::prepareForAllocation()
</ins><span class="cx"> {
</span><del>- if (Options::logGC())
- dataLog("Zombifying sweep...");
- m_heap->sweeper()->willFinishSweeping();
- forEachBlock(
- [&] (MarkedBlock::Handle* block) {
- if (block->needsSweeping())
- block->sweep();
- });
-}
-
-void MarkedSpace::resetAllocators()
-{
</del><span class="cx"> forEachAllocator(
</span><span class="cx"> [&] (MarkedAllocator& allocator) -> IterationStatus {
</span><del>- allocator.reset();
</del><ins>+ allocator.prepareForAllocation();
</ins><span class="cx"> return IterationStatus::Continue;
</span><span class="cx"> });
</span><span class="cx">
</span><del>- m_blocksWithNewObjects.clear();
</del><span class="cx"> m_activeWeakSets.takeFrom(m_newActiveWeakSets);
</span><ins>+
</ins><span class="cx"> if (m_heap->operationInProgress() == EdenCollection)
</span><span class="cx"> m_largeAllocationsNurseryOffsetForSweep = m_largeAllocationsNurseryOffset;
</span><span class="cx"> else
</span><span class="cx"> m_largeAllocationsNurseryOffsetForSweep = 0;
</span><span class="cx"> m_largeAllocationsNurseryOffset = m_largeAllocations.size();
</span><ins>+
+ m_allocatorForEmptyAllocation = m_firstAllocator;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::visitWeakSets(HeapRootVisitor& heapRootVisitor)
</span><span class="lines">@@ -410,11 +419,11 @@
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::shrink()
</span><span class="cx"> {
</span><del>- forEachBlock(
- [&] (MarkedBlock::Handle* block) {
- freeOrShrinkBlock(block);
</del><ins>+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.shrink();
+ return IterationStatus::Continue;
</ins><span class="cx"> });
</span><del>- // For LargeAllocations, we do the moral equivalent in sweepLargeAllocations().
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::clearNewlyAllocated()
</span><span class="lines">@@ -429,59 +438,63 @@
</span><span class="cx"> for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
</span><span class="cx"> m_largeAllocations[i]->clearNewlyAllocated();
</span><span class="cx">
</span><del>-#if !ASSERT_DISABLED
- forEachBlock(
- [&] (MarkedBlock::Handle* block) {
- ASSERT(!block->clearNewlyAllocated());
- });
-
- for (LargeAllocation* allocation : m_largeAllocations)
- ASSERT(!allocation->isNewlyAllocated());
-#endif // !ASSERT_DISABLED
</del><ins>+ if (!ASSERT_DISABLED) {
+ forEachBlock(
+ [&] (MarkedBlock::Handle* block) {
+ ASSERT_UNUSED(block, !block->clearNewlyAllocated());
+ });
+
+ for (LargeAllocation* allocation : m_largeAllocations)
+ ASSERT_UNUSED(allocation, !allocation->isNewlyAllocated());
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-#ifndef NDEBUG
-struct VerifyMarked : MarkedBlock::VoidFunctor {
- void operator()(MarkedBlock::Handle* block) const
- {
- if (block->needsFlip())
- return;
- switch (block->m_state) {
- case MarkedBlock::Marked:
- return;
- default:
- RELEASE_ASSERT_NOT_REACHED();
- }
- }
-};
-#endif
</del><ins>+void MarkedSpace::beginMarking()
+{
+ if (m_heap->operationInProgress() == FullCollection) {
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.beginMarkingForFullCollection();
+ return IterationStatus::Continue;
+ });
</ins><span class="cx">
</span><del>-void MarkedSpace::flip()
-{
- if (m_heap->operationInProgress() == EdenCollection) {
- for (unsigned i = 0; i < m_blocksWithNewObjects.size(); ++i)
- m_blocksWithNewObjects[i]->flipForEdenCollection();
- } else {
- HeapVersion nextVersion = m_version + 1;
- if (UNLIKELY(nextVersion == initialVersion)) {
- // Oh no! Version wrap-around! We handle this by flipping all blocks. This happens
- // super rarely, probably never for most users.
</del><ins>+ m_version = nextVersion(m_version);
+
+ if (UNLIKELY(m_version == initialVersion)) {
+ // Oh no! Version wrap-around! We handle this by setting all block versions to null.
</ins><span class="cx"> forEachBlock(
</span><span class="cx"> [&] (MarkedBlock::Handle* handle) {
</span><del>- handle->flipIfNecessary();
</del><ins>+ handle->block().resetVersion();
</ins><span class="cx"> });
</span><span class="cx"> }
</span><del>- m_version = nextVersion; // Henceforth, flipIfNecessary() will trigger on all blocks.
</del><ins>+
</ins><span class="cx"> for (LargeAllocation* allocation : m_largeAllocations)
</span><span class="cx"> allocation->flip();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-#ifndef NDEBUG
- VerifyMarked verifyFunctor;
- forEachBlock(verifyFunctor);
-#endif
</del><ins>+ if (!ASSERT_DISABLED) {
+ forEachBlock(
+ [&] (MarkedBlock::Handle* block) {
+ if (block->needsFlip())
+ return;
+ ASSERT(!block->isFreeListed());
+ });
+ }
+
+ m_isMarking = true;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkedSpace::endMarking()
+{
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.endMarking();
+ return IterationStatus::Continue;
+ });
+
+ m_isMarking = false;
+}
+
</ins><span class="cx"> void MarkedSpace::willStartIterating()
</span><span class="cx"> {
</span><span class="cx"> ASSERT(!isIterating());
</span><span class="lines">@@ -540,6 +553,9 @@
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::didAddBlock(MarkedBlock::Handle* block)
</span><span class="cx"> {
</span><ins>+ // WARNING: This function is called before block is fully initialized. The block will not know
+ // its cellSize() or attributes(). The latter implies that you can't ask things like
+ // needsDestruction().
</ins><span class="cx"> m_capacity += MarkedBlock::blockSize;
</span><span class="cx"> m_blocks.add(&block->block());
</span><span class="cx"> }
</span><span class="lines">@@ -546,9 +562,6 @@
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::didAllocateInBlock(MarkedBlock::Handle* block)
</span><span class="cx"> {
</span><del>- block->assertFlipped();
- m_blocksWithNewObjects.append(block);
-
</del><span class="cx"> if (block->weakSet().isOnList()) {
</span><span class="cx"> block->weakSet().remove();
</span><span class="cx"> m_newActiveWeakSets.append(&block->weakSet());
</span><span class="lines">@@ -555,4 +568,51 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+MarkedBlock::Handle* MarkedSpace::findEmptyBlockToSteal()
+{
+ for (; m_allocatorForEmptyAllocation; m_allocatorForEmptyAllocation = m_allocatorForEmptyAllocation->nextAllocator()) {
+ if (MarkedBlock::Handle* block = m_allocatorForEmptyAllocation->findEmptyBlockToSteal())
+ return block;
+ }
+ return nullptr;
+}
+
+void MarkedSpace::snapshotUnswept()
+{
+ if (m_heap->operationInProgress() == EdenCollection) {
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.snapshotUnsweptForEdenCollection();
+ return IterationStatus::Continue;
+ });
+ } else {
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.snapshotUnsweptForFullCollection();
+ return IterationStatus::Continue;
+ });
+ }
+}
+
+void MarkedSpace::assertNoUnswept()
+{
+ if (ASSERT_DISABLED)
+ return;
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ allocator.assertNoUnswept();
+ return IterationStatus::Continue;
+ });
+}
+
+void MarkedSpace::dumpBits(PrintStream& out)
+{
+ forEachAllocator(
+ [&] (MarkedAllocator& allocator) -> IterationStatus {
+ out.print("Bits for ", allocator, ":\n");
+ allocator.dumpBits(out);
+ return IterationStatus::Continue;
+ });
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpaceh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedSpace.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpace.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpace.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -65,8 +65,17 @@
</span><span class="cx">
</span><span class="cx"> static const size_t numSizeClasses = largeCutoff / sizeStep;
</span><span class="cx">
</span><del>- static const HeapVersion initialVersion = 42; // This can be any value, including random garbage, so long as it's consistent for the lifetime of the process.
</del><ins>+ static const HeapVersion nullVersion = 0; // The version of freshly allocated blocks.
+ static const HeapVersion initialVersion = 1; // The version that the heap starts out with.
</ins><span class="cx">
</span><ins>+ HeapVersion nextVersion(HeapVersion version)
+ {
+ version++;
+ if (version == nullVersion)
+ version++;
+ return version;
+ }
+
</ins><span class="cx"> static size_t sizeClassToIndex(size_t size)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(size);
</span><span class="lines">@@ -114,7 +123,7 @@
</span><span class="cx"> Subspace& subspaceForObjectsWithoutDestructor() { return m_normalSpace; }
</span><span class="cx"> Subspace& subspaceForAuxiliaryData() { return m_auxiliarySpace; }
</span><span class="cx">
</span><del>- void resetAllocators();
</del><ins>+ void prepareForAllocation();
</ins><span class="cx">
</span><span class="cx"> void visitWeakSets(HeapRootVisitor&);
</span><span class="cx"> void reapWeakSets();
</span><span class="lines">@@ -144,11 +153,13 @@
</span><span class="cx"> void didConsumeFreeList(MarkedBlock::Handle*);
</span><span class="cx"> void didAllocateInBlock(MarkedBlock::Handle*);
</span><span class="cx">
</span><del>- void flip();
</del><ins>+ void beginMarking();
+ void endMarking();
+ void snapshotUnswept();
</ins><span class="cx"> void clearNewlyAllocated();
</span><span class="cx"> void sweep();
</span><span class="cx"> void sweepLargeAllocations();
</span><del>- void zombifySweep();
</del><ins>+ void assertNoUnswept();
</ins><span class="cx"> size_t objectCount();
</span><span class="cx"> size_t size();
</span><span class="cx"> size_t capacity();
</span><span class="lines">@@ -157,8 +168,6 @@
</span><span class="cx">
</span><span class="cx"> HeapVersion version() const { return m_version; }
</span><span class="cx">
</span><del>- const Vector<MarkedBlock::Handle*>& blocksWithNewObjects() const { return m_blocksWithNewObjects; }
-
</del><span class="cx"> const Vector<LargeAllocation*>& largeAllocations() const { return m_largeAllocations; }
</span><span class="cx"> unsigned largeAllocationsNurseryOffset() const { return m_largeAllocationsNurseryOffset; }
</span><span class="cx"> unsigned largeAllocationsOffsetForThisCollection() const { return m_largeAllocationsOffsetForThisCollection; }
</span><span class="lines">@@ -169,13 +178,16 @@
</span><span class="cx"> LargeAllocation** largeAllocationsForThisCollectionEnd() const { return m_largeAllocationsForThisCollectionEnd; }
</span><span class="cx"> unsigned largeAllocationsForThisCollectionSize() const { return m_largeAllocationsForThisCollectionSize; }
</span><span class="cx">
</span><ins>+ MarkedAllocator* firstAllocator() const { return m_firstAllocator; }
+ MarkedAllocator* allocatorForEmptyAllocation() const { return m_allocatorForEmptyAllocation; }
+
+ MarkedBlock::Handle* findEmptyBlockToSteal();
+
</ins><span class="cx"> // When this is true it means that we have flipped but the mark bits haven't converged yet.
</span><span class="cx"> bool isMarking() const { return m_isMarking; }
</span><span class="cx">
</span><del>- // FIXME: After https://bugs.webkit.org/show_bug.cgi?id=161581, MarkedSpace will control this
- // flag directly.
- void setIsMarking(bool value) { m_isMarking = value; }
-
</del><ins>+ void dumpBits(PrintStream& = WTF::dataFile());
+
</ins><span class="cx"> private:
</span><span class="cx"> friend class LLIntOffsetsExtractor;
</span><span class="cx"> friend class JIT;
</span><span class="lines">@@ -190,8 +202,8 @@
</span><span class="cx">
</span><span class="cx"> void initializeSubspace(Subspace&);
</span><span class="cx">
</span><del>- template<typename Functor> void forEachAllocator(const Functor&);
- template<typename Functor> void forEachSubspace(const Functor&);
</del><ins>+ template<typename Functor> inline void forEachAllocator(const Functor&);
+ template<typename Functor> inline void forEachSubspace(const Functor&);
</ins><span class="cx">
</span><span class="cx"> void addActiveWeakSet(WeakSet*);
</span><span class="cx">
</span><span class="lines">@@ -205,7 +217,6 @@
</span><span class="cx"> bool m_isIterating;
</span><span class="cx"> bool m_isMarking { false };
</span><span class="cx"> MarkedBlockSet m_blocks;
</span><del>- Vector<MarkedBlock::Handle*> m_blocksWithNewObjects;
</del><span class="cx"> Vector<LargeAllocation*> m_largeAllocations;
</span><span class="cx"> unsigned m_largeAllocationsNurseryOffset { 0 };
</span><span class="cx"> unsigned m_largeAllocationsOffsetForThisCollection { 0 };
</span><span class="lines">@@ -215,40 +226,11 @@
</span><span class="cx"> unsigned m_largeAllocationsForThisCollectionSize { 0 };
</span><span class="cx"> SentinelLinkedList<WeakSet, BasicRawSentinelNode<WeakSet>> m_activeWeakSets;
</span><span class="cx"> SentinelLinkedList<WeakSet, BasicRawSentinelNode<WeakSet>> m_newActiveWeakSets;
</span><ins>+
+ MarkedAllocator* m_firstAllocator { nullptr };
+ MarkedAllocator* m_allocatorForEmptyAllocation { nullptr };
</ins><span class="cx"> };
</span><span class="cx">
</span><del>-template<typename Functor> inline void MarkedSpace::forEachLiveCell(HeapIterationScope&, const Functor& functor)
-{
- ASSERT(isIterating());
- BlockIterator end = m_blocks.set().end();
- for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
- if ((*it)->handle().forEachLiveCell(functor) == IterationStatus::Done)
- return;
- }
- for (LargeAllocation* allocation : m_largeAllocations) {
- if (allocation->isLive()) {
- if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
- return;
- }
- }
-}
-
-template<typename Functor> inline void MarkedSpace::forEachDeadCell(HeapIterationScope&, const Functor& functor)
-{
- ASSERT(isIterating());
- BlockIterator end = m_blocks.set().end();
- for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
- if ((*it)->handle().forEachDeadCell(functor) == IterationStatus::Done)
- return;
- }
- for (LargeAllocation* allocation : m_largeAllocations) {
- if (!allocation->isLive()) {
- if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
- return;
- }
- }
-}
-
</del><span class="cx"> inline MarkedAllocator* MarkedSpace::allocatorFor(Subspace& space, size_t bytes)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(bytes);
</span><span class="lines">@@ -304,15 +286,10 @@
</span><span class="cx"> template <typename Functor>
</span><span class="cx"> void MarkedSpace::forEachAllocator(const Functor& functor)
</span><span class="cx"> {
</span><del>- forEachSubspace(
- [&] (Subspace& subspace, AllocatorAttributes) -> IterationStatus {
- for (MarkedAllocator* allocator : subspace.bagOfAllocators) {
- if (functor(*allocator) == IterationStatus::Done)
- return IterationStatus::Done;
- }
-
- return IterationStatus::Continue;
- });
</del><ins>+ for (MarkedAllocator* allocator = m_firstAllocator; allocator; allocator = allocator->nextAllocator()) {
+ if (functor(*allocator) == IterationStatus::Done)
+ return;
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename Functor>
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpaceInlinesh"></a>
<div class="addfile"><h4>Added: trunk/Source/JavaScriptCore/heap/MarkedSpaceInlines.h (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpaceInlines.h (rev 0)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpaceInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,66 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "MarkedBlockInlines.h"
+#include "MarkedSpace.h"
+
+namespace JSC {
+
+template<typename Functor> inline void MarkedSpace::forEachLiveCell(HeapIterationScope&, const Functor& functor)
+{
+ ASSERT(isIterating());
+ BlockIterator end = m_blocks.set().end();
+ for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
+ if ((*it)->handle().forEachLiveCell(functor) == IterationStatus::Done)
+ return;
+ }
+ for (LargeAllocation* allocation : m_largeAllocations) {
+ if (allocation->isLive()) {
+ if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
+ return;
+ }
+ }
+}
+
+template<typename Functor> inline void MarkedSpace::forEachDeadCell(HeapIterationScope&, const Functor& functor)
+{
+ ASSERT(isIterating());
+ BlockIterator end = m_blocks.set().end();
+ for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
+ if ((*it)->handle().forEachDeadCell(functor) == IterationStatus::Done)
+ return;
+ }
+ for (LargeAllocation* allocation : m_largeAllocations) {
+ if (!allocation->isLive()) {
+ if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
+ return;
+ }
+ }
+}
+
+} // namespace JSC
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -189,6 +189,8 @@
</span><span class="cx"> validate(cell);
</span><span class="cx"> #endif
</span><span class="cx">
</span><ins>+ //dataLog(" Marking ", RawPointer(cell), "\n");
+
</ins><span class="cx"> if (cell->isLargeAllocation())
</span><span class="cx"> setMarkedAndAppendToMarkStack(cell->largeAllocation(), cell);
</span><span class="cx"> else
</span><span class="lines">@@ -198,7 +200,7 @@
</span><span class="cx"> template<typename ContainerType>
</span><span class="cx"> ALWAYS_INLINE void SlotVisitor::setMarkedAndAppendToMarkStack(ContainerType& container, JSCell* cell)
</span><span class="cx"> {
</span><del>- container.flipIfNecessaryDuringMarking(m_version);
</del><ins>+ container.aboutToMark(m_version);
</ins><span class="cx">
</span><span class="cx"> if (container.testAndSetMarked(cell))
</span><span class="cx"> return;
</span><span class="lines">@@ -246,10 +248,8 @@
</span><span class="cx">
</span><span class="cx"> ASSERT(cell->heap() == heap());
</span><span class="cx">
</span><del>- if (Heap::testAndSetMarked(m_version, cell)) {
- RELEASE_ASSERT(Heap::isMarkedConcurrently(cell));
</del><ins>+ if (Heap::testAndSetMarked(m_version, cell))
</ins><span class="cx"> return;
</span><del>- }
</del><span class="cx">
</span><span class="cx"> noteLiveAuxiliaryCell(cell);
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapWeakh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Weak.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Weak.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/Weak.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009, 2012, 2013 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009, 2012, 2013, 2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -28,7 +28,9 @@
</span><span class="cx">
</span><span class="cx"> #include "JSExportMacros.h"
</span><span class="cx"> #include <cstddef>
</span><ins>+#include <wtf/HashTraits.h>
</ins><span class="cx"> #include <wtf/Noncopyable.h>
</span><ins>+#include <wtf/VectorTraits.h>
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="lines">@@ -51,13 +53,13 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><del>- Weak(T*, WeakHandleOwner* = 0, void* context = 0);
</del><ins>+ inline Weak(T*, WeakHandleOwner* = 0, void* context = 0);
</ins><span class="cx">
</span><span class="cx"> enum HashTableDeletedValueTag { HashTableDeletedValue };
</span><del>- bool isHashTableDeletedValue() const;
- Weak(HashTableDeletedValueTag);
</del><ins>+ inline bool isHashTableDeletedValue() const;
+ inline Weak(HashTableDeletedValueTag);
</ins><span class="cx">
</span><del>- Weak(Weak&&);
</del><ins>+ inline Weak(Weak&&);
</ins><span class="cx">
</span><span class="cx"> ~Weak()
</span><span class="cx"> {
</span><span class="lines">@@ -64,20 +66,20 @@
</span><span class="cx"> clear();
</span><span class="cx"> }
</span><span class="cx">
</span><del>- void swap(Weak&);
</del><ins>+ inline void swap(Weak&);
</ins><span class="cx">
</span><del>- Weak& operator=(Weak&&);
</del><ins>+ inline Weak& operator=(Weak&&);
</ins><span class="cx">
</span><del>- bool operator!() const;
- T* operator->() const;
- T& operator*() const;
- T* get() const;
</del><ins>+ inline bool operator!() const;
+ inline T* operator->() const;
+ inline T& operator*() const;
+ inline T* get() const;
</ins><span class="cx">
</span><del>- bool was(T*) const;
</del><ins>+ inline bool was(T*) const;
</ins><span class="cx">
</span><del>- explicit operator bool() const;
</del><ins>+ inline explicit operator bool() const;
</ins><span class="cx">
</span><del>- WeakImpl* leakImpl() WARN_UNUSED_RETURN;
</del><ins>+ inline WeakImpl* leakImpl() WARN_UNUSED_RETURN;
</ins><span class="cx"> void clear()
</span><span class="cx"> {
</span><span class="cx"> if (!m_impl)
</span><span class="lines">@@ -86,7 +88,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private:
</span><del>- static WeakImpl* hashTableDeletedValue();
</del><ins>+ static inline WeakImpl* hashTableDeletedValue();
</ins><span class="cx">
</span><span class="cx"> WeakImpl* m_impl;
</span><span class="cx"> };
</span><span class="lines">@@ -93,4 +95,23 @@
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span><span class="cx">
</span><ins>+namespace WTF {
+
+template<typename T> struct VectorTraits<JSC::Weak<T>> : SimpleClassVectorTraits {
+ static const bool canCompareWithMemcmp = false;
+};
+
+template<typename T> struct HashTraits<JSC::Weak<T>> : SimpleClassHashTraits<JSC::Weak<T>> {
+ typedef JSC::Weak<T> StorageType;
+
+ typedef std::nullptr_t EmptyValueType;
+ static EmptyValueType emptyValue() { return nullptr; }
+
+ typedef T* PeekType;
+ static PeekType peek(const StorageType& value) { return value.get(); }
+ static PeekType peek(EmptyValueType) { return PeekType(); }
+};
+
+} // namespace WTF
+
</ins><span class="cx"> #endif // Weak_h
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapWeakBlockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/WeakBlock.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/WeakBlock.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/WeakBlock.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -113,7 +113,7 @@
</span><span class="cx"> continue;
</span><span class="cx">
</span><span class="cx"> const JSValue& jsValue = weakImpl->jsValue();
</span><del>- if (container.isMarkedDuringWeakVisiting(version, jsValue.asCell()))
</del><ins>+ if (container.isMarkedConcurrently(version, jsValue.asCell()))
</ins><span class="cx"> continue;
</span><span class="cx">
</span><span class="cx"> if (!weakHandleOwner->isReachableFromOpaqueRoots(Handle<Unknown>::wrapSlot(&const_cast<JSValue&>(jsValue)), weakImpl->context(), visitor))
</span><span class="lines">@@ -147,17 +147,14 @@
</span><span class="cx"> // If this WeakBlock doesn't belong to a CellContainer, we won't even be here.
</span><span class="cx"> ASSERT(m_container);
</span><span class="cx">
</span><del>- m_container.flipIfNecessary();
</del><ins>+ HeapVersion version = m_container.heap()->objectSpace().version();
</ins><span class="cx">
</span><del>- // We only reap after marking.
- ASSERT(m_container.isMarked());
-
</del><span class="cx"> for (size_t i = 0; i < weakImplCount(); ++i) {
</span><span class="cx"> WeakImpl* weakImpl = &weakImpls()[i];
</span><span class="cx"> if (weakImpl->state() > WeakImpl::Dead)
</span><span class="cx"> continue;
</span><span class="cx">
</span><del>- if (m_container.isMarked(weakImpl->jsValue().asCell())) {
</del><ins>+ if (m_container.isMarked(version, weakImpl->jsValue().asCell())) {
</ins><span class="cx"> ASSERT(weakImpl->state() == WeakImpl::Live);
</span><span class="cx"> continue;
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapWeakInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/WeakInlines.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/WeakInlines.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/heap/WeakInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009, 2012, 2013 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009, 2012, 2013, 2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -29,7 +29,6 @@
</span><span class="cx"> #include "JSCell.h"
</span><span class="cx"> #include "WeakSetInlines.h"
</span><span class="cx"> #include <wtf/Assertions.h>
</span><del>-#include <wtf/HashTraits.h>
</del><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="lines">@@ -149,23 +148,4 @@
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span><span class="cx">
</span><del>-namespace WTF {
-
-template<typename T> struct VectorTraits<JSC::Weak<T>> : SimpleClassVectorTraits {
- static const bool canCompareWithMemcmp = false;
-};
-
-template<typename T> struct HashTraits<JSC::Weak<T>> : SimpleClassHashTraits<JSC::Weak<T>> {
- typedef JSC::Weak<T> StorageType;
-
- typedef std::nullptr_t EmptyValueType;
- static EmptyValueType emptyValue() { return nullptr; }
-
- typedef T* PeekType;
- static PeekType peek(const StorageType& value) { return value.get(); }
- static PeekType peek(EmptyValueType) { return PeekType(); }
-};
-
-} // namespace WTF
-
</del><span class="cx"> #endif // WeakInlines_h
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITThunksh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITThunks.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITThunks.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/jit/JITThunks.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -34,10 +34,10 @@
</span><span class="cx"> #include "ThunkGenerator.h"
</span><span class="cx"> #include "Weak.h"
</span><span class="cx"> #include "WeakHandleOwner.h"
</span><del>-#include "WeakInlines.h"
</del><span class="cx"> #include <tuple>
</span><span class="cx"> #include <wtf/HashMap.h>
</span><span class="cx"> #include <wtf/ThreadingPrimitives.h>
</span><ins>+#include <wtf/text/StringHash.h>
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSGlobalObjectcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/JSGlobalObject.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -116,6 +116,7 @@
</span><span class="cx"> #include "MapConstructor.h"
</span><span class="cx"> #include "MapIteratorPrototype.h"
</span><span class="cx"> #include "MapPrototype.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "MathObject.h"
</span><span class="cx"> #include "Microtask.h"
</span><span class="cx"> #include "ModuleLoaderPrototype.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSObjectcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSObject.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSObject.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/JSObject.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -43,6 +43,7 @@
</span><span class="cx"> #include "ObjectPrototype.h"
</span><span class="cx"> #include "PropertyDescriptor.h"
</span><span class="cx"> #include "PropertyNameArray.h"
</span><ins>+#include "PrototypeMapInlines.h"
</ins><span class="cx"> #include "ProxyObject.h"
</span><span class="cx"> #include "Reject.h"
</span><span class="cx"> #include "SlotVisitorInlines.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSProxycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSProxy.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSProxy.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/JSProxy.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -28,6 +28,7 @@
</span><span class="cx">
</span><span class="cx"> #include "JSGlobalObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "PrototypeMapInlines.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeOptionsh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Options.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Options.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/Options.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -187,6 +187,7 @@
</span><span class="cx"> v(unsigned, largeAllocationCutoff, 100000, Normal, nullptr) \
</span><span class="cx"> v(bool, dumpSizeClasses, false, Normal, nullptr) \
</span><span class="cx"> v(bool, useBumpAllocator, true, Normal, nullptr) \
</span><ins>+ v(bool, stealEmptyBlocksFromOtherAllocators, true, Normal, nullptr) \
</ins><span class="cx"> v(bool, eagerlyUpdateTopCallFrame, false, Normal, nullptr) \
</span><span class="cx"> \
</span><span class="cx"> v(bool, useOSREntryToDFG, true, Normal, nullptr) \
</span><span class="lines">@@ -302,7 +303,6 @@
</span><span class="cx"> v(unsigned, numberOfGCMarkers, computeNumberOfGCMarkers(7), Normal, nullptr) \
</span><span class="cx"> v(unsigned, opaqueRootMergeThreshold, 1000, Normal, nullptr) \
</span><span class="cx"> v(double, minHeapUtilization, 0.8, Normal, nullptr) \
</span><del>- v(double, minCopiedBlockUtilization, 0.9, Normal, nullptr) \
</del><span class="cx"> v(double, minMarkedBlockUtilization, 0.9, Normal, nullptr) \
</span><span class="cx"> v(unsigned, slowPathAllocsBetweenGCs, 0, Normal, "force a GC on every Nth slow path alloc, where N is specified by this option") \
</span><span class="cx"> v(bool, deferGCShouldCollectWithProbability, false, Normal, "If true, we perform a collection based on flipping a coin according the probability in the 'deferGCProbability' option when DeferGC is destructed.") \
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimePrototypeMapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/PrototypeMap.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/PrototypeMap.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/PrototypeMap.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,6 +29,7 @@
</span><span class="cx"> #include "IndexingType.h"
</span><span class="cx"> #include "JSGlobalObject.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><ins>+#include "PrototypeMapInlines.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimePrototypeMaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/PrototypeMap.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/PrototypeMap.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/PrototypeMap.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -26,6 +26,8 @@
</span><span class="cx"> #ifndef PrototypeMap_h
</span><span class="cx"> #define PrototypeMap_h
</span><span class="cx">
</span><ins>+#include "IndexingType.h"
+#include "JSTypeInfo.h"
</ins><span class="cx"> #include "WeakGCMap.h"
</span><span class="cx"> #include <wtf/TriState.h>
</span><span class="cx">
</span><span class="lines">@@ -48,7 +50,7 @@
</span><span class="cx"> JS_EXPORT_PRIVATE Structure* emptyStructureForPrototypeFromBaseStructure(JSObject*, Structure*);
</span><span class="cx"> void clearEmptyObjectStructureForPrototype(JSObject*, unsigned inlineCapacity);
</span><span class="cx"> JS_EXPORT_PRIVATE void addPrototype(JSObject*);
</span><del>- TriState isPrototype(JSObject*) const; // Returns a conservative estimate.
</del><ins>+ inline TriState isPrototype(JSObject*) const; // Returns a conservative estimate.
</ins><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> Structure* createEmptyStructure(JSObject* prototype, const TypeInfo&, const ClassInfo*, IndexingType, unsigned inlineCapacity);
</span><span class="lines">@@ -58,18 +60,6 @@
</span><span class="cx"> StructureMap m_structures;
</span><span class="cx"> };
</span><span class="cx">
</span><del>-inline TriState PrototypeMap::isPrototype(JSObject* object) const
-{
- if (!m_prototypes.contains(object))
- return FalseTriState;
-
- // We know that 'object' was used as a prototype at one time, so be
- // conservative and say that it might still be so. (It would be expensive
- // to find out for sure, and we don't know of any cases where being precise
- // would improve performance.)
- return MixedTriState;
-}
-
</del><span class="cx"> } // namespace JSC
</span><span class="cx">
</span><span class="cx"> #endif // PrototypeMap_h
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimePrototypeMapInlinesh"></a>
<div class="addfile"><h4>Added: trunk/Source/JavaScriptCore/runtime/PrototypeMapInlines.h (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/PrototypeMapInlines.h (rev 0)
+++ trunk/Source/JavaScriptCore/runtime/PrototypeMapInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,46 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#include "PrototypeMap.h"
+#include "WeakGCMapInlines.h"
+
+namespace JSC {
+
+inline TriState PrototypeMap::isPrototype(JSObject* object) const
+{
+ if (!m_prototypes.contains(object))
+ return FalseTriState;
+
+ // We know that 'object' was used as a prototype at one time, so be
+ // conservative and say that it might still be so. (It would be expensive
+ // to find out for sure, and we don't know of any cases where being precise
+ // would improve performance.)
+ return MixedTriState;
+}
+
+} // namespace JSC
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeRegExpCacheh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/RegExpCache.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/RegExpCache.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/RegExpCache.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -29,7 +29,6 @@
</span><span class="cx"> #include "RegExpKey.h"
</span><span class="cx"> #include "Strong.h"
</span><span class="cx"> #include "Weak.h"
</span><del>-#include "WeakInlines.h"
</del><span class="cx"> #include <array>
</span><span class="cx"> #include <wtf/HashMap.h>
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeSamplingProfilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/SamplingProfiler.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -42,6 +42,7 @@
</span><span class="cx"> #include "LLIntPCRanges.h"
</span><span class="cx"> #include "MarkedBlock.h"
</span><span class="cx"> #include "MarkedBlockSet.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "PCToCodeOriginMap.h"
</span><span class="cx"> #include "SlotVisitor.h"
</span><span class="cx"> #include "SlotVisitorInlines.h"
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeWeakGCMaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/WeakGCMap.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/WeakGCMap.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/WeakGCMap.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009, 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009, 2015-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -27,7 +27,6 @@
</span><span class="cx"> #define WeakGCMap_h
</span><span class="cx">
</span><span class="cx"> #include "Weak.h"
</span><del>-#include "WeakInlines.h"
</del><span class="cx"> #include <wtf/HashMap.h>
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -81,19 +80,9 @@
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- iterator find(const KeyType& key)
- {
- iterator it = m_map.find(key);
- iterator end = m_map.end();
- if (it != end && !it->value) // Found a zombie value.
- return end;
- return it;
- }
</del><ins>+ inline iterator find(const KeyType& key);
</ins><span class="cx">
</span><del>- const_iterator find(const KeyType& key) const
- {
- return const_cast<WeakGCMap*>(this)->find(key);
- }
</del><ins>+ inline const_iterator find(const KeyType& key) const;
</ins><span class="cx">
</span><span class="cx"> template<typename Functor>
</span><span class="cx"> void forEach(Functor functor)
</span><span class="lines">@@ -104,10 +93,7 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>- bool contains(const KeyType& key) const
- {
- return find(key) != m_map.end();
- }
</del><ins>+ inline bool contains(const KeyType& key) const;
</ins><span class="cx">
</span><span class="cx"> void pruneStaleEntries();
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeWeakGCMapInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/WeakGCMapInlines.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/WeakGCMapInlines.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/runtime/WeakGCMapInlines.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -28,6 +28,7 @@
</span><span class="cx">
</span><span class="cx"> #include "HeapInlines.h"
</span><span class="cx"> #include "WeakGCMap.h"
</span><ins>+#include "WeakInlines.h"
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="lines">@@ -47,6 +48,28 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename KeyArg, typename ValueArg, typename HashArg, typename KeyTraitsArg>
</span><ins>+inline typename WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::iterator WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::find(const KeyType& key)
+{
+ iterator it = m_map.find(key);
+ iterator end = m_map.end();
+ if (it != end && !it->value) // Found a zombie value.
+ return end;
+ return it;
+}
+
+template<typename KeyArg, typename ValueArg, typename HashArg, typename KeyTraitsArg>
+inline typename WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::const_iterator WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::find(const KeyType& key) const
+{
+ return const_cast<WeakGCMap*>(this)->find(key);
+}
+
+template<typename KeyArg, typename ValueArg, typename HashArg, typename KeyTraitsArg>
+inline bool WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::contains(const KeyType& key) const
+{
+ return find(key) != m_map.end();
+}
+
+template<typename KeyArg, typename ValueArg, typename HashArg, typename KeyTraitsArg>
</ins><span class="cx"> NEVER_INLINE void WeakGCMap<KeyArg, ValueArg, HashArg, KeyTraitsArg>::pruneStaleEntries()
</span><span class="cx"> {
</span><span class="cx"> m_map.removeIf([](typename HashMapType::KeyValuePairType& entry) {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoretoolsJSDollarVMPrototypecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -31,6 +31,7 @@
</span><span class="cx"> #include "HeapIterationScope.h"
</span><span class="cx"> #include "JSCInlines.h"
</span><span class="cx"> #include "JSFunction.h"
</span><ins>+#include "MarkedSpaceInlines.h"
</ins><span class="cx"> #include "StackVisitor.h"
</span><span class="cx"> #include <wtf/DataLog.h>
</span><span class="cx"> #include <wtf/StringPrintStream.h>
</span></span></pre></div>
<a id="trunkSourceWTFChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/ChangeLog (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/ChangeLog 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/WTF/ChangeLog 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,3 +1,26 @@
</span><ins>+2016-09-20 Filip Pizlo <fpizlo@apple.com>
+
+ Make MarkedBlock state tracking support overlapped allocation and marking state
+ https://bugs.webkit.org/show_bug.cgi?id=161581
+
+ Reviewed by Geoffrey Garen.
+
+ The main change here is to bring back FastBitVector.cpp, so that I could outline some
+ large slow path functions. This also adds some utilities, like atomicSetAndCheck() and
+ isEmpty(). The GC uses these.
+
+ * WTF.xcodeproj/project.pbxproj:
+ * wtf/CMakeLists.txt:
+ * wtf/FastBitVector.cpp: Added.
+ (WTF::FastBitVectorWordOwner::setEqualsSlow):
+ (WTF::FastBitVectorWordOwner::resizeSlow):
+ * wtf/FastBitVector.h:
+ (WTF::FastBitVectorWordOwner::operator=):
+ (WTF::FastBitVectorWordOwner::resize):
+ (WTF::FastBitVectorImpl::isEmpty):
+ (WTF::FastBitVector::atomicSetAndCheck):
+ (WTF::FastBitVector::operator[]): Deleted.
+
</ins><span class="cx"> 2016-09-20 Jonathan Bedard <jbedard@apple.com>
</span><span class="cx">
</span><span class="cx"> Undefined behavior: Left shift negative number
</span></span></pre></div>
<a id="trunkSourceWTFWTFxcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/WTF.xcodeproj/project.pbxproj (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/WTF.xcodeproj/project.pbxproj 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/WTF/WTF.xcodeproj/project.pbxproj 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -28,6 +28,7 @@
</span><span class="cx"> 0F4570431BE5B58F0062A629 /* Dominators.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4570421BE5B58F0062A629 /* Dominators.h */; };
</span><span class="cx"> 0F4570451BE834410062A629 /* BubbleSort.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4570441BE834410062A629 /* BubbleSort.h */; };
</span><span class="cx"> 0F725CAC1C50461600AD943A /* RangeSet.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F725CAB1C50461600AD943A /* RangeSet.h */; };
</span><ins>+ 0F7C5FB61D885CF20044F5E2 /* FastBitVector.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F7C5FB51D885CF20044F5E2 /* FastBitVector.cpp */; };
</ins><span class="cx"> 0F824A681B7443A0002E345D /* ParkingLot.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F824A641B7443A0002E345D /* ParkingLot.cpp */; };
</span><span class="cx"> 0F824A691B7443A0002E345D /* ParkingLot.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F824A651B7443A0002E345D /* ParkingLot.h */; };
</span><span class="cx"> 0F87105A16643F190090B0AD /* RawPointer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F87105916643F190090B0AD /* RawPointer.h */; };
</span><span class="lines">@@ -366,6 +367,7 @@
</span><span class="cx"> 0F4570421BE5B58F0062A629 /* Dominators.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Dominators.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F4570441BE834410062A629 /* BubbleSort.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BubbleSort.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F725CAB1C50461600AD943A /* RangeSet.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RangeSet.h; sourceTree = "<group>"; };
</span><ins>+ 0F7C5FB51D885CF20044F5E2 /* FastBitVector.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FastBitVector.cpp; sourceTree = "<group>"; };
</ins><span class="cx"> 0F824A641B7443A0002E345D /* ParkingLot.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ParkingLot.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 0F824A651B7443A0002E345D /* ParkingLot.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ParkingLot.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F87105916643F190090B0AD /* RawPointer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RawPointer.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -880,6 +882,7 @@
</span><span class="cx"> A8A47298151A825A004123FF /* dtoa.h */,
</span><span class="cx"> 1AEA88E11D6BBCF400E5AD64 /* EnumTraits.h */,
</span><span class="cx"> A8A4729F151A825A004123FF /* ExportMacros.h */,
</span><ins>+ 0F7C5FB51D885CF20044F5E2 /* FastBitVector.cpp */,
</ins><span class="cx"> 0FD81AC4154FB22E00983E72 /* FastBitVector.h */,
</span><span class="cx"> A8A472A1151A825A004123FF /* FastMalloc.cpp */,
</span><span class="cx"> A8A472A2151A825A004123FF /* FastMalloc.h */,
</span><span class="lines">@@ -1594,6 +1597,7 @@
</span><span class="cx"> CD5497AC15857D0300B5BC30 /* MediaTime.cpp in Sources */,
</span><span class="cx"> A8A473EC151A825B004123FF /* MetaAllocator.cpp in Sources */,
</span><span class="cx"> A8A473F4151A825B004123FF /* NumberOfCores.cpp in Sources */,
</span><ins>+ 0F7C5FB61D885CF20044F5E2 /* FastBitVector.cpp in Sources */,
</ins><span class="cx"> A8A473F7151A825B004123FF /* OSAllocatorPosix.cpp in Sources */,
</span><span class="cx"> A8A473F9151A825B004123FF /* OSRandomSource.cpp in Sources */,
</span><span class="cx"> A8A47402151A825B004123FF /* PageBlock.cpp in Sources */,
</span></span></pre></div>
<a id="trunkSourceWTFwtfCMakeListstxt"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/CMakeLists.txt (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/CMakeLists.txt 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/WTF/wtf/CMakeLists.txt 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -181,6 +181,7 @@
</span><span class="cx"> DataLog.cpp
</span><span class="cx"> DateMath.cpp
</span><span class="cx"> DecimalNumber.cpp
</span><ins>+ FastBitVector.cpp
</ins><span class="cx"> FastMalloc.cpp
</span><span class="cx"> FilePrintStream.cpp
</span><span class="cx"> FunctionDispatcher.cpp
</span></span></pre></div>
<a id="trunkSourceWTFwtfFastBitVectorcpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WTF/wtf/FastBitVector.cpp (0 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/FastBitVector.cpp (rev 0)
+++ trunk/Source/WTF/wtf/FastBitVector.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -0,0 +1,57 @@
</span><ins>+/*
+ * Copyright (C) 2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "FastBitVector.h"
+
+namespace WTF {
+
+void FastBitVectorWordOwner::setEqualsSlow(const FastBitVectorWordOwner& other)
+{
+ uint32_t* newArray = static_cast<uint32_t*>(
+ fastCalloc(other.arrayLength(), sizeof(uint32_t)));
+ memcpy(newArray, other.m_words, other.arrayLength() * sizeof(uint32_t));
+ if (m_words)
+ fastFree(m_words);
+ m_words = newArray;
+ m_numBits = other.m_numBits;
+}
+
+void FastBitVectorWordOwner::resizeSlow(size_t numBits)
+{
+ size_t newLength = fastBitVectorArrayLength(numBits);
+
+ // Use fastCalloc instead of fastRealloc because we expect the common
+ // use case for this method to be initializing the size of the bitvector.
+
+ uint32_t* newArray = static_cast<uint32_t*>(fastCalloc(newLength, sizeof(uint32_t)));
+ memcpy(newArray, m_words, arrayLength() * sizeof(uint32_t));
+ if (m_words)
+ fastFree(m_words);
+ m_words = newArray;
+}
+
+} // namespace WTF
+
</ins></span></pre></div>
<a id="trunkSourceWTFwtfFastBitVectorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/FastBitVector.h (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/FastBitVector.h 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/WTF/wtf/FastBitVector.h 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -26,6 +26,7 @@
</span><span class="cx"> #pragma once
</span><span class="cx">
</span><span class="cx"> #include <string.h>
</span><ins>+#include <wtf/Atomics.h>
</ins><span class="cx"> #include <wtf/FastMalloc.h>
</span><span class="cx"> #include <wtf/PrintStream.h>
</span><span class="cx"> #include <wtf/StdLibExtras.h>
</span><span class="lines">@@ -91,17 +92,12 @@
</span><span class="cx">
</span><span class="cx"> FastBitVectorWordOwner& operator=(const FastBitVectorWordOwner& other)
</span><span class="cx"> {
</span><del>- size_t length = other.arrayLength();
- if (length == arrayLength()) {
- memcpy(m_words, other.m_words, length * sizeof(uint32_t));
- return *this;
</del><ins>+ if (arrayLength() != other.arrayLength())
+ setEqualsSlow(other);
+ else {
+ memcpy(m_words, other.m_words, arrayLength() * sizeof(uint32_t));
+ m_numBits = other.m_numBits;
</ins><span class="cx"> }
</span><del>- uint32_t* newArray = static_cast<uint32_t*>(fastCalloc(length, sizeof(uint32_t)));
- memcpy(newArray, other.m_words, length * sizeof(uint32_t));
- if (m_words)
- fastFree(m_words);
- m_words = newArray;
- m_numBits = other.m_numBits;
</del><span class="cx"> return *this;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -140,18 +136,8 @@
</span><span class="cx">
</span><span class="cx"> void resize(size_t numBits)
</span><span class="cx"> {
</span><del>- if (numBits == m_numBits)
- return;
-
- // Use fastCalloc instead of fastRealloc because we expect the common
- // use case for this method to be initializing the size of the bitvector.
-
- size_t newLength = fastBitVectorArrayLength(numBits);
- uint32_t* newArray = static_cast<uint32_t*>(fastCalloc(newLength, sizeof(uint32_t)));
- memcpy(newArray, m_words, arrayLength() * sizeof(uint32_t));
- if (m_words)
- fastFree(m_words);
- m_words = newArray;
</del><ins>+ if (arrayLength() != fastBitVectorArrayLength(numBits))
+ resizeSlow(numBits);
</ins><span class="cx"> m_numBits = numBits;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -171,6 +157,9 @@
</span><span class="cx"> uint32_t* words() { return m_words; }
</span><span class="cx">
</span><span class="cx"> private:
</span><ins>+ WTF_EXPORT_PRIVATE void setEqualsSlow(const FastBitVectorWordOwner& other);
+ WTF_EXPORT_PRIVATE void resizeSlow(size_t numBits);
+
</ins><span class="cx"> uint32_t* m_words { nullptr };
</span><span class="cx"> size_t m_numBits { 0 };
</span><span class="cx"> };
</span><span class="lines">@@ -320,6 +309,15 @@
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ bool isEmpty() const
+ {
+ for (size_t index = arrayLength(); index--;) {
+ if (m_words.word(index))
+ return false;
+ }
+ return true;
+ }
+
</ins><span class="cx"> template<typename OtherWords>
</span><span class="cx"> FastBitVectorImpl<FastBitVectorAndWords<typename Words::ViewType, typename OtherWords::ViewType>> operator&(const FastBitVectorImpl<OtherWords>& other) const
</span><span class="cx"> {
</span><span class="lines">@@ -380,7 +378,7 @@
</span><span class="cx"> // written this way so that it performs well regardless of whether value is a constant.
</span><span class="cx"> uint32_t skipValue = -(static_cast<uint32_t>(value) ^ 1);
</span><span class="cx">
</span><del>- size_t numWords = m_words.arrayLength();
</del><ins>+ size_t numWords = fastBitVectorArrayLength(m_words.numBits());
</ins><span class="cx">
</span><span class="cx"> size_t wordIndex = startIndex / 32;
</span><span class="cx"> size_t startIndexInWord = startIndex - wordIndex * 32;
</span><span class="lines">@@ -471,6 +469,7 @@
</span><span class="cx"> m_words.clearAll();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ // Returns true if the contents of this bitvector changed.
</ins><span class="cx"> template<typename OtherWords>
</span><span class="cx"> bool setAndCheck(const FastBitVectorImpl<OtherWords>& other)
</span><span class="cx"> {
</span><span class="lines">@@ -550,6 +549,28 @@
</span><span class="cx"> {
</span><span class="cx"> return at(index);
</span><span class="cx"> }
</span><ins>+
+ // Returns true if the contents changed.
+ ALWAYS_INLINE bool atomicSetAndCheck(size_t index, bool value)
+ {
+ uint32_t* pointer = &m_words.word(index >> 5);
+ uint32_t mask = 1 << (index & 31);
+ for (;;) {
+ uint32_t oldValue = *pointer;
+ uint32_t newValue;
+ if (value) {
+ if (oldValue & mask)
+ return false;
+ newValue = oldValue | mask;
+ } else {
+ if (!(oldValue & mask))
+ return false;
+ newValue = oldValue & ~mask;
+ }
+ if (weakCompareAndSwap(pointer, oldValue, newValue))
+ return true;
+ }
+ }
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> } // namespace WTF
</span></span></pre></div>
<a id="trunkSourceWebCoretestingGCObservationcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebCore/testing/GCObservation.cpp (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebCore/testing/GCObservation.cpp 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Source/WebCore/testing/GCObservation.cpp 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -27,6 +27,7 @@
</span><span class="cx"> #include "GCObservation.h"
</span><span class="cx">
</span><span class="cx"> #include <heap/HeapInlines.h>
</span><ins>+#include <heap/WeakInlines.h>
</ins><span class="cx">
</span><span class="cx"> namespace WebCore {
</span><span class="cx">
</span></span></pre></div>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Tools/ChangeLog 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -1,3 +1,14 @@
</span><ins>+2016-09-20 Filip Pizlo <fpizlo@apple.com>
+
+ Make MarkedBlock state tracking support overlapped allocation and marking state
+ https://bugs.webkit.org/show_bug.cgi?id=161581
+
+ Reviewed by Geoffrey Garen.
+
+ Remove the always-trigger-copy-phase configuration.
+
+ * Scripts/run-jsc-stress-tests:
+
</ins><span class="cx"> 2016-09-20 Don Olmstead <don.olmstead@am.sony.com>
</span><span class="cx">
</span><span class="cx"> [WinCairo] Use find_package cairo in build
</span></span></pre></div>
<a id="trunkToolsScriptsrunjscstresstests"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/run-jsc-stress-tests (206153 => 206154)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/run-jsc-stress-tests 2016-09-20 18:07:48 UTC (rev 206153)
+++ trunk/Tools/Scripts/run-jsc-stress-tests 2016-09-20 18:12:18 UTC (rev 206154)
</span><span class="lines">@@ -869,10 +869,6 @@
</span><span class="cx"> run("ftl-eager-no-cjit-osr-validation", "--validateFTLOSRExitLiveness=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + EAGER_OPTIONS))
</span><span class="cx"> end
</span><span class="cx">
</span><del>-def runAlwaysTriggerCopyPhase
- run("always-trigger-copy-phase", "--minHeapUtilization=2.0", "--minCopiedBlockUtilization=2.0")
-end
-
</del><span class="cx"> def runNoCJITNoASO
</span><span class="cx"> run("no-cjit-no-aso", "--useArchitectureSpecificOptimizations=false", *NO_CJIT_OPTIONS)
</span><span class="cx"> end
</span><span class="lines">@@ -910,7 +906,6 @@
</span><span class="cx"> defaultQuickRun
</span><span class="cx"> else
</span><span class="cx"> runDefault
</span><del>- runAlwaysTriggerCopyPhase
</del><span class="cx"> if $jitTests
</span><span class="cx"> runNoLLInt
</span><span class="cx"> runNoCJITValidatePhases
</span><span class="lines">@@ -936,7 +931,6 @@
</span><span class="cx"> defaultQuickRun
</span><span class="cx"> else
</span><span class="cx"> runDefault
</span><del>- runAlwaysTriggerCopyPhase
</del><span class="cx"> if $jitTests
</span><span class="cx"> runNoCJITValidatePhases
</span><span class="cx"> runDFGEager
</span><span class="lines">@@ -988,7 +982,6 @@
</span><span class="cx"> # by counting recompilations.
</span><span class="cx"> def defaultNoEagerRun
</span><span class="cx"> runDefault
</span><del>- runAlwaysTriggerCopyPhase
</del><span class="cx"> if $jitTests
</span><span class="cx"> runNoLLInt
</span><span class="cx"> runNoCJITValidatePhases
</span><span class="lines">@@ -1003,7 +996,6 @@
</span><span class="cx">
</span><span class="cx"> def defaultNoSamplingProfilerRun
</span><span class="cx"> runDefault
</span><del>- runAlwaysTriggerCopyPhase
</del><span class="cx"> if $jitTests
</span><span class="cx"> runNoLLInt
</span><span class="cx"> runNoCJITValidatePhases
</span><span class="lines">@@ -1141,7 +1133,6 @@
</span><span class="cx">
</span><span class="cx"> def runModules
</span><span class="cx"> run("default-modules", "-m")
</span><del>- run("always-trigger-copy-phase-modules", "-m", "--minHeapUtilization=2.0", "--minCopiedBlockUtilization=2.0")
</del><span class="cx">
</span><span class="cx"> if !$jitTests
</span><span class="cx"> return
</span></span></pre>
</div>
</div>
</body>
</html>