<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[208720] trunk</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/208720">208720</a></dd>
<dt>Author</dt> <dd>fpizlo@apple.com</dd>
<dt>Date</dt> <dd>2016-11-14 17:49:22 -0800 (Mon, 14 Nov 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>The GC should be optionally concurrent and disabled by default
https://bugs.webkit.org/show_bug.cgi?id=164454
Reviewed by Geoffrey Garen.
Source/JavaScriptCore:
This started out as a patch to have the GC scan the stack at the end, and then the
outage happened and I decided to pick a more aggresive target: give the GC a concurrent
mode that can be enabled at runtime, and whose only effect is that it turns on the
ResumeTheWorldScope. This gives our GC a really intuitive workflow: by default, the GC
thread is running solo with the world stopped and the parallel markers converged and
waiting. We have a parallel work scope to enable the parallel markers and now we have a
ResumeTheWorldScope that will optionally resume the world and then stop it again.
It's easy to make a concurrent GC that always instantly crashes. I can't promise that
this one won't do that when you run it. I set a specific goal: I wanted to do >10
concurrent GCs in debug mode with generations, optimizing JITs, and parallel marking
disabled.
To reach this milestone, I needed to do a bunch of stuff:
- The mutator needs a separate mark stack for the barrier, since it will mutate this
stack concurrently to the collector's slot visitors.
- The use of CellState to indicate whether an object is being scanned the first time or
a subsequent time was racy. It fails spectacularly when a barrier is fired at the same
time as visitChildren is running or if the barrier runs at the same time as the GC
marks the same object. So, I split SlotVisitor's mark stacks. It's now the case that
you know why you're being scanned by looking at which stack you came off of.
- All of root marking must be in the collector fixpoint. I renamed markRoots to
markToFixpoint. They say concurrency is hard, but the collector looks more intuitive
this way. We never gained anything from forcing people to make a choice between
scanning something in the fixpoint versus outside of it. Because root scanning is
cheap, we can afford to do it repeatedly, which means all root scanning can now do
constraint-based marking (like: I'll mark you if that thing is marked).
- JSObject::visitChildren's scanning of the butterfly raced with property additions,
indexed storage transitions and resizing, and a bunch of miscellaneous dirty butterfly
reshaping functions - like the one that flattens a dictionary and some sneaky
ArrayStorage transformations. Many of these can be fixed by using store-store fences
in the mutator and load-load fences in the collector. I've adopted the rule that the
collector must always see either a butterfly and structure that match or a newer
butterfly with an older structure, where their age is just one transition apart. This
can be achieved with fences. For the cases where it breaks down, I added a lock to
every JSCell. This is a full-fledged WTF lock that we sneak into two available bits in
the indexingType. See the WTF ChangeLog for details.
The mutator fencing rules are as follows:
- Store-store fence before and after setting the butterfly.
- Store-store fence before setting structure if you had changed the shape of the
butterfly.
- Store-store fence after initializing all fields in an allocation.
- A dictionary Structure can change in strange ways while the GC is trying to scan it.
So, JSObject::visitChildren will now grab the object's structure's lock if the
object's structure is a dictionary. Dictionary structures are 1:1 with their object,
so this does not reduce GC parallelism (super unlikely that the GC will simultaneously
scan an object from two threads).
- The GC can blow away a Structure's property table at any time. As a small consolation,
it's now holding the Structure's lock when it does so. But there was tons of code in
Structure that uses DeferGC to prevent the GC from blowing away the property table.
This doesn't work with concurrent GC, since DeferGC only means that the GC won't run
its safepoint (i.e. stop-the-world code) in the DeferGC region. It will still do
marking and it was the Structure::visitChildren that would delete the table. It turns
out that Structure's reliance on the property table not being deleted was the product
of code rot. We already had functions that would materialize the table on demand. We
were simply making the mistake of saying:
structure->materializePropertyMap();
...
structure->propertyTable()->things
Instead of saying:
PropertyTable* table = structure->ensurePropertyTable();
...
table->things
Switching the code to use the latter idiom allowed me to simplify the code a lot while
fixing the race.
- The LLInt's get_by_val handling was broken because the indexing shape constants were
wrong. Once I started putting more things into the IndexingType, that started causing
crashes for me. So I fixed LLInt. That turned out to be a lot of work, since that code
had rotted in subtle ways.
This is a speed-up in SunSpider, probably because of the LLInt fix. This is neutral on
Octane and Kraken. It's a smaller slow-down on LongSpider, but I think we can ignore
that (we don't view LongSpider as an official benchmark). By default, the concurrent GC
is disabled: in all of the places where it would have resumed the world to run marking
concurrently to the mutator, it will just skip the resume step. When you enable
concurrent GC (--useConcurrentGC=true), it can sometimes run Octane/splay to completion.
It seems to perform quite well: on my machine, it improves both splay-throughput and
splay-latency. It's probably unstable for other programs.
* API/JSVirtualMachine.mm:
(-[JSVirtualMachine isOldExternalObject:]):
* assembler/MacroAssemblerARMv7.h:
(JSC::MacroAssemblerARMv7::storeFence):
* bytecode/InlineAccess.cpp:
(JSC::InlineAccess::dumpCacheSizesAndCrash):
(JSC::InlineAccess::generateSelfPropertyAccess):
(JSC::InlineAccess::generateArrayLength):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::offsetOfInlineCapacity):
(JSC::ObjectAllocationProfile::ObjectAllocationProfile):
(JSC::ObjectAllocationProfile::initialize):
(JSC::ObjectAllocationProfile::inlineCapacity):
(JSC::ObjectAllocationProfile::clear):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateWithGuard):
(JSC::AccessCase::generateImpl):
* dfg/DFGArrayifySlowPathGenerator.h:
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGOSRExitCompiler32_64.cpp:
(JSC::DFG::OSRExitCompiler::compileExit):
* dfg/DFGOSRExitCompiler64.cpp:
(JSC::DFG::OSRExitCompiler::compileExit):
* dfg/DFGOperations.cpp:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::markCodeBlocks):
(JSC::DFG::Plan::rememberCodeBlocks):
* dfg/DFGPlan.h:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::checkArray):
(JSC::DFG::SpeculativeJIT::arrayify):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileNewFunctionCommon):
(JSC::DFG::SpeculativeJIT::compileCreateActivation):
(JSC::DFG::SpeculativeJIT::compileCreateDirectArguments):
(JSC::DFG::SpeculativeJIT::compileSpread):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileNewStringObject):
(JSC::DFG::SpeculativeJIT::compileNewTypedArray):
(JSC::DFG::SpeculativeJIT::compileStoreBarrier):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGTierUpCheckInjectionPhase.cpp:
(JSC::DFG::TierUpCheckInjectionPhase::run):
* dfg/DFGWorklist.cpp:
(JSC::DFG::Worklist::markCodeBlocks):
(JSC::DFG::Worklist::rememberCodeBlocks):
(JSC::DFG::markCodeBlocks):
(JSC::DFG::completeAllPlansForVM):
(JSC::DFG::rememberCodeBlocks):
* dfg/DFGWorklist.h:
* ftl/FTLAbstractHeapRepository.cpp:
(JSC::FTL::AbstractHeapRepository::AbstractHeapRepository):
(JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLJITCode.cpp:
(JSC::FTL::JITCode::~JITCode):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compilePutStructure):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateActivation):
(JSC::FTL::DFG::LowerDFGToB3::compileNewFunction):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::compileNewObject):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArray):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayBuffer):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMultiPutByOffset):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeCreateActivation):
(JSC::FTL::DFG::LowerDFGToB3::splatWords):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::reallocatePropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::isArrayType):
(JSC::FTL::DFG::LowerDFGToB3::emitStoreBarrier):
(JSC::FTL::DFG::LowerDFGToB3::mutatorFence):
(JSC::FTL::DFG::LowerDFGToB3::setButterfly):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStub):
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::signExt32ToPtr):
(JSC::FTL::Output::fence):
* ftl/FTLOutput.h:
* heap/CellState.h:
* heap/GCSegmentedArray.h:
* heap/Heap.cpp:
(JSC::Heap::ResumeTheWorldScope::ResumeTheWorldScope):
(JSC::Heap::ResumeTheWorldScope::~ResumeTheWorldScope):
(JSC::Heap::Heap):
(JSC::Heap::~Heap):
(JSC::Heap::harvestWeakReferences):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::completeAllJITPlans):
(JSC::Heap::markToFixpoint):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::beginMarking):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::visitCompilerWorklistWeakReferences):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::endMarking):
(JSC::Heap::addToRememberedSet):
(JSC::Heap::collectInThread):
(JSC::Heap::stopTheWorld):
(JSC::Heap::resumeTheWorld):
(JSC::Heap::setGCDidJIT):
(JSC::Heap::setNeedFinalize):
(JSC::Heap::setMutatorWaiting):
(JSC::Heap::clearMutatorWaiting):
(JSC::Heap::finalize):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::writeBarrierSlowPath):
(JSC::Heap::canCollect):
(JSC::Heap::reportExtraMemoryVisited):
(JSC::Heap::reportExternalMemoryVisited):
(JSC::Heap::notifyIsSafeToCollect):
(JSC::Heap::markRoots): Deleted.
(JSC::Heap::visitExternalRememberedSet): Deleted.
(JSC::Heap::visitSmallStrings): Deleted.
(JSC::Heap::visitProtectedObjects): Deleted.
(JSC::Heap::visitArgumentBuffers): Deleted.
(JSC::Heap::visitException): Deleted.
(JSC::Heap::visitStrongHandles): Deleted.
(JSC::Heap::visitHandleStack): Deleted.
(JSC::Heap::visitSamplingProfiler): Deleted.
(JSC::Heap::visitTypeProfiler): Deleted.
(JSC::Heap::visitShadowChicken): Deleted.
(JSC::Heap::traceCodeBlocksAndJITStubRoutines): Deleted.
(JSC::Heap::visitWeakHandles): Deleted.
(JSC::Heap::flushOldStructureIDTables): Deleted.
(JSC::Heap::stopAllocation): Deleted.
* heap/Heap.h:
(JSC::Heap::collectorSlotVisitor):
(JSC::Heap::mutatorMarkStack):
(JSC::Heap::mutatorShouldBeFenced):
(JSC::Heap::addressOfMutatorShouldBeFenced):
(JSC::Heap::slotVisitor): Deleted.
(JSC::Heap::notifyIsSafeToCollect): Deleted.
(JSC::Heap::barrierShouldBeFenced): Deleted.
(JSC::Heap::addressOfBarrierShouldBeFenced): Deleted.
* heap/MarkStack.cpp:
(JSC::MarkStackArray::transferTo):
* heap/MarkStack.h:
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateIn):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectMarksMode):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::aboutToMarkSlow):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): Deleted.
(JSC::SetNewlyAllocatedFunctor::operator()): Deleted.
* heap/MarkedBlock.h:
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::resumeAllocating):
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::~SlotVisitor):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::clearMarkStacks):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::appendToMutatorMarkStack):
(JSC::SlotVisitor::visitChildren):
(JSC::SlotVisitor::donateKnownParallel):
(JSC::SlotVisitor::drain):
(JSC::SlotVisitor::drainFromShared):
(JSC::SlotVisitor::containsOpaqueRoot):
(JSC::SlotVisitor::donateAndDrain):
(JSC::SlotVisitor::mergeOpaqueRoots):
(JSC::SlotVisitor::dump):
(JSC::SlotVisitor::clearMarkStack): Deleted.
(JSC::SlotVisitor::opaqueRootCount): Deleted.
* heap/SlotVisitor.h:
(JSC::SlotVisitor::collectorMarkStack):
(JSC::SlotVisitor::mutatorMarkStack):
(JSC::SlotVisitor::isEmpty):
(JSC::SlotVisitor::bytesVisited):
(JSC::SlotVisitor::markStack): Deleted.
(JSC::SlotVisitor::bytesCopied): Deleted.
* heap/SlotVisitorInlines.h:
(JSC::SlotVisitor::reportExtraMemoryVisited):
(JSC::SlotVisitor::reportExternalMemoryVisited):
* jit/AssemblyHelpers.cpp:
(JSC::AssemblyHelpers::emitStoreStructureWithTypeInfo):
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitStoreStructureWithTypeInfo):
(JSC::AssemblyHelpers::barrierStoreLoadFence):
(JSC::AssemblyHelpers::mutatorFence):
(JSC::AssemblyHelpers::storeButterfly):
(JSC::AssemblyHelpers::jumpIfMutatorFenceNotNeeded):
(JSC::AssemblyHelpers::emitInitializeInlineStorage):
(JSC::AssemblyHelpers::emitInitializeOutOfLineStorage):
(JSC::AssemblyHelpers::jumpIfBarrierStoreLoadFenceNotNeeded): Deleted.
* jit/JITInlines.h:
(JSC::JIT::emitArrayProfilingSiteWithCell):
* jit/JITOperations.cpp:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emit_op_put_to_arguments):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter64.asm:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::create):
(JSC::Butterfly::createOrGrowPropertyStorage):
* runtime/ConcurrentJITLock.h:
(JSC::GCSafeConcurrentJITLocker::NoDefer::NoDefer): Deleted.
* runtime/GenericArgumentsInlines.h:
(JSC::GenericArguments<Type>::getOwnPropertySlotByIndex):
(JSC::GenericArguments<Type>::putByIndex):
* runtime/IndexingType.h:
* runtime/JSArray.cpp:
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::unshiftCountWithArrayStorage):
* runtime/JSCell.h:
(JSC::JSCell::InternalLocker::InternalLocker):
(JSC::JSCell::InternalLocker::~InternalLocker):
(JSC::JSCell::atomicCompareExchangeCellStateWeakRelaxed):
(JSC::JSCell::atomicCompareExchangeCellStateStrong):
(JSC::JSCell::indexingTypeAndMiscOffset):
(JSC::JSCell::indexingTypeOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::JSCell::JSCell):
(JSC::JSCell::finishCreation):
(JSC::JSCell::indexingTypeAndMisc):
(JSC::JSCell::indexingType):
(JSC::JSCell::setStructure):
(JSC::JSCell::callDestructor):
(JSC::JSCell::lockInternalLock):
(JSC::JSCell::unlockInternalLock):
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::visitChildren):
(JSC::JSFinalObject::visitChildren):
(JSC::JSObject::enterDictionaryIndexingModeWhenArrayStorageAlreadyExists):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::deleteProperty):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::allocateMoreOutOfLineStorage):
(JSC::JSObject::shiftButterflyAfterFlattening):
(JSC::JSObject::growOutOfLineStorage): Deleted.
* runtime/JSObject.h:
(JSC::JSFinalObject::JSFinalObject):
(JSC::JSObject::setButterfly):
(JSC::JSObject::getOwnNonIndexPropertySlot):
(JSC::JSObject::fillCustomGetterPropertySlot):
(JSC::JSObject::getOwnPropertySlot):
(JSC::JSObject::getPropertySlot):
(JSC::JSObject::setStructureAndButterfly): Deleted.
(JSC::JSObject::setButterflyWithoutChangingStructure): Deleted.
(JSC::JSObject::putDirectInternal): Deleted.
(JSC::JSObject::putDirectWithoutTransition): Deleted.
* runtime/JSObjectInlines.h:
(JSC::JSObject::getPropertySlot):
(JSC::JSObject::getNonIndexPropertySlot):
(JSC::JSObject::putDirectWithoutTransition):
(JSC::JSObject::putDirectInternal):
* runtime/Options.h:
* runtime/SparseArrayValueMap.h:
* runtime/Structure.cpp:
(JSC::Structure::dumpStatistics):
(JSC::Structure::findStructuresAndMapForMaterialization):
(JSC::Structure::materializePropertyTable):
(JSC::Structure::addNewPropertyTransition):
(JSC::Structure::changePrototypeTransition):
(JSC::Structure::attributeChangeTransition):
(JSC::Structure::toDictionaryTransition):
(JSC::Structure::takePropertyTableOrCloneIfPinned):
(JSC::Structure::nonPropertyTransition):
(JSC::Structure::isSealed):
(JSC::Structure::isFrozen):
(JSC::Structure::flattenDictionaryStructure):
(JSC::Structure::pin):
(JSC::Structure::pinForCaching):
(JSC::Structure::willStoreValueSlow):
(JSC::Structure::copyPropertyTableForPinning):
(JSC::Structure::add):
(JSC::Structure::remove):
(JSC::Structure::getPropertyNamesFromStructure):
(JSC::Structure::visitChildren):
(JSC::Structure::materializePropertyMap): Deleted.
(JSC::Structure::addPropertyWithoutTransition): Deleted.
(JSC::Structure::removePropertyWithoutTransition): Deleted.
(JSC::Structure::copyPropertyTable): Deleted.
(JSC::Structure::createPropertyMap): Deleted.
(JSC::PropertyTable::checkConsistency): Deleted.
(JSC::Structure::checkConsistency): Deleted.
* runtime/Structure.h:
* runtime/StructureIDBlob.h:
(JSC::StructureIDBlob::StructureIDBlob):
(JSC::StructureIDBlob::indexingTypeIncludingHistory):
(JSC::StructureIDBlob::setIndexingTypeIncludingHistory):
(JSC::StructureIDBlob::indexingTypeIncludingHistoryOffset):
(JSC::StructureIDBlob::indexingType): Deleted.
(JSC::StructureIDBlob::setIndexingType): Deleted.
(JSC::StructureIDBlob::indexingTypeOffset): Deleted.
* runtime/StructureInlines.h:
(JSC::Structure::get):
(JSC::Structure::checkOffsetConsistency):
(JSC::Structure::checkConsistency):
(JSC::Structure::add):
(JSC::Structure::remove):
(JSC::Structure::addPropertyWithoutTransition):
(JSC::Structure::removePropertyWithoutTransition):
(JSC::Structure::setPropertyTable):
(JSC::Structure::putWillGrowOutOfLineStorage): Deleted.
(JSC::Structure::propertyTable): Deleted.
(JSC::Structure::suggestedNewOutOfLineStorageCapacity): Deleted.
Source/WTF:
The reason why I went to such great pains to make WTF::Lock fit in two bits is that I
knew that I would eventually need to stuff one into some miscellaneous bits of the
JSCell header. That time has come, because the concurrent GC has numerous race
conditions in visitChildren that can be trivially fixed if each object just has an
internal lock. Some cell types might use it to simply protect their entire visitChildren
function and anything that mutates the fields it touches, while other cell types might
use it as a "lock of last resort" to handle corner cases of an otherwise wait-free or
lock-free algorithm. Right now, it's used to protect certain transformations involving
indexing storage.
To make this happen, I factored the WTF::Lock algorithm into a LockAlgorithm struct that
is templatized on lock type (uint8_t for WTF::Lock), the isHeldBit value (1 for
WTF::Lock), and the hasParkedBit value (2 for WTF::Lock). This could have been done as
a templatized Lock class that basically contains Atomic<LockType>. You could then make
any field into a lock by bitwise_casting it to TemplateLock<field type, bit1, bit2>. But
this felt too dirty, so instead, LockAlgorithm has static methods that take
Atomic<LockType>& as their first argument. I think that this makes it more natural to
project a LockAlgorithm onto an existing Atomic<> field. Sadly, some places have to cast
their non-Atomic<> field to Atomic<> in order for this to work. Like so many other things
we do, this just shows that the C++ style of labeling fields that are subject to atomic
ops as atomic is counterproductive. Maybe some day I'll change LockAlgorithm to use our
other Atomics API, which does not require Atomic<>.
WTF::Lock now uses LockAlgorithm. The slow paths are still outlined. I don't feel too
bad about the LockAlgorithm.h header being included in so many places because we change
that algorithm so infrequently.
Also, I added a hasElapsed(time) function. This function makes it so much more natural
to write timeslicing code, which the concurrent GC has to do a lot of.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/ListDump.h:
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl): Deleted.
* wtf/Lock.h:
(WTF::LockBase::lock):
(WTF::LockBase::tryLock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(): Deleted.
* wtf/LockAlgorithm.h: Added.
(WTF::LockAlgorithm::lockFastAssumingZero):
(WTF::LockAlgorithm::lockFast):
(WTF::LockAlgorithm::lock):
(WTF::LockAlgorithm::tryLock):
(WTF::LockAlgorithm::unlockFastAssumingZero):
(WTF::LockAlgorithm::unlockFast):
(WTF::LockAlgorithm::unlock):
(WTF::LockAlgorithm::unlockFairly):
(WTF::LockAlgorithm::isLocked):
(WTF::LockAlgorithm::lockSlow):
(WTF::LockAlgorithm::unlockSlow):
* wtf/TimeWithDynamicClockType.cpp:
(WTF::hasElapsed):
* wtf/TimeWithDynamicClockType.h:</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkPerformanceTestsES6SampleBenchAirbenchmarkjs">trunk/PerformanceTests/ES6SampleBench/Air/benchmark.js</a></li>
<li><a href="#trunkSourceJavaScriptCoreAPIJSVirtualMachinemm">trunk/Source/JavaScriptCore/API/JSVirtualMachine.mm</a></li>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerMacroAssemblerARMv7h">trunk/Source/JavaScriptCore/assembler/MacroAssemblerARMv7.h</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeInlineAccesscpp">trunk/Source/JavaScriptCore/bytecode/InlineAccess.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeObjectAllocationProfileh">trunk/Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodePolymorphicAccesscpp">trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGArrayifySlowPathGeneratorh">trunk/Source/JavaScriptCore/dfg/DFGArrayifySlowPathGenerator.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGClobberizeh">trunk/Source/JavaScriptCore/dfg/DFGClobberize.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOSRExitCompiler32_64cpp">trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler32_64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOSRExitCompiler64cpp">trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOperationscpp">trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGPlancpp">trunk/Source/JavaScriptCore/dfg/DFGPlan.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGPlanh">trunk/Source/JavaScriptCore/dfg/DFGPlan.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJITcpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGTierUpCheckInjectionPhasecpp">trunk/Source/JavaScriptCore/dfg/DFGTierUpCheckInjectionPhase.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGWorklistcpp">trunk/Source/JavaScriptCore/dfg/DFGWorklist.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGWorklisth">trunk/Source/JavaScriptCore/dfg/DFGWorklist.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLAbstractHeapRepositorycpp">trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLAbstractHeapRepositoryh">trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLJITCodecpp">trunk/Source/JavaScriptCore/ftl/FTLJITCode.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp">trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLOSRExitCompilercpp">trunk/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLOutputcpp">trunk/Source/JavaScriptCore/ftl/FTLOutput.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLOutputh">trunk/Source/JavaScriptCore/ftl/FTLOutput.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapCellStateh">trunk/Source/JavaScriptCore/heap/CellState.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapGCSegmentedArrayh">trunk/Source/JavaScriptCore/heap/GCSegmentedArray.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapcpp">trunk/Source/JavaScriptCore/heap/Heap.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeaph">trunk/Source/JavaScriptCore/heap/Heap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkStackcpp">trunk/Source/JavaScriptCore/heap/MarkStack.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkStackh">trunk/Source/JavaScriptCore/heap/MarkStack.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedAllocatorcpp">trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockcpp">trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedBlockh">trunk/Source/JavaScriptCore/heap/MarkedBlock.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkedSpacecpp">trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorcpp">trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorh">trunk/Source/JavaScriptCore/heap/SlotVisitor.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorInlinesh">trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitAssemblyHelperscpp">trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitAssemblyHelpersh">trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITInlinesh">trunk/Source/JavaScriptCore/jit/JITInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITOperationscpp">trunk/Source/JavaScriptCore/jit/JITOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITPropertyAccesscpp">trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorellintLLIntDatacpp">trunk/Source/JavaScriptCore/llint/LLIntData.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorellintLowLevelInterpreterasm">trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm</a></li>
<li><a href="#trunkSourceJavaScriptCorellintLowLevelInterpreter64asm">trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeButterflyInlinesh">trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeConcurrentJITLockh">trunk/Source/JavaScriptCore/runtime/ConcurrentJITLock.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeGenericArgumentsInlinesh">trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeIndexingTypeh">trunk/Source/JavaScriptCore/runtime/IndexingType.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSArraycpp">trunk/Source/JavaScriptCore/runtime/JSArray.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSCellh">trunk/Source/JavaScriptCore/runtime/JSCell.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSCellInlinesh">trunk/Source/JavaScriptCore/runtime/JSCellInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSObjectcpp">trunk/Source/JavaScriptCore/runtime/JSObject.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSObjecth">trunk/Source/JavaScriptCore/runtime/JSObject.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeJSObjectInlinesh">trunk/Source/JavaScriptCore/runtime/JSObjectInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeOptionsh">trunk/Source/JavaScriptCore/runtime/Options.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeSparseArrayValueMaph">trunk/Source/JavaScriptCore/runtime/SparseArrayValueMap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeStructurecpp">trunk/Source/JavaScriptCore/runtime/Structure.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeStructureh">trunk/Source/JavaScriptCore/runtime/Structure.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeStructureIDBlobh">trunk/Source/JavaScriptCore/runtime/StructureIDBlob.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeStructureInlinesh">trunk/Source/JavaScriptCore/runtime/StructureInlines.h</a></li>
<li><a href="#trunkSourceWTFChangeLog">trunk/Source/WTF/ChangeLog</a></li>
<li><a href="#trunkSourceWTFWTFxcodeprojprojectpbxproj">trunk/Source/WTF/WTF.xcodeproj/project.pbxproj</a></li>
<li><a href="#trunkSourceWTFwtfAtomicsh">trunk/Source/WTF/wtf/Atomics.h</a></li>
<li><a href="#trunkSourceWTFwtfCMakeListstxt">trunk/Source/WTF/wtf/CMakeLists.txt</a></li>
<li><a href="#trunkSourceWTFwtfListDumph">trunk/Source/WTF/wtf/ListDump.h</a></li>
<li><a href="#trunkSourceWTFwtfLockcpp">trunk/Source/WTF/wtf/Lock.cpp</a></li>
<li><a href="#trunkSourceWTFwtfLockh">trunk/Source/WTF/wtf/Lock.h</a></li>
<li><a href="#trunkSourceWTFwtfTimeWithDynamicClockTypecpp">trunk/Source/WTF/wtf/TimeWithDynamicClockType.cpp</a></li>
<li><a href="#trunkSourceWTFwtfTimeWithDynamicClockTypeh">trunk/Source/WTF/wtf/TimeWithDynamicClockType.h</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkSourceWTFwtfLockAlgorithmh">trunk/Source/WTF/wtf/LockAlgorithm.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkPerformanceTestsES6SampleBenchAirbenchmarkjs"></a>
<div class="modfile"><h4>Modified: trunk/PerformanceTests/ES6SampleBench/Air/benchmark.js (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/PerformanceTests/ES6SampleBench/Air/benchmark.js 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/PerformanceTests/ES6SampleBench/Air/benchmark.js 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -75,10 +75,12 @@
</span><span class="cx">
</span><span class="cx"> let before = currentTime();
</span><span class="cx">
</span><del>- let benchmark = new Benchmark(verbose);
</del><ins>+ let benchmark = new AirBenchmark(verbose);
</ins><span class="cx">
</span><del>- for (let iteration = 0; iteration < numIterations; ++iteration)
</del><ins>+ for (let iteration = 0; iteration < numIterations; ++iteration) {
+ print("iteration " + iteration);
</ins><span class="cx"> benchmark.runIteration();
</span><ins>+ }
</ins><span class="cx">
</span><span class="cx"> let after = currentTime();
</span><span class="cx"> return after - before;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreAPIJSVirtualMachinemm"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/API/JSVirtualMachine.mm (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/API/JSVirtualMachine.mm 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/API/JSVirtualMachine.mm 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2013, 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2013-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -151,7 +151,7 @@
</span><span class="cx"> - (bool)isOldExternalObject:(id)object
</span><span class="cx"> {
</span><span class="cx"> JSC::VM* vm = toJS(m_group);
</span><del>- return vm->heap.slotVisitor().containsOpaqueRoot(object);
</del><ins>+ return vm->heap.collectorSlotVisitor().containsOpaqueRoot(object);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> - (void)addExternalRememberedObject:(id)object
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ChangeLog 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,3 +1,437 @@
</span><ins>+2016-11-14 Filip Pizlo <fpizlo@apple.com>
+
+ The GC should be optionally concurrent and disabled by default
+ https://bugs.webkit.org/show_bug.cgi?id=164454
+
+ Reviewed by Geoffrey Garen.
+
+ This started out as a patch to have the GC scan the stack at the end, and then the
+ outage happened and I decided to pick a more aggresive target: give the GC a concurrent
+ mode that can be enabled at runtime, and whose only effect is that it turns on the
+ ResumeTheWorldScope. This gives our GC a really intuitive workflow: by default, the GC
+ thread is running solo with the world stopped and the parallel markers converged and
+ waiting. We have a parallel work scope to enable the parallel markers and now we have a
+ ResumeTheWorldScope that will optionally resume the world and then stop it again.
+
+ It's easy to make a concurrent GC that always instantly crashes. I can't promise that
+ this one won't do that when you run it. I set a specific goal: I wanted to do >10
+ concurrent GCs in debug mode with generations, optimizing JITs, and parallel marking
+ disabled.
+
+ To reach this milestone, I needed to do a bunch of stuff:
+
+ - The mutator needs a separate mark stack for the barrier, since it will mutate this
+ stack concurrently to the collector's slot visitors.
+
+ - The use of CellState to indicate whether an object is being scanned the first time or
+ a subsequent time was racy. It fails spectacularly when a barrier is fired at the same
+ time as visitChildren is running or if the barrier runs at the same time as the GC
+ marks the same object. So, I split SlotVisitor's mark stacks. It's now the case that
+ you know why you're being scanned by looking at which stack you came off of.
+
+ - All of root marking must be in the collector fixpoint. I renamed markRoots to
+ markToFixpoint. They say concurrency is hard, but the collector looks more intuitive
+ this way. We never gained anything from forcing people to make a choice between
+ scanning something in the fixpoint versus outside of it. Because root scanning is
+ cheap, we can afford to do it repeatedly, which means all root scanning can now do
+ constraint-based marking (like: I'll mark you if that thing is marked).
+
+ - JSObject::visitChildren's scanning of the butterfly raced with property additions,
+ indexed storage transitions and resizing, and a bunch of miscellaneous dirty butterfly
+ reshaping functions - like the one that flattens a dictionary and some sneaky
+ ArrayStorage transformations. Many of these can be fixed by using store-store fences
+ in the mutator and load-load fences in the collector. I've adopted the rule that the
+ collector must always see either a butterfly and structure that match or a newer
+ butterfly with an older structure, where their age is just one transition apart. This
+ can be achieved with fences. For the cases where it breaks down, I added a lock to
+ every JSCell. This is a full-fledged WTF lock that we sneak into two available bits in
+ the indexingType. See the WTF ChangeLog for details.
+
+ The mutator fencing rules are as follows:
+
+ - Store-store fence before and after setting the butterfly.
+ - Store-store fence before setting structure if you had changed the shape of the
+ butterfly.
+ - Store-store fence after initializing all fields in an allocation.
+
+ - A dictionary Structure can change in strange ways while the GC is trying to scan it.
+ So, JSObject::visitChildren will now grab the object's structure's lock if the
+ object's structure is a dictionary. Dictionary structures are 1:1 with their object,
+ so this does not reduce GC parallelism (super unlikely that the GC will simultaneously
+ scan an object from two threads).
+
+ - The GC can blow away a Structure's property table at any time. As a small consolation,
+ it's now holding the Structure's lock when it does so. But there was tons of code in
+ Structure that uses DeferGC to prevent the GC from blowing away the property table.
+ This doesn't work with concurrent GC, since DeferGC only means that the GC won't run
+ its safepoint (i.e. stop-the-world code) in the DeferGC region. It will still do
+ marking and it was the Structure::visitChildren that would delete the table. It turns
+ out that Structure's reliance on the property table not being deleted was the product
+ of code rot. We already had functions that would materialize the table on demand. We
+ were simply making the mistake of saying:
+
+ structure->materializePropertyMap();
+ ...
+ structure->propertyTable()->things
+
+ Instead of saying:
+
+ PropertyTable* table = structure->ensurePropertyTable();
+ ...
+ table->things
+
+ Switching the code to use the latter idiom allowed me to simplify the code a lot while
+ fixing the race.
+
+ - The LLInt's get_by_val handling was broken because the indexing shape constants were
+ wrong. Once I started putting more things into the IndexingType, that started causing
+ crashes for me. So I fixed LLInt. That turned out to be a lot of work, since that code
+ had rotted in subtle ways.
+
+ This is a speed-up in SunSpider, probably because of the LLInt fix. This is neutral on
+ Octane and Kraken. It's a smaller slow-down on LongSpider, but I think we can ignore
+ that (we don't view LongSpider as an official benchmark). By default, the concurrent GC
+ is disabled: in all of the places where it would have resumed the world to run marking
+ concurrently to the mutator, it will just skip the resume step. When you enable
+ concurrent GC (--useConcurrentGC=true), it can sometimes run Octane/splay to completion.
+ It seems to perform quite well: on my machine, it improves both splay-throughput and
+ splay-latency. It's probably unstable for other programs.
+
+ * API/JSVirtualMachine.mm:
+ (-[JSVirtualMachine isOldExternalObject:]):
+ * assembler/MacroAssemblerARMv7.h:
+ (JSC::MacroAssemblerARMv7::storeFence):
+ * bytecode/InlineAccess.cpp:
+ (JSC::InlineAccess::dumpCacheSizesAndCrash):
+ (JSC::InlineAccess::generateSelfPropertyAccess):
+ (JSC::InlineAccess::generateArrayLength):
+ * bytecode/ObjectAllocationProfile.h:
+ (JSC::ObjectAllocationProfile::offsetOfInlineCapacity):
+ (JSC::ObjectAllocationProfile::ObjectAllocationProfile):
+ (JSC::ObjectAllocationProfile::initialize):
+ (JSC::ObjectAllocationProfile::inlineCapacity):
+ (JSC::ObjectAllocationProfile::clear):
+ * bytecode/PolymorphicAccess.cpp:
+ (JSC::AccessCase::generateWithGuard):
+ (JSC::AccessCase::generateImpl):
+ * dfg/DFGArrayifySlowPathGenerator.h:
+ * dfg/DFGClobberize.h:
+ (JSC::DFG::clobberize):
+ * dfg/DFGOSRExitCompiler32_64.cpp:
+ (JSC::DFG::OSRExitCompiler::compileExit):
+ * dfg/DFGOSRExitCompiler64.cpp:
+ (JSC::DFG::OSRExitCompiler::compileExit):
+ * dfg/DFGOperations.cpp:
+ * dfg/DFGPlan.cpp:
+ (JSC::DFG::Plan::markCodeBlocks):
+ (JSC::DFG::Plan::rememberCodeBlocks):
+ * dfg/DFGPlan.h:
+ * dfg/DFGSpeculativeJIT.cpp:
+ (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
+ (JSC::DFG::SpeculativeJIT::checkArray):
+ (JSC::DFG::SpeculativeJIT::arrayify):
+ (JSC::DFG::SpeculativeJIT::compileMakeRope):
+ (JSC::DFG::SpeculativeJIT::compileNewFunctionCommon):
+ (JSC::DFG::SpeculativeJIT::compileCreateActivation):
+ (JSC::DFG::SpeculativeJIT::compileCreateDirectArguments):
+ (JSC::DFG::SpeculativeJIT::compileSpread):
+ (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
+ (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
+ (JSC::DFG::SpeculativeJIT::compileNewStringObject):
+ (JSC::DFG::SpeculativeJIT::compileNewTypedArray):
+ (JSC::DFG::SpeculativeJIT::compileStoreBarrier):
+ * dfg/DFGSpeculativeJIT64.cpp:
+ (JSC::DFG::SpeculativeJIT::compile):
+ (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
+ * dfg/DFGTierUpCheckInjectionPhase.cpp:
+ (JSC::DFG::TierUpCheckInjectionPhase::run):
+ * dfg/DFGWorklist.cpp:
+ (JSC::DFG::Worklist::markCodeBlocks):
+ (JSC::DFG::Worklist::rememberCodeBlocks):
+ (JSC::DFG::markCodeBlocks):
+ (JSC::DFG::completeAllPlansForVM):
+ (JSC::DFG::rememberCodeBlocks):
+ * dfg/DFGWorklist.h:
+ * ftl/FTLAbstractHeapRepository.cpp:
+ (JSC::FTL::AbstractHeapRepository::AbstractHeapRepository):
+ (JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions):
+ * ftl/FTLAbstractHeapRepository.h:
+ * ftl/FTLJITCode.cpp:
+ (JSC::FTL::JITCode::~JITCode):
+ * ftl/FTLLowerDFGToB3.cpp:
+ (JSC::FTL::DFG::LowerDFGToB3::compilePutStructure):
+ (JSC::FTL::DFG::LowerDFGToB3::compileCreateActivation):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewFunction):
+ (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
+ (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewObject):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewArray):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSpread):
+ (JSC::FTL::DFG::LowerDFGToB3::compileSpread):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayBuffer):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
+ (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
+ (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
+ (JSC::FTL::DFG::LowerDFGToB3::compileMultiPutByOffset):
+ (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
+ (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeCreateActivation):
+ (JSC::FTL::DFG::LowerDFGToB3::splatWords):
+ (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorage):
+ (JSC::FTL::DFG::LowerDFGToB3::reallocatePropertyStorage):
+ (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
+ (JSC::FTL::DFG::LowerDFGToB3::isArrayType):
+ (JSC::FTL::DFG::LowerDFGToB3::emitStoreBarrier):
+ (JSC::FTL::DFG::LowerDFGToB3::mutatorFence):
+ (JSC::FTL::DFG::LowerDFGToB3::setButterfly):
+ * ftl/FTLOSRExitCompiler.cpp:
+ (JSC::FTL::compileStub):
+ * ftl/FTLOutput.cpp:
+ (JSC::FTL::Output::signExt32ToPtr):
+ (JSC::FTL::Output::fence):
+ * ftl/FTLOutput.h:
+ * heap/CellState.h:
+ * heap/GCSegmentedArray.h:
+ * heap/Heap.cpp:
+ (JSC::Heap::ResumeTheWorldScope::ResumeTheWorldScope):
+ (JSC::Heap::ResumeTheWorldScope::~ResumeTheWorldScope):
+ (JSC::Heap::Heap):
+ (JSC::Heap::~Heap):
+ (JSC::Heap::harvestWeakReferences):
+ (JSC::Heap::finalizeUnconditionalFinalizers):
+ (JSC::Heap::completeAllJITPlans):
+ (JSC::Heap::markToFixpoint):
+ (JSC::Heap::gatherStackRoots):
+ (JSC::Heap::beginMarking):
+ (JSC::Heap::visitConservativeRoots):
+ (JSC::Heap::visitCompilerWorklistWeakReferences):
+ (JSC::Heap::updateObjectCounts):
+ (JSC::Heap::endMarking):
+ (JSC::Heap::addToRememberedSet):
+ (JSC::Heap::collectInThread):
+ (JSC::Heap::stopTheWorld):
+ (JSC::Heap::resumeTheWorld):
+ (JSC::Heap::setGCDidJIT):
+ (JSC::Heap::setNeedFinalize):
+ (JSC::Heap::setMutatorWaiting):
+ (JSC::Heap::clearMutatorWaiting):
+ (JSC::Heap::finalize):
+ (JSC::Heap::flushWriteBarrierBuffer):
+ (JSC::Heap::writeBarrierSlowPath):
+ (JSC::Heap::canCollect):
+ (JSC::Heap::reportExtraMemoryVisited):
+ (JSC::Heap::reportExternalMemoryVisited):
+ (JSC::Heap::notifyIsSafeToCollect):
+ (JSC::Heap::markRoots): Deleted.
+ (JSC::Heap::visitExternalRememberedSet): Deleted.
+ (JSC::Heap::visitSmallStrings): Deleted.
+ (JSC::Heap::visitProtectedObjects): Deleted.
+ (JSC::Heap::visitArgumentBuffers): Deleted.
+ (JSC::Heap::visitException): Deleted.
+ (JSC::Heap::visitStrongHandles): Deleted.
+ (JSC::Heap::visitHandleStack): Deleted.
+ (JSC::Heap::visitSamplingProfiler): Deleted.
+ (JSC::Heap::visitTypeProfiler): Deleted.
+ (JSC::Heap::visitShadowChicken): Deleted.
+ (JSC::Heap::traceCodeBlocksAndJITStubRoutines): Deleted.
+ (JSC::Heap::visitWeakHandles): Deleted.
+ (JSC::Heap::flushOldStructureIDTables): Deleted.
+ (JSC::Heap::stopAllocation): Deleted.
+ * heap/Heap.h:
+ (JSC::Heap::collectorSlotVisitor):
+ (JSC::Heap::mutatorMarkStack):
+ (JSC::Heap::mutatorShouldBeFenced):
+ (JSC::Heap::addressOfMutatorShouldBeFenced):
+ (JSC::Heap::slotVisitor): Deleted.
+ (JSC::Heap::notifyIsSafeToCollect): Deleted.
+ (JSC::Heap::barrierShouldBeFenced): Deleted.
+ (JSC::Heap::addressOfBarrierShouldBeFenced): Deleted.
+ * heap/MarkStack.cpp:
+ (JSC::MarkStackArray::transferTo):
+ * heap/MarkStack.h:
+ * heap/MarkedAllocator.cpp:
+ (JSC::MarkedAllocator::tryAllocateIn):
+ * heap/MarkedBlock.cpp:
+ (JSC::MarkedBlock::MarkedBlock):
+ (JSC::MarkedBlock::Handle::specializedSweep):
+ (JSC::MarkedBlock::Handle::sweep):
+ (JSC::MarkedBlock::Handle::sweepHelperSelectMarksMode):
+ (JSC::MarkedBlock::Handle::stopAllocating):
+ (JSC::MarkedBlock::Handle::resumeAllocating):
+ (JSC::MarkedBlock::aboutToMarkSlow):
+ (JSC::MarkedBlock::Handle::didConsumeFreeList):
+ (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): Deleted.
+ (JSC::SetNewlyAllocatedFunctor::operator()): Deleted.
+ * heap/MarkedBlock.h:
+ * heap/MarkedSpace.cpp:
+ (JSC::MarkedSpace::resumeAllocating):
+ * heap/SlotVisitor.cpp:
+ (JSC::SlotVisitor::SlotVisitor):
+ (JSC::SlotVisitor::~SlotVisitor):
+ (JSC::SlotVisitor::reset):
+ (JSC::SlotVisitor::clearMarkStacks):
+ (JSC::SlotVisitor::appendJSCellOrAuxiliary):
+ (JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
+ (JSC::SlotVisitor::appendToMarkStack):
+ (JSC::SlotVisitor::appendToMutatorMarkStack):
+ (JSC::SlotVisitor::visitChildren):
+ (JSC::SlotVisitor::donateKnownParallel):
+ (JSC::SlotVisitor::drain):
+ (JSC::SlotVisitor::drainFromShared):
+ (JSC::SlotVisitor::containsOpaqueRoot):
+ (JSC::SlotVisitor::donateAndDrain):
+ (JSC::SlotVisitor::mergeOpaqueRoots):
+ (JSC::SlotVisitor::dump):
+ (JSC::SlotVisitor::clearMarkStack): Deleted.
+ (JSC::SlotVisitor::opaqueRootCount): Deleted.
+ * heap/SlotVisitor.h:
+ (JSC::SlotVisitor::collectorMarkStack):
+ (JSC::SlotVisitor::mutatorMarkStack):
+ (JSC::SlotVisitor::isEmpty):
+ (JSC::SlotVisitor::bytesVisited):
+ (JSC::SlotVisitor::markStack): Deleted.
+ (JSC::SlotVisitor::bytesCopied): Deleted.
+ * heap/SlotVisitorInlines.h:
+ (JSC::SlotVisitor::reportExtraMemoryVisited):
+ (JSC::SlotVisitor::reportExternalMemoryVisited):
+ * jit/AssemblyHelpers.cpp:
+ (JSC::AssemblyHelpers::emitStoreStructureWithTypeInfo):
+ * jit/AssemblyHelpers.h:
+ (JSC::AssemblyHelpers::emitStoreStructureWithTypeInfo):
+ (JSC::AssemblyHelpers::barrierStoreLoadFence):
+ (JSC::AssemblyHelpers::mutatorFence):
+ (JSC::AssemblyHelpers::storeButterfly):
+ (JSC::AssemblyHelpers::jumpIfMutatorFenceNotNeeded):
+ (JSC::AssemblyHelpers::emitInitializeInlineStorage):
+ (JSC::AssemblyHelpers::emitInitializeOutOfLineStorage):
+ (JSC::AssemblyHelpers::jumpIfBarrierStoreLoadFenceNotNeeded): Deleted.
+ * jit/JITInlines.h:
+ (JSC::JIT::emitArrayProfilingSiteWithCell):
+ * jit/JITOperations.cpp:
+ * jit/JITPropertyAccess.cpp:
+ (JSC::JIT::emit_op_put_to_scope):
+ (JSC::JIT::emit_op_put_to_arguments):
+ * llint/LLIntData.cpp:
+ (JSC::LLInt::Data::performAssertions):
+ * llint/LowLevelInterpreter.asm:
+ * llint/LowLevelInterpreter64.asm:
+ * runtime/ButterflyInlines.h:
+ (JSC::Butterfly::create):
+ (JSC::Butterfly::createOrGrowPropertyStorage):
+ * runtime/ConcurrentJITLock.h:
+ (JSC::GCSafeConcurrentJITLocker::NoDefer::NoDefer): Deleted.
+ * runtime/GenericArgumentsInlines.h:
+ (JSC::GenericArguments<Type>::getOwnPropertySlotByIndex):
+ (JSC::GenericArguments<Type>::putByIndex):
+ * runtime/IndexingType.h:
+ * runtime/JSArray.cpp:
+ (JSC::JSArray::unshiftCountSlowCase):
+ (JSC::JSArray::unshiftCountWithArrayStorage):
+ * runtime/JSCell.h:
+ (JSC::JSCell::InternalLocker::InternalLocker):
+ (JSC::JSCell::InternalLocker::~InternalLocker):
+ (JSC::JSCell::atomicCompareExchangeCellStateWeakRelaxed):
+ (JSC::JSCell::atomicCompareExchangeCellStateStrong):
+ (JSC::JSCell::indexingTypeAndMiscOffset):
+ (JSC::JSCell::indexingTypeOffset): Deleted.
+ * runtime/JSCellInlines.h:
+ (JSC::JSCell::JSCell):
+ (JSC::JSCell::finishCreation):
+ (JSC::JSCell::indexingTypeAndMisc):
+ (JSC::JSCell::indexingType):
+ (JSC::JSCell::setStructure):
+ (JSC::JSCell::callDestructor):
+ (JSC::JSCell::lockInternalLock):
+ (JSC::JSCell::unlockInternalLock):
+ * runtime/JSObject.cpp:
+ (JSC::JSObject::visitButterfly):
+ (JSC::JSObject::visitChildren):
+ (JSC::JSFinalObject::visitChildren):
+ (JSC::JSObject::enterDictionaryIndexingModeWhenArrayStorageAlreadyExists):
+ (JSC::JSObject::createInitialUndecided):
+ (JSC::JSObject::createInitialInt32):
+ (JSC::JSObject::createInitialDouble):
+ (JSC::JSObject::createInitialContiguous):
+ (JSC::JSObject::createArrayStorage):
+ (JSC::JSObject::convertUndecidedToArrayStorage):
+ (JSC::JSObject::convertInt32ToArrayStorage):
+ (JSC::JSObject::convertDoubleToArrayStorage):
+ (JSC::JSObject::convertContiguousToArrayStorage):
+ (JSC::JSObject::deleteProperty):
+ (JSC::JSObject::defineOwnIndexedProperty):
+ (JSC::JSObject::increaseVectorLength):
+ (JSC::JSObject::ensureLengthSlow):
+ (JSC::JSObject::reallocateAndShrinkButterfly):
+ (JSC::JSObject::allocateMoreOutOfLineStorage):
+ (JSC::JSObject::shiftButterflyAfterFlattening):
+ (JSC::JSObject::growOutOfLineStorage): Deleted.
+ * runtime/JSObject.h:
+ (JSC::JSFinalObject::JSFinalObject):
+ (JSC::JSObject::setButterfly):
+ (JSC::JSObject::getOwnNonIndexPropertySlot):
+ (JSC::JSObject::fillCustomGetterPropertySlot):
+ (JSC::JSObject::getOwnPropertySlot):
+ (JSC::JSObject::getPropertySlot):
+ (JSC::JSObject::setStructureAndButterfly): Deleted.
+ (JSC::JSObject::setButterflyWithoutChangingStructure): Deleted.
+ (JSC::JSObject::putDirectInternal): Deleted.
+ (JSC::JSObject::putDirectWithoutTransition): Deleted.
+ * runtime/JSObjectInlines.h:
+ (JSC::JSObject::getPropertySlot):
+ (JSC::JSObject::getNonIndexPropertySlot):
+ (JSC::JSObject::putDirectWithoutTransition):
+ (JSC::JSObject::putDirectInternal):
+ * runtime/Options.h:
+ * runtime/SparseArrayValueMap.h:
+ * runtime/Structure.cpp:
+ (JSC::Structure::dumpStatistics):
+ (JSC::Structure::findStructuresAndMapForMaterialization):
+ (JSC::Structure::materializePropertyTable):
+ (JSC::Structure::addNewPropertyTransition):
+ (JSC::Structure::changePrototypeTransition):
+ (JSC::Structure::attributeChangeTransition):
+ (JSC::Structure::toDictionaryTransition):
+ (JSC::Structure::takePropertyTableOrCloneIfPinned):
+ (JSC::Structure::nonPropertyTransition):
+ (JSC::Structure::isSealed):
+ (JSC::Structure::isFrozen):
+ (JSC::Structure::flattenDictionaryStructure):
+ (JSC::Structure::pin):
+ (JSC::Structure::pinForCaching):
+ (JSC::Structure::willStoreValueSlow):
+ (JSC::Structure::copyPropertyTableForPinning):
+ (JSC::Structure::add):
+ (JSC::Structure::remove):
+ (JSC::Structure::getPropertyNamesFromStructure):
+ (JSC::Structure::visitChildren):
+ (JSC::Structure::materializePropertyMap): Deleted.
+ (JSC::Structure::addPropertyWithoutTransition): Deleted.
+ (JSC::Structure::removePropertyWithoutTransition): Deleted.
+ (JSC::Structure::copyPropertyTable): Deleted.
+ (JSC::Structure::createPropertyMap): Deleted.
+ (JSC::PropertyTable::checkConsistency): Deleted.
+ (JSC::Structure::checkConsistency): Deleted.
+ * runtime/Structure.h:
+ * runtime/StructureIDBlob.h:
+ (JSC::StructureIDBlob::StructureIDBlob):
+ (JSC::StructureIDBlob::indexingTypeIncludingHistory):
+ (JSC::StructureIDBlob::setIndexingTypeIncludingHistory):
+ (JSC::StructureIDBlob::indexingTypeIncludingHistoryOffset):
+ (JSC::StructureIDBlob::indexingType): Deleted.
+ (JSC::StructureIDBlob::setIndexingType): Deleted.
+ (JSC::StructureIDBlob::indexingTypeOffset): Deleted.
+ * runtime/StructureInlines.h:
+ (JSC::Structure::get):
+ (JSC::Structure::checkOffsetConsistency):
+ (JSC::Structure::checkConsistency):
+ (JSC::Structure::add):
+ (JSC::Structure::remove):
+ (JSC::Structure::addPropertyWithoutTransition):
+ (JSC::Structure::removePropertyWithoutTransition):
+ (JSC::Structure::setPropertyTable):
+ (JSC::Structure::putWillGrowOutOfLineStorage): Deleted.
+ (JSC::Structure::propertyTable): Deleted.
+ (JSC::Structure::suggestedNewOutOfLineStorageCapacity): Deleted.
+
</ins><span class="cx"> 2016-11-14 Keith Miller <keith_miller@apple.com>
</span><span class="cx">
</span><span class="cx"> Add Wasm select
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerMacroAssemblerARMv7h"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/MacroAssemblerARMv7.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/MacroAssemblerARMv7.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/assembler/MacroAssemblerARMv7.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1344,6 +1344,13 @@
</span><span class="cx"> m_assembler.dmbSY();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ void storeFence()
+ {
+ // FIXME: We should actually implement this. The only current caller is related to
+ // concurrent GC, which is disabled on 32-bit systems.
+ // https://bugs.webkit.org/show_bug.cgi?id=164733
+ }
+
</ins><span class="cx"> static void replaceWithJump(CodeLocationLabel instructionStart, CodeLocationLabel destination)
</span><span class="cx"> {
</span><span class="cx"> ARMv7Assembler::replaceWithJump(instructionStart.dataLocation(), destination.dataLocation());
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeInlineAccesscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/InlineAccess.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/InlineAccess.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/bytecode/InlineAccess.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -53,7 +53,7 @@
</span><span class="cx"> CCallHelpers jit(&vm);
</span><span class="cx">
</span><span class="cx"> GPRReg scratchGPR = value;
</span><del>- jit.load8(CCallHelpers::Address(base, JSCell::indexingTypeOffset()), value);
</del><ins>+ jit.load8(CCallHelpers::Address(base, JSCell::indexingTypeAndMiscOffset()), value);
</ins><span class="cx"> jit.and32(CCallHelpers::TrustedImm32(IsArray | IndexingShapeMask), value);
</span><span class="cx"> jit.patchableBranch32(
</span><span class="cx"> CCallHelpers::NotEqual, value, CCallHelpers::TrustedImm32(IsArray | ContiguousShape));
</span><span class="lines">@@ -158,7 +158,7 @@
</span><span class="cx"> bool InlineAccess::generateSelfPropertyAccess(VM& vm, StructureStubInfo& stubInfo, Structure* structure, PropertyOffset offset)
</span><span class="cx"> {
</span><span class="cx"> CCallHelpers jit(&vm);
</span><del>-
</del><ins>+
</ins><span class="cx"> GPRReg base = static_cast<GPRReg>(stubInfo.patch.baseGPR);
</span><span class="cx"> JSValueRegs value = stubInfo.valueRegs();
</span><span class="cx">
</span><span class="lines">@@ -173,7 +173,7 @@
</span><span class="cx"> jit.loadPtr(CCallHelpers::Address(base, JSObject::butterflyOffset()), value.payloadGPR());
</span><span class="cx"> storage = value.payloadGPR();
</span><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> jit.loadValue(
</span><span class="cx"> MacroAssembler::Address(storage, offsetRelativeToBase(offset)), value);
</span><span class="cx">
</span><span class="lines">@@ -265,7 +265,7 @@
</span><span class="cx"> JSValueRegs value = stubInfo.valueRegs();
</span><span class="cx"> GPRReg scratch = getScratchRegister(stubInfo);
</span><span class="cx">
</span><del>- jit.load8(CCallHelpers::Address(base, JSCell::indexingTypeOffset()), scratch);
</del><ins>+ jit.load8(CCallHelpers::Address(base, JSCell::indexingTypeAndMiscOffset()), scratch);
</ins><span class="cx"> jit.and32(CCallHelpers::TrustedImm32(IsArray | IndexingShapeMask), scratch);
</span><span class="cx"> auto branchToSlowPath = jit.patchableBranch32(
</span><span class="cx"> CCallHelpers::NotEqual, scratch, CCallHelpers::TrustedImm32(array->indexingType()));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeObjectAllocationProfileh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -38,9 +38,11 @@
</span><span class="cx"> public:
</span><span class="cx"> static ptrdiff_t offsetOfAllocator() { return OBJECT_OFFSETOF(ObjectAllocationProfile, m_allocator); }
</span><span class="cx"> static ptrdiff_t offsetOfStructure() { return OBJECT_OFFSETOF(ObjectAllocationProfile, m_structure); }
</span><ins>+ static ptrdiff_t offsetOfInlineCapacity() { return OBJECT_OFFSETOF(ObjectAllocationProfile, m_inlineCapacity); }
</ins><span class="cx">
</span><span class="cx"> ObjectAllocationProfile()
</span><span class="cx"> : m_allocator(0)
</span><ins>+ , m_inlineCapacity(0)
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -50,6 +52,7 @@
</span><span class="cx"> {
</span><span class="cx"> ASSERT(!m_allocator);
</span><span class="cx"> ASSERT(!m_structure);
</span><ins>+ ASSERT(!m_inlineCapacity);
</ins><span class="cx">
</span><span class="cx"> unsigned inlineCapacity = 0;
</span><span class="cx"> if (inferredInlineCapacity < JSFinalObject::defaultInlineCapacity()) {
</span><span class="lines">@@ -96,6 +99,7 @@
</span><span class="cx">
</span><span class="cx"> m_allocator = allocator;
</span><span class="cx"> m_structure.set(vm, owner, structure);
</span><ins>+ m_inlineCapacity = inlineCapacity;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* structure()
</span><span class="lines">@@ -105,12 +109,13 @@
</span><span class="cx"> WTF::loadLoadFence();
</span><span class="cx"> return structure;
</span><span class="cx"> }
</span><del>- unsigned inlineCapacity() { return structure()->inlineCapacity(); }
</del><ins>+ unsigned inlineCapacity() { return m_inlineCapacity; }
</ins><span class="cx">
</span><span class="cx"> void clear()
</span><span class="cx"> {
</span><span class="cx"> m_allocator = 0;
</span><span class="cx"> m_structure.clear();
</span><ins>+ m_inlineCapacity = 0;
</ins><span class="cx"> ASSERT(isNull());
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -145,6 +150,7 @@
</span><span class="cx">
</span><span class="cx"> MarkedAllocator* m_allocator; // Precomputed to make things easier for generated code.
</span><span class="cx"> WriteBarrier<Structure> m_structure;
</span><ins>+ unsigned m_inlineCapacity;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodePolymorphicAccesscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -607,7 +607,7 @@
</span><span class="cx"> switch (m_type) {
</span><span class="cx"> case ArrayLength: {
</span><span class="cx"> ASSERT(!viaProxy());
</span><del>- jit.load8(CCallHelpers::Address(baseGPR, JSCell::indexingTypeOffset()), scratchGPR);
</del><ins>+ jit.load8(CCallHelpers::Address(baseGPR, JSCell::indexingTypeAndMiscOffset()), scratchGPR);
</ins><span class="cx"> fallThrough.append(
</span><span class="cx"> jit.branchTest32(
</span><span class="cx"> CCallHelpers::Zero, scratchGPR, CCallHelpers::TrustedImm32(IsArray)));
</span><span class="lines">@@ -1257,12 +1257,13 @@
</span><span class="cx"> jit.emitAllocate(scratchGPR, allocator, scratchGPR2, scratchGPR3, slowPath);
</span><span class="cx"> jit.addPtr(CCallHelpers::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR);
</span><span class="cx">
</span><ins>+ size_t oldSize = structure()->outOfLineCapacity() * sizeof(JSValue);
+ ASSERT(newSize > oldSize);
+
</ins><span class="cx"> if (reallocating) {
</span><span class="cx"> // Handle the case where we are reallocating (i.e. the old structure/butterfly
</span><span class="cx"> // already had out-of-line property storage).
</span><del>- size_t oldSize = structure()->outOfLineCapacity() * sizeof(JSValue);
- ASSERT(newSize > oldSize);
-
</del><ins>+
</ins><span class="cx"> jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR3);
</span><span class="cx">
</span><span class="cx"> // We have scratchGPR = new storage, scratchGPR3 = old storage,
</span><span class="lines">@@ -1281,6 +1282,9 @@
</span><span class="cx"> -static_cast<ptrdiff_t>(offset + sizeof(JSValue) + sizeof(void*))));
</span><span class="cx"> }
</span><span class="cx"> }
</span><ins>+
+ for (size_t offset = oldSize; offset < newSize; offset += sizeof(void*))
+ jit.storePtr(CCallHelpers::TrustedImmPtr(0), CCallHelpers::Address(scratchGPR, -static_cast<ptrdiff_t>(offset + sizeof(JSValue) + sizeof(void*))));
</ins><span class="cx"> } else {
</span><span class="cx"> // Handle the case where we are allocating out-of-line using an operation.
</span><span class="cx"> RegisterSet extraRegistersToPreserve;
</span><span class="lines">@@ -1353,8 +1357,7 @@
</span><span class="cx"> // We set the new butterfly and the structure last. Doing it this way ensures that
</span><span class="cx"> // whatever we had done up to this point is forgotten if we choose to branch to slow
</span><span class="cx"> // path.
</span><del>-
- jit.storePtr(scratchGPR, CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()));
</del><ins>+ jit.storeButterfly(scratchGPR, baseGPR);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> uint32_t structureBits = bitwise_cast<uint32_t>(newStructure()->id());
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGArrayifySlowPathGeneratorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGArrayifySlowPathGenerator.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGArrayifySlowPathGenerator.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGArrayifySlowPathGenerator.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -124,7 +124,8 @@
</span><span class="cx"> // bytecode operation corresponding to this arrayification being reexecuted.
</span><span class="cx"> // That's fine, since arrayification is not user-visible.
</span><span class="cx"> jit->m_jit.load8(
</span><del>- MacroAssembler::Address(m_baseGPR, JSCell::indexingTypeOffset()), m_structureGPR);
</del><ins>+ MacroAssembler::Address(m_baseGPR, JSCell::indexingTypeAndMiscOffset()),
+ m_structureGPR);
</ins><span class="cx"> m_badIndexingTypeJump.fill(
</span><span class="cx"> jit, jit->jumpSlowForUnwantedArrayMode(m_structureGPR, m_arrayMode));
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGClobberizeh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGClobberize.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGClobberize.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGClobberize.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -889,6 +889,7 @@
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> case PutStructure:
</span><ins>+ read(JSObject_butterfly); // This is a store-store fence.
</ins><span class="cx"> write(JSCell_structureID);
</span><span class="cx"> write(JSCell_typeInfoType);
</span><span class="cx"> write(JSCell_typeInfoFlags);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOSRExitCompiler32_64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler32_64.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler32_64.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler32_64.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -118,7 +118,7 @@
</span><span class="cx">
</span><span class="cx"> m_jit.loadPtr(AssemblyHelpers::Address(value, JSCell::structureIDOffset()), scratch1);
</span><span class="cx"> m_jit.storePtr(scratch1, arrayProfile->addressOfLastSeenStructureID());
</span><del>- m_jit.load8(AssemblyHelpers::Address(scratch1, Structure::indexingTypeOffset()), scratch1);
</del><ins>+ m_jit.load8(AssemblyHelpers::Address(scratch1, Structure::indexingTypeIncludingHistoryOffset()), scratch1);
</ins><span class="cx"> m_jit.move(AssemblyHelpers::TrustedImm32(1), scratch2);
</span><span class="cx"> m_jit.lshift32(scratch1, scratch2);
</span><span class="cx"> m_jit.or32(scratch2, AssemblyHelpers::AbsoluteAddress(arrayProfile->addressOfArrayModes()));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOSRExitCompiler64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler64.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler64.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompiler64.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -119,7 +119,7 @@
</span><span class="cx">
</span><span class="cx"> m_jit.load32(AssemblyHelpers::Address(value, JSCell::structureIDOffset()), scratch1);
</span><span class="cx"> m_jit.store32(scratch1, arrayProfile->addressOfLastSeenStructureID());
</span><del>- m_jit.load8(AssemblyHelpers::Address(value, JSCell::indexingTypeOffset()), scratch1);
</del><ins>+ m_jit.load8(AssemblyHelpers::Address(value, JSCell::indexingTypeAndMiscOffset()), scratch1);
</ins><span class="cx"> m_jit.move(AssemblyHelpers::TrustedImm32(1), scratch2);
</span><span class="cx"> m_jit.lshift32(scratch1, scratch2);
</span><span class="cx"> m_jit.or32(scratch2, AssemblyHelpers::AbsoluteAddress(arrayProfile->addressOfArrayModes()));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -506,9 +506,9 @@
</span><span class="cx"> if (LIKELY(baseValue.isCell())) {
</span><span class="cx"> JSCell* base = baseValue.asCell();
</span><span class="cx">
</span><del>- if (property.isUInt32()) {
</del><ins>+ if (property.isUInt32())
</ins><span class="cx"> return getByVal(exec, base, property.asUInt32());
</span><del>- } else if (property.isDouble()) {
</del><ins>+ else if (property.isDouble()) {
</ins><span class="cx"> double propertyAsDouble = property.asDouble();
</span><span class="cx"> uint32_t propertyAsUInt32 = static_cast<uint32_t>(propertyAsDouble);
</span><span class="cx"> if (propertyAsUInt32 == propertyAsDouble && isIndex(propertyAsUInt32))
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGPlancpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGPlan.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGPlan.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGPlan.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -610,18 +610,36 @@
</span><span class="cx"> return CompilationKey(codeBlock->alternative(), mode);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Plan::rememberCodeBlocks()
</del><ins>+void Plan::markCodeBlocks(SlotVisitor& slotVisitor)
</ins><span class="cx"> {
</span><ins>+ if (!isKnownToBeLiveDuringGC())
+ return;
+
</ins><span class="cx"> // Compilation writes lots of values to a CodeBlock without performing
</span><span class="cx"> // an explicit barrier. So, we need to be pessimistic and assume that
</span><span class="cx"> // all our CodeBlocks must be visited during GC.
</span><span class="cx">
</span><del>- Heap::heap(codeBlock)->writeBarrier(codeBlock);
- Heap::heap(codeBlock)->writeBarrier(codeBlock->alternative());
</del><ins>+ slotVisitor.appendUnbarrieredReadOnlyPointer(codeBlock);
+ slotVisitor.appendUnbarrieredReadOnlyPointer(codeBlock->alternative());
</ins><span class="cx"> if (profiledDFGCodeBlock)
</span><del>- Heap::heap(profiledDFGCodeBlock)->writeBarrier(profiledDFGCodeBlock);
</del><ins>+ slotVisitor.appendUnbarrieredReadOnlyPointer(profiledDFGCodeBlock);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void Plan::rememberCodeBlocks(VM& vm)
+{
+ if (!isKnownToBeLiveDuringGC())
+ return;
+
+ // Compilation writes lots of values to a CodeBlock without performing
+ // an explicit barrier. So, we need to be pessimistic and assume that
+ // all our CodeBlocks must be visited during GC.
+
+ vm.heap.writeBarrier(codeBlock);
+ vm.heap.writeBarrier(codeBlock->alternative());
+ if (profiledDFGCodeBlock)
+ vm.heap.writeBarrier(profiledDFGCodeBlock);
+}
+
</ins><span class="cx"> void Plan::checkLivenessAndVisitChildren(SlotVisitor& visitor)
</span><span class="cx"> {
</span><span class="cx"> if (!isKnownToBeLiveDuringGC())
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGPlanh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGPlan.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGPlan.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGPlan.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -69,7 +69,8 @@
</span><span class="cx">
</span><span class="cx"> CompilationKey key();
</span><span class="cx">
</span><del>- void rememberCodeBlocks();
</del><ins>+ void markCodeBlocks(SlotVisitor&);
+ void rememberCodeBlocks(VM&);
</ins><span class="cx"> void checkLivenessAndVisitChildren(SlotVisitor&);
</span><span class="cx"> bool isKnownToBeLiveDuringGC();
</span><span class="cx"> void cancel();
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJITcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -130,6 +130,7 @@
</span><span class="cx"> if (allocatorPtr) {
</span><span class="cx"> m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
</span><span class="cx"> emitAllocateJSObject(resultGPR, allocatorPtr, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
</span><ins>+ m_jit.emitInitializeInlineStorage(resultGPR, structure->inlineCapacity());
</ins><span class="cx"> } else
</span><span class="cx"> slowCases.append(m_jit.jump());
</span><span class="cx">
</span><span class="lines">@@ -163,6 +164,10 @@
</span><span class="cx">
</span><span class="cx"> if (hasIndexingHeader)
</span><span class="cx"> m_jit.store32(TrustedImm32(numElements), MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
</span><ins>+
+ m_jit.emitInitializeOutOfLineStorage(storageGPR, structure->outOfLineCapacity());
+
+ m_jit.mutatorFence();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SpeculativeJIT::emitGetLength(InlineCallFrame* inlineCallFrame, GPRReg lengthGPR, bool includeThis)
</span><span class="lines">@@ -839,7 +844,7 @@
</span><span class="cx"> case Array::SlowPutArrayStorage: {
</span><span class="cx"> GPRTemporary temp(this);
</span><span class="cx"> GPRReg tempGPR = temp.gpr();
</span><del>- m_jit.load8(MacroAssembler::Address(baseReg, JSCell::indexingTypeOffset()), tempGPR);
</del><ins>+ m_jit.load8(MacroAssembler::Address(baseReg, JSCell::indexingTypeAndMiscOffset()), tempGPR);
</ins><span class="cx"> speculationCheck(
</span><span class="cx"> BadIndexingType, JSValueSource::unboxedCell(baseReg), 0,
</span><span class="cx"> jumpSlowForUnwantedArrayMode(tempGPR, node->arrayMode()));
</span><span class="lines">@@ -903,7 +908,7 @@
</span><span class="cx"> node->structure()));
</span><span class="cx"> } else {
</span><span class="cx"> m_jit.load8(
</span><del>- MacroAssembler::Address(baseReg, JSCell::indexingTypeOffset()), tempGPR);
</del><ins>+ MacroAssembler::Address(baseReg, JSCell::indexingTypeAndMiscOffset()), tempGPR);
</ins><span class="cx">
</span><span class="cx"> slowPath.append(jumpSlowForUnwantedArrayMode(tempGPR, node->arrayMode()));
</span><span class="cx"> }
</span><span class="lines">@@ -3931,6 +3936,8 @@
</span><span class="cx"> }
</span><span class="cx"> m_jit.store32(allocatorGPR, JITCompiler::Address(resultGPR, JSString::offsetOfLength()));
</span><span class="cx">
</span><ins>+ m_jit.mutatorFence();
+
</ins><span class="cx"> switch (numOpGPRs) {
</span><span class="cx"> case 2:
</span><span class="cx"> addSlowPathGenerator(slowPathCall(
</span><span class="lines">@@ -3944,7 +3951,7 @@
</span><span class="cx"> RELEASE_ASSERT_NOT_REACHED();
</span><span class="cx"> break;
</span><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> cellResult(resultGPR, node);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -6356,6 +6363,8 @@
</span><span class="cx"> m_jit.storePtr(scopeGPR, JITCompiler::Address(resultGPR, offsetOfScopeChain));
</span><span class="cx"> m_jit.storePtr(TrustedImmPtr(executable), JITCompiler::Address(resultGPR, offsetOfExecutable));
</span><span class="cx"> m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(resultGPR, offsetOfRareData));
</span><ins>+
+ m_jit.mutatorFence();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SpeculativeJIT::compileNewFunction(Node* node)
</span><span class="lines">@@ -6556,6 +6565,8 @@
</span><span class="cx"> JITCompiler::Address(
</span><span class="cx"> resultGPR, JSLexicalEnvironment::offsetOfVariable(ScopeOffset(i))));
</span><span class="cx"> }
</span><ins>+
+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> #if USE(JSVALUE64)
</span><span class="cx"> addSlowPathGenerator(
</span><span class="lines">@@ -6635,7 +6646,7 @@
</span><span class="cx"> m_jit.move(TrustedImm32(DirectArguments::allocationSize(minCapacity)), scratch1GPR);
</span><span class="cx"> done.link(&m_jit);
</span><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> emitAllocateVariableSizedJSObject<DirectArguments>(
</span><span class="cx"> resultGPR, TrustedImmPtr(structure), scratch1GPR, scratch1GPR, scratch2GPR,
</span><span class="cx"> slowPath);
</span><span class="lines">@@ -6650,7 +6661,7 @@
</span><span class="cx">
</span><span class="cx"> m_jit.storePtr(
</span><span class="cx"> TrustedImmPtr(0), JITCompiler::Address(resultGPR, DirectArguments::offsetOfOverrides()));
</span><del>-
</del><ins>+
</ins><span class="cx"> if (lengthIsKnown) {
</span><span class="cx"> addSlowPathGenerator(
</span><span class="cx"> slowPathCall(
</span><span class="lines">@@ -6715,6 +6726,8 @@
</span><span class="cx"> done.link(&m_jit);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ m_jit.mutatorFence();
+
</ins><span class="cx"> cellResult(resultGPR, node);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -6923,7 +6936,7 @@
</span><span class="cx">
</span><span class="cx"> MacroAssembler::JumpList slowPath;
</span><span class="cx">
</span><del>- m_jit.load8(MacroAssembler::Address(argument, JSCell::indexingTypeOffset()), scratch1GPR);
</del><ins>+ m_jit.load8(MacroAssembler::Address(argument, JSCell::indexingTypeAndMiscOffset()), scratch1GPR);
</ins><span class="cx"> m_jit.and32(TrustedImm32(IndexingShapeMask), scratch1GPR);
</span><span class="cx"> m_jit.sub32(TrustedImm32(Int32Shape), scratch1GPR);
</span><span class="cx">
</span><span class="lines">@@ -6943,7 +6956,7 @@
</span><span class="cx">
</span><span class="cx"> MacroAssembler::JumpList done;
</span><span class="cx">
</span><del>- m_jit.load8(MacroAssembler::Address(argument, JSCell::indexingTypeOffset()), scratch2GPR);
</del><ins>+ m_jit.load8(MacroAssembler::Address(argument, JSCell::indexingTypeAndMiscOffset()), scratch2GPR);
</ins><span class="cx"> m_jit.and32(TrustedImm32(IndexingShapeMask), scratch2GPR);
</span><span class="cx"> auto isDoubleArray = m_jit.branch32(MacroAssembler::Equal, scratch2GPR, TrustedImm32(DoubleShape));
</span><span class="cx">
</span><span class="lines">@@ -6977,6 +6990,8 @@
</span><span class="cx"> m_jit.branchTest32(MacroAssembler::NonZero, lengthGPR).linkTo(loopStart, &m_jit);
</span><span class="cx"> done.append(m_jit.jump());
</span><span class="cx"> }
</span><ins>+
+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> slowPath.link(&m_jit);
</span><span class="cx"> addSlowPathGenerator(slowPathCall(m_jit.jump(), this, operationSpreadFastArray, resultGPR, argument));
</span><span class="lines">@@ -7395,11 +7410,17 @@
</span><span class="cx"> JITCompiler::JumpList slowPath;
</span><span class="cx"> m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
</span><span class="cx"> m_jit.addPtr(JITCompiler::TrustedImm32(size + sizeof(IndexingHeader)), scratchGPR1);
</span><ins>+
+ for (ptrdiff_t offset = 0; offset < static_cast<ptrdiff_t>(size); offset += sizeof(void*))
+ m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(scratchGPR1, -(offset + sizeof(JSValue) + sizeof(void*))));
+
+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> addSlowPathGenerator(
</span><span class="cx"> slowPathCall(slowPath, this, operationAllocatePropertyStorageWithInitialCapacity, scratchGPR1));
</span><span class="cx">
</span><span class="cx"> m_jit.storePtr(scratchGPR1, JITCompiler::Address(baseGPR, JSObject::butterflyOffset()));
</span><ins>+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> storageResult(scratchGPR1, node);
</span><span class="cx"> }
</span><span class="lines">@@ -7445,6 +7466,9 @@
</span><span class="cx">
</span><span class="cx"> m_jit.addPtr(JITCompiler::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR1);
</span><span class="cx">
</span><ins>+ for (ptrdiff_t offset = oldSize; offset < static_cast<ptrdiff_t>(newSize); offset += sizeof(void*))
+ m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(scratchGPR1, -(offset + sizeof(JSValue) + sizeof(void*))));
+
</ins><span class="cx"> addSlowPathGenerator(
</span><span class="cx"> slowPathCall(slowPath, this, operationAllocatePropertyStorage, scratchGPR1, newSize / sizeof(JSValue)));
</span><span class="cx">
</span><span class="lines">@@ -7453,7 +7477,8 @@
</span><span class="cx"> m_jit.loadPtr(JITCompiler::Address(oldStorageGPR, -(offset + sizeof(JSValue) + sizeof(void*))), scratchGPR2);
</span><span class="cx"> m_jit.storePtr(scratchGPR2, JITCompiler::Address(scratchGPR1, -(offset + sizeof(JSValue) + sizeof(void*))));
</span><span class="cx"> }
</span><del>- m_jit.storePtr(scratchGPR1, JITCompiler::Address(baseGPR, JSObject::butterflyOffset()));
</del><ins>+
+ m_jit.storeButterfly(scratchGPR1, baseGPR);
</ins><span class="cx">
</span><span class="cx"> storageResult(scratchGPR1, node);
</span><span class="cx"> }
</span><span class="lines">@@ -7741,6 +7766,8 @@
</span><span class="cx"> JITCompiler::Address(resultGPR, JSWrapperObject::internalValueOffset() + OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
</span><span class="cx"> #endif
</span><span class="cx">
</span><ins>+ m_jit.mutatorFence();
+
</ins><span class="cx"> addSlowPathGenerator(slowPathCall(
</span><span class="cx"> slowPath, this, operationNewStringObject, resultGPR, operandGPR, node->structure()));
</span><span class="cx">
</span><span class="lines">@@ -7818,6 +7845,8 @@
</span><span class="cx"> TrustedImm32(FastTypedArray),
</span><span class="cx"> MacroAssembler::Address(resultGPR, JSArrayBufferView::offsetOfMode()));
</span><span class="cx">
</span><ins>+ m_jit.mutatorFence();
+
</ins><span class="cx"> addSlowPathGenerator(slowPathCall(
</span><span class="cx"> slowCases, this, operationNewTypedArrayWithSizeForType(type),
</span><span class="cx"> resultGPR, structure, sizeGPR, storageGPR));
</span><span class="lines">@@ -8886,7 +8915,7 @@
</span><span class="cx"> if (isFenced) {
</span><span class="cx"> ok.append(m_jit.barrierBranch(baseGPR, scratch1GPR));
</span><span class="cx">
</span><del>- JITCompiler::Jump noFence = m_jit.jumpIfBarrierStoreLoadFenceNotNeeded();
</del><ins>+ JITCompiler::Jump noFence = m_jit.jumpIfMutatorFenceNotNeeded();
</ins><span class="cx"> m_jit.memoryFence();
</span><span class="cx"> ok.append(m_jit.barrierBranchWithoutFence(baseGPR));
</span><span class="cx"> noFence.link(&m_jit);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -4102,6 +4102,7 @@
</span><span class="cx"> // Rare data is only used to access the allocator & structure
</span><span class="cx"> // We can avoid using an additional GPR this way
</span><span class="cx"> GPRReg rareDataGPR = structureGPR;
</span><ins>+ GPRReg inlineCapacityGPR = rareDataGPR;
</ins><span class="cx">
</span><span class="cx"> MacroAssembler::JumpList slowPath;
</span><span class="cx">
</span><span class="lines">@@ -4113,6 +4114,11 @@
</span><span class="cx"> m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
</span><span class="cx"> slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
</span><span class="cx"> emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
</span><ins>+
+ m_jit.loadPtr(JITCompiler::Address(calleeGPR, JSFunction::offsetOfRareData()), rareDataGPR);
+ m_jit.load32(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfInlineCapacity()), inlineCapacityGPR);
+ m_jit.emitInitializeInlineStorage(resultGPR, inlineCapacityGPR);
+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
</span><span class="cx">
</span><span class="lines">@@ -4137,6 +4143,8 @@
</span><span class="cx">
</span><span class="cx"> m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
</span><span class="cx"> emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
</span><ins>+ m_jit.emitInitializeInlineStorage(resultGPR, structure->inlineCapacity());
+ m_jit.mutatorFence();
</ins><span class="cx">
</span><span class="cx"> addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
</span><span class="cx">
</span><span class="lines">@@ -5963,11 +5971,13 @@
</span><span class="cx"> m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
</span><span class="cx"> m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
</span><span class="cx"> done.link(&m_jit);
</span><del>-
</del><ins>+
</ins><span class="cx"> Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
</span><del>-
</del><ins>+
</ins><span class="cx"> emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
</span><span class="cx">
</span><ins>+ m_jit.mutatorFence();
+
</ins><span class="cx"> addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
</span><span class="cx"> slowCases, this, operationNewArrayWithSize, resultGPR,
</span><span class="cx"> structure,
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGTierUpCheckInjectionPhasecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGTierUpCheckInjectionPhase.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGTierUpCheckInjectionPhase.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGTierUpCheckInjectionPhase.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -64,7 +64,7 @@
</span><span class="cx">
</span><span class="cx"> if (!Options::useOSREntryToFTL())
</span><span class="cx"> level = FTL::CanCompile;
</span><del>-
</del><ins>+
</ins><span class="cx"> m_graph.ensureNaturalLoops();
</span><span class="cx"> NaturalLoops& naturalLoops = *m_graph.m_naturalLoops;
</span><span class="cx"> HashMap<const NaturalLoop*, unsigned> naturalLoopToLoopHint = buildNaturalLoopToLoopHintMap(naturalLoops);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGWorklistcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGWorklist.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGWorklist.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGWorklist.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -26,8 +26,6 @@
</span><span class="cx"> #include "config.h"
</span><span class="cx"> #include "DFGWorklist.h"
</span><span class="cx">
</span><del>-#if ENABLE(DFG_JIT)
-
</del><span class="cx"> #include "CodeBlock.h"
</span><span class="cx"> #include "DFGLongLivedState.h"
</span><span class="cx"> #include "DFGSafepoint.h"
</span><span class="lines">@@ -38,6 +36,8 @@
</span><span class="cx">
</span><span class="cx"> namespace JSC { namespace DFG {
</span><span class="cx">
</span><ins>+#if ENABLE(DFG_JIT)
+
</ins><span class="cx"> class Worklist::ThreadBody : public AutomaticThread {
</span><span class="cx"> public:
</span><span class="cx"> ThreadBody(const LockHolder& locker, Worklist& worklist, ThreadData& data, Box<Lock> lock, RefPtr<AutomaticThreadCondition> condition, int relativePriority)
</span><span class="lines">@@ -349,6 +349,17 @@
</span><span class="cx"> completeAllReadyPlansForVM(vm);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void Worklist::markCodeBlocks(VM& vm, SlotVisitor& slotVisitor)
+{
+ LockHolder locker(*m_lock);
+ for (PlanMap::iterator iter = m_plans.begin(); iter != m_plans.end(); ++iter) {
+ Plan* plan = iter->value.get();
+ if (plan->vm != &vm)
+ continue;
+ plan->markCodeBlocks(slotVisitor);
+ }
+}
+
</ins><span class="cx"> void Worklist::rememberCodeBlocks(VM& vm)
</span><span class="cx"> {
</span><span class="cx"> LockHolder locker(*m_lock);
</span><span class="lines">@@ -356,7 +367,7 @@
</span><span class="cx"> Plan* plan = iter->value.get();
</span><span class="cx"> if (plan->vm != &vm)
</span><span class="cx"> continue;
</span><del>- plan->rememberCodeBlocks();
</del><ins>+ plan->rememberCodeBlocks(vm);
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -553,6 +564,14 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void markCodeBlocks(VM& vm, SlotVisitor& slotVisitor)
+{
+ for (unsigned i = DFG::numberOfWorklists(); i--;) {
+ if (DFG::Worklist* worklist = DFG::existingWorklistForIndexOrNull(i))
+ worklist->markCodeBlocks(vm, slotVisitor);
+ }
+}
+
</ins><span class="cx"> void rememberCodeBlocks(VM& vm)
</span><span class="cx"> {
</span><span class="cx"> for (unsigned i = DFG::numberOfWorklists(); i--;) {
</span><span class="lines">@@ -561,7 +580,22 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-} } // namespace JSC::DFG
</del><ins>+#else // ENABLE(DFG_JIT)
</ins><span class="cx">
</span><ins>+void completeAllPlansForVM(VM&)
+{
+}
+
+void markCodeBlocks(VM&, SlotVisitor&)
+{
+}
+
+void rememberCodeBlocks(VM&)
+{
+}
+
</ins><span class="cx"> #endif // ENABLE(DFG_JIT)
</span><span class="cx">
</span><ins>+} } // namespace JSC::DFG
+
+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGWorklisth"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGWorklist.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGWorklist.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/dfg/DFGWorklist.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -25,8 +25,6 @@
</span><span class="cx">
</span><span class="cx"> #pragma once
</span><span class="cx">
</span><del>-#if ENABLE(DFG_JIT)
-
</del><span class="cx"> #include "DFGPlan.h"
</span><span class="cx"> #include "DFGThreadData.h"
</span><span class="cx"> #include <wtf/AutomaticThread.h>
</span><span class="lines">@@ -42,6 +40,8 @@
</span><span class="cx">
</span><span class="cx"> namespace DFG {
</span><span class="cx">
</span><ins>+#if ENABLE(DFG_JIT)
+
</ins><span class="cx"> class Worklist : public RefCounted<Worklist> {
</span><span class="cx"> public:
</span><span class="cx"> enum State { NotKnown, Compiling, Compiled };
</span><span class="lines">@@ -57,6 +57,7 @@
</span><span class="cx"> // worklist->completeAllReadyPlansForVM(vm);
</span><span class="cx"> void completeAllPlansForVM(VM&);
</span><span class="cx">
</span><ins>+ void markCodeBlocks(VM&, SlotVisitor&);
</ins><span class="cx"> void rememberCodeBlocks(VM&);
</span><span class="cx">
</span><span class="cx"> void waitUntilAllPlansForVMAreReady(VM&);
</span><span class="lines">@@ -163,9 +164,11 @@
</span><span class="cx"> return *result;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+#endif // ENABLE(DFG_JIT)
+
</ins><span class="cx"> void completeAllPlansForVM(VM&);
</span><ins>+void markCodeBlocks(VM&, SlotVisitor&);
</ins><span class="cx"> void rememberCodeBlocks(VM&);
</span><span class="cx">
</span><span class="cx"> } } // namespace JSC::DFG
</span><span class="cx">
</span><del>-#endif // ENABLE(DFG_JIT)
</del></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLAbstractHeapRepositorycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -73,14 +73,14 @@
</span><span class="cx"> , absolute(&root, "absolute")
</span><span class="cx"> {
</span><span class="cx"> // Make sure that our explicit assumptions about the StructureIDBlob match reality.
</span><del>- RELEASE_ASSERT(!(JSCell_indexingType.offset() & (sizeof(int32_t) - 1)));
- RELEASE_ASSERT(JSCell_indexingType.offset() + 1 == JSCell_typeInfoType.offset());
- RELEASE_ASSERT(JSCell_indexingType.offset() + 2 == JSCell_typeInfoFlags.offset());
- RELEASE_ASSERT(JSCell_indexingType.offset() + 3 == JSCell_cellState.offset());
</del><ins>+ RELEASE_ASSERT(!(JSCell_indexingTypeAndMisc.offset() & (sizeof(int32_t) - 1)));
+ RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 1 == JSCell_typeInfoType.offset());
+ RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 2 == JSCell_typeInfoFlags.offset());
+ RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 3 == JSCell_cellState.offset());
</ins><span class="cx">
</span><span class="cx"> JSCell_structureID.changeParent(&JSCell_header);
</span><span class="cx"> JSCell_usefulBytes.changeParent(&JSCell_header);
</span><del>- JSCell_indexingType.changeParent(&JSCell_usefulBytes);
</del><ins>+ JSCell_indexingTypeAndMisc.changeParent(&JSCell_usefulBytes);
</ins><span class="cx"> JSCell_typeInfoType.changeParent(&JSCell_usefulBytes);
</span><span class="cx"> JSCell_typeInfoFlags.changeParent(&JSCell_usefulBytes);
</span><span class="cx"> JSCell_cellState.changeParent(&JSCell_usefulBytes);
</span><span class="lines">@@ -135,21 +135,27 @@
</span><span class="cx"> dataLog("Abstract Heap Repository:\n");
</span><span class="cx"> root.deepDump(WTF::dataFile());
</span><span class="cx"> }
</span><ins>+
+ auto rangeFor = [&] (const AbstractHeap* heap) -> HeapRange {
+ if (heap)
+ return heap->range();
+ return HeapRange();
+ };
</ins><span class="cx">
</span><span class="cx"> for (HeapForValue entry : m_heapForMemory)
</span><del>- entry.value->as<MemoryValue>()->setRange(entry.heap->range());
</del><ins>+ entry.value->as<MemoryValue>()->setRange(rangeFor(entry.heap));
</ins><span class="cx"> for (HeapForValue entry : m_heapForCCallRead)
</span><del>- entry.value->as<CCallValue>()->effects.reads = entry.heap->range();
</del><ins>+ entry.value->as<CCallValue>()->effects.reads = rangeFor(entry.heap);
</ins><span class="cx"> for (HeapForValue entry : m_heapForCCallWrite)
</span><del>- entry.value->as<CCallValue>()->effects.writes = entry.heap->range();
</del><ins>+ entry.value->as<CCallValue>()->effects.writes = rangeFor(entry.heap);
</ins><span class="cx"> for (HeapForValue entry : m_heapForPatchpointRead)
</span><del>- entry.value->as<PatchpointValue>()->effects.reads = entry.heap->range();
</del><ins>+ entry.value->as<PatchpointValue>()->effects.reads = rangeFor(entry.heap);
</ins><span class="cx"> for (HeapForValue entry : m_heapForPatchpointWrite)
</span><del>- entry.value->as<PatchpointValue>()->effects.writes = entry.heap->range();
</del><ins>+ entry.value->as<PatchpointValue>()->effects.writes = rangeFor(entry.heap);
</ins><span class="cx"> for (HeapForValue entry : m_heapForFenceRead)
</span><del>- entry.value->as<FenceValue>()->read = entry.heap->range();
</del><ins>+ entry.value->as<FenceValue>()->read = rangeFor(entry.heap);
</ins><span class="cx"> for (HeapForValue entry : m_heapForFenceWrite)
</span><del>- entry.value->as<FenceValue>()->write = entry.heap->range();
</del><ins>+ entry.value->as<FenceValue>()->write = rangeFor(entry.heap);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> } } // namespace JSC::FTL
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLAbstractHeapRepositoryh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -59,11 +59,11 @@
</span><span class="cx"> macro(JSArrayBufferView_vector, JSArrayBufferView::offsetOfVector()) \
</span><span class="cx"> macro(JSCell_cellState, JSCell::cellStateOffset()) \
</span><span class="cx"> macro(JSCell_header, 0) \
</span><del>- macro(JSCell_indexingType, JSCell::indexingTypeOffset()) \
</del><ins>+ macro(JSCell_indexingTypeAndMisc, JSCell::indexingTypeAndMiscOffset()) \
</ins><span class="cx"> macro(JSCell_structureID, JSCell::structureIDOffset()) \
</span><span class="cx"> macro(JSCell_typeInfoFlags, JSCell::typeInfoFlagsOffset()) \
</span><span class="cx"> macro(JSCell_typeInfoType, JSCell::typeInfoTypeOffset()) \
</span><del>- macro(JSCell_usefulBytes, JSCell::indexingTypeOffset()) \
</del><ins>+ macro(JSCell_usefulBytes, JSCell::indexingTypeAndMiscOffset()) \
</ins><span class="cx"> macro(JSFunction_executable, JSFunction::offsetOfExecutable()) \
</span><span class="cx"> macro(JSFunction_scope, JSFunction::offsetOfScopeChain()) \
</span><span class="cx"> macro(JSFunction_rareData, JSFunction::offsetOfRareData()) \
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLJITCodecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLJITCode.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLJITCode.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLJITCode.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -46,6 +46,7 @@
</span><span class="cx"> CommaPrinter comma;
</span><span class="cx"> dataLog(comma, m_b3Code);
</span><span class="cx"> dataLog(comma, m_arityCheckEntrypoint);
</span><ins>+ dataLog("\n");
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -2788,7 +2788,7 @@
</span><span class="cx"> void compilePutStructure()
</span><span class="cx"> {
</span><span class="cx"> m_ftlState.jitCode->common.notifyCompilingStructureTransition(m_graph.m_plan, codeBlock(), m_node);
</span><del>-
</del><ins>+
</ins><span class="cx"> Structure* oldStructure = m_node->transition()->previous;
</span><span class="cx"> Structure* newStructure = m_node->transition()->next;
</span><span class="cx"> ASSERT_UNUSED(oldStructure, oldStructure->indexingType() == newStructure->indexingType());
</span><span class="lines">@@ -3947,6 +3947,8 @@
</span><span class="cx"> fastObject, m_heaps.JSEnvironmentRecord_variables[i]);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ mutatorFence();
+
</ins><span class="cx"> ValueFromBlock fastResult = m_out.anchor(fastObject);
</span><span class="cx"> m_out.jump(continuation);
</span><span class="cx">
</span><span class="lines">@@ -4006,6 +4008,8 @@
</span><span class="cx"> m_out.storePtr(weakPointer(executable), fastObject, m_heaps.JSFunction_executable);
</span><span class="cx"> m_out.storePtr(m_out.intPtrZero, fastObject, m_heaps.JSFunction_rareData);
</span><span class="cx">
</span><ins>+ mutatorFence();
+
</ins><span class="cx"> ValueFromBlock fastResult = m_out.anchor(fastObject);
</span><span class="cx"> m_out.jump(continuation);
</span><span class="cx">
</span><span class="lines">@@ -4140,6 +4144,8 @@
</span><span class="cx"> m_out.appendTo(end, lastNext);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ mutatorFence();
+
</ins><span class="cx"> setJSValue(result);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -4193,6 +4199,7 @@
</span><span class="cx"> m_out.branch(m_out.equal(currentOffset, m_out.constInt32(0)), unsure(continuation), unsure(loopStart));
</span><span class="cx">
</span><span class="cx"> m_out.appendTo(continuation, lastNext);
</span><ins>+ mutatorFence();
</ins><span class="cx"> setJSValue(array);
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="lines">@@ -4227,6 +4234,7 @@
</span><span class="cx"> void compileNewObject()
</span><span class="cx"> {
</span><span class="cx"> setJSValue(allocateObject(m_node->structure()));
</span><ins>+ mutatorFence();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void compileNewArray()
</span><span class="lines">@@ -4279,6 +4287,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> setJSValue(arrayValues.array);
</span><ins>+ mutatorFence();
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -4387,6 +4396,7 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ mutatorFence();
</ins><span class="cx"> setJSValue(result);
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="lines">@@ -4428,7 +4438,7 @@
</span><span class="cx"> LBasicBlock slowPath = m_out.newBlock();
</span><span class="cx"> LBasicBlock continuation = m_out.newBlock();
</span><span class="cx">
</span><del>- LValue indexingShape = m_out.load8ZeroExt32(argument, m_heaps.JSCell_indexingType);
</del><ins>+ LValue indexingShape = m_out.load8ZeroExt32(argument, m_heaps.JSCell_indexingTypeAndMisc);
</ins><span class="cx"> indexingShape = m_out.bitAnd(indexingShape, m_out.constInt32(IndexingShapeMask));
</span><span class="cx"> LValue isOKIndexingType = m_out.belowOrEqual(
</span><span class="cx"> m_out.sub(indexingShape, m_out.constInt32(Int32Shape)),
</span><span class="lines">@@ -4497,6 +4507,7 @@
</span><span class="cx">
</span><span class="cx"> m_out.appendTo(continuation, lastNext);
</span><span class="cx"> result = m_out.phi(Int64, fastResult, slowResult);
</span><ins>+ mutatorFence();
</ins><span class="cx"> } else
</span><span class="cx"> result = vmCall(Int64, m_out.operation(operationSpreadGeneric), m_callFrame, argument);
</span><span class="cx">
</span><span class="lines">@@ -4529,6 +4540,7 @@
</span><span class="cx"> m_heaps.forIndexingType(m_node->indexingType())->at(index));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ mutatorFence();
</ins><span class="cx"> setJSValue(arrayValues.array);
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="lines">@@ -4553,6 +4565,7 @@
</span><span class="cx"> publicLength,
</span><span class="cx"> globalObject->arrayStructureForIndexingTypeDuringAllocation(
</span><span class="cx"> m_node->indexingType())).array);
</span><ins>+ mutatorFence();
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -4620,6 +4633,7 @@
</span><span class="cx"> m_out.store32(size, fastResultValue, m_heaps.JSArrayBufferView_length);
</span><span class="cx"> m_out.store32(m_out.constInt32(FastTypedArray), fastResultValue, m_heaps.JSArrayBufferView_mode);
</span><span class="cx">
</span><ins>+ mutatorFence();
</ins><span class="cx"> ValueFromBlock fastResult = m_out.anchor(fastResultValue);
</span><span class="cx"> m_out.jump(continuation);
</span><span class="cx">
</span><span class="lines">@@ -4862,6 +4876,7 @@
</span><span class="cx"> result, m_heaps.JSString_flags);
</span><span class="cx"> m_out.store32(length, result, m_heaps.JSString_length);
</span><span class="cx">
</span><ins>+ mutatorFence();
</ins><span class="cx"> ValueFromBlock fastResult = m_out.anchor(result);
</span><span class="cx"> m_out.jump(continuation);
</span><span class="cx">
</span><span class="lines">@@ -5240,6 +5255,7 @@
</span><span class="cx"> else
</span><span class="cx"> storage = m_out.loadPtr(base, m_heaps.JSObject_butterfly);
</span><span class="cx"> } else {
</span><ins>+ DFG_ASSERT(m_graph, m_node, variant.kind() == PutByIdVariant::Transition);
</ins><span class="cx"> m_graph.m_plan.transitions.addLazily(
</span><span class="cx"> codeBlock(), m_node->origin.semantic.codeOriginOwner(),
</span><span class="cx"> variant.oldStructureForTransition(), variant.newStructure());
</span><span class="lines">@@ -5247,7 +5263,11 @@
</span><span class="cx"> storage = storageForTransition(
</span><span class="cx"> base, variant.offset(),
</span><span class="cx"> variant.oldStructureForTransition(), variant.newStructure());
</span><del>-
</del><ins>+ }
+
+ storeProperty(value, storage, data.identifierNumber, variant.offset());
+
+ if (variant.kind() == PutByIdVariant::Transition) {
</ins><span class="cx"> ASSERT(variant.oldStructureForTransition()->indexingType() == variant.newStructure()->indexingType());
</span><span class="cx"> ASSERT(variant.oldStructureForTransition()->typeInfo().inlineTypeFlags() == variant.newStructure()->typeInfo().inlineTypeFlags());
</span><span class="cx"> ASSERT(variant.oldStructureForTransition()->typeInfo().type() == variant.newStructure()->typeInfo().type());
</span><span class="lines">@@ -5255,7 +5275,6 @@
</span><span class="cx"> weakStructureID(variant.newStructure()), base, m_heaps.JSCell_structureID);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- storeProperty(value, storage, data.identifierNumber, variant.offset());
</del><span class="cx"> m_out.jump(continuation);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -8114,6 +8133,12 @@
</span><span class="cx"> structure->outOfLineCapacity() * sizeof(JSValue) + sizeof(IndexingHeader)));
</span><span class="cx">
</span><span class="cx"> ValueFromBlock haveButterfly = m_out.anchor(fastButterflyValue);
</span><ins>+
+ splatWords(
+ fastButterflyValue,
+ m_out.constInt32(-structure->outOfLineCapacity() - 1),
+ m_out.constInt32(-1),
+ m_out.int64Zero, m_heaps.properties.atAnyNumber());
</ins><span class="cx">
</span><span class="cx"> m_out.store32(vectorLength, fastButterflyValue, m_heaps.Butterfly_vectorLength);
</span><span class="cx">
</span><span class="lines">@@ -8268,7 +8293,8 @@
</span><span class="cx"> object = allocateObject(structure);
</span><span class="cx"> butterfly = nullptr; // Don't have one, don't need one.
</span><span class="cx"> }
</span><del>-
</del><ins>+
+ BitVector setInlineOffsets;
</ins><span class="cx"> for (PropertyMapEntry entry : structure->getPropertiesConcurrently()) {
</span><span class="cx"> for (unsigned i = data.m_properties.size(); i--;) {
</span><span class="cx"> PromotedLocationDescriptor descriptor = data.m_properties[i];
</span><span class="lines">@@ -8277,11 +8303,20 @@
</span><span class="cx"> if (m_graph.identifiers()[descriptor.info()] != entry.key)
</span><span class="cx"> continue;
</span><span class="cx">
</span><del>- LValue base = isInlineOffset(entry.offset) ? object : butterfly;
</del><ins>+ LValue base;
+ if (isInlineOffset(entry.offset)) {
+ setInlineOffsets.set(entry.offset);
+ base = object;
+ } else
+ base = butterfly;
</ins><span class="cx"> storeProperty(values[i], base, descriptor.info(), entry.offset);
</span><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> }
</span><ins>+ for (unsigned i = structure->inlineCapacity(); i--;) {
+ if (!setInlineOffsets.get(i))
+ m_out.store64(m_out.int64Zero, m_out.address(m_heaps.properties.atAnyNumber(), object, offsetRelativeToBase(i)));
+ }
</ins><span class="cx">
</span><span class="cx"> results.append(m_out.anchor(object));
</span><span class="cx"> m_out.jump(outerContinuation);
</span><span class="lines">@@ -8292,6 +8327,7 @@
</span><span class="cx">
</span><span class="cx"> m_out.appendTo(outerContinuation, outerLastNext);
</span><span class="cx"> setJSValue(m_out.phi(pointerType(), results));
</span><ins>+ mutatorFence();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void compileMaterializeCreateActivation()
</span><span class="lines">@@ -8367,6 +8403,7 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ mutatorFence();
</ins><span class="cx"> setJSValue(activation);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -8945,7 +8982,8 @@
</span><span class="cx"> LBasicBlock lastNext = m_out.insertNewBlocksBefore(initLoop);
</span><span class="cx">
</span><span class="cx"> ValueFromBlock originalIndex = m_out.anchor(end);
</span><del>- ValueFromBlock originalPointer = m_out.anchor(base);
</del><ins>+ ValueFromBlock originalPointer = m_out.anchor(
+ m_out.add(base, m_out.shl(m_out.signExt32ToPtr(begin), m_out.constInt32(3))));
</ins><span class="cx"> m_out.branch(m_out.notEqual(end, begin), unsure(initLoop), unsure(initDone));
</span><span class="cx">
</span><span class="cx"> m_out.appendTo(initLoop, initDone);
</span><span class="lines">@@ -8974,7 +9012,13 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> LValue result = allocatePropertyStorageWithSizeImpl(initialOutOfLineCapacity);
</span><del>- m_out.storePtr(result, object, m_heaps.JSObject_butterfly);
</del><ins>+
+ splatWords(
+ result,
+ m_out.constInt32(-initialOutOfLineCapacity - 1), m_out.constInt32(-1),
+ m_out.int64Zero, m_heaps.properties.atAnyNumber());
+
+ setButterfly(result, object);
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -9002,8 +9046,13 @@
</span><span class="cx"> m_out.storePtr(loaded, m_out.address(m_heaps.properties.atAnyNumber(), result, offset));
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_out.storePtr(result, m_out.address(object, m_heaps.JSObject_butterfly));
</del><ins>+ splatWords(
+ result,
+ m_out.constInt32(-newSize - 1), m_out.constInt32(-oldSize - 1),
+ m_out.int64Zero, m_heaps.properties.atAnyNumber());
</ins><span class="cx">
</span><ins>+ setButterfly(result, object);
+
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -9872,6 +9921,12 @@
</span><span class="cx"> LValue allocator, Structure* structure, LValue butterfly, LBasicBlock slowPath)
</span><span class="cx"> {
</span><span class="cx"> LValue result = allocateCell(allocator, structure, slowPath);
</span><ins>+ splatWords(
+ result,
+ m_out.constInt32(JSFinalObject::offsetOfInlineStorage() / 8),
+ m_out.constInt32(JSFinalObject::offsetOfInlineStorage() / 8 + structure->inlineCapacity()),
+ m_out.int64Zero,
+ m_heaps.properties.atAnyNumber());
</ins><span class="cx"> m_out.storePtr(butterfly, result, m_heaps.JSObject_butterfly);
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="lines">@@ -11954,7 +12009,7 @@
</span><span class="cx"> case Array::Int32:
</span><span class="cx"> case Array::Double:
</span><span class="cx"> case Array::Contiguous: {
</span><del>- LValue indexingType = m_out.load8ZeroExt32(cell, m_heaps.JSCell_indexingType);
</del><ins>+ LValue indexingType = m_out.load8ZeroExt32(cell, m_heaps.JSCell_indexingTypeAndMisc);
</ins><span class="cx">
</span><span class="cx"> switch (arrayMode.arrayClass()) {
</span><span class="cx"> case Array::OriginalArray:
</span><span class="lines">@@ -12447,7 +12502,7 @@
</span><span class="cx"> return LazySlowPath::createGenerator(
</span><span class="cx"> [=] (CCallHelpers& jit, LazySlowPath::GenerationParams& params) {
</span><span class="cx"> if (isFenced) {
</span><del>- CCallHelpers::Jump noFence = jit.jumpIfBarrierStoreLoadFenceNotNeeded();
</del><ins>+ CCallHelpers::Jump noFence = jit.jumpIfMutatorFenceNotNeeded();
</ins><span class="cx"> jit.memoryFence();
</span><span class="cx"> params.doneJumps.append(jit.barrierBranchWithoutFence(baseGPR));
</span><span class="cx"> noFence.link(&jit);
</span><span class="lines">@@ -12508,7 +12563,65 @@
</span><span class="cx">
</span><span class="cx"> m_out.appendTo(continuation, lastNext);
</span><span class="cx"> }
</span><ins>+
+ void mutatorFence()
+ {
+ if (isX86()) {
+ m_out.fence(&m_heaps.root, nullptr);
+ return;
+ }
+
+ LBasicBlock slowPath = m_out.newBlock();
+ LBasicBlock continuation = m_out.newBlock();
+
+ LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
+
+ m_out.branch(
+ m_out.load8ZeroExt32(m_out.absolute(vm().heap.addressOfMutatorShouldBeFenced())),
+ rarely(slowPath), usually(continuation));
+
+ m_out.appendTo(slowPath);
+
+ m_out.fence(&m_heaps.root, nullptr);
+ m_out.jump(continuation);
+
+ m_out.appendTo(continuation, lastNext);
+ }
+
+ void setButterfly(LValue butterfly, LValue object)
+ {
+ if (isX86()) {
+ m_out.fence(&m_heaps.root, nullptr);
+ m_out.storePtr(butterfly, object, m_heaps.JSObject_butterfly);
+ m_out.fence(&m_heaps.root, nullptr);
+ return;
+ }
+
+ LBasicBlock fastPath = m_out.newBlock();
+ LBasicBlock slowPath = m_out.newBlock();
+ LBasicBlock continuation = m_out.newBlock();
+
+ LBasicBlock lastNext = m_out.insertNewBlocksBefore(fastPath);
+
+ m_out.branch(
+ m_out.load8ZeroExt32(m_out.absolute(vm().heap.addressOfMutatorShouldBeFenced())),
+ rarely(slowPath), usually(fastPath));
</ins><span class="cx">
</span><ins>+ m_out.appendTo(fastPath);
+
+ m_out.storePtr(butterfly, object, m_heaps.JSObject_butterfly);
+ m_out.jump(continuation);
+
+ m_out.appendTo(slowPath);
+
+ m_out.fence(&m_heaps.root, nullptr);
+ m_out.storePtr(butterfly, object, m_heaps.JSObject_butterfly);
+ m_out.fence(&m_heaps.root, nullptr);
+ m_out.jump(continuation);
+
+ m_out.appendTo(continuation, lastNext);
+ }
+
</ins><span class="cx"> template<typename... Args>
</span><span class="cx"> LValue vmCall(LType type, LValue function, Args... args)
</span><span class="cx"> {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLOSRExitCompilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -277,7 +277,7 @@
</span><span class="cx"> if (ArrayProfile* arrayProfile = jit.baselineCodeBlockFor(codeOrigin)->getArrayProfile(codeOrigin.bytecodeIndex)) {
</span><span class="cx"> jit.load32(MacroAssembler::Address(GPRInfo::regT0, JSCell::structureIDOffset()), GPRInfo::regT1);
</span><span class="cx"> jit.store32(GPRInfo::regT1, arrayProfile->addressOfLastSeenStructureID());
</span><del>- jit.load8(MacroAssembler::Address(GPRInfo::regT0, JSCell::indexingTypeOffset()), GPRInfo::regT1);
</del><ins>+ jit.load8(MacroAssembler::Address(GPRInfo::regT0, JSCell::indexingTypeAndMiscOffset()), GPRInfo::regT1);
</ins><span class="cx"> jit.move(MacroAssembler::TrustedImm32(1), GPRInfo::regT2);
</span><span class="cx"> jit.lshift32(GPRInfo::regT1, GPRInfo::regT2);
</span><span class="cx"> jit.or32(GPRInfo::regT2, MacroAssembler::AbsoluteAddress(arrayProfile->addressOfArrayModes()));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLOutputcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLOutput.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLOutput.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLOutput.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -367,6 +367,11 @@
</span><span class="cx"> return m_block->appendNew<B3::Value>(m_proc, B3::SExt32, origin(), value);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+LValue Output::signExt32ToPtr(LValue value)
+{
+ return signExt32To64(value);
+}
+
</ins><span class="cx"> LValue Output::zeroExt(LValue value, LType type)
</span><span class="cx"> {
</span><span class="cx"> if (value->type() == type)
</span><span class="lines">@@ -446,9 +451,12 @@
</span><span class="cx"> m_heaps->decorateMemory(pointer.heap(), store);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-FenceValue* Output::fence()
</del><ins>+FenceValue* Output::fence(const AbstractHeap* read, const AbstractHeap* write)
</ins><span class="cx"> {
</span><del>- return m_block->appendNew<FenceValue>(m_proc, origin());
</del><ins>+ FenceValue* result = m_block->appendNew<FenceValue>(m_proc, origin());
+ m_heaps->decorateFenceRead(read, result);
+ m_heaps->decorateFenceWrite(write, result);
+ return result;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Output::store32As8(LValue value, TypedPointer pointer)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLOutputh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLOutput.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLOutput.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/ftl/FTLOutput.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -177,6 +177,7 @@
</span><span class="cx"> LValue doubleToUInt(LValue);
</span><span class="cx">
</span><span class="cx"> LValue signExt32To64(LValue);
</span><ins>+ LValue signExt32ToPtr(LValue);
</ins><span class="cx"> LValue zeroExt(LValue, LType);
</span><span class="cx"> LValue zeroExtPtr(LValue value) { return zeroExt(value, B3::Int64); }
</span><span class="cx"> LValue intToDouble(LValue);
</span><span class="lines">@@ -189,7 +190,7 @@
</span><span class="cx">
</span><span class="cx"> LValue load(TypedPointer, LType);
</span><span class="cx"> void store(LValue, TypedPointer);
</span><del>- B3::FenceValue* fence();
</del><ins>+ B3::FenceValue* fence(const AbstractHeap* read, const AbstractHeap* write);
</ins><span class="cx">
</span><span class="cx"> LValue load8SignExt32(TypedPointer);
</span><span class="cx"> LValue load8ZeroExt32(TypedPointer);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapCellStateh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/CellState.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/CellState.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/CellState.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -40,14 +40,8 @@
</span><span class="cx"> // The object is in eden. During GC, this means that the object has not been marked yet.
</span><span class="cx"> NewWhite = 1,
</span><span class="cx">
</span><del>- // The object is grey - i.e. it will be scanned - and this is the first time in this GC that we are
- // going to scan it. If this is an eden GC, this also means that the object is in eden.
- NewGrey = 2,
-
- // The object is grey - i.e. it will be scanned - but it either belongs to old gen (if this is eden
- // GC) or it is grey a second time in this current GC (because a concurrent store barrier requested
- // re-greying).
- OldGrey = 3
</del><ins>+ // The object is grey - i.e. it will be scanned.
+ Grey = 2,
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> static const unsigned blackThreshold = 0; // x <= blackThreshold means x is AnthraciteOrBlack.
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapGCSegmentedArrayh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/GCSegmentedArray.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/GCSegmentedArray.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/GCSegmentedArray.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2014 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -26,6 +26,8 @@
</span><span class="cx"> #pragma once
</span><span class="cx">
</span><span class="cx"> #include <wtf/DoublyLinkedList.h>
</span><ins>+#include <wtf/ListDump.h>
+#include <wtf/PrintStream.h>
</ins><span class="cx"> #include <wtf/Vector.h>
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -162,3 +164,4 @@
</span><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span><ins>+
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/Heap.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -78,6 +78,35 @@
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><ins>+class Heap::ResumeTheWorldScope {
+public:
+ ResumeTheWorldScope(Heap& heap)
+ : m_heap(heap)
+ {
+ if (!Options::useConcurrentGC())
+ return;
+
+ if (Options::logGC())
+ dataLog((MonotonicTime::now() - m_heap.m_stopTime).milliseconds(), " ms...]\n");
+
+ m_heap.resumeTheWorld();
+ }
+
+ ~ResumeTheWorldScope()
+ {
+ if (!Options::useConcurrentGC())
+ return;
+
+ m_heap.stopTheWorld();
+
+ if (Options::logGC())
+ dataLog("[GC: ");
+ }
+
+private:
+ Heap& m_heap;
+};
+
</ins><span class="cx"> namespace {
</span><span class="cx">
</span><span class="cx"> static const size_t largeHeapSize = 32 * MB; // About 1.5X the average webpage.
</span><span class="lines">@@ -244,7 +273,8 @@
</span><span class="cx"> , m_extraMemorySize(0)
</span><span class="cx"> , m_deprecatedExtraMemorySize(0)
</span><span class="cx"> , m_machineThreads(this)
</span><del>- , m_slotVisitor(*this)
</del><ins>+ , m_collectorSlotVisitor(std::make_unique<SlotVisitor>(*this))
+ , m_mutatorMarkStack(std::make_unique<MarkStackArray>())
</ins><span class="cx"> , m_handleSet(vm)
</span><span class="cx"> , m_codeBlocks(std::make_unique<CodeBlockSet>())
</span><span class="cx"> , m_jitStubRoutines(std::make_unique<JITStubRoutineSet>())
</span><span class="lines">@@ -266,6 +296,8 @@
</span><span class="cx"> #if USE(FOUNDATION)
</span><span class="cx"> , m_delayedReleaseRecursionCount(0)
</span><span class="cx"> #endif
</span><ins>+ , m_sharedCollectorMarkStack(std::make_unique<MarkStackArray>())
+ , m_sharedMutatorMarkStack(std::make_unique<MarkStackArray>())
</ins><span class="cx"> , m_helperClient(&heapHelperPool())
</span><span class="cx"> , m_threadLock(Box<Lock>::create())
</span><span class="cx"> , m_threadCondition(AutomaticThreadCondition::create())
</span><span class="lines">@@ -281,6 +313,11 @@
</span><span class="cx">
</span><span class="cx"> Heap::~Heap()
</span><span class="cx"> {
</span><ins>+ for (auto& slotVisitor : m_parallelSlotVisitors)
+ slotVisitor->clearMarkStacks();
+ m_collectorSlotVisitor->clearMarkStacks();
+ m_mutatorMarkStack->clear();
+
</ins><span class="cx"> for (WeakBlock* block : m_logicallyEmptyWeakBlocks)
</span><span class="cx"> WeakBlock::destroy(*this, block);
</span><span class="cx"> }
</span><span class="lines">@@ -416,12 +453,12 @@
</span><span class="cx">
</span><span class="cx"> void Heap::harvestWeakReferences()
</span><span class="cx"> {
</span><del>- m_slotVisitor.harvestWeakReferences();
</del><ins>+ m_collectorSlotVisitor->harvestWeakReferences();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::finalizeUnconditionalFinalizers()
</span><span class="cx"> {
</span><del>- m_slotVisitor.finalizeUnconditionalFinalizers();
</del><ins>+ m_collectorSlotVisitor->finalizeUnconditionalFinalizers();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::willStartIterating()
</span><span class="lines">@@ -439,85 +476,67 @@
</span><span class="cx"> #if ENABLE(JIT)
</span><span class="cx"> JITWorklist::instance()->completeAllForVM(*m_vm);
</span><span class="cx"> #endif // ENABLE(JIT)
</span><del>-#if ENABLE(DFG_JIT)
</del><span class="cx"> DFG::completeAllPlansForVM(*m_vm);
</span><del>-#endif
</del><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::markRoots(double gcStartTime)
</del><ins>+void Heap::markToFixpoint(double gcStartTime)
</ins><span class="cx"> {
</span><del>- TimingScope markRootsTimingScope(*this, "Heap::markRoots");
</del><ins>+ TimingScope markToFixpointTimingScope(*this, "Heap::markToFixpoint");
</ins><span class="cx">
</span><del>- HeapRootVisitor heapRootVisitor(m_slotVisitor);
</del><ins>+ HeapRootVisitor heapRootVisitor(*m_collectorSlotVisitor);
</ins><span class="cx">
</span><del>- {
- TimingScope preConvergenceTimingScope(*this, "Heap::markRoots before convergence");
</del><ins>+ if (m_collectionScope == CollectionScope::Full) {
+ m_opaqueRoots.clear();
+ m_collectorSlotVisitor->clearMarkStacks();
+ m_mutatorMarkStack->clear();
+ }
</ins><span class="cx">
</span><del>-#if ENABLE(DFG_JIT)
- DFG::rememberCodeBlocks(*m_vm);
-#endif
</del><ins>+ beginMarking();
</ins><span class="cx">
</span><del>-#if ENABLE(SAMPLING_PROFILER)
- if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
- // Note that we need to own the lock from now until we're done
- // marking the SamplingProfiler's data because once we verify the
- // SamplingProfiler's stack traces, we don't want it to accumulate
- // more stack traces before we get the chance to mark it.
- // This lock is released inside visitSamplingProfiler().
- samplingProfiler->getLock().lock();
- samplingProfiler->processUnverifiedStackTraces();
- }
-#endif // ENABLE(SAMPLING_PROFILER)
</del><ins>+ m_parallelMarkersShouldExit = false;
</ins><span class="cx">
</span><del>- if (m_collectionScope == CollectionScope::Full) {
- m_opaqueRoots.clear();
- m_slotVisitor.clearMarkStack();
- }
</del><ins>+ m_helperClient.setFunction(
+ [this] () {
+ SlotVisitor* slotVisitor;
+ {
+ LockHolder locker(m_parallelSlotVisitorLock);
+ if (m_availableParallelSlotVisitors.isEmpty()) {
+ std::unique_ptr<SlotVisitor> newVisitor =
+ std::make_unique<SlotVisitor>(*this);
+ slotVisitor = newVisitor.get();
+ m_parallelSlotVisitors.append(WTFMove(newVisitor));
+ } else
+ slotVisitor = m_availableParallelSlotVisitors.takeLast();
+ }
</ins><span class="cx">
</span><del>- beginMarking();
</del><ins>+ WTF::registerGCThread(GCThreadType::Helper);
</ins><span class="cx">
</span><del>- m_parallelMarkersShouldExit = false;
</del><ins>+ {
+ ParallelModeEnabler parallelModeEnabler(*slotVisitor);
+ slotVisitor->didStartMarking();
+ slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
+ }
</ins><span class="cx">
</span><del>- m_helperClient.setFunction(
- [this] () {
- SlotVisitor* slotVisitor;
- {
- LockHolder locker(m_parallelSlotVisitorLock);
- if (m_availableParallelSlotVisitors.isEmpty()) {
- std::unique_ptr<SlotVisitor> newVisitor =
- std::make_unique<SlotVisitor>(*this);
- slotVisitor = newVisitor.get();
- m_parallelSlotVisitors.append(WTFMove(newVisitor));
- } else
- slotVisitor = m_availableParallelSlotVisitors.takeLast();
- }
</del><ins>+ {
+ LockHolder locker(m_parallelSlotVisitorLock);
+ m_availableParallelSlotVisitors.append(slotVisitor);
+ }
+ });
</ins><span class="cx">
</span><del>- WTF::registerGCThread(GCThreadType::Helper);
</del><ins>+ m_collectorSlotVisitor->didStartMarking();
</ins><span class="cx">
</span><del>- {
- ParallelModeEnabler parallelModeEnabler(*slotVisitor);
- slotVisitor->didStartMarking();
- slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
- }
-
- {
- LockHolder locker(m_parallelSlotVisitorLock);
- m_availableParallelSlotVisitors.append(slotVisitor);
- }
- });
-
- m_slotVisitor.didStartMarking();
- }
-
- {
- SuperSamplerScope superSamplerScope(false);
- TimingScope convergenceTimingScope(*this, "Heap::markRoots convergence");
- ParallelModeEnabler enabler(m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-
</del><ins>+ Seconds concurrentTime;
+ double stoppedOverConcurrentRatio = 0.1;
+ const double ratioStep = 1.3;
+ MonotonicTime initialTime = MonotonicTime::now();
+ MonotonicTime topOfLoopTime = initialTime;
+ for (unsigned iteration = 1; ; ++iteration) {
+ if (Options::logGC())
+ dataLog("i#", iteration, " ");
+ MonotonicTime timeToResume = topOfLoopTime + concurrentTime * stoppedOverConcurrentRatio;
+ stoppedOverConcurrentRatio *= ratioStep;
</ins><span class="cx"> {
</span><del>- TimingScope preConvergenceTimingScope(*this, "Heap::markRoots conservative scan");
</del><ins>+ TimingScope preConvergenceTimingScope(*this, "Heap::markToFixpoint conservative scan");
</ins><span class="cx"> ConservativeRoots conservativeRoots(*this);
</span><span class="cx"> SuperSamplerScope superSamplerScope(false);
</span><span class="cx"> gatherStackRoots(conservativeRoots);
</span><span class="lines">@@ -524,33 +543,98 @@
</span><span class="cx"> gatherJSStackRoots(conservativeRoots);
</span><span class="cx"> gatherScratchBufferRoots(conservativeRoots);
</span><span class="cx"> visitConservativeRoots(conservativeRoots);
</span><ins>+ }
</ins><span class="cx">
</span><del>- // We want to do this to conservatively ensure that we rescan any code blocks that are
- // running right now. However, we need to be sure to do it *after* we mark the code block
- // so that we know for sure if it really needs a barrier.
- m_codeBlocks->writeBarrierCurrentlyExecuting(this);
</del><ins>+ // Now we visit roots that don't get barriered, so each fixpoint iteration just revisits
+ // all of them.
+#if JSC_OBJC_API_ENABLED
+ scanExternalRememberedSet(*m_vm, *m_collectorSlotVisitor);
+#endif
+
+ if (m_vm->smallStrings.needsToBeVisited(*m_collectionScope))
+ m_vm->smallStrings.visitStrongReferences(*m_collectorSlotVisitor);
+
+ for (auto& pair : m_protectedValues)
+ heapRootVisitor.visit(&pair.key);
+
+ if (m_markListSet && m_markListSet->size())
+ MarkedArgumentBuffer::markLists(heapRootVisitor, *m_markListSet);
+
+ if (m_vm->exception())
+ heapRootVisitor.visit(m_vm->addressOfException());
+ if (m_vm->lastException())
+ heapRootVisitor.visit(m_vm->addressOfLastException());
+
+ m_handleSet.visitStrongHandles(heapRootVisitor);
+ m_handleStack.visit(heapRootVisitor);
+
+#if ENABLE(SAMPLING_PROFILER)
+ if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
+ LockHolder locker(samplingProfiler->getLock());
+ samplingProfiler->processUnverifiedStackTraces();
+ samplingProfiler->visit(*m_collectorSlotVisitor);
+ if (Options::logGC() == GCLogging::Verbose)
+ dataLog("Sampling Profiler data:\n", *m_collectorSlotVisitor);
</ins><span class="cx"> }
</span><ins>+#endif // ENABLE(SAMPLING_PROFILER)
+
+ if (m_vm->typeProfiler())
+ m_vm->typeProfilerLog()->visit(*m_collectorSlotVisitor);
+
+ m_vm->shadowChicken().visitChildren(*m_collectorSlotVisitor);
+
+ m_jitStubRoutines->traceMarkedStubRoutines(*m_collectorSlotVisitor);
</ins><span class="cx">
</span><del>- visitExternalRememberedSet();
- visitSmallStrings();
- visitProtectedObjects(heapRootVisitor);
- visitArgumentBuffers(heapRootVisitor);
- visitException(heapRootVisitor);
- visitStrongHandles(heapRootVisitor);
- visitHandleStack(heapRootVisitor);
- visitSamplingProfiler();
- visitTypeProfiler();
- visitShadowChicken();
- traceCodeBlocksAndJITStubRoutines();
- m_slotVisitor.drainFromShared(SlotVisitor::MasterDrain);
</del><ins>+ m_collectorSlotVisitor->mergeOpaqueRootsIfNecessary();
+ for (auto& parallelVisitor : m_parallelSlotVisitors)
+ parallelVisitor->mergeOpaqueRootsIfNecessary();
+
+ m_objectSpace.visitWeakSets(heapRootVisitor);
+ harvestWeakReferences();
+ visitCompilerWorklistWeakReferences();
+ DFG::markCodeBlocks(*m_vm, *m_collectorSlotVisitor);
+ bool shouldTerminate = m_collectorSlotVisitor->isEmpty() && m_mutatorMarkStack->isEmpty();
+
+ // We want to do this to conservatively ensure that we rescan any code blocks that are
+ // running right now. However, we need to be sure to do it *after* we mark the code block
+ // so that we know for sure if it really needs a barrier. Also, this has to happen after the
+ // fixpoint check - otherwise we might loop forever. Incidentally, we also want to do this
+ // at the end of GC so that anything at the end of the last GC gets barriered in the next
+ // GC.
+ m_codeBlocks->writeBarrierCurrentlyExecuting(this);
+ DFG::rememberCodeBlocks(*m_vm);
+
+ if (shouldTerminate)
+ break;
+
+ // The SlotVisitor's mark stacks are accessed by the collector thread (i.e. this thread)
+ // without locks. That's why we double-buffer.
+ m_mutatorMarkStack->transferTo(m_collectorSlotVisitor->mutatorMarkStack());
+
+ if (Options::logGC() == GCLogging::Verbose)
+ dataLog("Live Weak Handles:\n", *m_collectorSlotVisitor);
+
+ if (Options::logGC())
+ dataLog(m_collectorSlotVisitor->collectorMarkStack().size(), "+", m_collectorSlotVisitor->mutatorMarkStack().size(), " ");
+
+ {
+ TimingScope traceTimingScope(*this, "Heap::markToFixpoint tracing");
+ ParallelModeEnabler enabler(*m_collectorSlotVisitor);
+ m_collectorSlotVisitor->donateAndDrain(timeToResume);
+ SlotVisitor::SharedDrainResult drainResult =
+ m_collectorSlotVisitor->drainFromShared(SlotVisitor::MasterDrain, timeToResume);
+ if (drainResult != SlotVisitor::SharedDrainResult::Done) {
+ ResumeTheWorldScope resumeTheWorldScope(*this);
+ m_collectorSlotVisitor->donateAndDrain();
+ m_collectorSlotVisitor->drainFromShared(SlotVisitor::MasterDrain);
+ }
+ }
+
+ MonotonicTime timeOfStop = MonotonicTime::now();
+ concurrentTime = timeOfStop - timeToResume;
+ topOfLoopTime = timeOfStop;
</ins><span class="cx"> }
</span><del>-
- TimingScope postConvergenceTimingScope(*this, "Heap::markRoots after convergence");
</del><span class="cx">
</span><del>- // Weak references must be marked last because their liveness depends on
- // the liveness of the rest of the object graph.
- visitWeakHandles(heapRootVisitor);
-
</del><span class="cx"> {
</span><span class="cx"> std::lock_guard<Lock> lock(m_markingMutex);
</span><span class="cx"> m_parallelMarkersShouldExit = true;
</span><span class="lines">@@ -563,7 +647,6 @@
</span><span class="cx">
</span><span class="cx"> void Heap::gatherStackRoots(ConservativeRoots& roots)
</span><span class="cx"> {
</span><del>- m_jitStubRoutines->clearMarks();
</del><span class="cx"> m_machineThreads.gatherConservativeRoots(roots, *m_jitStubRoutines, *m_codeBlocks);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -590,39 +673,18 @@
</span><span class="cx"> TimingScope timingScope(*this, "Heap::beginMarking");
</span><span class="cx"> if (m_collectionScope == CollectionScope::Full)
</span><span class="cx"> m_codeBlocks->clearMarksForFullCollection();
</span><del>-
- {
- TimingScope clearMarksTimingScope(*this, "m_objectSpace.beginMarking");
- m_objectSpace.beginMarking();
- }
</del><ins>+ m_jitStubRoutines->clearMarks();
+ m_objectSpace.beginMarking();
+ m_mutatorShouldBeFenced = true;
+ m_barrierThreshold = tautologicalThreshold;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::visitExternalRememberedSet()
-{
-#if JSC_OBJC_API_ENABLED
- scanExternalRememberedSet(*m_vm, m_slotVisitor);
-#endif
-}
-
-void Heap::visitSmallStrings()
-{
- if (!m_vm->smallStrings.needsToBeVisited(*m_collectionScope))
- return;
-
- m_vm->smallStrings.visitStrongReferences(m_slotVisitor);
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Small strings:\n", m_slotVisitor);
- m_slotVisitor.donateAndDrain();
-}
-
</del><span class="cx"> void Heap::visitConservativeRoots(ConservativeRoots& roots)
</span><span class="cx"> {
</span><del>- m_slotVisitor.append(roots);
</del><ins>+ m_collectorSlotVisitor->append(roots);
</ins><span class="cx">
</span><span class="cx"> if (Options::logGC() == GCLogging::Verbose)
</span><del>- dataLog("Conservative Roots:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
</del><ins>+ dataLog("Conservative Roots:\n", *m_collectorSlotVisitor);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::visitCompilerWorklistWeakReferences()
</span><span class="lines">@@ -629,10 +691,10 @@
</span><span class="cx"> {
</span><span class="cx"> #if ENABLE(DFG_JIT)
</span><span class="cx"> for (unsigned i = DFG::numberOfWorklists(); i--;)
</span><del>- DFG::existingWorklistForIndex(i).visitWeakReferences(m_slotVisitor);
</del><ins>+ DFG::existingWorklistForIndex(i).visitWeakReferences(*m_collectorSlotVisitor);
</ins><span class="cx">
</span><span class="cx"> if (Options::logGC() == GCLogging::Verbose)
</span><del>- dataLog("DFG Worklists:\n", m_slotVisitor);
</del><ins>+ dataLog("DFG Worklists:\n", *m_collectorSlotVisitor);
</ins><span class="cx"> #endif
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -705,132 +767,10 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::visitProtectedObjects(HeapRootVisitor& heapRootVisitor)
-{
- for (auto& pair : m_protectedValues)
- heapRootVisitor.visit(&pair.key);
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Protected Objects:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitArgumentBuffers(HeapRootVisitor& visitor)
-{
- if (!m_markListSet || !m_markListSet->size())
- return;
-
- MarkedArgumentBuffer::markLists(visitor, *m_markListSet);
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Argument Buffers:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitException(HeapRootVisitor& visitor)
-{
- if (!m_vm->exception() && !m_vm->lastException())
- return;
-
- visitor.visit(m_vm->addressOfException());
- visitor.visit(m_vm->addressOfLastException());
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Exceptions:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitStrongHandles(HeapRootVisitor& visitor)
-{
- m_handleSet.visitStrongHandles(visitor);
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Strong Handles:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitHandleStack(HeapRootVisitor& visitor)
-{
- m_handleStack.visit(visitor);
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Handle Stack:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitSamplingProfiler()
-{
-#if ENABLE(SAMPLING_PROFILER)
- if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
- ASSERT(samplingProfiler->getLock().isLocked());
- samplingProfiler->visit(m_slotVisitor);
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Sampling Profiler data:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
- samplingProfiler->getLock().unlock();
- }
-#endif // ENABLE(SAMPLING_PROFILER)
-}
-
-void Heap::visitTypeProfiler()
-{
- if (vm()->typeProfiler()) {
- vm()->typeProfilerLog()->visit(m_slotVisitor);
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Type Profiler visit data:\n", m_slotVisitor);
- m_slotVisitor.donateAndDrain();
- }
-}
-
-void Heap::visitShadowChicken()
-{
- m_vm->shadowChicken().visitChildren(m_slotVisitor);
-}
-
-void Heap::traceCodeBlocksAndJITStubRoutines()
-{
- m_jitStubRoutines->traceMarkedStubRoutines(m_slotVisitor);
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Code Blocks and JIT Stub Routines:\n", m_slotVisitor);
-
- m_slotVisitor.donateAndDrain();
-}
-
-void Heap::visitWeakHandles(HeapRootVisitor& visitor)
-{
- TimingScope timingScope(*this, "Heap::visitWeakHandles");
- while (true) {
- {
- TimingScope timingScope(*this, "m_objectSpace.visitWeakSets");
- m_objectSpace.visitWeakSets(visitor);
- }
- harvestWeakReferences();
- visitCompilerWorklistWeakReferences();
- if (m_slotVisitor.isEmpty())
- break;
-
- if (Options::logGC() == GCLogging::Verbose)
- dataLog("Live Weak Handles:\n", m_slotVisitor);
-
- {
- ParallelModeEnabler enabler(m_slotVisitor);
- m_slotVisitor.donateAndDrain();
- m_slotVisitor.drainFromShared(SlotVisitor::MasterDrain);
- }
- }
-}
-
</del><span class="cx"> void Heap::updateObjectCounts(double gcStartTime)
</span><span class="cx"> {
</span><span class="cx"> if (Options::logGC() == GCLogging::Verbose) {
</span><del>- size_t visitCount = m_slotVisitor.visitCount();
</del><ins>+ size_t visitCount = m_collectorSlotVisitor->visitCount();
</ins><span class="cx"> visitCount += threadVisitCount();
</span><span class="cx"> dataLogF("\nNumber of live Objects after GC %lu, took %.6f secs\n", static_cast<unsigned long>(visitCount), WTF::monotonicallyIncreasingTime() - gcStartTime);
</span><span class="cx"> }
</span><span class="lines">@@ -838,7 +778,9 @@
</span><span class="cx"> if (m_collectionScope == CollectionScope::Full)
</span><span class="cx"> m_totalBytesVisited = 0;
</span><span class="cx">
</span><del>- m_totalBytesVisitedThisCycle = m_slotVisitor.bytesVisited() + threadBytesVisited();
</del><ins>+ m_totalBytesVisitedThisCycle =
+ m_collectorSlotVisitor->bytesVisited() +
+ threadBytesVisited();
</ins><span class="cx">
</span><span class="cx"> m_totalBytesVisited += m_totalBytesVisitedThisCycle;
</span><span class="cx"> }
</span><span class="lines">@@ -845,15 +787,18 @@
</span><span class="cx">
</span><span class="cx"> void Heap::endMarking()
</span><span class="cx"> {
</span><del>- m_slotVisitor.reset();
</del><ins>+ m_collectorSlotVisitor->reset();
</ins><span class="cx">
</span><span class="cx"> for (auto& parallelVisitor : m_parallelSlotVisitors)
</span><span class="cx"> parallelVisitor->reset();
</span><span class="cx">
</span><del>- ASSERT(m_sharedMarkStack.isEmpty());
</del><ins>+ RELEASE_ASSERT(m_sharedCollectorMarkStack->isEmpty());
+ RELEASE_ASSERT(m_sharedMutatorMarkStack->isEmpty());
</ins><span class="cx"> m_weakReferenceHarvesters.removeAll();
</span><span class="cx">
</span><span class="cx"> m_objectSpace.endMarking();
</span><ins>+ m_mutatorShouldBeFenced = Options::forceFencedBarrier();
+ m_barrierThreshold = Options::forceFencedBarrier() ? tautologicalThreshold : blackThreshold;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> size_t Heap::objectCount()
</span><span class="lines">@@ -988,14 +933,20 @@
</span><span class="cx"> {
</span><span class="cx"> ASSERT(cell);
</span><span class="cx"> ASSERT(!Options::useConcurrentJIT() || !isCompilationThread());
</span><del>- ASSERT(cell->cellState() == CellState::AnthraciteOrBlack);
- // Indicate that this object is grey and that it's one of the following:
- // - A re-greyed object during a concurrent collection.
- // - An old remembered object.
- // "OldGrey" doesn't tell us which of these things is true, but we usually treat the two cases the
- // same.
- cell->setCellState(CellState::OldGrey);
- m_slotVisitor.appendToMarkStack(const_cast<JSCell*>(cell));
</del><ins>+ if (!Heap::isMarkedConcurrently(cell)) {
+ // During a full collection a store into an unmarked object that had surivived past
+ // collections will manifest as a store to an unmarked black object. If the object gets
+ // marked at some time after this then it will go down the normal marking path. We can
+ // safely ignore these stores.
+ return;
+ }
+ // It could be that the object was *just* marked. This means that the collector may set the
+ // state to Grey and then to AnthraciteOrBlack at any time. It's OK for us to race with the
+ // collector here. If we win then this is accurate because the object _will_ get scanned again.
+ // If we lose then someone else will barrier the object again. That would be unfortunate but not
+ // the end of the world.
+ cell->setCellState(CellState::Grey);
+ m_mutatorMarkStack->append(cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::collectAllGarbage()
</span><span class="lines">@@ -1094,24 +1045,14 @@
</span><span class="cx">
</span><span class="cx"> stopTheWorld();
</span><span class="cx">
</span><del>- double before = 0;
</del><ins>+ MonotonicTime before;
</ins><span class="cx"> if (Options::logGC()) {
</span><del>- dataLog("[GC: ", capacity() / 1024, " kb ");
- before = currentTimeMS();
</del><ins>+ dataLog("[GC: START ", capacity() / 1024, " kb ");
+ before = MonotonicTime::now();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> double gcStartTime;
</span><span class="cx">
</span><del>-#if ENABLE(JIT)
- {
- DeferGCForAWhile awhile(*this);
- if (JITWorklist::instance()->completeAllForVM(*m_vm))
- setGCDidJIT();
- }
-#endif // ENABLE(JIT)
-
- vm()->shadowChicken().update(*vm(), vm()->topCallFrame);
-
</del><span class="cx"> ASSERT(m_isSafeToCollect);
</span><span class="cx"> if (m_collectionScope) {
</span><span class="cx"> dataLog("Collection scope already set during GC: ", m_collectionScope, "\n");
</span><span class="lines">@@ -1131,12 +1072,9 @@
</span><span class="cx"> m_verifier->gatherLiveObjects(HeapVerifier::Phase::BeforeMarking);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- flushOldStructureIDTables();
- stopAllocation();
</del><span class="cx"> prepareForMarking();
</span><del>- flushWriteBarrierBuffer();
</del><span class="cx">
</span><del>- markRoots(gcStartTime);
</del><ins>+ markToFixpoint(gcStartTime);
</ins><span class="cx">
</span><span class="cx"> if (m_verifier) {
</span><span class="cx"> m_verifier->gatherLiveObjects(HeapVerifier::Phase::AfterMarking);
</span><span class="lines">@@ -1173,8 +1111,8 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> if (Options::logGC()) {
</span><del>- double after = currentTimeMS();
- dataLog(after - before, " ms]\n");
</del><ins>+ MonotonicTime after = MonotonicTime::now();
+ dataLog((after - m_stopTime).milliseconds(), " ms, cycle ", (after - before).milliseconds(), " ms END]\n");
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> {
</span><span class="lines">@@ -1196,10 +1134,33 @@
</span><span class="cx"> stopTheMutator();
</span><span class="cx"> suspendCompilerThreads();
</span><span class="cx"> m_collectorBelievesThatTheWorldIsStopped = true;
</span><ins>+
+#if ENABLE(JIT)
+ {
+ DeferGCForAWhile awhile(*this);
+ if (JITWorklist::instance()->completeAllForVM(*m_vm))
+ setGCDidJIT();
+ }
+#endif // ENABLE(JIT)
+
+ vm()->shadowChicken().update(*vm(), vm()->topCallFrame);
+
+ flushWriteBarrierBuffer();
+ m_structureIDTable.flushOldTables();
+ m_objectSpace.stopAllocating();
+
+ m_stopTime = MonotonicTime::now();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::resumeTheWorld()
</span><span class="cx"> {
</span><ins>+ // Calling resumeAllocating does the Right Thing depending on whether this is the end of a
+ // collection cycle or this is just a concurrent phase within a collection cycle:
+ // - At end of collection cycle: it's a no-op because prepareForAllocation already cleared the
+ // last active block.
+ // - During collection cycle: it reinstates the last active block.
+ m_objectSpace.resumeAllocating();
+
</ins><span class="cx"> RELEASE_ASSERT(m_collectorBelievesThatTheWorldIsStopped);
</span><span class="cx"> m_collectorBelievesThatTheWorldIsStopped = false;
</span><span class="cx"> resumeCompilerThreads();
</span><span class="lines">@@ -1408,23 +1369,18 @@
</span><span class="cx">
</span><span class="cx"> void Heap::setGCDidJIT()
</span><span class="cx"> {
</span><del>- for (;;) {
- unsigned oldState = m_worldState.load();
- RELEASE_ASSERT(oldState & stoppedBit);
- if (m_worldState.compareExchangeWeak(oldState, oldState | gcDidJITBit))
- return;
- }
</del><ins>+ m_worldState.transaction(
+ [&] (unsigned& state) {
+ RELEASE_ASSERT(state & stoppedBit);
+ state |= gcDidJITBit;
+ });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::setNeedFinalize()
</span><span class="cx"> {
</span><del>- for (;;) {
- unsigned oldState = m_worldState.load();
- if (m_worldState.compareExchangeWeak(oldState, oldState | needFinalizeBit)) {
- m_stopIfNecessaryTimer->scheduleSoon();
- return;
- }
- }
</del><ins>+ m_worldState.exchangeOr(needFinalizeBit);
+ ParkingLot::unparkAll(&m_worldState);
+ m_stopIfNecessaryTimer->scheduleSoon();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::waitWhileNeedFinalize()
</span><span class="lines">@@ -1441,23 +1397,14 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-unsigned Heap::setMutatorWaiting()
</del><ins>+void Heap::setMutatorWaiting()
</ins><span class="cx"> {
</span><del>- for (;;) {
- unsigned oldState = m_worldState.load();
- unsigned newState = oldState | mutatorWaitingBit;
- if (m_worldState.compareExchangeWeak(oldState, newState))
- return newState;
- }
</del><ins>+ m_worldState.exchangeOr(mutatorWaitingBit);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::clearMutatorWaiting()
</span><span class="cx"> {
</span><del>- for (;;) {
- unsigned oldState = m_worldState.load();
- if (m_worldState.compareExchangeWeak(oldState, oldState & ~mutatorWaitingBit))
- return;
- }
</del><ins>+ m_worldState.exchangeAnd(~mutatorWaitingBit);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Heap::notifyThreadStopping(const LockHolder&)
</span><span class="lines">@@ -1469,10 +1416,15 @@
</span><span class="cx">
</span><span class="cx"> void Heap::finalize()
</span><span class="cx"> {
</span><del>- HelpingGCScope helpingGCScope(*this);
- deleteUnmarkedCompiledCode();
- deleteSourceProviderCaches();
- sweepLargeAllocations();
</del><ins>+ {
+ HelpingGCScope helpingGCScope(*this);
+ deleteUnmarkedCompiledCode();
+ deleteSourceProviderCaches();
+ sweepLargeAllocations();
+ }
+
+ if (Options::collectContinuously())
+ collectAsync();
</ins><span class="cx"> if (HasOwnPropertyCache* cache = vm()->hasOwnPropertyCache())
</span><span class="cx"> cache->clear();
</span><span class="cx"> }
</span><span class="lines">@@ -1554,25 +1506,11 @@
</span><span class="cx"> observer->willGarbageCollect();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::flushOldStructureIDTables()
-{
- m_structureIDTable.flushOldTables();
-}
-
</del><span class="cx"> void Heap::flushWriteBarrierBuffer()
</span><span class="cx"> {
</span><del>- if (m_collectionScope == CollectionScope::Eden) {
- m_writeBarrierBuffer.flush(*this);
- return;
- }
- m_writeBarrierBuffer.reset();
</del><ins>+ m_writeBarrierBuffer.flush(*this);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::stopAllocation()
-{
- m_objectSpace.stopAllocating();
-}
-
</del><span class="cx"> void Heap::prepareForMarking()
</span><span class="cx"> {
</span><span class="cx"> m_objectSpace.prepareForMarking();
</span><span class="lines">@@ -1937,7 +1875,7 @@
</span><span class="cx">
</span><span class="cx"> void Heap::writeBarrierSlowPath(const JSCell* from)
</span><span class="cx"> {
</span><del>- if (UNLIKELY(barrierShouldBeFenced())) {
</del><ins>+ if (UNLIKELY(mutatorShouldBeFenced())) {
</ins><span class="cx"> // In this case, the barrierThreshold is the tautological threshold, so from could still be
</span><span class="cx"> // not black. But we can't know for sure until we fire off a fence.
</span><span class="cx"> WTF::storeLoadFence();
</span><span class="lines">@@ -1954,7 +1892,7 @@
</span><span class="cx"> return false;
</span><span class="cx"> if (!m_isSafeToCollect)
</span><span class="cx"> return false;
</span><del>- if (collectionScope() || mutatorState() == MutatorState::HelpingGC)
</del><ins>+ if (mutatorState() == MutatorState::HelpingGC)
</ins><span class="cx"> return false;
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="lines">@@ -1976,12 +1914,8 @@
</span><span class="cx"> return mayBeGCThread() || mutatorState() != MutatorState::Running;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Heap::reportExtraMemoryVisited(CellState oldState, size_t size)
</del><ins>+void Heap::reportExtraMemoryVisited(size_t size)
</ins><span class="cx"> {
</span><del>- // We don't want to double-count the extra memory that was reported in previous collections.
- if (collectionScope() == CollectionScope::Eden && oldState == CellState::OldGrey)
- return;
-
</del><span class="cx"> size_t* counter = &m_extraMemorySize;
</span><span class="cx">
</span><span class="cx"> for (;;) {
</span><span class="lines">@@ -1992,12 +1926,8 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #if ENABLE(RESOURCE_USAGE)
</span><del>-void Heap::reportExternalMemoryVisited(CellState oldState, size_t size)
</del><ins>+void Heap::reportExternalMemoryVisited(size_t size)
</ins><span class="cx"> {
</span><del>- // We don't want to double-count the external memory that was reported in previous collections.
- if (collectionScope() == CollectionScope::Eden && oldState == CellState::OldGrey)
- return;
-
</del><span class="cx"> size_t* counter = &m_externalMemorySize;
</span><span class="cx">
</span><span class="cx"> for (;;) {
</span><span class="lines">@@ -2091,4 +2021,11 @@
</span><span class="cx"> }
</span><span class="cx"> #endif // USE(CF)
</span><span class="cx">
</span><ins>+void Heap::notifyIsSafeToCollect()
+{
+ m_isSafeToCollect = true;
+ if (Options::collectContinuously())
+ collectAsync();
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/Heap.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -22,6 +22,7 @@
</span><span class="cx"> #pragma once
</span><span class="cx">
</span><span class="cx"> #include "ArrayBuffer.h"
</span><ins>+#include "CellState.h"
</ins><span class="cx"> #include "CollectionScope.h"
</span><span class="cx"> #include "GCIncomingRefCountedSet.h"
</span><span class="cx"> #include "HandleSet.h"
</span><span class="lines">@@ -35,7 +36,6 @@
</span><span class="cx"> #include "MarkedSpace.h"
</span><span class="cx"> #include "MutatorState.h"
</span><span class="cx"> #include "Options.h"
</span><del>-#include "SlotVisitor.h"
</del><span class="cx"> #include "StructureIDTable.h"
</span><span class="cx"> #include "TinyBloomFilter.h"
</span><span class="cx"> #include "UnconditionalFinalizer.h"
</span><span class="lines">@@ -70,7 +70,9 @@
</span><span class="cx"> class JSCell;
</span><span class="cx"> class JSValue;
</span><span class="cx"> class LLIntOffsetsExtractor;
</span><ins>+class MarkStackArray;
</ins><span class="cx"> class MarkedArgumentBuffer;
</span><ins>+class SlotVisitor;
</ins><span class="cx"> class StopIfNecessaryTimer;
</span><span class="cx"> class VM;
</span><span class="cx">
</span><span class="lines">@@ -129,7 +131,8 @@
</span><span class="cx"> MarkedSpace& objectSpace() { return m_objectSpace; }
</span><span class="cx"> MachineThreads& machineThreads() { return m_machineThreads; }
</span><span class="cx">
</span><del>- const SlotVisitor& slotVisitor() const { return m_slotVisitor; }
</del><ins>+ SlotVisitor& collectorSlotVisitor() { return *m_collectorSlotVisitor; }
+ MarkStackArray& mutatorMarkStack() { return *m_mutatorMarkStack; }
</ins><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE GCActivityCallback* fullActivityCallback();
</span><span class="cx"> JS_EXPORT_PRIVATE GCActivityCallback* edenActivityCallback();
</span><span class="lines">@@ -168,7 +171,7 @@
</span><span class="cx"> JS_EXPORT_PRIVATE void addFinalizer(JSCell*, Finalizer);
</span><span class="cx"> void addExecutable(ExecutableBase*);
</span><span class="cx">
</span><del>- void notifyIsSafeToCollect() { m_isSafeToCollect = true; }
</del><ins>+ void notifyIsSafeToCollect();
</ins><span class="cx"> bool isSafeToCollect() const { return m_isSafeToCollect; }
</span><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE bool isHeapSnapshotting() const;
</span><span class="lines">@@ -203,11 +206,11 @@
</span><span class="cx"> // call both of these functions: Calling only one may trigger catastropic
</span><span class="cx"> // memory growth.
</span><span class="cx"> void reportExtraMemoryAllocated(size_t);
</span><del>- JS_EXPORT_PRIVATE void reportExtraMemoryVisited(CellState oldState, size_t);
</del><ins>+ JS_EXPORT_PRIVATE void reportExtraMemoryVisited(size_t);
</ins><span class="cx">
</span><span class="cx"> #if ENABLE(RESOURCE_USAGE)
</span><span class="cx"> // Use this API to report the subset of extra memory that lives outside this process.
</span><del>- JS_EXPORT_PRIVATE void reportExternalMemoryVisited(CellState oldState, size_t);
</del><ins>+ JS_EXPORT_PRIVATE void reportExternalMemoryVisited(size_t);
</ins><span class="cx"> size_t externalMemorySize() { return m_externalMemorySize; }
</span><span class="cx"> #endif
</span><span class="cx">
</span><span class="lines">@@ -283,8 +286,8 @@
</span><span class="cx"> void didAllocateBlock(size_t capacity);
</span><span class="cx"> void didFreeBlock(size_t capacity);
</span><span class="cx">
</span><del>- bool barrierShouldBeFenced() const { return m_barrierShouldBeFenced; }
- const bool* addressOfBarrierShouldBeFenced() const { return &m_barrierShouldBeFenced; }
</del><ins>+ bool mutatorShouldBeFenced() const { return m_mutatorShouldBeFenced; }
+ const bool* addressOfMutatorShouldBeFenced() const { return &m_mutatorShouldBeFenced; }
</ins><span class="cx">
</span><span class="cx"> unsigned barrierThreshold() const { return m_barrierThreshold; }
</span><span class="cx"> const unsigned* addressOfBarrierThreshold() const { return &m_barrierThreshold; }
</span><span class="lines">@@ -391,6 +394,9 @@
</span><span class="cx"> void stopTheWorld();
</span><span class="cx"> void resumeTheWorld();
</span><span class="cx">
</span><ins>+ class ResumeTheWorldScope;
+ friend class ResumeTheWorldScope;
+
</ins><span class="cx"> void stopTheMutator();
</span><span class="cx"> void resumeTheMutator();
</span><span class="cx">
</span><span class="lines">@@ -412,7 +418,7 @@
</span><span class="cx"> void setNeedFinalize();
</span><span class="cx"> void waitWhileNeedFinalize();
</span><span class="cx">
</span><del>- unsigned setMutatorWaiting();
</del><ins>+ void setMutatorWaiting();
</ins><span class="cx"> void clearMutatorWaiting();
</span><span class="cx"> void notifyThreadStopping(const LockHolder&);
</span><span class="cx">
</span><span class="lines">@@ -422,31 +428,18 @@
</span><span class="cx">
</span><span class="cx"> void suspendCompilerThreads();
</span><span class="cx"> void willStartCollection(Optional<CollectionScope>);
</span><del>- void flushOldStructureIDTables();
</del><span class="cx"> void flushWriteBarrierBuffer();
</span><del>- void stopAllocation();
</del><span class="cx"> void prepareForMarking();
</span><span class="cx">
</span><del>- void markRoots(double gcStartTime);
</del><ins>+ void markToFixpoint(double gcStartTime);
</ins><span class="cx"> void gatherStackRoots(ConservativeRoots&);
</span><span class="cx"> void gatherJSStackRoots(ConservativeRoots&);
</span><span class="cx"> void gatherScratchBufferRoots(ConservativeRoots&);
</span><span class="cx"> void beginMarking();
</span><del>- void visitExternalRememberedSet();
- void visitSmallStrings();
</del><span class="cx"> void visitConservativeRoots(ConservativeRoots&);
</span><span class="cx"> void visitCompilerWorklistWeakReferences();
</span><span class="cx"> void removeDeadCompilerWorklistEntries();
</span><del>- void visitProtectedObjects(HeapRootVisitor&);
- void visitArgumentBuffers(HeapRootVisitor&);
- void visitException(HeapRootVisitor&);
- void visitStrongHandles(HeapRootVisitor&);
- void visitHandleStack(HeapRootVisitor&);
- void visitSamplingProfiler();
- void visitTypeProfiler();
- void visitShadowChicken();
- void traceCodeBlocksAndJITStubRoutines();
- void visitWeakHandles(HeapRootVisitor&);
</del><ins>+ void markToFixpoint(HeapRootVisitor&);
</ins><span class="cx"> void updateObjectCounts(double gcStartTime);
</span><span class="cx"> void endMarking();
</span><span class="cx">
</span><span class="lines">@@ -518,7 +511,8 @@
</span><span class="cx">
</span><span class="cx"> MachineThreads m_machineThreads;
</span><span class="cx">
</span><del>- SlotVisitor m_slotVisitor;
</del><ins>+ std::unique_ptr<SlotVisitor> m_collectorSlotVisitor;
+ std::unique_ptr<MarkStackArray> m_mutatorMarkStack;
</ins><span class="cx">
</span><span class="cx"> // We pool the slot visitors used by parallel marking threads. It's useful to be able to
</span><span class="cx"> // enumerate over them, and it's useful to have them cache some small amount of memory from
</span><span class="lines">@@ -537,7 +531,7 @@
</span><span class="cx"> bool m_isSafeToCollect;
</span><span class="cx">
</span><span class="cx"> WriteBarrierBuffer m_writeBarrierBuffer;
</span><del>- bool m_barrierShouldBeFenced { Options::forceFencedBarrier() };
</del><ins>+ bool m_mutatorShouldBeFenced { Options::forceFencedBarrier() };
</ins><span class="cx"> unsigned m_barrierThreshold { Options::forceFencedBarrier() ? tautologicalThreshold : blackThreshold };
</span><span class="cx">
</span><span class="cx"> VM* m_vm;
</span><span class="lines">@@ -572,7 +566,8 @@
</span><span class="cx">
</span><span class="cx"> Lock m_markingMutex;
</span><span class="cx"> Condition m_markingConditionVariable;
</span><del>- MarkStackArray m_sharedMarkStack;
</del><ins>+ std::unique_ptr<MarkStackArray> m_sharedCollectorMarkStack;
+ std::unique_ptr<MarkStackArray> m_sharedMutatorMarkStack;
</ins><span class="cx"> unsigned m_numberOfActiveParallelMarkers { 0 };
</span><span class="cx"> unsigned m_numberOfWaitingParallelMarkers { 0 };
</span><span class="cx"> bool m_parallelMarkersShouldExit { false };
</span><span class="lines">@@ -600,6 +595,7 @@
</span><span class="cx"> static const unsigned mutatorWaitingBit = 1u << 5u; // Allows the mutator to use this as a condition variable.
</span><span class="cx"> Atomic<unsigned> m_worldState;
</span><span class="cx"> bool m_collectorBelievesThatTheWorldIsStopped { false };
</span><ins>+ MonotonicTime m_stopTime;
</ins><span class="cx">
</span><span class="cx"> Deque<Optional<CollectionScope>> m_requests;
</span><span class="cx"> Ticket m_lastServedTicket { 0 };
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkStackcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkStack.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkStack.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkStack.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009, 2011, 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -36,6 +36,34 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void MarkStackArray::transferTo(MarkStackArray& other)
+{
+ RELEASE_ASSERT(this != &other);
+
+ // Remove our head and the head of the other list.
+ GCArraySegment<const JSCell*>* myHead = m_segments.removeHead();
+ GCArraySegment<const JSCell*>* otherHead = other.m_segments.removeHead();
+ m_numberOfSegments--;
+ other.m_numberOfSegments--;
+
+ other.m_segments.append(m_segments);
+
+ other.m_numberOfSegments += m_numberOfSegments;
+ m_numberOfSegments = 0;
+
+ // Put the original heads back in their places.
+ m_segments.push(myHead);
+ other.m_segments.push(otherHead);
+ m_numberOfSegments++;
+ other.m_numberOfSegments++;
+
+ while (!isEmpty()) {
+ refill();
+ while (canRemoveLast())
+ other.append(removeLast());
+ }
+}
+
</ins><span class="cx"> void MarkStackArray::donateSomeCellsTo(MarkStackArray& other)
</span><span class="cx"> {
</span><span class="cx"> // Try to donate about 1 / 2 of our cells. To reduce copying costs,
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkStackh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkStack.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkStack.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkStack.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009, 2011 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -35,8 +35,9 @@
</span><span class="cx"> public:
</span><span class="cx"> MarkStackArray();
</span><span class="cx">
</span><del>- void donateSomeCellsTo(MarkStackArray& other);
- void stealSomeCellsFrom(MarkStackArray& other, size_t idleThreadCount);
</del><ins>+ void transferTo(MarkStackArray&);
+ void donateSomeCellsTo(MarkStackArray&);
+ void stealSomeCellsFrom(MarkStackArray&, size_t idleThreadCount);
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedAllocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkedAllocator.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -137,7 +137,6 @@
</span><span class="cx"> void* MarkedAllocator::tryAllocateIn(MarkedBlock::Handle* block)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(block);
</span><del>- ASSERT(!block->hasAnyNewlyAllocated());
</del><span class="cx"> ASSERT(!block->isFreeListed());
</span><span class="cx">
</span><span class="cx"> FreeList freeList = block->sweep(MarkedBlock::Handle::SweepToFreeList);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -82,12 +82,14 @@
</span><span class="cx"> , m_handle(handle)
</span><span class="cx"> , m_vm(&vm)
</span><span class="cx"> {
</span><ins>+ if (false)
+ dataLog(RawPointer(this), ": Allocated.\n");
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode, MarkedBlock::Handle::MarksMode marksMode>
</del><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, MarkedBlock::Handle::SweepDestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode, MarkedBlock::Handle::MarksMode marksMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::specializedSweep()
</span><span class="cx"> {
</span><del>- RELEASE_ASSERT(!(destructionMode == DoesNotNeedDestruction && sweepMode == SweepOnly));
</del><ins>+ RELEASE_ASSERT(!(destructionMode == BlockHasNoDestructors && sweepMode == SweepOnly));
</ins><span class="cx">
</span><span class="cx"> SuperSamplerScope superSamplerScope(false);
</span><span class="cx">
</span><span class="lines">@@ -99,7 +101,7 @@
</span><span class="cx"> if (Options::useBumpAllocator()
</span><span class="cx"> && emptyMode == IsEmpty
</span><span class="cx"> && newlyAllocatedMode == DoesNotHaveNewlyAllocated) {
</span><del>- ASSERT(marksMode == MarksStale);
</del><ins>+ ASSERT(marksMode == MarksStale || block.m_marks.isEmpty());
</ins><span class="cx">
</span><span class="cx"> char* startOfLastCell = static_cast<char*>(cellAlign(block.atoms() + m_endAtom - 1));
</span><span class="cx"> char* payloadEnd = startOfLastCell + cellSize();
</span><span class="lines">@@ -111,6 +113,8 @@
</span><span class="cx"> setIsFreeListed();
</span><span class="cx"> else
</span><span class="cx"> m_allocator->setIsEmpty(this, true);
</span><ins>+ if (space()->isMarking())
+ block.m_lock.unlock();
</ins><span class="cx"> FreeList result = FreeList::bump(payloadEnd, payloadEnd - payloadBegin);
</span><span class="cx"> if (false)
</span><span class="cx"> dataLog("Quickly swept block ", RawPointer(this), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
</span><span class="lines">@@ -123,17 +127,11 @@
</span><span class="cx"> FreeCell* head = 0;
</span><span class="cx"> size_t count = 0;
</span><span class="cx"> bool isEmpty = true;
</span><del>- for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
- if (emptyMode == NotEmpty
- && ((marksMode == MarksNotStale && block.m_marks.get(i))
- || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated.get(i)))) {
- isEmpty = false;
- continue;
- }
-
</del><ins>+ Vector<size_t> deadCells;
+ auto handleDeadCell = [&] (size_t i) {
</ins><span class="cx"> HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&block.atoms()[i]);
</span><span class="cx">
</span><del>- if (destructionMode == NeedsDestruction && emptyMode == NotEmpty)
</del><ins>+ if (destructionMode != BlockHasNoDestructors && emptyMode == NotEmpty)
</ins><span class="cx"> static_cast<JSCell*>(cell)->callDestructor(*vm());
</span><span class="cx">
</span><span class="cx"> if (sweepMode == SweepToFreeList) {
</span><span class="lines">@@ -144,12 +142,33 @@
</span><span class="cx"> head = freeCell;
</span><span class="cx"> ++count;
</span><span class="cx"> }
</span><ins>+ };
+ for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
+ if (emptyMode == NotEmpty
+ && ((marksMode == MarksNotStale && block.m_marks.get(i))
+ || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated.get(i)))) {
+ isEmpty = false;
+ continue;
+ }
+
+ if (destructionMode == BlockHasDestructorsAndCollectorIsRunning)
+ deadCells.append(i);
+ else
+ handleDeadCell(i);
</ins><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> // We only want to discard the newlyAllocated bits if we're creating a FreeList,
</span><span class="cx"> // otherwise we would lose information on what's currently alive.
</span><span class="cx"> if (sweepMode == SweepToFreeList && newlyAllocatedMode == HasNewlyAllocated)
</span><span class="cx"> m_newlyAllocatedVersion = MarkedSpace::nullVersion;
</span><ins>+
+ if (space()->isMarking())
+ block.m_lock.unlock();
+
+ if (destructionMode == BlockHasDestructorsAndCollectorIsRunning) {
+ for (size_t i : deadCells)
+ handleDeadCell(i);
+ }
</ins><span class="cx">
</span><span class="cx"> FreeList result = FreeList::list(head, count * cellSize());
</span><span class="cx"> if (sweepMode == SweepToFreeList)
</span><span class="lines">@@ -177,12 +196,18 @@
</span><span class="cx">
</span><span class="cx"> ASSERT(!m_allocator->isAllocated(this));
</span><span class="cx">
</span><del>- if (m_attributes.destruction == NeedsDestruction)
- return sweepHelperSelectScribbleMode<NeedsDestruction>(sweepMode);
- return sweepHelperSelectScribbleMode<DoesNotNeedDestruction>(sweepMode);
</del><ins>+ if (space()->isMarking())
+ block().m_lock.lock();
+
+ if (m_attributes.destruction == NeedsDestruction) {
+ if (space()->isMarking())
+ return sweepHelperSelectScribbleMode<BlockHasDestructorsAndCollectorIsRunning>(sweepMode);
+ return sweepHelperSelectScribbleMode<BlockHasDestructors>(sweepMode);
+ }
+ return sweepHelperSelectScribbleMode<BlockHasNoDestructors>(sweepMode);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-template<DestructionMode destructionMode>
</del><ins>+template<MarkedBlock::Handle::SweepDestructionMode destructionMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectScribbleMode(SweepMode sweepMode)
</span><span class="cx"> {
</span><span class="cx"> if (scribbleFreeCells())
</span><span class="lines">@@ -190,7 +215,7 @@
</span><span class="cx"> return sweepHelperSelectEmptyMode<destructionMode, DontScribble>(sweepMode);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</del><ins>+template<MarkedBlock::Handle::SweepDestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectEmptyMode(SweepMode sweepMode)
</span><span class="cx"> {
</span><span class="cx"> // It's not obvious, but this is the only way to know if the block is empty. It's the only
</span><span class="lines">@@ -205,7 +230,7 @@
</span><span class="cx"> return sweepHelperSelectHasNewlyAllocated<NotEmpty, destructionMode, scribbleMode>(sweepMode);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<MarkedBlock::Handle::EmptyMode emptyMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</del><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepDestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated(SweepMode sweepMode)
</span><span class="cx"> {
</span><span class="cx"> if (hasAnyNewlyAllocated())
</span><span class="lines">@@ -213,7 +238,7 @@
</span><span class="cx"> return sweepHelperSelectSweepMode<emptyMode, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>(sweepMode);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<MarkedBlock::Handle::EmptyMode emptyMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
</del><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepDestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectSweepMode(SweepMode sweepMode)
</span><span class="cx"> {
</span><span class="cx"> if (sweepMode == SweepToFreeList)
</span><span class="lines">@@ -221,10 +246,16 @@
</span><span class="cx"> return sweepHelperSelectMarksMode<emptyMode, SweepOnly, destructionMode, scribbleMode, newlyAllocatedMode>();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
</del><ins>+template<MarkedBlock::Handle::EmptyMode emptyMode, MarkedBlock::Handle::SweepMode sweepMode, MarkedBlock::Handle::SweepDestructionMode destructionMode, MarkedBlock::Handle::ScribbleMode scribbleMode, MarkedBlock::Handle::NewlyAllocatedMode newlyAllocatedMode>
</ins><span class="cx"> FreeList MarkedBlock::Handle::sweepHelperSelectMarksMode()
</span><span class="cx"> {
</span><del>- if (areMarksStale())
</del><ins>+ HeapVersion markingVersion = space()->markingVersion();
+ bool marksAreUseful = !block().areMarksStale(markingVersion);
+
+ if (space()->isMarking())
+ marksAreUseful |= block().marksConveyLivenessDuringMarking(markingVersion);
+
+ if (!marksAreUseful)
</ins><span class="cx"> return specializedSweep<emptyMode, sweepMode, destructionMode, scribbleMode, newlyAllocatedMode, MarksStale>();
</span><span class="cx"> return specializedSweep<emptyMode, sweepMode, destructionMode, scribbleMode, newlyAllocatedMode, MarksNotStale>();
</span><span class="cx"> }
</span><span class="lines">@@ -241,24 +272,6 @@
</span><span class="cx"> m_isFreeListed = true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-class SetNewlyAllocatedFunctor : public MarkedBlock::VoidFunctor {
-public:
- SetNewlyAllocatedFunctor(MarkedBlock::Handle* block)
- : m_block(block)
- {
- }
-
- IterationStatus operator()(HeapCell* cell, HeapCell::Kind) const
- {
- ASSERT(MarkedBlock::blockFor(cell) == &m_block->block());
- m_block->setNewlyAllocated(cell);
- return IterationStatus::Continue;
- }
-
-private:
- MarkedBlock::Handle* m_block;
-};
-
</del><span class="cx"> void MarkedBlock::Handle::stopAllocating(const FreeList& freeList)
</span><span class="cx"> {
</span><span class="cx"> if (false)
</span><span class="lines">@@ -266,6 +279,8 @@
</span><span class="cx"> ASSERT(!allocator()->isAllocated(this));
</span><span class="cx">
</span><span class="cx"> if (!isFreeListed()) {
</span><ins>+ if (false)
+ dataLog("There ain't no newly allocated.\n");
</ins><span class="cx"> // This means that we either didn't use this block at all for allocation since last GC,
</span><span class="cx"> // or someone had already done stopAllocating() before.
</span><span class="cx"> ASSERT(freeList.allocationWillFail());
</span><span class="lines">@@ -272,6 +287,9 @@
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ if (false)
+ dataLog("Free list: ", freeList, "\n");
+
</ins><span class="cx"> // Roll back to a coherent state for Heap introspection. Cells newly
</span><span class="cx"> // allocated from our free list are not currently marked, so we need another
</span><span class="cx"> // way to tell what's live vs dead.
</span><span class="lines">@@ -279,12 +297,17 @@
</span><span class="cx"> m_newlyAllocated.clearAll();
</span><span class="cx"> m_newlyAllocatedVersion = heap()->objectSpace().newlyAllocatedVersion();
</span><span class="cx">
</span><del>- SetNewlyAllocatedFunctor functor(this);
- forEachCell(functor);
</del><ins>+ forEachCell(
+ [&] (HeapCell* cell, HeapCell::Kind) -> IterationStatus {
+ setNewlyAllocated(cell);
+ return IterationStatus::Continue;
+ });
</ins><span class="cx">
</span><span class="cx"> forEachFreeCell(
</span><span class="cx"> freeList,
</span><span class="cx"> [&] (HeapCell* cell) {
</span><ins>+ if (false)
+ dataLog("Free cell: ", RawPointer(cell), "\n");
</ins><span class="cx"> if (m_attributes.destruction == NeedsDestruction)
</span><span class="cx"> cell->zap();
</span><span class="cx"> clearNewlyAllocated(cell);
</span><span class="lines">@@ -307,10 +330,14 @@
</span><span class="cx">
</span><span class="cx"> FreeList MarkedBlock::Handle::resumeAllocating()
</span><span class="cx"> {
</span><ins>+ if (false)
+ dataLog(RawPointer(this), ": MarkedBlock::Handle::resumeAllocating!\n");
</ins><span class="cx"> ASSERT(!allocator()->isAllocated(this));
</span><span class="cx"> ASSERT(!isFreeListed());
</span><span class="cx">
</span><span class="cx"> if (!hasAnyNewlyAllocated()) {
</span><ins>+ if (false)
+ dataLog("There ain't no newly allocated.\n");
</ins><span class="cx"> // This means we had already exhausted the block when we stopped allocation.
</span><span class="cx"> return FreeList();
</span><span class="cx"> }
</span><span class="lines">@@ -354,6 +381,8 @@
</span><span class="cx">
</span><span class="cx"> if (handle().allocator()->isAllocated(&handle())
</span><span class="cx"> || !marksConveyLivenessDuringMarking(markingVersion)) {
</span><ins>+ if (false)
+ dataLog(RawPointer(this), ": Clearing marks without doing anything else.\n");
</ins><span class="cx"> // We already know that the block is full and is already recognized as such, or that the
</span><span class="cx"> // block did not survive the previous GC. So, we can clear mark bits the old fashioned
</span><span class="cx"> // way. Note that it's possible for such a block to have newlyAllocated with an up-to-
</span><span class="lines">@@ -362,6 +391,8 @@
</span><span class="cx"> // we created a newlyAllocated.
</span><span class="cx"> m_marks.clearAll();
</span><span class="cx"> } else {
</span><ins>+ if (false)
+ dataLog(RawPointer(this), ": Doing things.\n");
</ins><span class="cx"> HeapVersion newlyAllocatedVersion = space()->newlyAllocatedVersion();
</span><span class="cx"> if (handle().m_newlyAllocatedVersion == newlyAllocatedVersion) {
</span><span class="cx"> // Merge the contents of marked into newlyAllocated. If we get the full set of bits
</span><span class="lines">@@ -429,7 +460,7 @@
</span><span class="cx"> dataLog(RawPointer(this), ": MarkedBlock::Handle::didConsumeFreeList!\n");
</span><span class="cx"> ASSERT(isFreeListed());
</span><span class="cx"> m_isFreeListed = false;
</span><del>- allocator()->setIsAllocated(this, true);
</del><ins>+ allocator()->atomicSetAndCheckIsAllocated(this, true);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> size_t MarkedBlock::markCount()
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedBlockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedBlock.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkedBlock.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -190,30 +190,32 @@
</span><span class="cx"> private:
</span><span class="cx"> Handle(Heap&, void*);
</span><span class="cx">
</span><del>- template<DestructionMode>
</del><ins>+ enum SweepDestructionMode { BlockHasNoDestructors, BlockHasDestructors, BlockHasDestructorsAndCollectorIsRunning };
+
+ template<SweepDestructionMode>
</ins><span class="cx"> FreeList sweepHelperSelectScribbleMode(SweepMode = SweepOnly);
</span><span class="cx">
</span><span class="cx"> enum ScribbleMode { DontScribble, Scribble };
</span><span class="cx">
</span><del>- template<DestructionMode, ScribbleMode>
</del><ins>+ template<SweepDestructionMode, ScribbleMode>
</ins><span class="cx"> FreeList sweepHelperSelectEmptyMode(SweepMode = SweepOnly);
</span><span class="cx">
</span><span class="cx"> enum EmptyMode { IsEmpty, NotEmpty };
</span><span class="cx">
</span><del>- template<EmptyMode, DestructionMode, ScribbleMode>
</del><ins>+ template<EmptyMode, SweepDestructionMode, ScribbleMode>
</ins><span class="cx"> FreeList sweepHelperSelectHasNewlyAllocated(SweepMode = SweepOnly);
</span><span class="cx">
</span><span class="cx"> enum NewlyAllocatedMode { HasNewlyAllocated, DoesNotHaveNewlyAllocated };
</span><span class="cx">
</span><del>- template<EmptyMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
</del><ins>+ template<EmptyMode, SweepDestructionMode, ScribbleMode, NewlyAllocatedMode>
</ins><span class="cx"> FreeList sweepHelperSelectSweepMode(SweepMode = SweepOnly);
</span><span class="cx">
</span><del>- template<EmptyMode, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
</del><ins>+ template<EmptyMode, SweepMode, SweepDestructionMode, ScribbleMode, NewlyAllocatedMode>
</ins><span class="cx"> FreeList sweepHelperSelectMarksMode();
</span><span class="cx">
</span><span class="cx"> enum MarksMode { MarksStale, MarksNotStale };
</span><span class="cx">
</span><del>- template<EmptyMode, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode, MarksMode>
</del><ins>+ template<EmptyMode, SweepMode, SweepDestructionMode, ScribbleMode, NewlyAllocatedMode, MarksMode>
</ins><span class="cx"> FreeList specializedSweep();
</span><span class="cx">
</span><span class="cx"> template<typename Func>
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkedSpacecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/MarkedSpace.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -412,7 +412,6 @@
</span><span class="cx">
</span><span class="cx"> void MarkedSpace::resumeAllocating()
</span><span class="cx"> {
</span><del>- ASSERT(isIterating());
</del><span class="cx"> forEachAllocator(
</span><span class="cx"> [&] (MarkedAllocator& allocator) -> IterationStatus {
</span><span class="cx"> allocator.resumeAllocating();
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -75,9 +75,7 @@
</span><span class="cx"> #endif
</span><span class="cx">
</span><span class="cx"> SlotVisitor::SlotVisitor(Heap& heap)
</span><del>- : m_stack()
- , m_bytesVisited(0)
- , m_bytesCopied(0)
</del><ins>+ : m_bytesVisited(0)
</ins><span class="cx"> , m_visitCount(0)
</span><span class="cx"> , m_isInParallelMode(false)
</span><span class="cx"> , m_markingVersion(MarkedSpace::initialVersion)
</span><span class="lines">@@ -91,7 +89,7 @@
</span><span class="cx">
</span><span class="cx"> SlotVisitor::~SlotVisitor()
</span><span class="cx"> {
</span><del>- clearMarkStack();
</del><ins>+ clearMarkStacks();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::didStartMarking()
</span><span class="lines">@@ -109,16 +107,17 @@
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::reset()
</span><span class="cx"> {
</span><ins>+ RELEASE_ASSERT(!m_opaqueRoots.size());
</ins><span class="cx"> m_bytesVisited = 0;
</span><del>- m_bytesCopied = 0;
</del><span class="cx"> m_visitCount = 0;
</span><span class="cx"> m_heapSnapshotBuilder = nullptr;
</span><del>- ASSERT(!m_currentCell);
</del><ins>+ RELEASE_ASSERT(!m_currentCell);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::clearMarkStack()
</del><ins>+void SlotVisitor::clearMarkStacks()
</ins><span class="cx"> {
</span><del>- m_stack.clear();
</del><ins>+ m_collectorStack.clear();
+ m_mutatorStack.clear();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::append(ConservativeRoots& conservativeRoots)
</span><span class="lines">@@ -148,7 +147,7 @@
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- jsCell->setCellState(CellState::NewGrey);
</del><ins>+ jsCell->setCellState(CellState::Grey);
</ins><span class="cx">
</span><span class="cx"> appendToMarkStack(jsCell);
</span><span class="cx"> return;
</span><span class="lines">@@ -210,7 +209,7 @@
</span><span class="cx"> // Indicate that the object is grey and that:
</span><span class="cx"> // In case of concurrent GC: it's the first time it is grey in this GC cycle.
</span><span class="cx"> // In case of eden collection: it's a new object that became grey rather than an old remembered object.
</span><del>- cell->setCellState(CellState::NewGrey);
</del><ins>+ cell->setCellState(CellState::Grey);
</ins><span class="cx">
</span><span class="cx"> appendToMarkStack(container, cell);
</span><span class="cx"> }
</span><span class="lines">@@ -228,18 +227,21 @@
</span><span class="cx"> {
</span><span class="cx"> ASSERT(Heap::isMarkedConcurrently(cell));
</span><span class="cx"> ASSERT(!cell->isZapped());
</span><del>- ASSERT(cell->cellState() == CellState::NewGrey || cell->cellState() == CellState::OldGrey);
</del><ins>+ ASSERT(cell->cellState() == CellState::Grey);
</ins><span class="cx">
</span><span class="cx"> container.noteMarked();
</span><span class="cx">
</span><del>- // FIXME: These "just work" because the GC resets these fields before doing anything else. But
- // that won't be the case when we do concurrent GC.
</del><span class="cx"> m_visitCount++;
</span><span class="cx"> m_bytesVisited += container.cellSize();
</span><span class="cx">
</span><del>- m_stack.append(cell);
</del><ins>+ m_collectorStack.append(cell);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void SlotVisitor::appendToMutatorMarkStack(const JSCell* cell)
+{
+ m_mutatorStack.append(cell);
+}
+
</ins><span class="cx"> void SlotVisitor::markAuxiliary(const void* base)
</span><span class="cx"> {
</span><span class="cx"> HeapCell* cell = bitwise_cast<HeapCell*>(base);
</span><span class="lines">@@ -294,12 +296,17 @@
</span><span class="cx">
</span><span class="cx"> SetCurrentCellScope currentCellScope(*this, cell);
</span><span class="cx">
</span><del>- m_oldCellState = cell->cellState();
</del><ins>+ if (false) {
+ dataLog("Visiting ", RawPointer(cell));
+ if (m_isVisitingMutatorStack)
+ dataLog(" (mutator)");
+ dataLog("\n");
+ }
</ins><span class="cx">
</span><del>- // There is no race here - the cell state cannot change right now. Grey objects can only be
- // visited by one marking thread. Neither the barrier nor marking will change the state of an
- // object that is already grey.
- ASSERT(m_oldCellState == CellState::OldGrey || m_oldCellState == CellState::NewGrey);
</del><ins>+ // Funny story: it's possible for the object to be black already, if we barrier the object at
+ // about the same time that it's marked. That's fine. It's a gnarly and super-rare race. It's
+ // not clear to me that it would be correct or profitable to bail here if the object is already
+ // black.
</ins><span class="cx">
</span><span class="cx"> cell->setCellState(CellState::AnthraciteOrBlack);
</span><span class="cx">
</span><span class="lines">@@ -326,23 +333,23 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> if (UNLIKELY(m_heapSnapshotBuilder)) {
</span><del>- if (m_oldCellState == CellState::NewGrey)
</del><ins>+ if (!m_isVisitingMutatorStack)
</ins><span class="cx"> m_heapSnapshotBuilder->appendNode(const_cast<JSCell*>(cell));
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::donateKnownParallel()
</del><ins>+void SlotVisitor::donateKnownParallel(MarkStackArray& from, MarkStackArray& to)
</ins><span class="cx"> {
</span><span class="cx"> // NOTE: Because we re-try often, we can afford to be conservative, and
</span><span class="cx"> // assume that donating is not profitable.
</span><span class="cx">
</span><span class="cx"> // Avoid locking when a thread reaches a dead end in the object graph.
</span><del>- if (m_stack.size() < 2)
</del><ins>+ if (from.size() < 2)
</ins><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> // If there's already some shared work queued up, be conservative and assume
</span><span class="cx"> // that donating more is not profitable.
</span><del>- if (m_heap.m_sharedMarkStack.size())
</del><ins>+ if (to.size())
</ins><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> // If we're contending on the lock, be conservative and assume that another
</span><span class="lines">@@ -352,19 +359,36 @@
</span><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> // Otherwise, assume that a thread will go idle soon, and donate.
</span><del>- m_stack.donateSomeCellsTo(m_heap.m_sharedMarkStack);
</del><ins>+ from.donateSomeCellsTo(to);
</ins><span class="cx">
</span><span class="cx"> m_heap.m_markingConditionVariable.notifyAll();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::drain()
</del><ins>+void SlotVisitor::donateKnownParallel()
</ins><span class="cx"> {
</span><ins>+ donateKnownParallel(m_collectorStack, *m_heap.m_sharedCollectorMarkStack);
+ donateKnownParallel(m_mutatorStack, *m_heap.m_sharedMutatorMarkStack);
+}
+
+void SlotVisitor::drain(MonotonicTime timeout)
+{
</ins><span class="cx"> ASSERT(m_isInParallelMode);
</span><span class="cx">
</span><del>- while (!m_stack.isEmpty()) {
- m_stack.refill();
- for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); m_stack.canRemoveLast() && countdown--;)
- visitChildren(m_stack.removeLast());
</del><ins>+ while ((!m_collectorStack.isEmpty() || !m_mutatorStack.isEmpty()) && !hasElapsed(timeout)) {
+ if (!m_collectorStack.isEmpty()) {
+ m_collectorStack.refill();
+ m_isVisitingMutatorStack = false;
+ for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); m_collectorStack.canRemoveLast() && countdown--;)
+ visitChildren(m_collectorStack.removeLast());
+ } else if (!m_mutatorStack.isEmpty()) {
+ m_mutatorStack.refill();
+ // We know for sure that we are visiting objects because of the barrier, not because of
+ // marking. Marking will visit an object exactly once. The barrier will visit it
+ // possibly many times, and always after it was already marked.
+ m_isVisitingMutatorStack = true;
+ for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); m_mutatorStack.canRemoveLast() && countdown--;)
+ visitChildren(m_mutatorStack.removeLast());
+ }
</ins><span class="cx"> donateKnownParallel();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -371,7 +395,7 @@
</span><span class="cx"> mergeOpaqueRootsIfNecessary();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::drainFromShared(SharedDrainMode sharedDrainMode)
</del><ins>+SlotVisitor::SharedDrainResult SlotVisitor::drainFromShared(SharedDrainMode sharedDrainMode, MonotonicTime timeout)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(m_isInParallelMode);
</span><span class="cx">
</span><span class="lines">@@ -392,48 +416,60 @@
</span><span class="cx"> // Wait until either termination is reached, or until there is some work
</span><span class="cx"> // for us to do.
</span><span class="cx"> while (true) {
</span><ins>+ if (hasElapsed(timeout))
+ return SharedDrainResult::TimedOut;
+
</ins><span class="cx"> // Did we reach termination?
</span><span class="cx"> if (!m_heap.m_numberOfActiveParallelMarkers
</span><del>- && m_heap.m_sharedMarkStack.isEmpty()) {
</del><ins>+ && m_heap.m_sharedCollectorMarkStack->isEmpty()
+ && m_heap.m_sharedMutatorMarkStack->isEmpty()) {
</ins><span class="cx"> // Let any sleeping slaves know it's time for them to return;
</span><span class="cx"> m_heap.m_markingConditionVariable.notifyAll();
</span><del>- return;
</del><ins>+ return SharedDrainResult::Done;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // Is there work to be done?
</span><del>- if (!m_heap.m_sharedMarkStack.isEmpty())
</del><ins>+ if (!m_heap.m_sharedCollectorMarkStack->isEmpty()
+ || !m_heap.m_sharedMutatorMarkStack->isEmpty())
</ins><span class="cx"> break;
</span><span class="cx">
</span><span class="cx"> // Otherwise wait.
</span><del>- m_heap.m_markingConditionVariable.wait(lock);
</del><ins>+ m_heap.m_markingConditionVariable.waitUntil(lock, timeout);
</ins><span class="cx"> }
</span><span class="cx"> } else {
</span><span class="cx"> ASSERT(sharedDrainMode == SlaveDrain);
</span><ins>+
+ if (hasElapsed(timeout))
+ return SharedDrainResult::TimedOut;
</ins><span class="cx">
</span><span class="cx"> // Did we detect termination? If so, let the master know.
</span><span class="cx"> if (!m_heap.m_numberOfActiveParallelMarkers
</span><del>- && m_heap.m_sharedMarkStack.isEmpty())
</del><ins>+ && m_heap.m_sharedCollectorMarkStack->isEmpty()
+ && m_heap.m_sharedMutatorMarkStack->isEmpty())
</ins><span class="cx"> m_heap.m_markingConditionVariable.notifyAll();
</span><span class="cx">
</span><del>- m_heap.m_markingConditionVariable.wait(
- lock,
</del><ins>+ m_heap.m_markingConditionVariable.waitUntil(
+ lock, timeout,
</ins><span class="cx"> [this] {
</span><del>- return !m_heap.m_sharedMarkStack.isEmpty()
</del><ins>+ return !m_heap.m_sharedCollectorMarkStack->isEmpty()
+ || !m_heap.m_sharedMutatorMarkStack->isEmpty()
</ins><span class="cx"> || m_heap.m_parallelMarkersShouldExit;
</span><span class="cx"> });
</span><span class="cx">
</span><span class="cx"> // Is the current phase done? If so, return from this function.
</span><span class="cx"> if (m_heap.m_parallelMarkersShouldExit)
</span><del>- return;
</del><ins>+ return SharedDrainResult::Done;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- m_stack.stealSomeCellsFrom(
- m_heap.m_sharedMarkStack, m_heap.m_numberOfWaitingParallelMarkers);
</del><ins>+ m_collectorStack.stealSomeCellsFrom(
+ *m_heap.m_sharedCollectorMarkStack, m_heap.m_numberOfWaitingParallelMarkers);
+ m_mutatorStack.stealSomeCellsFrom(
+ *m_heap.m_sharedMutatorMarkStack, m_heap.m_numberOfWaitingParallelMarkers);
</ins><span class="cx"> m_heap.m_numberOfActiveParallelMarkers++;
</span><span class="cx"> m_heap.m_numberOfWaitingParallelMarkers--;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- drain();
</del><ins>+ drain(timeout);
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -453,7 +489,6 @@
</span><span class="cx"> bool SlotVisitor::containsOpaqueRoot(void* root) const
</span><span class="cx"> {
</span><span class="cx"> ASSERT(!m_isInParallelMode);
</span><del>- ASSERT(m_opaqueRoots.isEmpty());
</del><span class="cx"> return m_heap.m_opaqueRoots.contains(root);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -467,13 +502,6 @@
</span><span class="cx"> return MixedTriState;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-int SlotVisitor::opaqueRootCount()
-{
- ASSERT(!m_isInParallelMode);
- ASSERT(m_opaqueRoots.isEmpty());
- return m_heap.m_opaqueRoots.size();
-}
-
</del><span class="cx"> void SlotVisitor::mergeOpaqueRootsIfNecessary()
</span><span class="cx"> {
</span><span class="cx"> if (m_opaqueRoots.isEmpty())
</span><span class="lines">@@ -497,15 +525,14 @@
</span><span class="cx"> donateKnownParallel();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::donateAndDrain()
</del><ins>+void SlotVisitor::donateAndDrain(MonotonicTime timeout)
</ins><span class="cx"> {
</span><span class="cx"> donate();
</span><del>- drain();
</del><ins>+ drain(timeout);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::mergeOpaqueRoots()
</span><span class="cx"> {
</span><del>- ASSERT(!m_opaqueRoots.isEmpty()); // Should only be called when opaque roots are non-empty.
</del><span class="cx"> {
</span><span class="cx"> std::lock_guard<Lock> lock(m_heap.m_opaqueRootsMutex);
</span><span class="cx"> for (auto* root : m_opaqueRoots)
</span><span class="lines">@@ -526,10 +553,9 @@
</span><span class="cx"> m_heap.m_unconditionalFinalizers.removeNext()->finalizeUnconditionally();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void SlotVisitor::dump(PrintStream&) const
</del><ins>+void SlotVisitor::dump(PrintStream& out) const
</ins><span class="cx"> {
</span><del>- for (const JSCell* cell : markStack())
- dataLog(*cell, "\n");
</del><ins>+ out.print("Collector: [", pointerListDump(collectorMarkStack()), "], Mutator: [", pointerListDump(mutatorMarkStack()), "]");
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -29,6 +29,7 @@
</span><span class="cx"> #include "HandleTypes.h"
</span><span class="cx"> #include "MarkStack.h"
</span><span class="cx"> #include "OpaqueRootSet.h"
</span><ins>+#include <wtf/MonotonicTime.h>
</ins><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="cx">
</span><span class="lines">@@ -57,9 +58,11 @@
</span><span class="cx"> SlotVisitor(Heap&);
</span><span class="cx"> ~SlotVisitor();
</span><span class="cx">
</span><del>- MarkStackArray& markStack() { return m_stack; }
- const MarkStackArray& markStack() const { return m_stack; }
-
</del><ins>+ MarkStackArray& collectorMarkStack() { return m_collectorStack; }
+ MarkStackArray& mutatorMarkStack() { return m_mutatorStack; }
+ const MarkStackArray& collectorMarkStack() const { return m_collectorStack; }
+ const MarkStackArray& mutatorMarkStack() const { return m_mutatorStack; }
+
</ins><span class="cx"> VM& vm();
</span><span class="cx"> const VM& vm() const;
</span><span class="cx"> Heap* heap() const;
</span><span class="lines">@@ -84,24 +87,23 @@
</span><span class="cx"> JS_EXPORT_PRIVATE void addOpaqueRoot(void*);
</span><span class="cx"> JS_EXPORT_PRIVATE bool containsOpaqueRoot(void*) const;
</span><span class="cx"> TriState containsOpaqueRootTriState(void*) const;
</span><del>- int opaqueRootCount();
</del><span class="cx">
</span><del>- bool isEmpty() { return m_stack.isEmpty(); }
</del><ins>+ bool isEmpty() { return m_collectorStack.isEmpty() && m_mutatorStack.isEmpty(); }
</ins><span class="cx">
</span><span class="cx"> void didStartMarking();
</span><span class="cx"> void reset();
</span><del>- void clearMarkStack();
</del><ins>+ void clearMarkStacks();
</ins><span class="cx">
</span><span class="cx"> size_t bytesVisited() const { return m_bytesVisited; }
</span><del>- size_t bytesCopied() const { return m_bytesCopied; }
</del><span class="cx"> size_t visitCount() const { return m_visitCount; }
</span><span class="cx">
</span><span class="cx"> void donate();
</span><del>- void drain();
- void donateAndDrain();
</del><ins>+ void drain(MonotonicTime timeout = MonotonicTime::infinity());
+ void donateAndDrain(MonotonicTime timeout = MonotonicTime::infinity());
</ins><span class="cx">
</span><span class="cx"> enum SharedDrainMode { SlaveDrain, MasterDrain };
</span><del>- void drainFromShared(SharedDrainMode);
</del><ins>+ enum class SharedDrainResult { Done, TimedOut };
+ SharedDrainResult drainFromShared(SharedDrainMode, MonotonicTime timeout = MonotonicTime::infinity());
</ins><span class="cx">
</span><span class="cx"> void harvestWeakReferences();
</span><span class="cx"> void finalizeUnconditionalFinalizers();
</span><span class="lines">@@ -124,6 +126,8 @@
</span><span class="cx">
</span><span class="cx"> HeapVersion markingVersion() const { return m_markingVersion; }
</span><span class="cx">
</span><ins>+ void mergeOpaqueRootsIfNecessary();
+
</ins><span class="cx"> private:
</span><span class="cx"> friend class ParallelModeEnabler;
</span><span class="cx">
</span><span class="lines">@@ -141,21 +145,23 @@
</span><span class="cx"> template<typename ContainerType>
</span><span class="cx"> void appendToMarkStack(ContainerType&, JSCell*);
</span><span class="cx">
</span><ins>+ void appendToMutatorMarkStack(const JSCell*);
+
</ins><span class="cx"> void noteLiveAuxiliaryCell(HeapCell*);
</span><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE void mergeOpaqueRoots();
</span><del>- void mergeOpaqueRootsIfNecessary();
</del><span class="cx"> void mergeOpaqueRootsIfProfitable();
</span><span class="cx">
</span><span class="cx"> void visitChildren(const JSCell*);
</span><span class="cx">
</span><span class="cx"> void donateKnownParallel();
</span><ins>+ void donateKnownParallel(MarkStackArray& from, MarkStackArray& to);
</ins><span class="cx">
</span><del>- MarkStackArray m_stack;
</del><ins>+ MarkStackArray m_collectorStack;
+ MarkStackArray m_mutatorStack;
</ins><span class="cx"> OpaqueRootSet m_opaqueRoots; // Handle-owning data structures not visible to the garbage collector.
</span><span class="cx">
</span><span class="cx"> size_t m_bytesVisited;
</span><del>- size_t m_bytesCopied;
</del><span class="cx"> size_t m_visitCount;
</span><span class="cx"> bool m_isInParallelMode;
</span><span class="cx">
</span><span class="lines">@@ -165,7 +171,7 @@
</span><span class="cx">
</span><span class="cx"> HeapSnapshotBuilder* m_heapSnapshotBuilder { nullptr };
</span><span class="cx"> JSCell* m_currentCell { nullptr };
</span><del>- CellState m_oldCellState;
</del><ins>+ bool m_isVisitingMutatorStack { false };
</ins><span class="cx">
</span><span class="cx"> public:
</span><span class="cx"> #if !ASSERT_DISABLED
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -105,13 +105,15 @@
</span><span class="cx">
</span><span class="cx"> inline void SlotVisitor::reportExtraMemoryVisited(size_t size)
</span><span class="cx"> {
</span><del>- heap()->reportExtraMemoryVisited(m_oldCellState, size);
</del><ins>+ if (!m_isVisitingMutatorStack)
+ heap()->reportExtraMemoryVisited(size);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #if ENABLE(RESOURCE_USAGE)
</span><span class="cx"> inline void SlotVisitor::reportExternalMemoryVisited(size_t size)
</span><span class="cx"> {
</span><del>- heap()->reportExternalMemoryVisited(m_oldCellState, size);
</del><ins>+ if (!m_isVisitingMutatorStack)
+ heap()->reportExternalMemoryVisited(size);
</ins><span class="cx"> }
</span><span class="cx"> #endif
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitAssemblyHelperscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/jit/AssemblyHelpers.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -401,7 +401,7 @@
</span><span class="cx"> jit.abortWithReason(AHStructureIDIsValid);
</span><span class="cx"> correctStructure.link(&jit);
</span><span class="cx">
</span><del>- Jump correctIndexingType = jit.branch8(Equal, MacroAssembler::Address(dest, JSCell::indexingTypeOffset()), TrustedImm32(structurePtr->indexingType()));
</del><ins>+ Jump correctIndexingType = jit.branch8(Equal, MacroAssembler::Address(dest, JSCell::indexingTypeAndMiscOffset()), TrustedImm32(structurePtr->indexingTypeIncludingHistory()));
</ins><span class="cx"> jit.abortWithReason(AHIndexingTypeIsValid);
</span><span class="cx"> correctIndexingType.link(&jit);
</span><span class="cx">
</span><span class="lines">@@ -415,7 +415,7 @@
</span><span class="cx"> }
</span><span class="cx"> #else
</span><span class="cx"> // Do a 32-bit wide store to initialize the cell's fields.
</span><del>- jit.store32(TrustedImm32(structurePtr->objectInitializationBlob()), MacroAssembler::Address(dest, JSCell::indexingTypeOffset()));
</del><ins>+ jit.store32(TrustedImm32(structurePtr->objectInitializationBlob()), MacroAssembler::Address(dest, JSCell::indexingTypeAndMiscOffset()));
</ins><span class="cx"> jit.storePtr(structure, MacroAssembler::Address(dest, JSCell::structureIDOffset()));
</span><span class="cx"> #endif
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitAssemblyHelpersh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/jit/AssemblyHelpers.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1318,8 +1318,8 @@
</span><span class="cx"> store64(scratch, MacroAssembler::Address(dest, JSCell::structureIDOffset()));
</span><span class="cx"> #else
</span><span class="cx"> // Store all the info flags using a single 32-bit wide load and store.
</span><del>- load32(MacroAssembler::Address(structure, Structure::indexingTypeOffset()), scratch);
- store32(scratch, MacroAssembler::Address(dest, JSCell::indexingTypeOffset()));
</del><ins>+ load32(MacroAssembler::Address(structure, Structure::indexingTypeIncludingHistoryOffset()), scratch);
+ store32(scratch, MacroAssembler::Address(dest, JSCell::indexingTypeAndMiscOffset()));
</ins><span class="cx">
</span><span class="cx"> // Store the StructureID
</span><span class="cx"> storePtr(structure, MacroAssembler::Address(dest, JSCell::structureIDOffset()));
</span><span class="lines">@@ -1356,16 +1356,42 @@
</span><span class="cx"> {
</span><span class="cx"> if (!Options::useConcurrentBarriers())
</span><span class="cx"> return;
</span><del>- Jump ok = jumpIfBarrierStoreLoadFenceNotNeeded();
</del><ins>+ Jump ok = jumpIfMutatorFenceNotNeeded();
</ins><span class="cx"> memoryFence();
</span><span class="cx"> ok.link(this);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- Jump jumpIfBarrierStoreLoadFenceNotNeeded()
</del><ins>+ void mutatorFence()
</ins><span class="cx"> {
</span><del>- return branchTest8(Zero, AbsoluteAddress(vm()->heap.addressOfBarrierShouldBeFenced()));
</del><ins>+ if (isX86())
+ return;
+ Jump ok = jumpIfMutatorFenceNotNeeded();
+ storeFence();
+ ok.link(this);
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+ void storeButterfly(GPRReg butterfly, GPRReg object)
+ {
+ if (isX86()) {
+ storePtr(butterfly, Address(object, JSObject::butterflyOffset()));
+ return;
+ }
+
+ Jump ok = jumpIfMutatorFenceNotNeeded();
+ storeFence();
+ storePtr(butterfly, Address(object, JSObject::butterflyOffset()));
+ storeFence();
+ Jump done = jump();
+ ok.link(this);
+ storePtr(butterfly, Address(object, JSObject::butterflyOffset()));
+ done.link(this);
+ }
+
+ Jump jumpIfMutatorFenceNotNeeded()
+ {
+ return branchTest8(Zero, AbsoluteAddress(vm()->heap.addressOfMutatorShouldBeFenced()));
+ }
+
</ins><span class="cx"> // Emits the branch structure for typeof. The code emitted by this doesn't fall through. The
</span><span class="cx"> // functor is called at those points where we have pinpointed a type. One way to use this is to
</span><span class="cx"> // have the functor emit the code to put the type string into an appropriate register and then
</span><span class="lines">@@ -1589,7 +1615,29 @@
</span><span class="cx"> emitAllocateJSObject<ClassType>(resultGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR1, scratchGPR2, slowPath);
</span><span class="cx"> storePtr(TrustedImmPtr(structure->classInfo()), Address(resultGPR, JSDestructibleObject::classInfoOffset()));
</span><span class="cx"> }
</span><ins>+
+ void emitInitializeInlineStorage(GPRReg baseGPR, unsigned inlineCapacity)
+ {
+ for (unsigned i = 0; i < inlineCapacity; ++i)
+ storeTrustedValue(JSValue(), Address(baseGPR, JSObject::offsetOfInlineStorage() + i * sizeof(EncodedJSValue)));
+ }
</ins><span class="cx">
</span><ins>+ void emitInitializeInlineStorage(GPRReg baseGPR, GPRReg inlineCapacity)
+ {
+ Jump empty = branchTest32(Zero, inlineCapacity);
+ Label loop = label();
+ sub32(TrustedImm32(1), inlineCapacity);
+ storeTrustedValue(JSValue(), BaseIndex(baseGPR, inlineCapacity, TimesEight, JSObject::offsetOfInlineStorage()));
+ branchTest32(NonZero, inlineCapacity).linkTo(loop, this);
+ empty.link(this);
+ }
+
+ void emitInitializeOutOfLineStorage(GPRReg butterflyGPR, unsigned outOfLineCapacity)
+ {
+ for (unsigned i = 0; i < outOfLineCapacity; ++i)
+ storeTrustedValue(JSValue(), Address(butterflyGPR, -sizeof(IndexingHeader) - (i + 1) * sizeof(EncodedJSValue)));
+ }
+
</ins><span class="cx"> #if USE(JSVALUE64)
</span><span class="cx"> void wangsInt64Hash(GPRReg inputAndResult, GPRReg scratch);
</span><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/jit/JITInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -996,7 +996,7 @@
</span><span class="cx"> store32(indexingType, arrayProfile->addressOfLastSeenStructureID());
</span><span class="cx"> }
</span><span class="cx">
</span><del>- load8(Address(cell, JSCell::indexingTypeOffset()), indexingType);
</del><ins>+ load8(Address(cell, JSCell::indexingTypeAndMiscOffset()), indexingType);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void JIT::emitArrayProfilingSiteForBytecodeIndexWithCell(RegisterID cell, RegisterID indexingType, unsigned bytecodeIndex)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITOperations.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITOperations.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/jit/JITOperations.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -2201,8 +2201,10 @@
</span><span class="cx"> NativeCallFrameTracer tracer(&vm, exec);
</span><span class="cx">
</span><span class="cx"> ASSERT(!object->structure()->outOfLineCapacity());
</span><del>- Butterfly* result = object->growOutOfLineStorage(vm, 0, initialOutOfLineCapacity);
- object->setButterflyWithoutChangingStructure(vm, result);
</del><ins>+ Butterfly* result = object->allocateMoreOutOfLineStorage(vm, 0, initialOutOfLineCapacity);
+ WTF::storeStoreFence();
+ object->setButterfly(vm, result);
+ WTF::storeStoreFence();
</ins><span class="cx"> return reinterpret_cast<char*>(result);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -2211,8 +2213,10 @@
</span><span class="cx"> VM& vm = exec->vm();
</span><span class="cx"> NativeCallFrameTracer tracer(&vm, exec);
</span><span class="cx">
</span><del>- Butterfly* result = object->growOutOfLineStorage(vm, object->structure()->outOfLineCapacity(), newSize);
- object->setButterflyWithoutChangingStructure(vm, result);
</del><ins>+ Butterfly* result = object->allocateMoreOutOfLineStorage(vm, object->structure()->outOfLineCapacity(), newSize);
+ WTF::storeStoreFence();
+ object->setButterfly(vm, result);
+ WTF::storeStoreFence();
</ins><span class="cx"> return reinterpret_cast<char*>(result);
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITPropertyAccesscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/jit/JITPropertyAccess.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1008,7 +1008,6 @@
</span><span class="cx"> switch (resolveType) {
</span><span class="cx"> case GlobalProperty:
</span><span class="cx"> case GlobalPropertyWithVarInjectionChecks: {
</span><del>- emitWriteBarrier(m_codeBlock->globalObject(), value, ShouldFilterValue);
</del><span class="cx"> emitLoadWithStructureCheck(scope, structureSlot); // Structure check covers var injection.
</span><span class="cx"> emitGetVirtualRegister(value, regT2);
</span><span class="cx">
</span><span class="lines">@@ -1016,6 +1015,7 @@
</span><span class="cx"> loadPtr(operandSlot, regT1);
</span><span class="cx"> negPtr(regT1);
</span><span class="cx"> storePtr(regT2, BaseIndex(regT0, regT1, TimesEight, (firstOutOfLineOffset - 2) * sizeof(EncodedJSValue)));
</span><ins>+ emitWriteBarrier(m_codeBlock->globalObject(), value, ShouldFilterValue);
</ins><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> case GlobalVar:
</span><span class="lines">@@ -1024,7 +1024,6 @@
</span><span class="cx"> case GlobalLexicalVarWithVarInjectionChecks: {
</span><span class="cx"> JSScope* constantScope = JSScope::constantScopeForCodeBlock(resolveType, m_codeBlock);
</span><span class="cx"> RELEASE_ASSERT(constantScope);
</span><del>- emitWriteBarrier(constantScope, value, ShouldFilterValue);
</del><span class="cx"> emitVarInjectionCheck(needsVarInjectionChecks(resolveType));
</span><span class="cx"> if (!isInitialization(getPutInfo.initializationMode()) && (resolveType == GlobalLexicalVar || resolveType == GlobalLexicalVarWithVarInjectionChecks)) {
</span><span class="cx"> // We need to do a TDZ check here because we can't always prove we need to emit TDZ checks statically.
</span><span class="lines">@@ -1038,14 +1037,15 @@
</span><span class="cx"> emitPutGlobalVariableIndirect(bitwise_cast<JSValue**>(operandSlot), value, bitwise_cast<WatchpointSet**>(&currentInstruction[5]));
</span><span class="cx"> else
</span><span class="cx"> emitPutGlobalVariable(bitwise_cast<JSValue*>(*operandSlot), value, currentInstruction[5].u.watchpointSet);
</span><ins>+ emitWriteBarrier(constantScope, value, ShouldFilterValue);
</ins><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> case LocalClosureVar:
</span><span class="cx"> case ClosureVar:
</span><span class="cx"> case ClosureVarWithVarInjectionChecks:
</span><del>- emitWriteBarrier(scope, value, ShouldFilterValue);
</del><span class="cx"> emitVarInjectionCheck(needsVarInjectionChecks(resolveType));
</span><span class="cx"> emitPutClosureVar(scope, *operandSlot, value, currentInstruction[5].u.watchpointSet);
</span><ins>+ emitWriteBarrier(scope, value, ShouldFilterValue);
</ins><span class="cx"> break;
</span><span class="cx"> case ModuleVar:
</span><span class="cx"> case Dynamic:
</span><span class="lines">@@ -1157,11 +1157,11 @@
</span><span class="cx"> int index = currentInstruction[2].u.operand;
</span><span class="cx"> int value = currentInstruction[3].u.operand;
</span><span class="cx">
</span><del>- emitWriteBarrier(arguments, value, ShouldFilterValue);
-
</del><span class="cx"> emitGetVirtualRegister(arguments, regT0);
</span><span class="cx"> emitGetVirtualRegister(value, regT1);
</span><span class="cx"> store64(regT1, Address(regT0, DirectArguments::storageOffset() + index * sizeof(WriteBarrier<Unknown>)));
</span><ins>+
+ emitWriteBarrier(arguments, value, ShouldFilterValue);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #endif // USE(JSVALUE64)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorellintLLIntDatacpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/llint/LLIntData.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/llint/LLIntData.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/llint/LLIntData.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -178,6 +178,15 @@
</span><span class="cx"> STATIC_ASSERT(EvalCode == 1);
</span><span class="cx"> STATIC_ASSERT(FunctionCode == 2);
</span><span class="cx"> STATIC_ASSERT(ModuleCode == 3);
</span><ins>+
+ STATIC_ASSERT(IsArray == 0x01);
+ STATIC_ASSERT(IndexingShapeMask == 0x0E);
+ STATIC_ASSERT(NoIndexingShape == 0x00);
+ STATIC_ASSERT(Int32Shape == 0x04);
+ STATIC_ASSERT(DoubleShape == 0x06);
+ STATIC_ASSERT(ContiguousShape == 0x08);
+ STATIC_ASSERT(ArrayStorageShape == 0x0A);
+ STATIC_ASSERT(SlowPutArrayStorageShape == 0x0C);
</ins><span class="cx">
</span><span class="cx"> ASSERT(!(reinterpret_cast<ptrdiff_t>((reinterpret_cast<WriteBarrier<JSCell>*>(0x4000)->slot())) - 0x4000));
</span><span class="cx"> static_assert(PutByIdPrimaryTypeMask == 0x6, "LLInt assumes PutByIdPrimaryTypeMask is == 0x6");
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorellintLowLevelInterpreterasm"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/llint/LowLevelInterpreter.asm 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -333,14 +333,14 @@
</span><span class="cx"> end
</span><span class="cx">
</span><span class="cx"> # Constant for reasoning about butterflies.
</span><del>-const IsArray = 1
-const IndexingShapeMask = 30
-const NoIndexingShape = 0
-const Int32Shape = 20
-const DoubleShape = 22
-const ContiguousShape = 26
-const ArrayStorageShape = 28
-const SlowPutArrayStorageShape = 30
</del><ins>+const IsArray = 0x01
+const IndexingShapeMask = 0x0E
+const NoIndexingShape = 0x00
+const Int32Shape = 0x04
+const DoubleShape = 0x06
+const ContiguousShape = 0x08
+const ArrayStorageShape = 0x0A
+const SlowPutArrayStorageShape = 0x0C
</ins><span class="cx">
</span><span class="cx"> # Type constants.
</span><span class="cx"> const StringType = 6
</span><span class="lines">@@ -887,7 +887,7 @@
</span><span class="cx"> const indexingType = cellAndIndexingType
</span><span class="cx"> loadi JSCell::m_structureID[cell], scratch
</span><span class="cx"> storei scratch, ArrayProfile::m_lastSeenStructureID[profile]
</span><del>- loadb JSCell::m_indexingType[cell], indexingType
</del><ins>+ loadb JSCell::m_indexingTypeAndMisc[cell], indexingType
</ins><span class="cx"> end
</span><span class="cx">
</span><span class="cx"> macro skipIfIsRememberedOrInEden(cell, slowPath)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorellintLowLevelInterpreter64asm"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1505,6 +1505,7 @@
</span><span class="cx"> .opGetByValOutOfBounds:
</span><span class="cx"> loadpFromInstruction(4, t0)
</span><span class="cx"> storeb 1, ArrayProfile::m_outOfBounds[t0]
</span><ins>+ jmp .opGetByValSlow
</ins><span class="cx">
</span><span class="cx"> .opGetByValNotIndexedStorage:
</span><span class="cx"> # First lets check if we even have a typed array. This lets us do some boilerplate up front.
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeButterflyInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/ButterflyInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -74,6 +74,7 @@
</span><span class="cx"> indexingPayloadSizeInBytes);
</span><span class="cx"> if (hasIndexingHeader)
</span><span class="cx"> *result->indexingHeader() = indexingHeader;
</span><ins>+ memset(result->propertyStorage() - propertyCapacity, 0, propertyCapacity * sizeof(EncodedJSValue));
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -105,6 +106,10 @@
</span><span class="cx"> result->propertyStorage() - oldPropertyCapacity,
</span><span class="cx"> oldButterfly->propertyStorage() - oldPropertyCapacity,
</span><span class="cx"> totalSize(0, oldPropertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes));
</span><ins>+ memset(
+ result->propertyStorage() - newPropertyCapacity,
+ 0,
+ (newPropertyCapacity - oldPropertyCapacity) * sizeof(EncodedJSValue));
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeConcurrentJITLockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/ConcurrentJITLock.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/ConcurrentJITLock.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/ConcurrentJITLock.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -94,15 +94,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private:
</span><del>-#if ENABLE(CONCURRENT_JIT)
</del><span class="cx"> DeferGC m_deferGC;
</span><del>-#else
- struct NoDefer {
- NoDefer(Heap& heap) : m_heap(heap) { }
- Heap& m_heap;
- };
- NoDefer m_deferGC;
-#endif
</del><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> class ConcurrentJITLocker : public ConcurrentJITLockerBase {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeGenericArgumentsInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/GenericArgumentsInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -66,11 +66,13 @@
</span><span class="cx"> Type* thisObject = jsCast<Type*>(object);
</span><span class="cx">
</span><span class="cx"> if (thisObject->canAccessIndexQuickly(index)) {
</span><del>- slot.setValue(thisObject, None, thisObject->getIndexQuickly(index));
</del><ins>+ JSValue result = thisObject->getIndexQuickly(index);
+ slot.setValue(thisObject, None, result);
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- return Base::getOwnPropertySlotByIndex(object, exec, index, slot);
</del><ins>+ bool result = Base::getOwnPropertySlotByIndex(object, exec, index, slot);
+ return result;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename Type>
</span><span class="lines">@@ -129,7 +131,7 @@
</span><span class="cx"> {
</span><span class="cx"> Type* thisObject = jsCast<Type*>(cell);
</span><span class="cx"> VM& vm = exec->vm();
</span><del>-
</del><ins>+
</ins><span class="cx"> if (thisObject->canAccessIndexQuickly(index)) {
</span><span class="cx"> thisObject->setIndexQuickly(vm, index, value);
</span><span class="cx"> return true;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeIndexingTypeh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/IndexingType.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/IndexingType.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/IndexingType.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -26,6 +26,7 @@
</span><span class="cx"> #pragma once
</span><span class="cx">
</span><span class="cx"> #include "SpeculatedType.h"
</span><ins>+#include <wtf/LockAlgorithm.h>
</ins><span class="cx"> #include <wtf/StdLibExtras.h>
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -52,22 +53,26 @@
</span><span class="cx"> static const IndexingType IsArray = 0x01;
</span><span class="cx">
</span><span class="cx"> // The shape of the indexed property storage.
</span><del>-static const IndexingType IndexingShapeMask = 0x0E;
-static const IndexingType NoIndexingShape = 0x00;
-static const IndexingType UndecidedShape = 0x02; // Only useful for arrays.
-static const IndexingType Int32Shape = 0x04;
-static const IndexingType DoubleShape = 0x06;
-static const IndexingType ContiguousShape = 0x08;
-static const IndexingType ArrayStorageShape = 0x0A;
-static const IndexingType SlowPutArrayStorageShape = 0x0C;
</del><ins>+static const IndexingType IndexingShapeMask = 0x0E;
+static const IndexingType NoIndexingShape = 0x00;
+static const IndexingType UndecidedShape = 0x02; // Only useful for arrays.
+static const IndexingType Int32Shape = 0x04;
+static const IndexingType DoubleShape = 0x06;
+static const IndexingType ContiguousShape = 0x08;
+static const IndexingType ArrayStorageShape = 0x0A;
+static const IndexingType SlowPutArrayStorageShape = 0x0C;
</ins><span class="cx">
</span><del>-static const IndexingType IndexingShapeShift = 1;
-static const IndexingType NumberOfIndexingShapes = 7;
</del><ins>+static const IndexingType IndexingShapeShift = 1;
+static const IndexingType NumberOfIndexingShapes = 7;
</ins><span class="cx">
</span><span class="cx"> // Additional flags for tracking the history of the type. These are usually
</span><span class="cx"> // masked off unless you ask for them directly.
</span><del>-static const IndexingType MayHaveIndexedAccessors = 0x10;
</del><ins>+static const IndexingType MayHaveIndexedAccessors = 0x10;
</ins><span class="cx">
</span><ins>+// The IndexingType field of JSCells is stolen for locks.
+static const IndexingType IndexingTypeLockIsHeld = 0x20;
+static const IndexingType IndexingTypeLockHasParked = 0x40;
+
</ins><span class="cx"> // List of acceptable array types.
</span><span class="cx"> static const IndexingType NonArray = 0x0;
</span><span class="cx"> static const IndexingType NonArrayWithInt32 = Int32Shape;
</span><span class="lines">@@ -177,4 +182,6 @@
</span><span class="cx"> // Mask of all possible types including the history.
</span><span class="cx"> static const IndexingType AllArrayTypesAndHistory = AllArrayTypes | MayHaveIndexedAccessors;
</span><span class="cx">
</span><ins>+typedef LockAlgorithm<IndexingType, IndexingTypeLockIsHeld, IndexingTypeLockHasParked> IndexingTypeLockAlgorithm;
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSArraycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSArray.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSArray.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSArray.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -390,7 +390,9 @@
</span><span class="cx"> newButterfly->arrayStorage()->setVectorLength(newVectorLength);
</span><span class="cx"> newButterfly->arrayStorage()->m_indexBias = newIndexBias;
</span><span class="cx">
</span><del>- setButterflyWithoutChangingStructure(vm, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+
+ setButterfly(vm, newButterfly);
</ins><span class="cx">
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="lines">@@ -1063,6 +1065,7 @@
</span><span class="cx"> // Need to have GC deferred around the unshiftCountSlowCase(), since that leaves the butterfly in
</span><span class="cx"> // a weird state: some parts of it will be left uninitialized, which we will fill in here.
</span><span class="cx"> DeferGC deferGC(vm.heap);
</span><ins>+ JSCell::InternalLocker locker(this);
</ins><span class="cx">
</span><span class="cx"> if (moveFront && storage->m_indexBias >= count) {
</span><span class="cx"> Butterfly* newButterfly = storage->butterfly()->unshift(structure(), count);
</span><span class="lines">@@ -1069,7 +1072,8 @@
</span><span class="cx"> storage = newButterfly->arrayStorage();
</span><span class="cx"> storage->m_indexBias -= count;
</span><span class="cx"> storage->setVectorLength(vectorLength + count);
</span><del>- setButterflyWithoutChangingStructure(vm, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
</ins><span class="cx"> } else if (!moveFront && vectorLength - length >= count)
</span><span class="cx"> storage = storage->butterfly()->arrayStorage();
</span><span class="cx"> else if (unshiftCountSlowCase(vm, deferGC, moveFront, count))
</span><span class="lines">@@ -1090,6 +1094,7 @@
</span><span class="cx">
</span><span class="cx"> for (unsigned i = 0; i < count; i++)
</span><span class="cx"> vector[i + startIndex].clear();
</span><ins>+
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSCellh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSCell.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSCell.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSCell.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -100,8 +100,32 @@
</span><span class="cx"> bool isProxy() const;
</span><span class="cx"> bool inherits(const ClassInfo*) const;
</span><span class="cx"> bool isAPIValueWrapper() const;
</span><ins>+
+ // Each cell has a built-in lock. Currently it's simply available for use if you need it. It's
+ // a full-blown WTF::Lock. Note that this lock is currently used in JSArray and that lock's
+ // ordering with the Structure lock is that the Structure lock must be acquired first.
+ void lockInternalLock();
+ void unlockInternalLock();
+
+ class InternalLocker {
+ public:
+ InternalLocker(JSCell* cell)
+ : m_cell(cell)
+ {
+ m_cell->lockInternalLock();
+ }
+
+ ~InternalLocker()
+ {
+ m_cell->unlockInternalLock();
+ }
+
+ private:
+ JSCell* m_cell;
+ };
</ins><span class="cx">
</span><span class="cx"> JSType type() const;
</span><ins>+ IndexingType indexingTypeAndMisc() const;
</ins><span class="cx"> IndexingType indexingType() const;
</span><span class="cx"> StructureID structureID() const { return m_structureID; }
</span><span class="cx"> Structure* structure() const;
</span><span class="lines">@@ -167,7 +191,17 @@
</span><span class="cx"> CellState cellState() const { return m_cellState; }
</span><span class="cx">
</span><span class="cx"> void setCellState(CellState data) const { const_cast<JSCell*>(this)->m_cellState = data; }
</span><ins>+
+ bool atomicCompareExchangeCellStateWeakRelaxed(CellState oldState, CellState newState)
+ {
+ return WTF::atomicCompareExchangeWeakRelaxed(&m_cellState, oldState, newState);
+ }
</ins><span class="cx">
</span><ins>+ CellState atomicCompareExchangeCellStateStrong(CellState oldState, CellState newState)
+ {
+ return WTF::atomicCompareExchangeStrong(&m_cellState, oldState, newState);
+ }
+
</ins><span class="cx"> static ptrdiff_t structureIDOffset()
</span><span class="cx"> {
</span><span class="cx"> return OBJECT_OFFSETOF(JSCell, m_structureID);
</span><span class="lines">@@ -183,9 +217,12 @@
</span><span class="cx"> return OBJECT_OFFSETOF(JSCell, m_type);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- static ptrdiff_t indexingTypeOffset()
</del><ins>+ // DO NOT store to this field. Always use a CAS loop, since some bits are flipped using CAS
+ // from other threads due to the internal lock. One exception: you don't need the CAS if the
+ // object has not escaped yet.
+ static ptrdiff_t indexingTypeAndMiscOffset()
</ins><span class="cx"> {
</span><del>- return OBJECT_OFFSETOF(JSCell, m_indexingType);
</del><ins>+ return OBJECT_OFFSETOF(JSCell, m_indexingTypeAndMisc);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> static ptrdiff_t cellStateOffset()
</span><span class="lines">@@ -228,7 +265,7 @@
</span><span class="cx"> friend class LLIntOffsetsExtractor;
</span><span class="cx">
</span><span class="cx"> StructureID m_structureID;
</span><del>- IndexingType m_indexingType;
</del><ins>+ IndexingType m_indexingTypeAndMisc; // DO NOT store to this field. Always CAS.
</ins><span class="cx"> JSType m_type;
</span><span class="cx"> TypeInfo::InlineTypeFlags m_flags;
</span><span class="cx"> CellState m_cellState;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSCellInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSCellInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSCellInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSCellInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2012, 2013, 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2012-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -47,7 +47,7 @@
</span><span class="cx">
</span><span class="cx"> inline JSCell::JSCell(VM&, Structure* structure)
</span><span class="cx"> : m_structureID(structure->id())
</span><del>- , m_indexingType(structure->indexingType())
</del><ins>+ , m_indexingTypeAndMisc(structure->indexingTypeIncludingHistory())
</ins><span class="cx"> , m_type(structure->typeInfo().type())
</span><span class="cx"> , m_flags(structure->typeInfo().inlineTypeFlags())
</span><span class="cx"> , m_cellState(CellState::NewWhite)
</span><span class="lines">@@ -57,6 +57,10 @@
</span><span class="cx">
</span><span class="cx"> inline void JSCell::finishCreation(VM& vm)
</span><span class="cx"> {
</span><ins>+ // This object is ready to be escaped so the concurrent GC may see it at any time. We have
+ // to make sure that none of our stores sink below here.
+ if (isX86() || UNLIKELY(vm.heap.mutatorShouldBeFenced()))
+ WTF::storeStoreFence();
</ins><span class="cx"> #if ENABLE(GC_VALIDATION)
</span><span class="cx"> ASSERT(vm.isInitializingObject());
</span><span class="cx"> vm.setInitializingObjectClass(0);
</span><span class="lines">@@ -74,7 +78,7 @@
</span><span class="cx"> if (structure) {
</span><span class="cx"> #endif
</span><span class="cx"> m_structureID = structure->id();
</span><del>- m_indexingType = structure->indexingType();
</del><ins>+ m_indexingTypeAndMisc = structure->indexingTypeIncludingHistory();
</ins><span class="cx"> m_type = structure->typeInfo().type();
</span><span class="cx"> m_flags = structure->typeInfo().inlineTypeFlags();
</span><span class="cx"> #if ENABLE(GC_VALIDATION)
</span><span class="lines">@@ -91,9 +95,14 @@
</span><span class="cx"> return m_type;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline IndexingType JSCell::indexingTypeAndMisc() const
+{
+ return m_indexingTypeAndMisc;
+}
+
</ins><span class="cx"> inline IndexingType JSCell::indexingType() const
</span><span class="cx"> {
</span><del>- return m_indexingType;
</del><ins>+ return indexingTypeAndMisc() & AllArrayTypes;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline Structure* JSCell::structure() const
</span><span class="lines">@@ -195,17 +204,26 @@
</span><span class="cx"> return m_type == APIValueWrapperType;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void JSCell::setStructure(VM& vm, Structure* structure)
</del><ins>+ALWAYS_INLINE void JSCell::setStructure(VM& vm, Structure* structure)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(structure->classInfo() == this->structure()->classInfo());
</span><span class="cx"> ASSERT(!this->structure()
</span><span class="cx"> || this->structure()->transitionWatchpointSetHasBeenInvalidated()
</span><span class="cx"> || Heap::heap(this)->structureIDTable().get(structure->id()) == structure);
</span><del>- vm.heap.writeBarrier(this, structure);
</del><span class="cx"> m_structureID = structure->id();
</span><span class="cx"> m_flags = structure->typeInfo().inlineTypeFlags();
</span><span class="cx"> m_type = structure->typeInfo().type();
</span><del>- m_indexingType = structure->indexingType();
</del><ins>+ IndexingType newIndexingType = structure->indexingTypeIncludingHistory();
+ if (m_indexingTypeAndMisc != newIndexingType) {
+ ASSERT(!(newIndexingType & ~AllArrayTypesAndHistory));
+ for (;;) {
+ IndexingType oldValue = m_indexingTypeAndMisc;
+ IndexingType newValue = (oldValue & ~AllArrayTypesAndHistory) | structure->indexingTypeIncludingHistory();
+ if (WTF::atomicCompareExchangeWeakRelaxed(&m_indexingTypeAndMisc, oldValue, newValue))
+ break;
+ }
+ }
+ vm.heap.writeBarrier(this, structure);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline const MethodTable* JSCell::methodTable() const
</span><span class="lines">@@ -284,11 +302,26 @@
</span><span class="cx"> if (isZapped())
</span><span class="cx"> return;
</span><span class="cx"> ASSERT(structureID());
</span><del>- if (inlineTypeFlags() & StructureIsImmortal)
- structure(vm)->classInfo()->methodTable.destroy(this);
- else
</del><ins>+ if (inlineTypeFlags() & StructureIsImmortal) {
+ Structure* structure = this->structure(vm);
+ const ClassInfo* classInfo = structure->classInfo();
+ MethodTable::DestroyFunctionPtr destroy = classInfo->methodTable.destroy;
+ destroy(this);
+ } else
</ins><span class="cx"> jsCast<JSDestructibleObject*>(this)->classInfo()->methodTable.destroy(this);
</span><span class="cx"> zap();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline void JSCell::lockInternalLock()
+{
+ Atomic<IndexingType>* lock = bitwise_cast<Atomic<IndexingType>*>(&m_indexingTypeAndMisc);
+ IndexingTypeLockAlgorithm::lock(*lock);
+}
+
+inline void JSCell::unlockInternalLock()
+{
+ Atomic<IndexingType>* lock = bitwise_cast<Atomic<IndexingType>*>(&m_indexingTypeAndMisc);
+ IndexingTypeLockAlgorithm::unlock(*lock);
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSObjectcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSObject.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSObject.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSObject.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -112,17 +112,28 @@
</span><span class="cx">
</span><span class="cx"> // Mark the properties.
</span><span class="cx"> visitor.appendValuesHidden(butterfly->propertyStorage() - storageSize, storageSize);
</span><del>-
- // Mark the array if appropriate.
- switch (this->indexingType()) {
</del><ins>+
+ IndexingType oldType = structure->indexingType();
+ switch (oldType) {
</ins><span class="cx"> case ALL_CONTIGUOUS_INDEXING_TYPES:
</span><del>- visitor.appendValuesHidden(butterfly->contiguous().data(), butterfly->publicLength());
</del><ins>+ case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
+ // This lock is here to protect Contiguous->ArrayStorage transitions, but we could make that
+ // race work if we needed to.
+ JSCell::InternalLocker locker(this);
+ IndexingType newType = this->indexingType();
+ butterfly = this->butterfly();
+ switch (newType) {
+ case ALL_CONTIGUOUS_INDEXING_TYPES:
+ visitor.appendValuesHidden(butterfly->contiguous().data(), butterfly->publicLength());
+ break;
+ default: // ALL_ARRAY_STORAGE_INDEXING_TYPES
+ visitor.appendValuesHidden(butterfly->arrayStorage()->m_vector, butterfly->arrayStorage()->vectorLength());
+ if (butterfly->arrayStorage()->m_sparseMap)
+ visitor.append(&butterfly->arrayStorage()->m_sparseMap);
+ break;
+ }
</ins><span class="cx"> break;
</span><del>- case ALL_ARRAY_STORAGE_INDEXING_TYPES:
- visitor.appendValuesHidden(butterfly->arrayStorage()->m_vector, butterfly->arrayStorage()->vectorLength());
- if (butterfly->arrayStorage()->m_sparseMap)
- visitor.append(&butterfly->arrayStorage()->m_sparseMap);
- break;
</del><ins>+ }
</ins><span class="cx"> default:
</span><span class="cx"> break;
</span><span class="cx"> }
</span><span class="lines">@@ -146,10 +157,34 @@
</span><span class="cx">
</span><span class="cx"> JSCell::visitChildren(thisObject, visitor);
</span><span class="cx">
</span><del>- Butterfly* butterfly = thisObject->m_butterfly.get();
- if (butterfly)
- thisObject->visitButterfly(visitor, butterfly, thisObject->structure(visitor.vm()));
-
</del><ins>+ // We want to defend against multiple possible races with the mutator:
+ // - Mutator could install a bigger butterfly and then install a structure that claims more
+ // properties. This trivially works.
+ // - Mutator could mutate the dictionary structure. Hence, we hold the structure lock for
+ // dictionary structures.
+ // - The butterfly could shrink, but this could only happen if the object was a dictionary.
+ // Usually, this means that we would get the dictionary structure and acquire a lock. That
+ // trivially works. But we could load the structure before the object became a dictionary
+ // and then load the butterfly that was shrunk. We protect against this case by checking
+ // the structure a second time before scanning the butterfly. This means that the butterfly
+ // is at worst the butterfly that the mutator had installed just prior to flipping the
+ // sturcture. So if the structure we see is a non-dictionary then the butterfly cannot yet
+ // have been shrunk.
+ VM& vm = visitor.vm();
+ Structure* structure = thisObject->structure(vm);
+ bool wasDictionary = structure->isDictionary();
+ if (wasDictionary) {
+ structure->lock().lock();
+ } else
+ WTF::loadLoadFence();
+ if (Butterfly* butterfly = thisObject->m_butterfly.get()) {
+ WTF::loadLoadFence();
+ if (thisObject->structureID() == structure->id())
+ thisObject->visitButterfly(visitor, butterfly, structure);
+ }
+ if (wasDictionary)
+ structure->lock().unlock();
+
</ins><span class="cx"> #if !ASSERT_DISABLED
</span><span class="cx"> visitor.m_isCheckingForDefaultMarkViolation = wasCheckingForDefaultMarkViolation;
</span><span class="cx"> #endif
</span><span class="lines">@@ -204,15 +239,25 @@
</span><span class="cx">
</span><span class="cx"> JSCell::visitChildren(thisObject, visitor);
</span><span class="cx">
</span><del>- Structure* structure = thisObject->structure(visitor.vm());
- Butterfly* butterfly = thisObject->butterfly();
- if (butterfly)
- thisObject->visitButterfly(visitor, butterfly, structure);
</del><ins>+ VM& vm = visitor.vm();
+ Structure* structure = thisObject->structure(vm);
+ bool wasDictionary = structure->isDictionary();
+ if (wasDictionary)
+ structure->lock().lock();
+ else
+ WTF::loadLoadFence();
+ if (Butterfly* butterfly = thisObject->butterfly()) {
+ WTF::loadLoadFence();
+ if (structure->id() == thisObject->structureID())
+ thisObject->visitButterfly(visitor, butterfly, structure);
+ }
</ins><span class="cx">
</span><span class="cx"> size_t storageSize = structure->inlineSize();
</span><span class="cx"> if (storageSize)
</span><span class="cx"> visitor.appendValuesHidden(thisObject->inlineStorage(), storageSize);
</span><del>-
</del><ins>+ if (wasDictionary)
+ structure->lock().unlock();
+
</ins><span class="cx"> #if !ASSERT_DISABLED
</span><span class="cx"> visitor.m_isCheckingForDefaultMarkViolation = wasCheckingForDefaultMarkViolation;
</span><span class="cx"> #endif
</span><span class="lines">@@ -666,7 +711,8 @@
</span><span class="cx"> newButterfly->arrayStorage()->m_indexBias = 0;
</span><span class="cx"> newButterfly->arrayStorage()->setVectorLength(0);
</span><span class="cx"> newButterfly->arrayStorage()->m_sparseMap.set(vm, this, map);
</span><del>- setButterflyWithoutChangingStructure(vm, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
</ins><span class="cx">
</span><span class="cx"> return newButterfly->arrayStorage();
</span><span class="cx"> }
</span><span class="lines">@@ -731,7 +777,10 @@
</span><span class="cx"> DeferGC deferGC(vm.heap);
</span><span class="cx"> Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateUndecided);
</span><del>- setStructureAndButterfly(vm, newStructure, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newButterfly;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -742,7 +791,10 @@
</span><span class="cx"> for (unsigned i = newButterfly->vectorLength(); i--;)
</span><span class="cx"> newButterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32);
</span><del>- setStructureAndButterfly(vm, newStructure, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newButterfly->contiguousInt32();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -753,7 +805,10 @@
</span><span class="cx"> for (unsigned i = newButterfly->vectorLength(); i--;)
</span><span class="cx"> newButterfly->contiguousDouble()[i] = PNaN;
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble);
</span><del>- setStructureAndButterfly(vm, newStructure, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newButterfly->contiguousDouble();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -764,7 +819,10 @@
</span><span class="cx"> for (unsigned i = newButterfly->vectorLength(); i--;)
</span><span class="cx"> newButterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous);
</span><del>- setStructureAndButterfly(vm, newStructure, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newButterfly->contiguous();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -797,7 +855,10 @@
</span><span class="cx"> Butterfly* newButterfly = createArrayStorageButterfly(vm, this, structure, length, vectorLength, m_butterfly.get());
</span><span class="cx"> ArrayStorage* result = newButterfly->arrayStorage();
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure, structure->suggestedArrayStorageTransition());
</span><del>- setStructureAndButterfly(vm, newStructure, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -880,7 +941,10 @@
</span><span class="cx"> storage->m_vector[i].setWithoutWriteBarrier(JSValue());
</span><span class="cx">
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
</span><del>- setStructureAndButterfly(vm, newStructure, storage->butterfly());
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, storage->butterfly());
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return storage;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -935,7 +999,10 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
</span><del>- setStructureAndButterfly(vm, newStructure, newStorage->butterfly());
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newStorage->butterfly());
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newStorage;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -981,7 +1048,10 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
</span><del>- setStructureAndButterfly(vm, newStructure, newStorage->butterfly());
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newStorage->butterfly());
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newStorage;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -1006,7 +1076,23 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
</span><del>- setStructureAndButterfly(vm, newStructure, newStorage->butterfly());
</del><ins>+
+ // This has a crazy race with the garbage collector. When changing the butterfly and structure,
+ // the mutator always sets the structure last. The collector will always read the structure
+ // first. We probably have to follow that convention here as well. This means that the collector
+ // will sometimes see the new butterfly (the one with ArrayStorage) with the only structure (the
+ // one that says that the butterfly is Contiguous). When scanning Contiguous, the collector will
+ // mark word at addresses greater than or equal to the butterfly pointer, up to the publicLength
+ // in the butterfly. But an ArrayStorage has some non-pointer header data at low positive
+ // offsets from the butterfly - so when this race happens, the collector will surely crash
+ // because it will fail to decode two consecutive int32s as if it was a JSValue.
+ //
+ // Fortunately, we have the JSCell lock for this purpose!
+
+ InternalLocker locker(this);
+ setButterfly(vm, newStorage->butterfly());
+ WTF::storeStoreFence();
+ setStructure(vm, newStructure);
</ins><span class="cx"> return newStorage;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -1468,7 +1554,7 @@
</span><span class="cx">
</span><span class="cx"> PropertyOffset offset;
</span><span class="cx"> if (structure->isUncacheableDictionary())
</span><del>- offset = structure->removePropertyWithoutTransition(vm, propertyName);
</del><ins>+ offset = structure->removePropertyWithoutTransition(vm, propertyName, [] (const ConcurrentJITLocker&, PropertyOffset) { });
</ins><span class="cx"> else
</span><span class="cx"> thisObject->setStructure(vm, Structure::removePropertyTransition(vm, structure, propertyName, offset));
</span><span class="cx">
</span><span class="lines">@@ -2019,7 +2105,7 @@
</span><span class="cx">
</span><span class="cx"> SparseArrayValueMap* map = m_butterfly.get()->arrayStorage()->m_sparseMap.get();
</span><span class="cx"> RELEASE_ASSERT(map);
</span><del>-
</del><ins>+
</ins><span class="cx"> // 1. Let current be the result of calling the [[GetOwnProperty]] internal method of O with property name P.
</span><span class="cx"> SparseArrayValueMap::AddResult result = map->add(this, index);
</span><span class="cx"> SparseArrayEntry* entryInMap = &result.iterator->value;
</span><span class="lines">@@ -2725,7 +2811,8 @@
</span><span class="cx"> for (unsigned i = vectorLength; i < newVectorLength; ++i)
</span><span class="cx"> newButterfly->arrayStorage()->m_vector[i].clear();
</span><span class="cx"> newButterfly->arrayStorage()->setVectorLength(newVectorLength);
</span><del>- setButterflyWithoutChangingStructure(vm, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -2742,7 +2829,8 @@
</span><span class="cx"> newButterfly->arrayStorage()->m_vector[i].clear();
</span><span class="cx"> newButterfly->arrayStorage()->setVectorLength(newVectorLength);
</span><span class="cx"> newButterfly->arrayStorage()->m_indexBias = newIndexBias;
</span><del>- setButterflyWithoutChangingStructure(vm, newButterfly);
</del><ins>+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -2762,6 +2850,7 @@
</span><span class="cx">
</span><span class="cx"> unsigned availableOldLength =
</span><span class="cx"> Butterfly::availableContiguousVectorLength(propertyCapacity, oldVectorLength);
</span><ins>+ Butterfly* newButterfly = nullptr;
</ins><span class="cx"> if (availableOldLength >= length) {
</span><span class="cx"> // This is the case where someone else selected a vector length that caused internal
</span><span class="cx"> // fragmentation. If we did our jobs right, this would never happen. But I bet we will mess
</span><span class="lines">@@ -2776,19 +2865,26 @@
</span><span class="cx"> newVectorLength * sizeof(EncodedJSValue));
</span><span class="cx"> if (!butterfly)
</span><span class="cx"> return false;
</span><del>- m_butterfly.set(vm, this, butterfly);
</del><ins>+ newButterfly = butterfly;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- butterfly->setVectorLength(newVectorLength);
-
</del><span class="cx"> if (hasDouble(indexingType())) {
</span><span class="cx"> for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
</span><del>- butterfly->contiguousDouble()[i] = PNaN;
</del><ins>+ butterfly->indexingPayload<double>()[i] = PNaN;
</ins><span class="cx"> } else {
</span><span class="cx"> for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
</span><del>- butterfly->contiguous()[i].clear();
</del><ins>+ butterfly->indexingPayload<WriteBarrier<Unknown>>()[i].clear();
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+ if (newButterfly) {
+ butterfly->setVectorLength(newVectorLength);
+ WTF::storeStoreFence();
+ m_butterfly.set(vm, this, newButterfly);
+ } else {
+ WTF::storeStoreFence();
+ butterfly->setVectorLength(newVectorLength);
+ }
+
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -2802,12 +2898,13 @@
</span><span class="cx">
</span><span class="cx"> DeferGC deferGC(vm.heap);
</span><span class="cx"> Butterfly* newButterfly = m_butterfly.get()->resizeArray(vm, this, structure(), 0, ArrayStorage::sizeFor(length));
</span><del>- m_butterfly.set(vm, this, newButterfly);
</del><span class="cx"> newButterfly->setVectorLength(length);
</span><span class="cx"> newButterfly->setPublicLength(length);
</span><ins>+ WTF::storeStoreFence();
+ m_butterfly.set(vm, this, newButterfly);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-Butterfly* JSObject::growOutOfLineStorage(VM& vm, size_t oldSize, size_t newSize)
</del><ins>+Butterfly* JSObject::allocateMoreOutOfLineStorage(VM& vm, size_t oldSize, size_t newSize)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(newSize > oldSize);
</span><span class="cx">
</span><span class="lines">@@ -3092,15 +3189,34 @@
</span><span class="cx"> vm, Structure::toCacheableDictionaryTransition(vm, structure(vm), &deferredWatchpointFire));
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void JSObject::shiftButterflyAfterFlattening(VM& vm, size_t outOfLineCapacityBefore, size_t outOfLineCapacityAfter)
</del><ins>+void JSObject::shiftButterflyAfterFlattening(const GCSafeConcurrentJITLocker&, VM& vm, Structure* structure, size_t outOfLineCapacityAfter)
</ins><span class="cx"> {
</span><del>- Butterfly* butterfly = this->butterfly();
- size_t preCapacity = this->butterflyPreCapacity();
- void* currentBase = butterfly->base(preCapacity, outOfLineCapacityAfter);
- void* newBase = butterfly->base(preCapacity, outOfLineCapacityBefore);
</del><ins>+ // This could interleave visitChildren because some old structure could have been a non
+ // dictionary structure. We have to be crazy careful. But, we are guaranteed to be holding
+ // the structure's lock right now, and that helps a bit.
+
+ Butterfly* oldButterfly = this->butterfly();
+ size_t preCapacity;
+ size_t indexingPayloadSizeInBytes;
+ bool hasIndexingHeader = this->hasIndexingHeader();
+ if (UNLIKELY(hasIndexingHeader)) {
+ preCapacity = oldButterfly->indexingHeader()->preCapacity(structure);
+ indexingPayloadSizeInBytes = oldButterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
+ } else {
+ preCapacity = 0;
+ indexingPayloadSizeInBytes = 0;
+ }
</ins><span class="cx">
</span><del>- memmove(newBase, currentBase, this->butterflyTotalSize());
- setButterflyWithoutChangingStructure(vm, Butterfly::fromBase(newBase, preCapacity, outOfLineCapacityAfter));
</del><ins>+ Butterfly* newButterfly = Butterfly::createUninitialized(vm, this, preCapacity, outOfLineCapacityAfter, hasIndexingHeader, indexingPayloadSizeInBytes);
+
+ // No need to copy the precapacity.
+ void* currentBase = oldButterfly->base(0, outOfLineCapacityAfter);
+ void* newBase = newButterfly->base(0, outOfLineCapacityAfter);
+
+ memcpy(newBase, currentBase, Butterfly::totalSize(0, outOfLineCapacityAfter, hasIndexingHeader, indexingPayloadSizeInBytes));
+
+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> uint32_t JSObject::getEnumerableLength(ExecState* exec, JSObject* object)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSObjecth"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSObject.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSObject.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSObject.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -747,11 +747,10 @@
</span><span class="cx"> bool staticPropertiesReified() { return structure()->staticPropertiesReified(); }
</span><span class="cx"> void reifyAllStaticProperties(ExecState*);
</span><span class="cx">
</span><del>- JS_EXPORT_PRIVATE Butterfly* growOutOfLineStorage(VM&, size_t oldSize, size_t newSize);
- void setButterflyWithoutChangingStructure(VM&, Butterfly*);
</del><ins>+ JS_EXPORT_PRIVATE Butterfly* allocateMoreOutOfLineStorage(VM&, size_t oldSize, size_t newSize);
</ins><span class="cx">
</span><ins>+ void setButterfly(VM&, Butterfly*); // Always change the butterfly before changing the structure!
</ins><span class="cx"> void setStructure(VM&, Structure*);
</span><del>- void setStructureAndButterfly(VM&, Structure*, Butterfly*);
</del><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE void convertToDictionary(VM&);
</span><span class="cx">
</span><span class="lines">@@ -759,7 +758,7 @@
</span><span class="cx"> {
</span><span class="cx"> structure(vm)->flattenDictionaryStructure(vm, this);
</span><span class="cx"> }
</span><del>- void shiftButterflyAfterFlattening(VM&, size_t outOfLineCapacityBefore, size_t outOfLineCapacityAfter);
</del><ins>+ void shiftButterflyAfterFlattening(const GCSafeConcurrentJITLocker&, VM&, Structure* structure, size_t outOfLineCapacityAfter);
</ins><span class="cx">
</span><span class="cx"> JSGlobalObject* globalObject() const
</span><span class="cx"> {
</span><span class="lines">@@ -990,9 +989,9 @@
</span><span class="cx"> JS_EXPORT_PRIVATE NEVER_INLINE bool putInlineSlow(ExecState*, PropertyName, JSValue, PutPropertySlot&);
</span><span class="cx">
</span><span class="cx"> bool getNonIndexPropertySlot(ExecState*, PropertyName, PropertySlot&);
</span><del>- bool getOwnNonIndexPropertySlot(VM&, Structure&, PropertyName, PropertySlot&);
</del><ins>+ bool getOwnNonIndexPropertySlot(VM&, Structure*, PropertyName, PropertySlot&);
</ins><span class="cx"> JS_EXPORT_PRIVATE void fillGetterPropertySlot(PropertySlot&, JSValue, unsigned, PropertyOffset);
</span><del>- void fillCustomGetterPropertySlot(PropertySlot&, JSValue, unsigned, Structure&);
</del><ins>+ void fillCustomGetterPropertySlot(PropertySlot&, JSValue, unsigned, Structure*);
</ins><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE bool getOwnStaticPropertySlot(VM&, PropertyName, PropertySlot&);
</span><span class="cx"> JS_EXPORT_PRIVATE const HashTableValue* findPropertyHashEntry(PropertyName) const;
</span><span class="lines">@@ -1113,6 +1112,7 @@
</span><span class="cx"> explicit JSFinalObject(VM& vm, Structure* structure, Butterfly* butterfly = nullptr)
</span><span class="cx"> : JSObject(vm, structure, butterfly)
</span><span class="cx"> {
</span><ins>+ memset(inlineStorage(), 0, structure->inlineCapacity() * sizeof(EncodedJSValue));
</ins><span class="cx"> }
</span><span class="cx"> };
</span><span class="cx">
</span><span class="lines">@@ -1205,14 +1205,6 @@
</span><span class="cx"> return type() == WithScopeType;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void JSObject::setStructureAndButterfly(VM& vm, Structure* structure, Butterfly* butterfly)
-{
- ASSERT(structure);
- ASSERT(!butterfly == (!structure->outOfLineCapacity() && !structure->hasIndexingHeader(this)));
- m_butterfly.set(vm, this, butterfly);
- setStructure(vm, structure);
-}
-
</del><span class="cx"> inline void JSObject::setStructure(VM& vm, Structure* structure)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(structure);
</span><span class="lines">@@ -1220,7 +1212,7 @@
</span><span class="cx"> JSCell::setStructure(vm, structure);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void JSObject::setButterflyWithoutChangingStructure(VM& vm, Butterfly* butterfly)
</del><ins>+inline void JSObject::setButterfly(VM& vm, Butterfly* butterfly)
</ins><span class="cx"> {
</span><span class="cx"> m_butterfly.set(vm, this, butterfly);
</span><span class="cx"> }
</span><span class="lines">@@ -1273,16 +1265,16 @@
</span><span class="cx">
</span><span class="cx"> // It is safe to call this method with a PropertyName that is actually an index,
</span><span class="cx"> // but if so will always return false (doesn't search index storage).
</span><del>-ALWAYS_INLINE bool JSObject::getOwnNonIndexPropertySlot(VM& vm, Structure& structure, PropertyName propertyName, PropertySlot& slot)
</del><ins>+ALWAYS_INLINE bool JSObject::getOwnNonIndexPropertySlot(VM& vm, Structure* structure, PropertyName propertyName, PropertySlot& slot)
</ins><span class="cx"> {
</span><span class="cx"> unsigned attributes;
</span><del>- PropertyOffset offset = structure.get(vm, propertyName, attributes);
</del><ins>+ PropertyOffset offset = structure->get(vm, propertyName, attributes);
</ins><span class="cx"> if (!isValidOffset(offset)) {
</span><span class="cx"> if (!TypeInfo::hasStaticPropertyTable(inlineTypeFlags()))
</span><span class="cx"> return false;
</span><span class="cx"> return getOwnStaticPropertySlot(vm, propertyName, slot);
</span><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> // getPropertySlot relies on this method never returning index properties!
</span><span class="cx"> ASSERT(!parseIndex(propertyName));
</span><span class="cx">
</span><span class="lines">@@ -1307,9 +1299,9 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-ALWAYS_INLINE void JSObject::fillCustomGetterPropertySlot(PropertySlot& slot, JSValue customGetterSetter, unsigned attributes, Structure& structure)
</del><ins>+ALWAYS_INLINE void JSObject::fillCustomGetterPropertySlot(PropertySlot& slot, JSValue customGetterSetter, unsigned attributes, Structure* structure)
</ins><span class="cx"> {
</span><del>- if (structure.isUncacheableDictionary()) {
</del><ins>+ if (structure->isUncacheableDictionary()) {
</ins><span class="cx"> slot.setCustom(this, attributes, jsCast<CustomGetterSetter*>(customGetterSetter)->getter());
</span><span class="cx"> return;
</span><span class="cx"> }
</span><span class="lines">@@ -1325,7 +1317,7 @@
</span><span class="cx"> ALWAYS_INLINE bool JSObject::getOwnPropertySlot(JSObject* object, ExecState* exec, PropertyName propertyName, PropertySlot& slot)
</span><span class="cx"> {
</span><span class="cx"> VM& vm = exec->vm();
</span><del>- Structure& structure = *object->structure(vm);
</del><ins>+ Structure* structure = object->structure(vm);
</ins><span class="cx"> if (object->getOwnNonIndexPropertySlot(vm, structure, propertyName, slot))
</span><span class="cx"> return true;
</span><span class="cx"> if (Optional<uint32_t> index = parseIndex(propertyName))
</span><span class="lines">@@ -1353,10 +1345,10 @@
</span><span class="cx"> return object->getNonIndexPropertySlot(exec, propertyName, slot);
</span><span class="cx"> }
</span><span class="cx"> ASSERT(object->type() != ProxyObjectType);
</span><del>- Structure& structure = *structureIDTable.get(object->structureID());
</del><ins>+ Structure* structure = structureIDTable.get(object->structureID());
</ins><span class="cx"> if (object->getOwnNonIndexPropertySlot(vm, structure, propertyName, slot))
</span><span class="cx"> return true;
</span><del>- JSValue prototype = structure.storedPrototype();
</del><ins>+ JSValue prototype = structure->storedPrototype();
</ins><span class="cx"> if (!prototype.isObject())
</span><span class="cx"> break;
</span><span class="cx"> object = asObject(prototype);
</span><span class="lines">@@ -1385,132 +1377,6 @@
</span><span class="cx"> return jsUndefined();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<JSObject::PutMode mode>
-ALWAYS_INLINE bool JSObject::putDirectInternal(VM& vm, PropertyName propertyName, JSValue value, unsigned attributes, PutPropertySlot& slot)
-{
- ASSERT(value);
- ASSERT(value.isGetterSetter() == !!(attributes & Accessor));
- ASSERT(!Heap::heap(value) || Heap::heap(value) == Heap::heap(this));
- ASSERT(!parseIndex(propertyName));
-
- Structure* structure = this->structure(vm);
- if (structure->isDictionary()) {
- ASSERT(!structure->hasInferredTypes());
-
- unsigned currentAttributes;
- PropertyOffset offset = structure->get(vm, propertyName, currentAttributes);
- if (offset != invalidOffset) {
- if ((mode == PutModePut) && currentAttributes & ReadOnly)
- return false;
-
- putDirect(vm, offset, value);
- structure->didReplaceProperty(offset);
- slot.setExistingProperty(this, offset);
-
- if ((attributes & Accessor) != (currentAttributes & Accessor) || (attributes & CustomAccessor) != (currentAttributes & CustomAccessor)) {
- ASSERT(!(attributes & ReadOnly));
- setStructure(vm, Structure::attributeChangeTransition(vm, structure, propertyName, attributes));
- }
- return true;
- }
-
- if ((mode == PutModePut) && !isStructureExtensible())
- return false;
-
- DeferGC deferGC(vm.heap);
- Butterfly* newButterfly = butterfly();
- if (this->structure()->putWillGrowOutOfLineStorage())
- newButterfly = growOutOfLineStorage(vm, this->structure()->outOfLineCapacity(), this->structure()->suggestedNewOutOfLineStorageCapacity());
- offset = this->structure()->addPropertyWithoutTransition(vm, propertyName, attributes);
- setStructureAndButterfly(vm, this->structure(), newButterfly);
-
- validateOffset(offset);
- ASSERT(this->structure()->isValidOffset(offset));
- putDirect(vm, offset, value);
- slot.setNewProperty(this, offset);
- if (attributes & ReadOnly)
- this->structure()->setContainsReadOnlyProperties();
- return true;
- }
-
- PropertyOffset offset;
- size_t currentCapacity = this->structure()->outOfLineCapacity();
- Structure* newStructure = Structure::addPropertyTransitionToExistingStructure(
- structure, propertyName, attributes, offset);
- if (newStructure) {
- newStructure->willStoreValueForExistingTransition(
- vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
-
- DeferGC deferGC(vm.heap);
- Butterfly* newButterfly = butterfly();
- if (currentCapacity != newStructure->outOfLineCapacity()) {
- ASSERT(newStructure != this->structure());
- newButterfly = growOutOfLineStorage(vm, currentCapacity, newStructure->outOfLineCapacity());
- }
-
- validateOffset(offset);
- ASSERT(newStructure->isValidOffset(offset));
- setStructureAndButterfly(vm, newStructure, newButterfly);
- putDirect(vm, offset, value);
- slot.setNewProperty(this, offset);
- return true;
- }
-
- unsigned currentAttributes;
- bool hasInferredType;
- offset = structure->get(vm, propertyName, currentAttributes, hasInferredType);
- if (offset != invalidOffset) {
- if ((mode == PutModePut) && currentAttributes & ReadOnly)
- return false;
-
- structure->didReplaceProperty(offset);
- if (UNLIKELY(hasInferredType)) {
- structure->willStoreValueForReplace(
- vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
- }
-
- slot.setExistingProperty(this, offset);
- putDirect(vm, offset, value);
-
- if ((attributes & Accessor) != (currentAttributes & Accessor) || (attributes & CustomAccessor) != (currentAttributes & CustomAccessor)) {
- ASSERT(!(attributes & ReadOnly));
- setStructure(vm, Structure::attributeChangeTransition(vm, structure, propertyName, attributes));
- }
- return true;
- }
-
- if ((mode == PutModePut) && !isStructureExtensible())
- return false;
-
- // We want the structure transition watchpoint to fire after this object has switched
- // structure. This allows adaptive watchpoints to observe if the new structure is the one
- // we want.
- DeferredStructureTransitionWatchpointFire deferredWatchpointFire;
-
- newStructure = Structure::addNewPropertyTransition(
- vm, structure, propertyName, attributes, offset, slot.context(), &deferredWatchpointFire);
- newStructure->willStoreValueForNewTransition(
- vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
-
- validateOffset(offset);
- ASSERT(newStructure->isValidOffset(offset));
- DeferGC deferGC(vm.heap);
- size_t oldCapacity = structure->outOfLineCapacity();
- size_t newCapacity = newStructure->outOfLineCapacity();
- ASSERT(oldCapacity <= newCapacity);
- if (oldCapacity == newCapacity)
- setStructure(vm, newStructure);
- else {
- Butterfly* newButterfly = growOutOfLineStorage(vm, oldCapacity, newCapacity);
- setStructureAndButterfly(vm, newStructure, newButterfly);
- }
- putDirect(vm, offset, value);
- slot.setNewProperty(this, offset);
- if (attributes & ReadOnly)
- newStructure->setContainsReadOnlyProperties();
- return true;
-}
-
</del><span class="cx"> inline bool JSObject::putOwnDataProperty(VM& vm, PropertyName propertyName, JSValue value, PutPropertySlot& slot)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(value);
</span><span class="lines">@@ -1549,24 +1415,6 @@
</span><span class="cx"> return putDirectInternal<PutModeDefineOwnProperty>(vm, propertyName, value, 0, slot);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void JSObject::putDirectWithoutTransition(VM& vm, PropertyName propertyName, JSValue value, unsigned attributes)
-{
- DeferGC deferGC(vm.heap);
- ASSERT(!value.isGetterSetter() && !(attributes & Accessor));
- ASSERT(!value.isCustomGetterSetter());
- Butterfly* newButterfly = m_butterfly.get();
- if (structure()->putWillGrowOutOfLineStorage())
- newButterfly = growOutOfLineStorage(vm, structure()->outOfLineCapacity(), structure()->suggestedNewOutOfLineStorageCapacity());
- Structure* structure = this->structure();
- PropertyOffset offset = structure->addPropertyWithoutTransition(vm, propertyName, attributes);
- if (attributes & ReadOnly)
- structure->setContainsReadOnlyProperties();
- bool shouldOptimize = false;
- structure->willStoreValueForNewTransition(vm, propertyName, value, shouldOptimize);
- setStructureAndButterfly(vm, structure, newButterfly);
- putDirect(vm, offset, value);
-}
-
</del><span class="cx"> ALWAYS_INLINE JSObject* Register::object() const
</span><span class="cx"> {
</span><span class="cx"> return asObject(jsValue());
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeJSObjectInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/JSObjectInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/JSObjectInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/JSObjectInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -107,13 +107,13 @@
</span><span class="cx"> JSObject* object = this;
</span><span class="cx"> MethodTable::GetPrototypeFunctionPtr defaultGetPrototype = JSObject::getPrototype;
</span><span class="cx"> while (true) {
</span><del>- Structure& structure = *structureIDTable.get(object->structureID());
- if (structure.classInfo()->methodTable.getOwnPropertySlotByIndex(object, exec, propertyName, slot))
</del><ins>+ Structure* structure = structureIDTable.get(object->structureID());
+ if (structure->classInfo()->methodTable.getOwnPropertySlotByIndex(object, exec, propertyName, slot))
</ins><span class="cx"> return true;
</span><span class="cx"> RETURN_IF_EXCEPTION(scope, false);
</span><span class="cx"> JSValue prototype;
</span><del>- if (LIKELY(structure.classInfo()->methodTable.getPrototype == defaultGetPrototype || slot.internalMethodType() == PropertySlot::InternalMethodType::VMInquiry))
- prototype = structure.storedPrototype();
</del><ins>+ if (LIKELY(structure->classInfo()->methodTable.getPrototype == defaultGetPrototype || slot.internalMethodType() == PropertySlot::InternalMethodType::VMInquiry))
+ prototype = structure->storedPrototype();
</ins><span class="cx"> else {
</span><span class="cx"> prototype = object->getPrototype(vm, exec);
</span><span class="cx"> RETURN_IF_EXCEPTION(scope, false);
</span><span class="lines">@@ -135,18 +135,18 @@
</span><span class="cx"> JSObject* object = this;
</span><span class="cx"> MethodTable::GetPrototypeFunctionPtr defaultGetPrototype = JSObject::getPrototype;
</span><span class="cx"> while (true) {
</span><del>- Structure& structure = *structureIDTable.get(object->structureID());
</del><ins>+ Structure* structure = structureIDTable.get(object->structureID());
</ins><span class="cx"> if (LIKELY(!TypeInfo::overridesGetOwnPropertySlot(object->inlineTypeFlags()))) {
</span><span class="cx"> if (object->getOwnNonIndexPropertySlot(vm, structure, propertyName, slot))
</span><span class="cx"> return true;
</span><span class="cx"> } else {
</span><del>- if (structure.classInfo()->methodTable.getOwnPropertySlot(object, exec, propertyName, slot))
</del><ins>+ if (structure->classInfo()->methodTable.getOwnPropertySlot(object, exec, propertyName, slot))
</ins><span class="cx"> return true;
</span><span class="cx"> RETURN_IF_EXCEPTION(scope, false);
</span><span class="cx"> }
</span><span class="cx"> JSValue prototype;
</span><del>- if (LIKELY(structure.classInfo()->methodTable.getPrototype == defaultGetPrototype || slot.internalMethodType() == PropertySlot::InternalMethodType::VMInquiry))
- prototype = structure.storedPrototype();
</del><ins>+ if (LIKELY(structure->classInfo()->methodTable.getPrototype == defaultGetPrototype || slot.internalMethodType() == PropertySlot::InternalMethodType::VMInquiry))
+ prototype = structure->storedPrototype();
</ins><span class="cx"> else {
</span><span class="cx"> prototype = object->getPrototype(vm, exec);
</span><span class="cx"> RETURN_IF_EXCEPTION(scope, false);
</span><span class="lines">@@ -157,6 +157,30 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline void JSObject::putDirectWithoutTransition(VM& vm, PropertyName propertyName, JSValue value, unsigned attributes)
+{
+ DeferGC deferGC(vm.heap);
+ ASSERT(!value.isGetterSetter() && !(attributes & Accessor));
+ ASSERT(!value.isCustomGetterSetter());
+ Butterfly* butterfly = m_butterfly.get();
+ Structure* structure = this->structure(vm);
+ unsigned oldOutOfLineCapacity = structure->outOfLineCapacity();
+ structure->addPropertyWithoutTransition(
+ vm, propertyName, attributes,
+ [&] (const GCSafeConcurrentJITLocker&, PropertyOffset offset) {
+ if (structure->outOfLineCapacity() != oldOutOfLineCapacity) {
+ butterfly = allocateMoreOutOfLineStorage(vm, oldOutOfLineCapacity, structure->outOfLineCapacity());
+ WTF::storeStoreFence();
+ setButterfly(vm, butterfly);
+ }
+ if (attributes & ReadOnly)
+ structure->setContainsReadOnlyProperties();
+ bool shouldOptimize = false;
+ structure->willStoreValueForNewTransition(vm, propertyName, value, shouldOptimize);
+ putDirect(vm, offset, value);
+ });
+}
+
</ins><span class="cx"> // ECMA 8.6.2.2
</span><span class="cx"> ALWAYS_INLINE bool JSObject::putInline(JSCell* cell, ExecState* exec, PropertyName propertyName, JSValue value, PutPropertySlot& slot)
</span><span class="cx"> {
</span><span class="lines">@@ -207,4 +231,139 @@
</span><span class="cx"> return const_cast<JSObject*>(this)->methodTable(exec->vm())->getOwnPropertySlotByIndex(const_cast<JSObject*>(this), exec, propertyName, slot);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+template<JSObject::PutMode mode>
+ALWAYS_INLINE bool JSObject::putDirectInternal(VM& vm, PropertyName propertyName, JSValue value, unsigned attributes, PutPropertySlot& slot)
+{
+ ASSERT(value);
+ ASSERT(value.isGetterSetter() == !!(attributes & Accessor));
+ ASSERT(!Heap::heap(value) || Heap::heap(value) == Heap::heap(this));
+ ASSERT(!parseIndex(propertyName));
+
+ Structure* structure = this->structure(vm);
+ if (structure->isDictionary()) {
+ ASSERT(!structure->hasInferredTypes());
+
+ unsigned currentAttributes;
+ PropertyOffset offset = structure->get(vm, propertyName, currentAttributes);
+ if (offset != invalidOffset) {
+ if ((mode == PutModePut) && currentAttributes & ReadOnly)
+ return false;
+
+ putDirect(vm, offset, value);
+ structure->didReplaceProperty(offset);
+ slot.setExistingProperty(this, offset);
+
+ if ((attributes & Accessor) != (currentAttributes & Accessor) || (attributes & CustomAccessor) != (currentAttributes & CustomAccessor)) {
+ ASSERT(!(attributes & ReadOnly));
+ setStructure(vm, Structure::attributeChangeTransition(vm, structure, propertyName, attributes));
+ }
+ return true;
+ }
+
+ if ((mode == PutModePut) && !isStructureExtensible())
+ return false;
+
+ unsigned oldOutOfLineCapacity = structure->outOfLineCapacity();
+ offset = structure->addPropertyWithoutTransition(
+ vm, propertyName, attributes,
+ [&] (const GCSafeConcurrentJITLocker&, PropertyOffset offset) {
+ Butterfly* butterfly = this->butterfly();
+ if (structure->outOfLineCapacity() != oldOutOfLineCapacity) {
+ butterfly = allocateMoreOutOfLineStorage(vm, oldOutOfLineCapacity, structure->outOfLineCapacity());
+ WTF::storeStoreFence();
+ setButterfly(vm, butterfly);
+ }
+ validateOffset(offset);
+ ASSERT(structure->isValidOffset(offset));
+ putDirect(vm, offset, value);
+ });
+
+ slot.setNewProperty(this, offset);
+ if (attributes & ReadOnly)
+ this->structure()->setContainsReadOnlyProperties();
+ return true;
+ }
+
+ PropertyOffset offset;
+ size_t currentCapacity = this->structure()->outOfLineCapacity();
+ Structure* newStructure = Structure::addPropertyTransitionToExistingStructure(
+ structure, propertyName, attributes, offset);
+ if (newStructure) {
+ newStructure->willStoreValueForExistingTransition(
+ vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
+
+ DeferGC deferGC(vm.heap);
+ Butterfly* newButterfly = butterfly();
+ if (currentCapacity != newStructure->outOfLineCapacity()) {
+ ASSERT(newStructure != this->structure());
+ newButterfly = allocateMoreOutOfLineStorage(vm, currentCapacity, newStructure->outOfLineCapacity());
+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ }
+
+ validateOffset(offset);
+ ASSERT(newStructure->isValidOffset(offset));
+ putDirect(vm, offset, value);
+ setStructure(vm, newStructure);
+ slot.setNewProperty(this, offset);
+ return true;
+ }
+
+ unsigned currentAttributes;
+ bool hasInferredType;
+ offset = structure->get(vm, propertyName, currentAttributes, hasInferredType);
+ if (offset != invalidOffset) {
+ if ((mode == PutModePut) && currentAttributes & ReadOnly)
+ return false;
+
+ structure->didReplaceProperty(offset);
+ if (UNLIKELY(hasInferredType)) {
+ structure->willStoreValueForReplace(
+ vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
+ }
+
+ slot.setExistingProperty(this, offset);
+ putDirect(vm, offset, value);
+
+ if ((attributes & Accessor) != (currentAttributes & Accessor) || (attributes & CustomAccessor) != (currentAttributes & CustomAccessor)) {
+ ASSERT(!(attributes & ReadOnly));
+ setStructure(vm, Structure::attributeChangeTransition(vm, structure, propertyName, attributes));
+ }
+ return true;
+ }
+
+ if ((mode == PutModePut) && !isStructureExtensible())
+ return false;
+
+ // We want the structure transition watchpoint to fire after this object has switched
+ // structure. This allows adaptive watchpoints to observe if the new structure is the one
+ // we want.
+ DeferredStructureTransitionWatchpointFire deferredWatchpointFire;
+
+ newStructure = Structure::addNewPropertyTransition(
+ vm, structure, propertyName, attributes, offset, slot.context(), &deferredWatchpointFire);
+ newStructure->willStoreValueForNewTransition(
+ vm, propertyName, value, slot.context() == PutPropertySlot::PutById);
+
+ validateOffset(offset);
+ ASSERT(newStructure->isValidOffset(offset));
+ DeferGC deferGC(vm.heap);
+ size_t oldCapacity = structure->outOfLineCapacity();
+ size_t newCapacity = newStructure->outOfLineCapacity();
+ ASSERT(oldCapacity <= newCapacity);
+ if (oldCapacity != newCapacity) {
+ Butterfly* newButterfly = allocateMoreOutOfLineStorage(vm, oldCapacity, newCapacity);
+ WTF::storeStoreFence();
+ setButterfly(vm, newButterfly);
+ WTF::storeStoreFence();
+ }
+ putDirect(vm, offset, value);
+ setStructure(vm, newStructure);
+ slot.setNewProperty(this, offset);
+ if (attributes & ReadOnly)
+ newStructure->setContainsReadOnlyProperties();
+ return true;
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeOptionsh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Options.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Options.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/Options.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -184,6 +184,8 @@
</span><span class="cx"> v(bool, verboseSanitizeStack, false, Normal, nullptr) \
</span><span class="cx"> v(bool, useGenerationalGC, true, Normal, nullptr) \
</span><span class="cx"> v(bool, useConcurrentBarriers, true, Normal, nullptr) \
</span><ins>+ v(bool, useConcurrentGC, false, Normal, nullptr) \
+ v(bool, collectContinuously, false, Normal, nullptr) \
</ins><span class="cx"> v(bool, forceFencedBarrier, false, Normal, nullptr) \
</span><span class="cx"> v(bool, scribbleFreeCells, false, Normal, nullptr) \
</span><span class="cx"> v(double, sizeClassProgression, 1.4, Normal, nullptr) \
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeSparseArrayValueMaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/SparseArrayValueMap.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/SparseArrayValueMap.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/SparseArrayValueMap.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -49,6 +49,7 @@
</span><span class="cx"> unsigned attributes;
</span><span class="cx"> };
</span><span class="cx">
</span><ins>+// FIXME: This needs to have a lock in it.
</ins><span class="cx"> class SparseArrayValueMap final : public JSCell {
</span><span class="cx"> public:
</span><span class="cx"> typedef JSCell Base;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeStructurecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Structure.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Structure.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/Structure.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -45,12 +45,6 @@
</span><span class="cx">
</span><span class="cx"> #define DUMP_STRUCTURE_ID_STATISTICS 0
</span><span class="cx">
</span><del>-#ifndef NDEBUG
-#define DO_PROPERTYMAP_CONSTENCY_CHECK 0
-#else
-#define DO_PROPERTYMAP_CONSTENCY_CHECK 0
-#endif
-
</del><span class="cx"> using namespace std;
</span><span class="cx"> using namespace WTF;
</span><span class="cx">
</span><span class="lines">@@ -163,9 +157,9 @@
</span><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- if (structure->propertyTable()) {
</del><ins>+ if (PropertyTable* table = structure->propertyTableOrNull()) {
</ins><span class="cx"> ++numberWithPropertyMaps;
</span><del>- totalPropertyMapsSize += structure->propertyTable()->sizeInMemory();
</del><ins>+ totalPropertyMapsSize += table->sizeInMemory();
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -306,7 +300,7 @@
</span><span class="cx"> for (structure = this; structure; structure = structure->previousID()) {
</span><span class="cx"> structure->m_lock.lock();
</span><span class="cx">
</span><del>- table = structure->propertyTable().get();
</del><ins>+ table = structure->propertyTableOrNull();
</ins><span class="cx"> if (table) {
</span><span class="cx"> // Leave the structure locked, so that the caller can do things to it atomically
</span><span class="cx"> // before it loses its property table.
</span><span class="lines">@@ -321,11 +315,12 @@
</span><span class="cx"> ASSERT(!table);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Structure::materializePropertyMap(VM& vm)
</del><ins>+PropertyTable* Structure::materializePropertyTable(VM& vm, bool setPropertyTable)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(structure()->classInfo() == info());
</span><del>- ASSERT(!propertyTable());
-
</del><ins>+
+ DeferGC deferGC(vm.heap);
+
</ins><span class="cx"> Vector<Structure*, 8> structures;
</span><span class="cx"> Structure* structure;
</span><span class="cx"> PropertyTable* table;
</span><span class="lines">@@ -332,19 +327,19 @@
</span><span class="cx">
</span><span class="cx"> findStructuresAndMapForMaterialization(structures, structure, table);
</span><span class="cx">
</span><ins>+ unsigned capacity = numberOfSlotsForLastOffset(m_offset, m_inlineCapacity);
</ins><span class="cx"> if (table) {
</span><del>- table = table->copy(vm, numberOfSlotsForLastOffset(m_offset, m_inlineCapacity));
</del><ins>+ table = table->copy(vm, capacity);
</ins><span class="cx"> structure->m_lock.unlock();
</span><del>- }
</del><ins>+ } else
+ table = PropertyTable::create(vm, capacity);
</ins><span class="cx">
</span><span class="cx"> // Must hold the lock on this structure, since we will be modifying this structure's
</span><span class="cx"> // property map. We don't want getConcurrently() to see the property map in a half-baked
</span><span class="cx"> // state.
</span><span class="cx"> GCSafeConcurrentJITLocker locker(m_lock, vm.heap);
</span><del>- if (!table)
- createPropertyMap(locker, vm, numberOfSlotsForLastOffset(m_offset, m_inlineCapacity));
- else
- propertyTable().set(vm, this, table);
</del><ins>+ if (setPropertyTable)
+ this->setPropertyTable(vm, table);
</ins><span class="cx">
</span><span class="cx"> InferredTypeTable* typeTable = m_inferredTypeTable.get();
</span><span class="cx">
</span><span class="lines">@@ -355,10 +350,12 @@
</span><span class="cx"> PropertyMapEntry entry(structure->m_nameInPrevious.get(), structure->m_offset, structure->attributesInPrevious());
</span><span class="cx"> if (typeTable && typeTable->get(structure->m_nameInPrevious.get()))
</span><span class="cx"> entry.hasInferredType = true;
</span><del>- propertyTable()->add(entry, m_offset, PropertyTable::PropertyOffsetMustNotChange);
</del><ins>+ table->add(entry, m_offset, PropertyTable::PropertyOffsetMustNotChange);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> checkOffsetConsistency();
</span><ins>+
+ return table;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* Structure::addPropertyTransitionToExistingStructureImpl(Structure* structure, UniquedStringImpl* uid, unsigned attributes, PropertyOffset& offset)
</span><span class="lines">@@ -473,7 +470,7 @@
</span><span class="cx"> transition->m_cachedPrototypeChain.setMayBeNull(vm, transition, structure->m_cachedPrototypeChain.get());
</span><span class="cx"> transition->m_nameInPrevious = propertyName.uid();
</span><span class="cx"> transition->setAttributesInPrevious(attributes);
</span><del>- transition->propertyTable().set(vm, transition, structure->takePropertyTableOrCloneIfPinned(vm));
</del><ins>+ transition->setPropertyTable(vm, structure->takePropertyTableOrCloneIfPinned(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><span class="cx"> transition->m_inferredTypeTable.setMayBeNull(vm, transition, structure->m_inferredTypeTable.get());
</span><span class="cx">
</span><span class="lines">@@ -521,16 +518,14 @@
</span><span class="cx">
</span><span class="cx"> Structure* Structure::changePrototypeTransition(VM& vm, Structure* structure, JSValue prototype)
</span><span class="cx"> {
</span><ins>+ DeferGC deferGC(vm.heap);
</ins><span class="cx"> Structure* transition = create(vm, structure);
</span><span class="cx">
</span><span class="cx"> transition->m_prototype.set(vm, transition, prototype);
</span><del>-
- DeferGC deferGC(vm.heap);
- structure->materializePropertyMapIfNecessary(vm, deferGC);
- transition->propertyTable().set(vm, transition, structure->copyPropertyTableForPinning(vm));
</del><ins>+
+ transition->pin(vm, structure->copyPropertyTableForPinning(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><del>- transition->pin();
-
</del><ins>+
</ins><span class="cx"> transition->checkOffsetConsistency();
</span><span class="cx"> return transition;
</span><span class="cx"> }
</span><span class="lines">@@ -537,20 +532,16 @@
</span><span class="cx">
</span><span class="cx"> Structure* Structure::attributeChangeTransition(VM& vm, Structure* structure, PropertyName propertyName, unsigned attributes)
</span><span class="cx"> {
</span><del>- DeferGC deferGC(vm.heap);
</del><span class="cx"> if (!structure->isUncacheableDictionary()) {
</span><span class="cx"> Structure* transition = create(vm, structure);
</span><del>-
- structure->materializePropertyMapIfNecessary(vm, deferGC);
- transition->propertyTable().set(vm, transition, structure->copyPropertyTableForPinning(vm));
</del><ins>+
+ transition->pin(vm, structure->copyPropertyTableForPinning(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><del>- transition->pin();
</del><span class="cx">
</span><span class="cx"> structure = transition;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- ASSERT(structure->propertyTable());
- PropertyMapEntry* entry = structure->propertyTable()->get(propertyName.uid());
</del><ins>+ PropertyMapEntry* entry = structure->ensurePropertyTable(vm)->get(propertyName.uid());
</ins><span class="cx"> ASSERT(entry);
</span><span class="cx"> entry->attributes = attributes;
</span><span class="cx">
</span><span class="lines">@@ -561,17 +552,15 @@
</span><span class="cx"> Structure* Structure::toDictionaryTransition(VM& vm, Structure* structure, DictionaryKind kind, DeferredStructureTransitionWatchpointFire* deferred)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(!structure->isUncacheableDictionary());
</span><ins>+ DeferGC deferGC(vm.heap);
</ins><span class="cx">
</span><span class="cx"> Structure* transition = create(vm, structure, deferred);
</span><span class="cx">
</span><del>- DeferGC deferGC(vm.heap);
- structure->materializePropertyMapIfNecessary(vm, deferGC);
- transition->propertyTable().set(vm, transition, structure->copyPropertyTableForPinning(vm));
</del><ins>+ transition->pin(vm, structure->copyPropertyTableForPinning(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><span class="cx"> transition->setDictionaryKind(kind);
</span><del>- transition->pin();
</del><span class="cx"> transition->setHasBeenDictionary(true);
</span><del>-
</del><ins>+
</ins><span class="cx"> transition->checkOffsetConsistency();
</span><span class="cx"> return transition;
</span><span class="cx"> }
</span><span class="lines">@@ -603,31 +592,29 @@
</span><span class="cx">
</span><span class="cx"> PropertyTable* Structure::takePropertyTableOrCloneIfPinned(VM& vm)
</span><span class="cx"> {
</span><del>- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessaryForPinning(vm, deferGC);
-
- if (isPinnedPropertyTable())
- return propertyTable()->copy(vm, propertyTable()->size() + 1);
-
- // Hold the lock while stealing the table - so that getConcurrently() on another thread
- // will either have to bypass this structure, or will get to use the property table
- // before it is stolen.
- ConcurrentJITLocker locker(m_lock);
- PropertyTable* takenPropertyTable = propertyTable().get();
- propertyTable().clear();
- return takenPropertyTable;
</del><ins>+ // This must always return a property table. It can't return null.
+ PropertyTable* result = propertyTableOrNull();
+ if (result) {
+ if (isPinnedPropertyTable())
+ return result->copy(vm, result->size() + 1);
+ ConcurrentJITLocker locker(m_lock);
+ setPropertyTable(vm, nullptr);
+ return result;
+ }
+ bool setPropertyTable = false;
+ return materializePropertyTable(vm, setPropertyTable);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Structure* Structure::nonPropertyTransition(VM& vm, Structure* structure, NonPropertyTransition transitionKind)
</span><span class="cx"> {
</span><span class="cx"> unsigned attributes = toAttributes(transitionKind);
</span><del>- IndexingType indexingType = newIndexingType(structure->indexingTypeIncludingHistory(), transitionKind);
</del><ins>+ IndexingType indexingTypeIncludingHistory = newIndexingType(structure->indexingTypeIncludingHistory(), transitionKind);
</ins><span class="cx">
</span><span class="cx"> if (changesIndexingType(transitionKind)) {
</span><span class="cx"> if (JSGlobalObject* globalObject = structure->m_globalObject.get()) {
</span><span class="cx"> if (globalObject->isOriginalArrayStructure(structure)) {
</span><del>- Structure* result = globalObject->originalArrayStructureForIndexingType(indexingType);
- if (result->indexingTypeIncludingHistory() == indexingType) {
</del><ins>+ Structure* result = globalObject->originalArrayStructureForIndexingType(indexingTypeIncludingHistory);
+ if (result->indexingTypeIncludingHistory() == indexingTypeIncludingHistory) {
</ins><span class="cx"> structure->didTransitionFromThisStructure();
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="lines">@@ -638,7 +625,7 @@
</span><span class="cx"> Structure* existingTransition;
</span><span class="cx"> if (!structure->isDictionary() && (existingTransition = structure->m_transitionTable.get(0, attributes))) {
</span><span class="cx"> ASSERT(existingTransition->attributesInPrevious() == attributes);
</span><del>- ASSERT(existingTransition->indexingTypeIncludingHistory() == indexingType);
</del><ins>+ ASSERT(existingTransition->indexingTypeIncludingHistory() == indexingTypeIncludingHistory);
</ins><span class="cx"> return existingTransition;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -646,7 +633,7 @@
</span><span class="cx">
</span><span class="cx"> Structure* transition = create(vm, structure);
</span><span class="cx"> transition->setAttributesInPrevious(attributes);
</span><del>- transition->m_blob.setIndexingType(indexingType);
</del><ins>+ transition->m_blob.setIndexingTypeIncludingHistory(indexingTypeIncludingHistory);
</ins><span class="cx">
</span><span class="cx"> if (preventsExtensions(transitionKind))
</span><span class="cx"> transition->setDidPreventExtensions(true);
</span><span class="lines">@@ -657,32 +644,29 @@
</span><span class="cx"> // table, since our logic for walking the property transition chain to rematerialize the
</span><span class="cx"> // table doesn't know how to take into account such wholesale edits.
</span><span class="cx">
</span><del>- structure->materializePropertyMapIfNecessary(vm, deferGC);
- transition->propertyTable().set(vm, transition, structure->copyPropertyTableForPinning(vm));
</del><ins>+ transition->pinForCaching(vm, structure->copyPropertyTableForPinning(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><del>- transition->pinForCaching();
</del><span class="cx">
</span><del>- if (transition->propertyTable()) {
- for (auto& entry : *transition->propertyTable().get()) {
- if (setsDontDeleteOnAllProperties(transitionKind))
- entry.attributes |= DontDelete;
- if (setsReadOnlyOnNonAccessorProperties(transitionKind) && !(entry.attributes & Accessor))
- entry.attributes |= ReadOnly;
- }
</del><ins>+ PropertyTable* table = transition->propertyTableOrNull();
+ RELEASE_ASSERT(table);
+ for (auto& entry : *table) {
+ if (setsDontDeleteOnAllProperties(transitionKind))
+ entry.attributes |= DontDelete;
+ if (setsReadOnlyOnNonAccessorProperties(transitionKind) && !(entry.attributes & Accessor))
+ entry.attributes |= ReadOnly;
</ins><span class="cx"> }
</span><span class="cx"> } else {
</span><del>- transition->propertyTable().set(vm, transition, structure->takePropertyTableOrCloneIfPinned(vm));
</del><ins>+ transition->setPropertyTable(vm, structure->takePropertyTableOrCloneIfPinned(vm));
</ins><span class="cx"> transition->m_offset = structure->m_offset;
</span><span class="cx"> checkOffset(transition->m_offset, transition->inlineCapacity());
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> if (setsReadOnlyOnNonAccessorProperties(transitionKind)
</span><del>- && transition->propertyTable()
- && !transition->propertyTable()->isEmpty())
</del><ins>+ && !transition->propertyTableOrNull()->isEmpty())
</ins><span class="cx"> transition->setHasReadOnlyOrGetterSetterPropertiesExcludingProto(true);
</span><span class="cx">
</span><span class="cx"> if (structure->isDictionary())
</span><del>- transition->pin();
</del><ins>+ transition->pin(vm, transition->ensurePropertyTable(vm));
</ins><span class="cx"> else {
</span><span class="cx"> ConcurrentJITLocker locker(structure->m_lock);
</span><span class="cx"> structure->m_transitionTable.add(vm, transition);
</span><span class="lines">@@ -698,13 +682,12 @@
</span><span class="cx"> if (isStructureExtensible())
</span><span class="cx"> return false;
</span><span class="cx">
</span><del>- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessary(vm, deferGC);
- if (!propertyTable())
</del><ins>+ PropertyTable* table = ensurePropertyTableIfNotEmpty(vm);
+ if (!table)
</ins><span class="cx"> return true;
</span><del>-
- PropertyTable::iterator end = propertyTable()->end();
- for (PropertyTable::iterator iter = propertyTable()->begin(); iter != end; ++iter) {
</del><ins>+
+ PropertyTable::iterator end = table->end();
+ for (PropertyTable::iterator iter = table->begin(); iter != end; ++iter) {
</ins><span class="cx"> if ((iter->attributes & DontDelete) != DontDelete)
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="lines">@@ -717,13 +700,12 @@
</span><span class="cx"> if (isStructureExtensible())
</span><span class="cx"> return false;
</span><span class="cx">
</span><del>- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessary(vm, deferGC);
- if (!propertyTable())
</del><ins>+ PropertyTable* table = ensurePropertyTableIfNotEmpty(vm);
+ if (!table)
</ins><span class="cx"> return true;
</span><del>-
- PropertyTable::iterator end = propertyTable()->end();
- for (PropertyTable::iterator iter = propertyTable()->begin(); iter != end; ++iter) {
</del><ins>+
+ PropertyTable::iterator end = table->end();
+ for (PropertyTable::iterator iter = table->begin(); iter != end; ++iter) {
</ins><span class="cx"> if (!(iter->attributes & DontDelete))
</span><span class="cx"> return false;
</span><span class="cx"> if (!(iter->attributes & (ReadOnly | Accessor)))
</span><span class="lines">@@ -736,12 +718,15 @@
</span><span class="cx"> {
</span><span class="cx"> checkOffsetConsistency();
</span><span class="cx"> ASSERT(isDictionary());
</span><ins>+
+ GCSafeConcurrentJITLocker locker(m_lock, vm.heap);
</ins><span class="cx">
</span><span class="cx"> size_t beforeOutOfLineCapacity = this->outOfLineCapacity();
</span><span class="cx"> if (isUncacheableDictionary()) {
</span><del>- ASSERT(propertyTable());
</del><ins>+ PropertyTable* table = propertyTableOrNull();
+ ASSERT(table);
</ins><span class="cx">
</span><del>- size_t propertyCount = propertyTable()->size();
</del><ins>+ size_t propertyCount = table->size();
</ins><span class="cx">
</span><span class="cx"> // Holds our values compacted by insertion order.
</span><span class="cx"> Vector<JSValue> values(propertyCount);
</span><span class="lines">@@ -748,9 +733,9 @@
</span><span class="cx">
</span><span class="cx"> // Copies out our values from their hashed locations, compacting property table offsets as we go.
</span><span class="cx"> unsigned i = 0;
</span><del>- PropertyTable::iterator end = propertyTable()->end();
</del><ins>+ PropertyTable::iterator end = table->end();
</ins><span class="cx"> m_offset = invalidOffset;
</span><del>- for (PropertyTable::iterator iter = propertyTable()->begin(); iter != end; ++iter, ++i) {
</del><ins>+ for (PropertyTable::iterator iter = table->begin(); iter != end; ++iter, ++i) {
</ins><span class="cx"> values[i] = object->getDirect(iter->offset);
</span><span class="cx"> m_offset = iter->offset = offsetForPropertyNumber(i, m_inlineCapacity);
</span><span class="cx"> }
</span><span class="lines">@@ -759,7 +744,7 @@
</span><span class="cx"> for (unsigned i = 0; i < propertyCount; i++)
</span><span class="cx"> object->putDirect(vm, offsetForPropertyNumber(i, m_inlineCapacity), values[i]);
</span><span class="cx">
</span><del>- propertyTable()->clearDeletedOffsets();
</del><ins>+ table->clearDeletedOffsets();
</ins><span class="cx"> checkOffsetConsistency();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -768,55 +753,34 @@
</span><span class="cx">
</span><span class="cx"> size_t afterOutOfLineCapacity = this->outOfLineCapacity();
</span><span class="cx">
</span><del>- if (beforeOutOfLineCapacity != afterOutOfLineCapacity) {
</del><ins>+ if (object->butterfly() && beforeOutOfLineCapacity != afterOutOfLineCapacity) {
</ins><span class="cx"> ASSERT(beforeOutOfLineCapacity > afterOutOfLineCapacity);
</span><span class="cx"> // If the object had a Butterfly but after flattening/compacting we no longer have need of it,
</span><span class="cx"> // we need to zero it out because the collector depends on the Structure to know the size for copying.
</span><del>- if (object->butterfly() && !afterOutOfLineCapacity && !this->hasIndexingHeader(object))
- object->setStructureAndButterfly(vm, this, 0);
</del><ins>+ if (!afterOutOfLineCapacity && !this->hasIndexingHeader(object))
+ object->setButterfly(vm, nullptr);
</ins><span class="cx"> // If the object was down-sized to the point where the base of the Butterfly is no longer within the
</span><span class="cx"> // first CopiedBlock::blockSize bytes, we'll get the wrong answer if we try to mask the base back to
</span><span class="cx"> // the CopiedBlock header. To prevent this case we need to memmove the Butterfly down.
</span><del>- else if (object->butterfly())
- object->shiftButterflyAfterFlattening(vm, beforeOutOfLineCapacity, afterOutOfLineCapacity);
</del><ins>+ else
+ object->shiftButterflyAfterFlattening(locker, vm, this, afterOutOfLineCapacity);
</ins><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> return this;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-PropertyOffset Structure::addPropertyWithoutTransition(VM& vm, PropertyName propertyName, unsigned attributes)
</del><ins>+void Structure::pin(VM& vm, PropertyTable* table)
</ins><span class="cx"> {
</span><del>- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessaryForPinning(vm, deferGC);
-
- pin();
-
- return add(vm, propertyName, attributes);
-}
-
-PropertyOffset Structure::removePropertyWithoutTransition(VM& vm, PropertyName propertyName)
-{
- ASSERT(isUncacheableDictionary());
-
- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessaryForPinning(vm, deferGC);
-
- pin();
- return remove(propertyName);
-}
-
-void Structure::pin()
-{
- ASSERT(propertyTable());
</del><span class="cx"> setIsPinnedPropertyTable(true);
</span><ins>+ setPropertyTable(vm, table);
</ins><span class="cx"> clearPreviousID();
</span><span class="cx"> m_nameInPrevious = nullptr;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Structure::pinForCaching()
</del><ins>+void Structure::pinForCaching(VM& vm, PropertyTable* table)
</ins><span class="cx"> {
</span><del>- ASSERT(propertyTable());
</del><span class="cx"> setIsPinnedPropertyTable(true);
</span><ins>+ setPropertyTable(vm, table);
</ins><span class="cx"> m_nameInPrevious = nullptr;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -891,17 +855,13 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // This only works if we've got a property table.
</span><del>- PropertyTable* propertyTable;
- materializePropertyMapIfNecessary(vm, propertyTable);
-
</del><ins>+ PropertyTable* propertyTable = ensurePropertyTable(vm);
+
</ins><span class="cx"> // We must be calling this after having created the given property or confirmed that it was present
</span><del>- // already, so we must have a property table now.
- ASSERT(propertyTable);
-
- // ... and the property must be present.
</del><ins>+ // already, so the property must be present.
</ins><span class="cx"> PropertyMapEntry* entry = propertyTable->get(propertyName.uid());
</span><span class="cx"> ASSERT(entry);
</span><del>-
</del><ins>+
</ins><span class="cx"> if (shouldOptimize)
</span><span class="cx"> entry->hasInferredType = table->willStoreValue(vm, propertyName, value, age);
</span><span class="cx"> else {
</span><span class="lines">@@ -943,18 +903,12 @@
</span><span class="cx">
</span><span class="cx"> #endif
</span><span class="cx">
</span><del>-PropertyTable* Structure::copyPropertyTable(VM& vm)
-{
- if (!propertyTable())
- return 0;
- return PropertyTable::clone(vm, *propertyTable().get());
-}
-
</del><span class="cx"> PropertyTable* Structure::copyPropertyTableForPinning(VM& vm)
</span><span class="cx"> {
</span><del>- if (propertyTable())
- return PropertyTable::clone(vm, *propertyTable().get());
- return PropertyTable::create(vm, numberOfSlotsForLastOffset(m_offset, m_inlineCapacity));
</del><ins>+ if (PropertyTable* table = propertyTableOrNull())
+ return PropertyTable::clone(vm, *table);
+ bool setPropertyTable = false;
+ return materializePropertyTable(vm, setPropertyTable);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> PropertyOffset Structure::getConcurrently(UniquedStringImpl* uid, unsigned& attributes)
</span><span class="lines">@@ -989,70 +943,24 @@
</span><span class="cx">
</span><span class="cx"> PropertyOffset Structure::add(VM& vm, PropertyName propertyName, unsigned attributes)
</span><span class="cx"> {
</span><del>- GCSafeConcurrentJITLocker locker(m_lock, vm.heap);
-
- ASSERT(!JSC::isValidOffset(get(vm, propertyName)));
-
- checkConsistency();
- if (attributes & DontEnum || propertyName.isSymbol())
- setIsQuickPropertyAccessAllowedForEnumeration(false);
-
- auto rep = propertyName.uid();
-
- if (!propertyTable())
- createPropertyMap(locker, vm);
-
- PropertyOffset newOffset = propertyTable()->nextOffset(m_inlineCapacity);
-
- propertyTable()->add(PropertyMapEntry(rep, newOffset, attributes), m_offset, PropertyTable::PropertyOffsetMayChange);
-
- checkConsistency();
- return newOffset;
</del><ins>+ return add(vm, propertyName, attributes, [] (const GCSafeConcurrentJITLocker&, PropertyOffset) { });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> PropertyOffset Structure::remove(PropertyName propertyName)
</span><span class="cx"> {
</span><del>- ConcurrentJITLocker locker(m_lock);
-
- checkConsistency();
-
- auto rep = propertyName.uid();
-
- if (!propertyTable())
- return invalidOffset;
-
- PropertyTable::find_iterator position = propertyTable()->find(rep);
- if (!position.first)
- return invalidOffset;
-
- PropertyOffset offset = position.first->offset;
-
- propertyTable()->remove(position);
- propertyTable()->addDeletedOffset(offset);
-
- checkConsistency();
- return offset;
</del><ins>+ return remove(propertyName, [] (const ConcurrentJITLocker&, PropertyOffset) { });
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-void Structure::createPropertyMap(const GCSafeConcurrentJITLocker&, VM& vm, unsigned capacity)
-{
- ASSERT(!propertyTable());
-
- checkConsistency();
- propertyTable().set(vm, this, PropertyTable::create(vm, capacity));
-}
-
</del><span class="cx"> void Structure::getPropertyNamesFromStructure(VM& vm, PropertyNameArray& propertyNames, EnumerationMode mode)
</span><span class="cx"> {
</span><del>- DeferGC deferGC(vm.heap);
- materializePropertyMapIfNecessary(vm, deferGC);
- if (!propertyTable())
</del><ins>+ PropertyTable* table = ensurePropertyTableIfNotEmpty(vm);
+ if (!table)
</ins><span class="cx"> return;
</span><del>-
</del><ins>+
</ins><span class="cx"> bool knownUnique = propertyNames.canAddKnownUniqueForStructure();
</span><del>-
- PropertyTable::iterator end = propertyTable()->end();
- for (PropertyTable::iterator iter = propertyTable()->begin(); iter != end; ++iter) {
</del><ins>+
+ PropertyTable::iterator end = table->end();
+ for (PropertyTable::iterator iter = table->begin(); iter != end; ++iter) {
</ins><span class="cx"> ASSERT(!isQuickPropertyAccessAllowedForEnumeration() || !(iter->attributes & DontEnum));
</span><span class="cx"> ASSERT(!isQuickPropertyAccessAllowedForEnumeration() || !iter->key->isSymbol());
</span><span class="cx"> if (!(iter->attributes & DontEnum) || mode.includeDontEnumProperties()) {
</span><span class="lines">@@ -1114,6 +1022,9 @@
</span><span class="cx"> ASSERT_GC_OBJECT_INHERITS(thisObject, info());
</span><span class="cx">
</span><span class="cx"> JSCell::visitChildren(thisObject, visitor);
</span><ins>+
+ ConcurrentJITLocker locker(thisObject->m_lock);
+
</ins><span class="cx"> visitor.append(&thisObject->m_globalObject);
</span><span class="cx"> if (!thisObject->isObject())
</span><span class="cx"> thisObject->m_cachedPrototypeChain.clear();
</span><span class="lines">@@ -1276,92 +1187,6 @@
</span><span class="cx"> out.print("Structures:");
</span><span class="cx"> }
</span><span class="cx">
</span><del>-#if DO_PROPERTYMAP_CONSTENCY_CHECK
-
-void PropertyTable::checkConsistency()
-{
- ASSERT(m_indexSize >= PropertyTable::MinimumTableSize);
- ASSERT(m_indexMask);
- ASSERT(m_indexSize == m_indexMask + 1);
- ASSERT(!(m_indexSize & m_indexMask));
-
- ASSERT(m_keyCount <= m_indexSize / 2);
- ASSERT(m_keyCount + m_deletedCount <= m_indexSize / 2);
- ASSERT(m_deletedCount <= m_indexSize / 4);
-
- unsigned indexCount = 0;
- unsigned deletedIndexCount = 0;
- for (unsigned a = 0; a != m_indexSize; ++a) {
- unsigned entryIndex = m_index[a];
- if (entryIndex == PropertyTable::EmptyEntryIndex)
- continue;
- if (entryIndex == deletedEntryIndex()) {
- ++deletedIndexCount;
- continue;
- }
- ASSERT(entryIndex < deletedEntryIndex());
- ASSERT(entryIndex - 1 <= usedCount());
- ++indexCount;
-
- for (unsigned b = a + 1; b != m_indexSize; ++b)
- ASSERT(m_index[b] != entryIndex);
- }
- ASSERT(indexCount == m_keyCount);
- ASSERT(deletedIndexCount == m_deletedCount);
-
- ASSERT(!table()[deletedEntryIndex() - 1].key);
-
- unsigned nonEmptyEntryCount = 0;
- for (unsigned c = 0; c < usedCount(); ++c) {
- StringImpl* rep = table()[c].key;
- if (rep == PROPERTY_MAP_DELETED_ENTRY_KEY)
- continue;
- ++nonEmptyEntryCount;
- unsigned i = IdentifierRepHash::hash(rep);
- unsigned k = 0;
- unsigned entryIndex;
- while (1) {
- entryIndex = m_index[i & m_indexMask];
- ASSERT(entryIndex != PropertyTable::EmptyEntryIndex);
- if (rep == table()[entryIndex - 1].key)
- break;
- if (k == 0)
- k = 1 | doubleHash(IdentifierRepHash::hash(rep));
- i += k;
- }
- ASSERT(entryIndex == c + 1);
- }
-
- ASSERT(nonEmptyEntryCount == m_keyCount);
-}
-
-void Structure::checkConsistency()
-{
- checkOffsetConsistency();
-
- if (!propertyTable())
- return;
-
- if (isQuickPropertyAccessAllowedForEnumeration()) {
- PropertyTable::iterator end = propertyTable()->end();
- for (PropertyTable::iterator iter = propertyTable()->begin(); iter != end; ++iter) {
- ASSERT(!(iter->attributes & DontEnum));
- ASSERT(!iter->key->isSymbol());
- }
- }
-
- propertyTable()->checkConsistency();
-}
-
-#else
-
-inline void Structure::checkConsistency()
-{
- checkOffsetConsistency();
-}
-
-#endif // DO_PROPERTYMAP_CONSTENCY_CHECK
-
</del><span class="cx"> bool ClassInfo::hasStaticSetterOrReadonlyProperties() const
</span><span class="cx"> {
</span><span class="cx"> for (const ClassInfo* ci = this; ci; ci = ci->parentClass) {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeStructureh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Structure.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Structure.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/Structure.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -183,8 +183,6 @@
</span><span class="cx"> JS_EXPORT_PRIVATE bool isSealed(VM&);
</span><span class="cx"> JS_EXPORT_PRIVATE bool isFrozen(VM&);
</span><span class="cx"> bool isStructureExtensible() const { return !didPreventExtensions(); }
</span><del>- bool putWillGrowOutOfLineStorage();
- size_t suggestedNewOutOfLineStorageCapacity();
</del><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE Structure* flattenDictionaryStructure(VM&, JSObject*);
</span><span class="cx">
</span><span class="lines">@@ -191,9 +189,13 @@
</span><span class="cx"> static const bool needsDestruction = true;
</span><span class="cx"> static void destroy(JSCell*);
</span><span class="cx">
</span><del>- // These should be used with caution.
- JS_EXPORT_PRIVATE PropertyOffset addPropertyWithoutTransition(VM&, PropertyName, unsigned attributes);
- PropertyOffset removePropertyWithoutTransition(VM&, PropertyName);
</del><ins>+ // Versions that take a func will call it after making the change but while still holding
+ // the lock. The callback is not called if there is no change being made, like if you call
+ // removePropertyWithoutTransition() and the property is not found.
+ template<typename Func>
+ PropertyOffset addPropertyWithoutTransition(VM&, PropertyName, unsigned attributes, const Func&);
+ template<typename Func>
+ PropertyOffset removePropertyWithoutTransition(VM&, PropertyName, const Func&);
</ins><span class="cx"> void setPrototypeWithoutTransition(VM& vm, JSValue prototype) { m_prototype.set(vm, this, prototype); }
</span><span class="cx">
</span><span class="cx"> bool isDictionary() const { return dictionaryKind() != NoneDictionaryKind; }
</span><span class="lines">@@ -229,8 +231,8 @@
</span><span class="cx"> TypeInfo typeInfo() const { ASSERT(structure()->classInfo() == info()); return m_blob.typeInfo(m_outOfLineTypeFlags); }
</span><span class="cx"> bool isObject() const { return typeInfo().isObject(); }
</span><span class="cx">
</span><del>- IndexingType indexingType() const { return m_blob.indexingType() & AllArrayTypes; }
- IndexingType indexingTypeIncludingHistory() const { return m_blob.indexingType(); }
</del><ins>+ IndexingType indexingType() const { return m_blob.indexingTypeIncludingHistory() & AllArrayTypes; }
+ IndexingType indexingTypeIncludingHistory() const { return m_blob.indexingTypeIncludingHistory(); }
</ins><span class="cx">
</span><span class="cx"> bool mayInterceptIndexedAccesses() const
</span><span class="cx"> {
</span><span class="lines">@@ -433,9 +435,9 @@
</span><span class="cx"> return OBJECT_OFFSETOF(Structure, m_classInfo);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- static ptrdiff_t indexingTypeOffset()
</del><ins>+ static ptrdiff_t indexingTypeIncludingHistoryOffset()
</ins><span class="cx"> {
</span><del>- return OBJECT_OFFSETOF(Structure, m_blob) + StructureIDBlob::indexingTypeOffset();
</del><ins>+ return OBJECT_OFFSETOF(Structure, m_blob) + StructureIDBlob::indexingTypeIncludingHistoryOffset();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> static ptrdiff_t propertyTableUnsafeOffset()
</span><span class="lines">@@ -579,6 +581,8 @@
</span><span class="cx">
</span><span class="cx"> static void dumpContextHeader(PrintStream&);
</span><span class="cx">
</span><ins>+ ConcurrentJITLock& lock() { return m_lock; }
+
</ins><span class="cx"> DECLARE_EXPORT_INFO;
</span><span class="cx">
</span><span class="cx"> private:
</span><span class="lines">@@ -633,44 +637,45 @@
</span><span class="cx">
</span><span class="cx"> static Structure* toDictionaryTransition(VM&, Structure*, DictionaryKind, DeferredStructureTransitionWatchpointFire* = nullptr);
</span><span class="cx">
</span><ins>+ template<typename Func>
+ PropertyOffset add(VM&, PropertyName, unsigned attributes, const Func&);
</ins><span class="cx"> PropertyOffset add(VM&, PropertyName, unsigned attributes);
</span><ins>+ template<typename Func>
+ PropertyOffset remove(PropertyName, const Func&);
</ins><span class="cx"> PropertyOffset remove(PropertyName);
</span><span class="cx">
</span><del>- void createPropertyMap(const GCSafeConcurrentJITLocker&, VM&, unsigned keyCount = 0);
</del><span class="cx"> void checkConsistency();
</span><span class="cx">
</span><del>- WriteBarrier<PropertyTable>& propertyTable();
- PropertyTable* takePropertyTableOrCloneIfPinned(VM&);
- PropertyTable* copyPropertyTable(VM&);
- PropertyTable* copyPropertyTableForPinning(VM&);
- JS_EXPORT_PRIVATE void materializePropertyMap(VM&);
- ALWAYS_INLINE void materializePropertyMapIfNecessary(VM& vm, DeferGC&)
</del><ins>+ // This may grab the lock, or not. Do not call when holding the Structure's lock.
+ PropertyTable* ensurePropertyTableIfNotEmpty(VM& vm)
</ins><span class="cx"> {
</span><del>- ASSERT(!isCompilationThread());
- ASSERT(structure()->classInfo() == info());
- ASSERT(checkOffsetConsistency());
- if (!propertyTable() && previousID())
- materializePropertyMap(vm);
</del><ins>+ if (PropertyTable* result = m_propertyTableUnsafe.get())
+ return result;
+ if (!previousID())
+ return nullptr;
+ return materializePropertyTable(vm);
</ins><span class="cx"> }
</span><del>- ALWAYS_INLINE void materializePropertyMapIfNecessary(VM& vm, PropertyTable*& table)
</del><ins>+
+ // This may grab the lock, or not. Do not call when holding the Structure's lock.
+ PropertyTable* ensurePropertyTable(VM& vm)
</ins><span class="cx"> {
</span><del>- ASSERT(!isCompilationThread());
- ASSERT(structure()->classInfo() == info());
- ASSERT(checkOffsetConsistency());
- table = propertyTable().get();
- if (!table && previousID()) {
- DeferGC deferGC(vm.heap);
- materializePropertyMap(vm);
- table = propertyTable().get();
- }
</del><ins>+ if (PropertyTable* result = m_propertyTableUnsafe.get())
+ return result;
+ return materializePropertyTable(vm);
</ins><span class="cx"> }
</span><del>- void materializePropertyMapIfNecessaryForPinning(VM& vm, DeferGC&)
</del><ins>+
+ PropertyTable* propertyTableOrNull() const
</ins><span class="cx"> {
</span><del>- ASSERT(structure()->classInfo() == info());
- checkOffsetConsistency();
- if (!propertyTable())
- materializePropertyMap(vm);
</del><ins>+ return m_propertyTableUnsafe.get();
</ins><span class="cx"> }
</span><ins>+
+ // This will grab the lock. Do not call when holding the Structure's lock.
+ JS_EXPORT_PRIVATE PropertyTable* materializePropertyTable(VM&, bool setPropertyTable = true);
+
+ void setPropertyTable(VM& vm, PropertyTable* table);
+
+ PropertyTable* takePropertyTableOrCloneIfPinned(VM&);
+ PropertyTable* copyPropertyTableForPinning(VM&);
</ins><span class="cx">
</span><span class="cx"> void setPreviousID(VM& vm, Structure* structure)
</span><span class="cx"> {
</span><span class="lines">@@ -697,8 +702,8 @@
</span><span class="cx"> bool isValid(JSGlobalObject*, StructureChain* cachedPrototypeChain) const;
</span><span class="cx"> bool isValid(ExecState*, StructureChain* cachedPrototypeChain) const;
</span><span class="cx">
</span><del>- void pin();
- void pinForCaching();
</del><ins>+ JS_EXPORT_PRIVATE void pin(VM&, PropertyTable*);
+ void pinForCaching(VM&, PropertyTable*);
</ins><span class="cx">
</span><span class="cx"> bool isRareData(JSCell* cell) const
</span><span class="cx"> {
</span><span class="lines">@@ -710,7 +715,7 @@
</span><span class="cx"> ASSERT(hasRareData());
</span><span class="cx"> return static_cast<StructureRareData*>(m_previousOrRareData.get());
</span><span class="cx"> }
</span><del>-
</del><ins>+
</ins><span class="cx"> bool checkOffsetConsistency() const;
</span><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE void allocateRareData(VM&);
</span><span class="lines">@@ -740,7 +745,7 @@
</span><span class="cx">
</span><span class="cx"> StructureTransitionTable m_transitionTable;
</span><span class="cx">
</span><del>- // Should be accessed through propertyTable(). During GC, it may be set to 0 by another thread.
</del><ins>+ // Should be accessed through ensurePropertyTable(). During GC, it may be set to 0 by another thread.
</ins><span class="cx"> // During a Heap Snapshot GC we avoid clearing the table so it is safe to use.
</span><span class="cx"> WriteBarrier<PropertyTable> m_propertyTableUnsafe;
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeStructureIDBlobh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/StructureIDBlob.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/StructureIDBlob.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/StructureIDBlob.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -40,10 +40,10 @@
</span><span class="cx"> u.doubleWord = 0xbbadbeef;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- StructureIDBlob(StructureID structureID, IndexingType indexingType, const TypeInfo& typeInfo)
</del><ins>+ StructureIDBlob(StructureID structureID, IndexingType indexingTypeIncludingHistory, const TypeInfo& typeInfo)
</ins><span class="cx"> {
</span><span class="cx"> u.fields.structureID = structureID;
</span><del>- u.fields.indexingType = indexingType;
</del><ins>+ u.fields.indexingTypeIncludingHistory = indexingTypeIncludingHistory;
</ins><span class="cx"> u.fields.type = typeInfo.type();
</span><span class="cx"> u.fields.inlineTypeFlags = typeInfo.inlineTypeFlags();
</span><span class="cx"> u.fields.defaultCellState = CellState::NewWhite;
</span><span class="lines">@@ -52,8 +52,8 @@
</span><span class="cx"> void operator=(const StructureIDBlob& other) { u.doubleWord = other.u.doubleWord; }
</span><span class="cx">
</span><span class="cx"> StructureID structureID() const { return u.fields.structureID; }
</span><del>- IndexingType indexingType() const { return u.fields.indexingType; }
- void setIndexingType(IndexingType indexingType) { u.fields.indexingType = indexingType; }
</del><ins>+ IndexingType indexingTypeIncludingHistory() const { return u.fields.indexingTypeIncludingHistory; }
+ void setIndexingTypeIncludingHistory(IndexingType indexingTypeIncludingHistory) { u.fields.indexingTypeIncludingHistory = indexingTypeIncludingHistory; }
</ins><span class="cx"> JSType type() const { return u.fields.type; }
</span><span class="cx"> TypeInfo::InlineTypeFlags inlineTypeFlags() const { return u.fields.inlineTypeFlags; }
</span><span class="cx">
</span><span class="lines">@@ -67,9 +67,9 @@
</span><span class="cx"> return OBJECT_OFFSETOF(StructureIDBlob, u.fields.structureID);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- static ptrdiff_t indexingTypeOffset()
</del><ins>+ static ptrdiff_t indexingTypeIncludingHistoryOffset()
</ins><span class="cx"> {
</span><del>- return OBJECT_OFFSETOF(StructureIDBlob, u.fields.indexingType);
</del><ins>+ return OBJECT_OFFSETOF(StructureIDBlob, u.fields.indexingTypeIncludingHistory);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private:
</span><span class="lines">@@ -76,7 +76,7 @@
</span><span class="cx"> union {
</span><span class="cx"> struct {
</span><span class="cx"> StructureID structureID;
</span><del>- IndexingType indexingType;
</del><ins>+ IndexingType indexingTypeIncludingHistory;
</ins><span class="cx"> JSType type;
</span><span class="cx"> TypeInfo::InlineTypeFlags inlineTypeFlags;
</span><span class="cx"> CellState defaultCellState;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeStructureInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/StructureInlines.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/StructureInlines.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/JavaScriptCore/runtime/StructureInlines.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2013-2015 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2013-2016 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -93,8 +93,7 @@
</span><span class="cx"> ASSERT(!isCompilationThread());
</span><span class="cx"> ASSERT(structure()->classInfo() == info());
</span><span class="cx">
</span><del>- PropertyTable* propertyTable;
- materializePropertyMapIfNecessary(vm, propertyTable);
</del><ins>+ PropertyTable* propertyTable = ensurePropertyTableIfNotEmpty(vm);
</ins><span class="cx"> if (!propertyTable)
</span><span class="cx"> return invalidOffset;
</span><span class="cx">
</span><span class="lines">@@ -219,31 +218,6 @@
</span><span class="cx"> return isValid(exec->lexicalGlobalObject(), cachedPrototypeChain);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline bool Structure::putWillGrowOutOfLineStorage()
-{
- checkOffsetConsistency();
-
- ASSERT(outOfLineCapacity() >= outOfLineSize());
-
- if (!propertyTable()) {
- unsigned currentSize = numberOfOutOfLineSlotsForLastOffset(m_offset);
- ASSERT(outOfLineCapacity() >= currentSize);
- return currentSize == outOfLineCapacity();
- }
-
- ASSERT(totalStorageCapacity() >= propertyTable()->propertyStorageSize());
- if (propertyTable()->hasDeletedOffset())
- return false;
-
- ASSERT(totalStorageCapacity() >= propertyTable()->size());
- return propertyTable()->size() == totalStorageCapacity();
-}
-
-ALWAYS_INLINE WriteBarrier<PropertyTable>& Structure::propertyTable()
-{
- return m_propertyTableUnsafe;
-}
-
</del><span class="cx"> inline void Structure::didReplaceProperty(PropertyOffset offset)
</span><span class="cx"> {
</span><span class="cx"> if (LIKELY(!hasRareData()))
</span><span class="lines">@@ -271,7 +245,7 @@
</span><span class="cx">
</span><span class="cx"> ALWAYS_INLINE bool Structure::checkOffsetConsistency() const
</span><span class="cx"> {
</span><del>- PropertyTable* propertyTable = m_propertyTableUnsafe.get();
</del><ins>+ PropertyTable* propertyTable = propertyTableOrNull();
</ins><span class="cx">
</span><span class="cx"> if (!propertyTable) {
</span><span class="cx"> ASSERT(!isPinnedPropertyTable());
</span><span class="lines">@@ -292,6 +266,11 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+inline void Structure::checkConsistency()
+{
+ checkOffsetConsistency();
+}
+
</ins><span class="cx"> inline size_t nextOutOfLineStorageCapacity(size_t currentCapacity)
</span><span class="cx"> {
</span><span class="cx"> if (!currentCapacity)
</span><span class="lines">@@ -299,11 +278,6 @@
</span><span class="cx"> return currentCapacity * outOfLineGrowthFactor;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline size_t Structure::suggestedNewOutOfLineStorageCapacity()
-{
- return nextOutOfLineStorageCapacity(outOfLineCapacity());
-}
-
</del><span class="cx"> inline void Structure::setObjectToStringValue(ExecState* exec, VM& vm, JSString* value, PropertySlot toStringTagSymbolSlot)
</span><span class="cx"> {
</span><span class="cx"> if (!hasRareData())
</span><span class="lines">@@ -311,4 +285,85 @@
</span><span class="cx"> rareData()->setObjectToStringValue(exec, vm, this, value, toStringTagSymbolSlot);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+template<typename Func>
+inline PropertyOffset Structure::add(VM& vm, PropertyName propertyName, unsigned attributes, const Func& func)
+{
+ PropertyTable* table = ensurePropertyTable(vm);
+
+ GCSafeConcurrentJITLocker locker(m_lock, vm.heap);
+
+ setPropertyTable(vm, table);
+
+ ASSERT(!JSC::isValidOffset(get(vm, propertyName)));
+
+ checkConsistency();
+ if (attributes & DontEnum || propertyName.isSymbol())
+ setIsQuickPropertyAccessAllowedForEnumeration(false);
+
+ auto rep = propertyName.uid();
+
+ PropertyOffset newOffset = table->nextOffset(m_inlineCapacity);
+
+ table->add(PropertyMapEntry(rep, newOffset, attributes), m_offset, PropertyTable::PropertyOffsetMayChange);
+
+ checkConsistency();
+
+ func(locker, newOffset);
+ return newOffset;
+}
+
+template<typename Func>
+inline PropertyOffset Structure::remove(PropertyName propertyName, const Func& func)
+{
+ ConcurrentJITLocker locker(m_lock);
+
+ checkConsistency();
+
+ auto rep = propertyName.uid();
+
+ // We ONLY remove from uncacheable dictionaries, which will have a pinned property table.
+ // The only way for them not to have a table is if they are empty.
+ PropertyTable* table = propertyTableOrNull();
+
+ if (!table)
+ return invalidOffset;
+
+ PropertyTable::find_iterator position = table->find(rep);
+ if (!position.first)
+ return invalidOffset;
+
+ PropertyOffset offset = position.first->offset;
+
+ table->remove(position);
+ table->addDeletedOffset(offset);
+
+ checkConsistency();
+
+ func(locker, offset);
+ return offset;
+}
+
+template<typename Func>
+inline PropertyOffset Structure::addPropertyWithoutTransition(VM& vm, PropertyName propertyName, unsigned attributes, const Func& func)
+{
+ pin(vm, ensurePropertyTable(vm));
+
+ return add(vm, propertyName, attributes, func);
+}
+
+template<typename Func>
+inline PropertyOffset Structure::removePropertyWithoutTransition(VM&, PropertyName propertyName, const Func& func)
+{
+ ASSERT(isUncacheableDictionary());
+ ASSERT(isPinnedPropertyTable());
+ ASSERT(propertyTableOrNull());
+
+ return remove(propertyName, func);
+}
+
+inline void Structure::setPropertyTable(VM& vm, PropertyTable* table)
+{
+ m_propertyTableUnsafe.setMayBeNull(vm, this, table);
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceWTFChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/ChangeLog (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/ChangeLog 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/ChangeLog 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -1,3 +1,71 @@
</span><ins>+2016-11-11 Filip Pizlo <fpizlo@apple.com>
+
+ The GC should be optionally concurrent and disabled by default
+ https://bugs.webkit.org/show_bug.cgi?id=164454
+
+ Reviewed by Geoffrey Garen.
+
+ The reason why I went to such great pains to make WTF::Lock fit in two bits is that I
+ knew that I would eventually need to stuff one into some miscellaneous bits of the
+ JSCell header. That time has come, because the concurrent GC has numerous race
+ conditions in visitChildren that can be trivially fixed if each object just has an
+ internal lock. Some cell types might use it to simply protect their entire visitChildren
+ function and anything that mutates the fields it touches, while other cell types might
+ use it as a "lock of last resort" to handle corner cases of an otherwise wait-free or
+ lock-free algorithm. Right now, it's used to protect certain transformations involving
+ indexing storage.
+
+ To make this happen, I factored the WTF::Lock algorithm into a LockAlgorithm struct that
+ is templatized on lock type (uint8_t for WTF::Lock), the isHeldBit value (1 for
+ WTF::Lock), and the hasParkedBit value (2 for WTF::Lock). This could have been done as
+ a templatized Lock class that basically contains Atomic<LockType>. You could then make
+ any field into a lock by bitwise_casting it to TemplateLock<field type, bit1, bit2>. But
+ this felt too dirty, so instead, LockAlgorithm has static methods that take
+ Atomic<LockType>& as their first argument. I think that this makes it more natural to
+ project a LockAlgorithm onto an existing Atomic<> field. Sadly, some places have to cast
+ their non-Atomic<> field to Atomic<> in order for this to work. Like so many other things
+ we do, this just shows that the C++ style of labeling fields that are subject to atomic
+ ops as atomic is counterproductive. Maybe some day I'll change LockAlgorithm to use our
+ other Atomics API, which does not require Atomic<>.
+
+ WTF::Lock now uses LockAlgorithm. The slow paths are still outlined. I don't feel too
+ bad about the LockAlgorithm.h header being included in so many places because we change
+ that algorithm so infrequently.
+
+ Also, I added a hasElapsed(time) function. This function makes it so much more natural
+ to write timeslicing code, which the concurrent GC has to do a lot of.
+
+ * WTF.xcodeproj/project.pbxproj:
+ * wtf/CMakeLists.txt:
+ * wtf/ListDump.h:
+ * wtf/Lock.cpp:
+ (WTF::LockBase::lockSlow):
+ (WTF::LockBase::unlockSlow):
+ (WTF::LockBase::unlockFairlySlow):
+ (WTF::LockBase::unlockSlowImpl): Deleted.
+ * wtf/Lock.h:
+ (WTF::LockBase::lock):
+ (WTF::LockBase::tryLock):
+ (WTF::LockBase::unlock):
+ (WTF::LockBase::unlockFairly):
+ (WTF::LockBase::isHeld):
+ (): Deleted.
+ * wtf/LockAlgorithm.h: Added.
+ (WTF::LockAlgorithm::lockFastAssumingZero):
+ (WTF::LockAlgorithm::lockFast):
+ (WTF::LockAlgorithm::lock):
+ (WTF::LockAlgorithm::tryLock):
+ (WTF::LockAlgorithm::unlockFastAssumingZero):
+ (WTF::LockAlgorithm::unlockFast):
+ (WTF::LockAlgorithm::unlock):
+ (WTF::LockAlgorithm::unlockFairly):
+ (WTF::LockAlgorithm::isLocked):
+ (WTF::LockAlgorithm::lockSlow):
+ (WTF::LockAlgorithm::unlockSlow):
+ * wtf/TimeWithDynamicClockType.cpp:
+ (WTF::hasElapsed):
+ * wtf/TimeWithDynamicClockType.h:
+
</ins><span class="cx"> 2016-11-14 JF Bastien <jfbastien@apple.com>
</span><span class="cx">
</span><span class="cx"> Expected: add missing `inline`
</span></span></pre></div>
<a id="trunkSourceWTFWTFxcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/WTF.xcodeproj/project.pbxproj (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/WTF.xcodeproj/project.pbxproj 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/WTF.xcodeproj/project.pbxproj 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -22,6 +22,7 @@
</span><span class="cx">
</span><span class="cx"> /* Begin PBXBuildFile section */
</span><span class="cx"> 0F0D85B417234CC100338210 /* NoLock.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0D85B317234CB100338210 /* NoLock.h */; };
</span><ins>+ 0F0FCDDE1DD167F900CCAB53 /* LockAlgorithm.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0FCDDD1DD167F900CCAB53 /* LockAlgorithm.h */; };
</ins><span class="cx"> 0F2B66A617B6B4FB00A7AE3F /* DeferrableRefCounted.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F2B66A417B6B4F700A7AE3F /* DeferrableRefCounted.h */; };
</span><span class="cx"> 0F2B66A717B6B4FD00A7AE3F /* FlipBytes.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F2B66A517B6B4F700A7AE3F /* FlipBytes.h */; };
</span><span class="cx"> 0F3501641BB258D500F0A2A3 /* WeakRandom.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3501631BB258C800F0A2A3 /* WeakRandom.h */; };
</span><span class="lines">@@ -375,6 +376,7 @@
</span><span class="cx">
</span><span class="cx"> /* Begin PBXFileReference section */
</span><span class="cx"> 0F0D85B317234CB100338210 /* NoLock.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = NoLock.h; sourceTree = "<group>"; };
</span><ins>+ 0F0FCDDD1DD167F900CCAB53 /* LockAlgorithm.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LockAlgorithm.h; sourceTree = "<group>"; };
</ins><span class="cx"> 0F2B66A417B6B4F700A7AE3F /* DeferrableRefCounted.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = DeferrableRefCounted.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F2B66A517B6B4F700A7AE3F /* FlipBytes.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = FlipBytes.h; sourceTree = "<group>"; };
</span><span class="cx"> 0F300B7D18AB48B400A6D72E /* HashMethod.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = HashMethod.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -956,6 +958,7 @@
</span><span class="cx"> A8A472C1151A825A004123FF /* ListHashSet.h */,
</span><span class="cx"> 0FE164681B6FFC9600400E7C /* Lock.cpp */,
</span><span class="cx"> 0FE164691B6FFC9600400E7C /* Lock.h */,
</span><ins>+ 0F0FCDDD1DD167F900CCAB53 /* LockAlgorithm.h */,
</ins><span class="cx"> A8A472C3151A825A004123FF /* Locker.h */,
</span><span class="cx"> 513E170A1CD7D5BF00E3650B /* LoggingAccumulator.h */,
</span><span class="cx"> A8A472C6151A825A004123FF /* MainThread.cpp */,
</span><span class="lines">@@ -1326,6 +1329,7 @@
</span><span class="cx"> A8A4739B151A825B004123FF /* CryptographicallyRandomNumber.h in Headers */,
</span><span class="cx"> E15556F618A0CC18006F48FB /* CryptographicUtilities.h in Headers */,
</span><span class="cx"> A8A4743A151A825B004123FF /* CString.h in Headers */,
</span><ins>+ 0F0FCDDE1DD167F900CCAB53 /* LockAlgorithm.h in Headers */,
</ins><span class="cx"> A8A4739D151A825B004123FF /* CurrentTime.h in Headers */,
</span><span class="cx"> A8A4739F151A825B004123FF /* DataLog.h in Headers */,
</span><span class="cx"> A8A473A1151A825B004123FF /* DateMath.h in Headers */,
</span></span></pre></div>
<a id="trunkSourceWTFwtfAtomicsh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/Atomics.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/Atomics.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/Atomics.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -106,7 +106,37 @@
</span><span class="cx"> ALWAYS_INLINE T exchangeXor(U operand, std::memory_order order = std::memory_order_seq_cst) { return value.fetch_xor(operand, order); }
</span><span class="cx">
</span><span class="cx"> ALWAYS_INLINE T exchange(T newValue, std::memory_order order = std::memory_order_seq_cst) { return value.exchange(newValue, order); }
</span><ins>+
+ template<typename Func>
+ ALWAYS_INLINE bool tryTransactionRelaxed(const Func& func)
+ {
+ T oldValue = load(std::memory_order_relaxed);
+ T newValue = oldValue;
+ func(newValue);
+ return compareExchangeWeakRelaxed(oldValue, newValue);
+ }
</ins><span class="cx">
</span><ins>+ template<typename Func>
+ ALWAYS_INLINE void transactionRelaxed(const Func& func)
+ {
+ while (!tryTransationRelaxed(func)) { }
+ }
+
+ template<typename Func>
+ ALWAYS_INLINE bool tryTransaction(const Func& func)
+ {
+ T oldValue = load(std::memory_order_relaxed);
+ T newValue = oldValue;
+ func(newValue);
+ return compareExchangeWeak(oldValue, newValue);
+ }
+
+ template<typename Func>
+ ALWAYS_INLINE void transaction(const Func& func)
+ {
+ while (!tryTransaction(func)) { }
+ }
+
</ins><span class="cx"> std::atomic<T> value;
</span><span class="cx"> };
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceWTFwtfCMakeListstxt"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/CMakeLists.txt (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/CMakeLists.txt 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/CMakeLists.txt 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -56,6 +56,7 @@
</span><span class="cx"> IteratorRange.h
</span><span class="cx"> ListHashSet.h
</span><span class="cx"> Lock.h
</span><ins>+ LockAlgorithm.h
</ins><span class="cx"> Locker.h
</span><span class="cx"> MD5.h
</span><span class="cx"> MainThread.h
</span></span></pre></div>
<a id="trunkSourceWTFwtfListDumph"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/ListDump.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/ListDump.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/ListDump.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -177,10 +177,11 @@
</span><span class="cx"> } // namespace WTF
</span><span class="cx">
</span><span class="cx"> using WTF::listDump;
</span><ins>+using WTF::listDumpInContext;
+using WTF::mapDump;
+using WTF::pointerListDump;
</ins><span class="cx"> using WTF::sortedListDump;
</span><del>-using WTF::mapDump;
</del><span class="cx"> using WTF::sortedMapDump;
</span><del>-using WTF::listDumpInContext;
</del><span class="cx">
</span><span class="cx"> #endif // ListDump_h
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceWTFwtfLockcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/Lock.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/Lock.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/Lock.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -26,121 +26,22 @@
</span><span class="cx"> #include "config.h"
</span><span class="cx"> #include "Lock.h"
</span><span class="cx">
</span><del>-#include "DataLog.h"
-#include "ParkingLot.h"
-#include "StringPrintStream.h"
-#include "ThreadingPrimitives.h"
-#include <thread>
-
</del><span class="cx"> namespace WTF {
</span><span class="cx">
</span><del>-static const bool verbose = false;
-
-NEVER_INLINE void LockBase::lockSlow()
</del><ins>+void LockBase::lockSlow()
</ins><span class="cx"> {
</span><del>- unsigned spinCount = 0;
-
- // This magic number turns out to be optimal based on past JikesRVM experiments.
- const unsigned spinLimit = 40;
-
- for (;;) {
- uint8_t currentByteValue = m_byte.load();
- if (verbose)
- dataLog(toString(currentThread(), ": locking with ", currentByteValue, "\n"));
-
- // We allow ourselves to barge in.
- if (!(currentByteValue & isHeldBit)
- && m_byte.compareExchangeWeak(currentByteValue, currentByteValue | isHeldBit))
- return;
-
- // If there is nobody parked and we haven't spun too much, we can just try to spin around.
- if (!(currentByteValue & hasParkedBit) && spinCount < spinLimit) {
- spinCount++;
- std::this_thread::yield();
- continue;
- }
-
- // Need to park. We do this by setting the parked bit first, and then parking. We spin around
- // if the parked bit wasn't set and we failed at setting it.
- if (!(currentByteValue & hasParkedBit)
- && !m_byte.compareExchangeWeak(currentByteValue, currentByteValue | hasParkedBit))
- continue;
-
- // We now expect the value to be isHeld|hasParked. So long as that's the case, we can park.
- ParkingLot::ParkResult parkResult =
- ParkingLot::compareAndPark(&m_byte, isHeldBit | hasParkedBit);
- if (parkResult.wasUnparked) {
- switch (static_cast<Token>(parkResult.token)) {
- case DirectHandoff:
- // The lock was never released. It was handed to us directly by the thread that did
- // unlock(). This means we're done!
- RELEASE_ASSERT(isHeld());
- return;
- case BargingOpportunity:
- // This is the common case. The thread that called unlock() has released the lock,
- // and we have been woken up so that we may get an opportunity to grab the lock. But
- // other threads may barge, so the best that we can do is loop around and try again.
- break;
- }
- }
-
- // We have awoken, or we never parked because the byte value changed. Either way, we loop
- // around and try again.
- }
</del><ins>+ DefaultLockAlgorithm::lockSlow(m_byte);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void LockBase::unlockSlow()
</span><span class="cx"> {
</span><del>- unlockSlowImpl(Unfair);
</del><ins>+ DefaultLockAlgorithm::unlockSlow(m_byte, DefaultLockAlgorithm::Unfair);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void LockBase::unlockFairlySlow()
</span><span class="cx"> {
</span><del>- unlockSlowImpl(Fair);
</del><ins>+ DefaultLockAlgorithm::unlockSlow(m_byte, DefaultLockAlgorithm::Fair);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-NEVER_INLINE void LockBase::unlockSlowImpl(Fairness fairness)
-{
- // We could get here because the weak CAS in unlock() failed spuriously, or because there is
- // someone parked. So, we need a CAS loop: even if right now the lock is just held, it could
- // be held and parked if someone attempts to lock just as we are unlocking.
- for (;;) {
- uint8_t oldByteValue = m_byte.load();
- RELEASE_ASSERT(oldByteValue == isHeldBit || oldByteValue == (isHeldBit | hasParkedBit));
-
- if (oldByteValue == isHeldBit) {
- if (m_byte.compareExchangeWeak(isHeldBit, 0))
- return;
- continue;
- }
-
- // Someone is parked. Unpark exactly one thread. We may hand the lock to that thread
- // directly, or we will unlock the lock at the same time as we unpark to allow for barging.
- // When we unlock, we may leave the parked bit set if there is a chance that there are still
- // other threads parked.
- ASSERT(oldByteValue == (isHeldBit | hasParkedBit));
- ParkingLot::unparkOne(
- &m_byte,
- [&] (ParkingLot::UnparkResult result) -> intptr_t {
- // We are the only ones that can clear either the isHeldBit or the hasParkedBit,
- // so we should still see both bits set right now.
- ASSERT(m_byte.load() == (isHeldBit | hasParkedBit));
-
- if (result.didUnparkThread && (fairness == Fair || result.timeToBeFair)) {
- // We don't unlock anything. Instead, we hand the lock to the thread that was
- // waiting.
- return DirectHandoff;
- }
-
- if (result.mayHaveMoreThreads)
- m_byte.store(hasParkedBit);
- else
- m_byte.store(0);
- return BargingOpportunity;
- });
- return;
- }
-}
-
</del><span class="cx"> } // namespace WTF
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceWTFwtfLockh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/Lock.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/Lock.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/Lock.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -26,8 +26,7 @@
</span><span class="cx"> #ifndef WTF_Lock_h
</span><span class="cx"> #define WTF_Lock_h
</span><span class="cx">
</span><del>-#include <wtf/Atomics.h>
-#include <wtf/Compiler.h>
</del><ins>+#include <wtf/LockAlgorithm.h>
</ins><span class="cx"> #include <wtf/Locker.h>
</span><span class="cx"> #include <wtf/Noncopyable.h>
</span><span class="cx">
</span><span class="lines">@@ -37,6 +36,8 @@
</span><span class="cx">
</span><span class="cx"> namespace WTF {
</span><span class="cx">
</span><ins>+typedef LockAlgorithm<uint8_t, 1, 2> DefaultLockAlgorithm;
+
</ins><span class="cx"> // This is a fully adaptive mutex that only requires 1 byte of storage. It has fast paths that are
</span><span class="cx"> // competetive to a spinlock (uncontended locking is inlined and is just a CAS, microcontention is
</span><span class="cx"> // handled by spinning and yielding), and a slow path that is competetive to std::mutex (if a lock
</span><span class="lines">@@ -53,23 +54,13 @@
</span><span class="cx"> struct LockBase {
</span><span class="cx"> void lock()
</span><span class="cx"> {
</span><del>- if (LIKELY(m_byte.compareExchangeWeak(0, isHeldBit, std::memory_order_acquire))) {
- // Lock acquired!
- return;
- }
-
- lockSlow();
</del><ins>+ if (UNLIKELY(!DefaultLockAlgorithm::lockFastAssumingZero(m_byte)))
+ lockSlow();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool tryLock()
</span><span class="cx"> {
</span><del>- for (;;) {
- uint8_t currentByteValue = m_byte.load();
- if (currentByteValue & isHeldBit)
- return false;
- if (m_byte.compareExchangeWeak(currentByteValue, currentByteValue | isHeldBit))
- return true;
- }
</del><ins>+ return DefaultLockAlgorithm::tryLock(m_byte);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // Need this version for std::unique_lock.
</span><span class="lines">@@ -88,12 +79,8 @@
</span><span class="cx"> // guarantees that long critical sections always get a fair lock.
</span><span class="cx"> void unlock()
</span><span class="cx"> {
</span><del>- if (LIKELY(m_byte.compareExchangeWeak(isHeldBit, 0, std::memory_order_release))) {
- // Lock released and nobody was waiting!
- return;
- }
-
- unlockSlow();
</del><ins>+ if (UNLIKELY(!DefaultLockAlgorithm::unlockFastAssumingZero(m_byte)))
+ unlockSlow();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // This is like unlock() but it guarantees that we unlock the lock fairly. For short critical
</span><span class="lines">@@ -103,17 +90,13 @@
</span><span class="cx"> // want.
</span><span class="cx"> void unlockFairly()
</span><span class="cx"> {
</span><del>- if (LIKELY(m_byte.compareExchangeWeak(isHeldBit, 0, std::memory_order_release))) {
- // Lock released and nobody was waiting!
- return;
- }
-
- unlockFairlySlow();
</del><ins>+ if (UNLIKELY(!DefaultLockAlgorithm::unlockFastAssumingZero(m_byte)))
+ unlockFairlySlow();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool isHeld() const
</span><span class="cx"> {
</span><del>- return m_byte.load(std::memory_order_acquire) & isHeldBit;
</del><ins>+ return DefaultLockAlgorithm::isLocked(m_byte);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool isLocked() const
</span><span class="lines">@@ -126,21 +109,10 @@
</span><span class="cx">
</span><span class="cx"> static const uint8_t isHeldBit = 1;
</span><span class="cx"> static const uint8_t hasParkedBit = 2;
</span><del>-
</del><ins>+
</ins><span class="cx"> WTF_EXPORT_PRIVATE void lockSlow();
</span><span class="cx"> WTF_EXPORT_PRIVATE void unlockSlow();
</span><span class="cx"> WTF_EXPORT_PRIVATE void unlockFairlySlow();
</span><del>-
- enum Fairness {
- Fair,
- Unfair
- };
- void unlockSlowImpl(Fairness);
-
- enum Token {
- BargingOpportunity,
- DirectHandoff
- };
</del><span class="cx">
</span><span class="cx"> // Method used for testing only.
</span><span class="cx"> bool isFullyReset() const
</span></span></pre></div>
<a id="trunkSourceWTFwtfLockAlgorithmh"></a>
<div class="addfile"><h4>Added: trunk/Source/WTF/wtf/LockAlgorithm.h (0 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/LockAlgorithm.h (rev 0)
+++ trunk/Source/WTF/wtf/LockAlgorithm.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -0,0 +1,218 @@
</span><ins>+/*
+ * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef WTF_LockAlgorithm_h
+#define WTF_LockAlgorithm_h
+
+#include <thread>
+#include <wtf/Atomics.h>
+#include <wtf/Compiler.h>
+#include <wtf/ParkingLot.h>
+
+namespace WTF {
+
+// This is the algorithm used by WTF::Lock.
+
+template<typename LockType, LockType isHeldBit, LockType hasParkedBit>
+class LockAlgorithm {
+ static const bool verbose = false;
+ static const LockType mask = isHeldBit | hasParkedBit;
+
+public:
+ static bool lockFastAssumingZero(Atomic<LockType>& lock)
+ {
+ return lock.compareExchangeWeak(0, isHeldBit, std::memory_order_acquire);
+ }
+
+ static bool lockFast(Atomic<LockType>& lock)
+ {
+ LockType oldValue = lock.load(std::memory_order_relaxed);
+ if (oldValue & isHeldBit)
+ return false;
+ return lock.compareExchangeWeak(oldValue, oldValue | isHeldBit, std::memory_order_acquire);
+ }
+
+ static void lock(Atomic<LockType>& lock)
+ {
+ if (UNLIKELY(!lockFast(lock)))
+ lockSlow(lock);
+ }
+
+ static bool tryLock(Atomic<LockType>& lock)
+ {
+ for (;;) {
+ uint8_t currentByteValue = lock.load(std::memory_order_relaxed);
+ if (currentByteValue & isHeldBit)
+ return false;
+ if (lock.compareExchangeWeak(currentByteValue, currentByteValue | isHeldBit, std::memory_order_acquire))
+ return true;
+ }
+ }
+
+ static bool unlockFastAssumingZero(Atomic<LockType>& lock)
+ {
+ return lock.compareExchangeWeak(isHeldBit, 0, std::memory_order_release);
+ }
+
+ static bool unlockFast(Atomic<LockType>& lock)
+ {
+ LockType oldValue = lock.load(std::memory_order_relaxed);
+ if ((oldValue & mask) != isHeldBit)
+ return false;
+ return lock.compareExchangeWeak(oldValue, oldValue & ~isHeldBit, std::memory_order_release);
+ }
+
+ static void unlock(Atomic<LockType>& lock)
+ {
+ if (UNLIKELY(!unlockFast(lock)))
+ unlockSlow(lock, Unfair);
+ }
+
+ static void unlockFairly(Atomic<LockType>& lock)
+ {
+ if (UNLIKELY(!unlockFast(lock)))
+ unlockSlow(lock, Fair);
+ }
+
+ static bool isLocked(const Atomic<LockType>& lock)
+ {
+ return lock.load(std::memory_order_acquire) & isHeldBit;
+ }
+
+ NEVER_INLINE static void lockSlow(Atomic<LockType>& lock)
+ {
+ unsigned spinCount = 0;
+
+ // This magic number turns out to be optimal based on past JikesRVM experiments.
+ const unsigned spinLimit = 40;
+
+ for (;;) {
+ uint8_t currentByteValue = lock.load();
+
+ // We allow ourselves to barge in.
+ if (!(currentByteValue & isHeldBit)
+ && lock.compareExchangeWeak(currentByteValue, currentByteValue | isHeldBit))
+ return;
+
+ // If there is nobody parked and we haven't spun too much, we can just try to spin around.
+ if (!(currentByteValue & hasParkedBit) && spinCount < spinLimit) {
+ spinCount++;
+ std::this_thread::yield();
+ continue;
+ }
+
+ // Need to park. We do this by setting the parked bit first, and then parking. We spin around
+ // if the parked bit wasn't set and we failed at setting it.
+ if (!(currentByteValue & hasParkedBit)
+ && !lock.compareExchangeWeak(currentByteValue, currentByteValue | hasParkedBit))
+ continue;
+
+ // We now expect the value to be isHeld|hasParked. So long as that's the case, we can park.
+ ParkingLot::ParkResult parkResult =
+ ParkingLot::compareAndPark(&lock, currentByteValue | isHeldBit | hasParkedBit);
+ if (parkResult.wasUnparked) {
+ switch (static_cast<Token>(parkResult.token)) {
+ case DirectHandoff:
+ // The lock was never released. It was handed to us directly by the thread that did
+ // unlock(). This means we're done!
+ RELEASE_ASSERT(isLocked(lock));
+ return;
+ case BargingOpportunity:
+ // This is the common case. The thread that called unlock() has released the lock,
+ // and we have been woken up so that we may get an opportunity to grab the lock. But
+ // other threads may barge, so the best that we can do is loop around and try again.
+ break;
+ }
+ }
+
+ // We have awoken, or we never parked because the byte value changed. Either way, we loop
+ // around and try again.
+ }
+ }
+
+ enum Fairness {
+ Fair,
+ Unfair
+ };
+ NEVER_INLINE static void unlockSlow(Atomic<LockType>& lock, Fairness fairness)
+ {
+ // We could get here because the weak CAS in unlock() failed spuriously, or because there is
+ // someone parked. So, we need a CAS loop: even if right now the lock is just held, it could
+ // be held and parked if someone attempts to lock just as we are unlocking.
+ for (;;) {
+ uint8_t oldByteValue = lock.load();
+ RELEASE_ASSERT(
+ (oldByteValue & mask) == isHeldBit
+ || (oldByteValue & mask) == (isHeldBit | hasParkedBit));
+
+ if ((oldByteValue & mask) == isHeldBit) {
+ if (lock.compareExchangeWeak(oldByteValue, oldByteValue & ~isHeldBit))
+ return;
+ continue;
+ }
+
+ // Someone is parked. Unpark exactly one thread. We may hand the lock to that thread
+ // directly, or we will unlock the lock at the same time as we unpark to allow for barging.
+ // When we unlock, we may leave the parked bit set if there is a chance that there are still
+ // other threads parked.
+ ASSERT((oldByteValue & mask) == (isHeldBit | hasParkedBit));
+ ParkingLot::unparkOne(
+ &lock,
+ [&] (ParkingLot::UnparkResult result) -> intptr_t {
+ // We are the only ones that can clear either the isHeldBit or the hasParkedBit,
+ // so we should still see both bits set right now.
+ ASSERT((lock.load() & mask) == (isHeldBit | hasParkedBit));
+
+ if (result.didUnparkThread && (fairness == Fair || result.timeToBeFair)) {
+ // We don't unlock anything. Instead, we hand the lock to the thread that was
+ // waiting.
+ return DirectHandoff;
+ }
+
+ lock.transaction(
+ [&] (LockType& value) {
+ value &= ~mask;
+ if (result.mayHaveMoreThreads)
+ value |= hasParkedBit;
+ });
+ return BargingOpportunity;
+ });
+ return;
+ }
+ }
+
+private:
+ enum Token {
+ BargingOpportunity,
+ DirectHandoff
+ };
+};
+
+} // namespace WTF
+
+using WTF::LockAlgorithm;
+
+#endif // WTF_LockAlgorithm_h
+
</ins></span></pre></div>
<a id="trunkSourceWTFwtfTimeWithDynamicClockTypecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/TimeWithDynamicClockType.cpp (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/TimeWithDynamicClockType.cpp 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/TimeWithDynamicClockType.cpp 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -29,6 +29,9 @@
</span><span class="cx"> #include "Condition.h"
</span><span class="cx"> #include "Lock.h"
</span><span class="cx"> #include "PrintStream.h"
</span><ins>+#include <cfloat>
+#include <cmath>
+#include <wtf/DataLog.h>
</ins><span class="cx">
</span><span class="cx"> namespace WTF {
</span><span class="cx">
</span><span class="lines">@@ -128,6 +131,17 @@
</span><span class="cx"> fakeCondition.waitUntil(fakeLock, time);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+bool hasElapsed(const TimeWithDynamicClockType& time)
+{
+ // Avoid doing now().
+ if (!(time > time.withSameClockAndRawSeconds(0)))
+ return true;
+ if (std::isinf(time.secondsSinceEpoch().value()))
+ return false;
+
+ return time <= time.nowWithSameClock();
+}
+
</ins><span class="cx"> } // namespace WTF
</span><span class="cx">
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceWTFwtfTimeWithDynamicClockTypeh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/TimeWithDynamicClockType.h (208719 => 208720)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/TimeWithDynamicClockType.h 2016-11-15 01:09:33 UTC (rev 208719)
+++ trunk/Source/WTF/wtf/TimeWithDynamicClockType.h 2016-11-15 01:49:22 UTC (rev 208720)
</span><span class="lines">@@ -134,8 +134,12 @@
</span><span class="cx">
</span><span class="cx"> WTF_EXPORT_PRIVATE void sleep(const TimeWithDynamicClockType&);
</span><span class="cx">
</span><ins>+WTF_EXPORT_PRIVATE bool hasElapsed(const TimeWithDynamicClockType&);
+
</ins><span class="cx"> } // namespace WTF
</span><span class="cx">
</span><span class="cx"> using WTF::TimeWithDynamicClockType;
</span><ins>+using WTF::hasElapsed;
+using WTF::sleep;
</ins><span class="cx">
</span><span class="cx"> #endif // WTF_TimeWithDynamicClockType_h
</span></span></pre>
</div>
</div>
</body>
</html>