<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[211448] trunk/Source</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/211448">211448</a></dd>
<dt>Author</dt> <dd>fpizlo@apple.com</dd>
<dt>Date</dt> <dd>2017-01-31 14:31:24 -0800 (Tue, 31 Jan 2017)</dd>
</dl>
<h3>Log Message</h3>
<pre>The mutator should be able to perform increments of GC work
https://bugs.webkit.org/show_bug.cgi?id=167528
Reviewed by Keith Miller and Geoffrey Garen.
Source/JavaScriptCore:
The cool thing about having a concurrent and parallel collector is that it's easy to also make
it incremental, because the load balancer can also hand over work to anyone (including the
mutator) and since the collector is running concurrently anyway, the mutator can usually rely
on the balancer having some spare work.
This change adds a classic work-based incremental mode to the GC. When you allocate K bytes,
you have to do Options::gcIncrementScale() * K "bytes" of draining. This is ammortized so that
it only happens in allocation slow paths.
On computers that have a lot of CPUs, this mode is not profitable and we set gcIncrementScale
to zero. On such computers, Riptide was already performing great because there was no way that
one mutator thread could outpace many GC threads. But on computers with fewer CPUs, there were
problems having to do with making the collector progress quickly enough so that the heap
doesn't grow too much. The stochastic scheduler actually made things worse, because it relies
a lot on the fact that the GC will simply be faster than the mutator anyway. The old scheduler
claimed to address the problem of GC pace, but it used a time-based scheduler, which is not as
precise at keeping pase as the new work-based incremental mode.
In theory, the work-based mode guarantees a bound on how much the heap can grow during a
collection just because each byte allocated means some number of bytes visited. We don't try
to create such a theoretical bound. We're just trying to give the collector an unfair advantage
in any race with the mutator.
Turning on incremental mode, the stochastic scheduler, and passive draining in combination with
each other is a huge splay-latency speed-up on my iPad. It's also a CDjs progression. It does
regress splay-throughput, but I think that's fine (the regression is 11%, the progression is
3x).
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::~Heap):
(JSC::Heap::markToFixpoint):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::endMarking):
(JSC::Heap::finalize):
(JSC::Heap::didAllocate):
(JSC::Heap::visitCount):
(JSC::Heap::bytesVisited):
(JSC::Heap::forEachSlotVisitor):
(JSC::Heap::performIncrement):
(JSC::Heap::threadVisitCount): Deleted.
(JSC::Heap::threadBytesVisited): Deleted.
* heap/Heap.h:
* heap/MarkStack.cpp:
(JSC::MarkStackArray::transferTo):
* heap/MarkStack.h:
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::clearMarkStacks):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::donateKnownParallel):
(JSC::SlotVisitor::drain):
(JSC::SlotVisitor::performIncrementOfDraining):
(JSC::SlotVisitor::didReachTermination):
(JSC::SlotVisitor::hasWork):
(JSC::SlotVisitor::drainFromShared):
(JSC::SlotVisitor::drainInParallelPassively):
(JSC::SlotVisitor::donateAll):
(JSC::SlotVisitor::correspondingGlobalStack):
* heap/SlotVisitor.h:
* heap/SlotVisitorInlines.h:
(JSC::SlotVisitor::reportExtraMemoryVisited):
(JSC::SlotVisitor::forEachMarkStack):
* heap/SpaceTimeMutatorScheduler.cpp:
(JSC::SpaceTimeMutatorScheduler::log):
* heap/StochasticSpaceTimeMutatorScheduler.cpp:
(JSC::StochasticSpaceTimeMutatorScheduler::log):
* jsc.cpp:
(GlobalObject::finishCreation):
(functionHeapCapacity):
* runtime/Options.cpp:
(JSC::overrideDefaults):
* runtime/Options.h:
Source/WTF:
We want dataLog to be locked even if you're not logging to a file!
* wtf/DataLog.cpp:
(WTF::initializeLogFileOnce):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeapcpp">trunk/Source/JavaScriptCore/heap/Heap.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapHeaph">trunk/Source/JavaScriptCore/heap/Heap.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkStackcpp">trunk/Source/JavaScriptCore/heap/MarkStack.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapMarkStackh">trunk/Source/JavaScriptCore/heap/MarkStack.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorcpp">trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorh">trunk/Source/JavaScriptCore/heap/SlotVisitor.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSlotVisitorInlinesh">trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapSpaceTimeMutatorSchedulercpp">trunk/Source/JavaScriptCore/heap/SpaceTimeMutatorScheduler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreheapStochasticSpaceTimeMutatorSchedulercpp">trunk/Source/JavaScriptCore/heap/StochasticSpaceTimeMutatorScheduler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejsccpp">trunk/Source/JavaScriptCore/jsc.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeOptionscpp">trunk/Source/JavaScriptCore/runtime/Options.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreruntimeOptionsh">trunk/Source/JavaScriptCore/runtime/Options.h</a></li>
<li><a href="#trunkSourceWTFChangeLog">trunk/Source/WTF/ChangeLog</a></li>
<li><a href="#trunkSourceWTFwtfDataLogcpp">trunk/Source/WTF/wtf/DataLog.cpp</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/ChangeLog 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -1,3 +1,85 @@
</span><ins>+2017-01-31 Filip Pizlo <fpizlo@apple.com>
+
+ The mutator should be able to perform increments of GC work
+ https://bugs.webkit.org/show_bug.cgi?id=167528
+
+ Reviewed by Keith Miller and Geoffrey Garen.
+
+ The cool thing about having a concurrent and parallel collector is that it's easy to also make
+ it incremental, because the load balancer can also hand over work to anyone (including the
+ mutator) and since the collector is running concurrently anyway, the mutator can usually rely
+ on the balancer having some spare work.
+
+ This change adds a classic work-based incremental mode to the GC. When you allocate K bytes,
+ you have to do Options::gcIncrementScale() * K "bytes" of draining. This is ammortized so that
+ it only happens in allocation slow paths.
+
+ On computers that have a lot of CPUs, this mode is not profitable and we set gcIncrementScale
+ to zero. On such computers, Riptide was already performing great because there was no way that
+ one mutator thread could outpace many GC threads. But on computers with fewer CPUs, there were
+ problems having to do with making the collector progress quickly enough so that the heap
+ doesn't grow too much. The stochastic scheduler actually made things worse, because it relies
+ a lot on the fact that the GC will simply be faster than the mutator anyway. The old scheduler
+ claimed to address the problem of GC pace, but it used a time-based scheduler, which is not as
+ precise at keeping pase as the new work-based incremental mode.
+
+ In theory, the work-based mode guarantees a bound on how much the heap can grow during a
+ collection just because each byte allocated means some number of bytes visited. We don't try
+ to create such a theoretical bound. We're just trying to give the collector an unfair advantage
+ in any race with the mutator.
+
+ Turning on incremental mode, the stochastic scheduler, and passive draining in combination with
+ each other is a huge splay-latency speed-up on my iPad. It's also a CDjs progression. It does
+ regress splay-throughput, but I think that's fine (the regression is 11%, the progression is
+ 3x).
+
+ * heap/Heap.cpp:
+ (JSC::Heap::Heap):
+ (JSC::Heap::~Heap):
+ (JSC::Heap::markToFixpoint):
+ (JSC::Heap::updateObjectCounts):
+ (JSC::Heap::endMarking):
+ (JSC::Heap::finalize):
+ (JSC::Heap::didAllocate):
+ (JSC::Heap::visitCount):
+ (JSC::Heap::bytesVisited):
+ (JSC::Heap::forEachSlotVisitor):
+ (JSC::Heap::performIncrement):
+ (JSC::Heap::threadVisitCount): Deleted.
+ (JSC::Heap::threadBytesVisited): Deleted.
+ * heap/Heap.h:
+ * heap/MarkStack.cpp:
+ (JSC::MarkStackArray::transferTo):
+ * heap/MarkStack.h:
+ * heap/SlotVisitor.cpp:
+ (JSC::SlotVisitor::didStartMarking):
+ (JSC::SlotVisitor::clearMarkStacks):
+ (JSC::SlotVisitor::appendToMarkStack):
+ (JSC::SlotVisitor::noteLiveAuxiliaryCell):
+ (JSC::SlotVisitor::donateKnownParallel):
+ (JSC::SlotVisitor::drain):
+ (JSC::SlotVisitor::performIncrementOfDraining):
+ (JSC::SlotVisitor::didReachTermination):
+ (JSC::SlotVisitor::hasWork):
+ (JSC::SlotVisitor::drainFromShared):
+ (JSC::SlotVisitor::drainInParallelPassively):
+ (JSC::SlotVisitor::donateAll):
+ (JSC::SlotVisitor::correspondingGlobalStack):
+ * heap/SlotVisitor.h:
+ * heap/SlotVisitorInlines.h:
+ (JSC::SlotVisitor::reportExtraMemoryVisited):
+ (JSC::SlotVisitor::forEachMarkStack):
+ * heap/SpaceTimeMutatorScheduler.cpp:
+ (JSC::SpaceTimeMutatorScheduler::log):
+ * heap/StochasticSpaceTimeMutatorScheduler.cpp:
+ (JSC::StochasticSpaceTimeMutatorScheduler::log):
+ * jsc.cpp:
+ (GlobalObject::finishCreation):
+ (functionHeapCapacity):
+ * runtime/Options.cpp:
+ (JSC::overrideDefaults):
+ * runtime/Options.h:
+
</ins><span class="cx"> 2017-01-31 Tomas Popela <tpopela@redhat.com>
</span><span class="cx">
</span><span class="cx"> Compilation error in JSArrayBufferView.h
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/Heap.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -259,6 +259,7 @@
</span><span class="cx"> , m_deprecatedExtraMemorySize(0)
</span><span class="cx"> , m_machineThreads(this)
</span><span class="cx"> , m_collectorSlotVisitor(std::make_unique<SlotVisitor>(*this))
</span><ins>+ , m_mutatorSlotVisitor(std::make_unique<SlotVisitor>(*this))
</ins><span class="cx"> , m_mutatorMarkStack(std::make_unique<MarkStackArray>())
</span><span class="cx"> , m_raceMarkStack(std::make_unique<MarkStackArray>())
</span><span class="cx"> , m_constraintSet(std::make_unique<MarkingConstraintSet>())
</span><span class="lines">@@ -305,7 +306,7 @@
</span><span class="cx"> m_verifier = std::make_unique<HeapVerifier>(this, Options::numberOfGCCyclesToRecordForVerification());
</span><span class="cx">
</span><span class="cx"> m_collectorSlotVisitor->optimizeForStoppedMutator();
</span><del>-
</del><ins>+
</ins><span class="cx"> LockHolder locker(*m_threadLock);
</span><span class="cx"> m_thread = adoptRef(new Thread(locker, *this));
</span><span class="cx"> }
</span><span class="lines">@@ -312,9 +313,10 @@
</span><span class="cx">
</span><span class="cx"> Heap::~Heap()
</span><span class="cx"> {
</span><del>- for (auto& slotVisitor : m_parallelSlotVisitors)
- slotVisitor->clearMarkStacks();
- m_collectorSlotVisitor->clearMarkStacks();
</del><ins>+ forEachSlotVisitor(
+ [&] (SlotVisitor& visitor) {
+ visitor.clearMarkStacks();
+ });
</ins><span class="cx"> m_mutatorMarkStack->clear();
</span><span class="cx"> m_raceMarkStack->clear();
</span><span class="cx">
</span><span class="lines">@@ -537,6 +539,11 @@
</span><span class="cx">
</span><span class="cx"> beginMarking();
</span><span class="cx">
</span><ins>+ forEachSlotVisitor(
+ [&] (SlotVisitor& visitor) {
+ visitor.didStartMarking();
+ });
+
</ins><span class="cx"> m_parallelMarkersShouldExit = false;
</span><span class="cx">
</span><span class="cx"> m_helperClient.setFunction(
</span><span class="lines">@@ -551,6 +558,8 @@
</span><span class="cx"> if (Options::optimizeParallelSlotVisitorsForStoppedMutator())
</span><span class="cx"> newVisitor->optimizeForStoppedMutator();
</span><span class="cx">
</span><ins>+ newVisitor->didStartMarking();
+
</ins><span class="cx"> slotVisitor = newVisitor.get();
</span><span class="cx"> m_parallelSlotVisitors.append(WTFMove(newVisitor));
</span><span class="cx"> } else
</span><span class="lines">@@ -561,7 +570,6 @@
</span><span class="cx">
</span><span class="cx"> {
</span><span class="cx"> ParallelModeEnabler parallelModeEnabler(*slotVisitor);
</span><del>- slotVisitor->didStartMarking();
</del><span class="cx"> slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -572,7 +580,7 @@
</span><span class="cx"> });
</span><span class="cx">
</span><span class="cx"> SlotVisitor& slotVisitor = *m_collectorSlotVisitor;
</span><del>- slotVisitor.didStartMarking();
</del><ins>+
</ins><span class="cx"> m_constraintSet->didStartMarking();
</span><span class="cx">
</span><span class="cx"> m_scheduler->beginCollection();
</span><span class="lines">@@ -653,6 +661,10 @@
</span><span class="cx"> double thisPauseMS = (MonotonicTime::now() - m_stopTime).milliseconds();
</span><span class="cx"> dataLog("p=", thisPauseMS, "ms (max ", maxPauseMS(thisPauseMS), ")...]\n");
</span><span class="cx"> }
</span><ins>+
+ // Forgive the mutator for its past failures to keep up.
+ // FIXME: Figure out if moving this to different places results in perf changes.
+ m_incrementBalance = 0;
</ins><span class="cx">
</span><span class="cx"> resumeTheWorld();
</span><span class="cx">
</span><span class="lines">@@ -795,9 +807,7 @@
</span><span class="cx"> void Heap::updateObjectCounts(double gcStartTime)
</span><span class="cx"> {
</span><span class="cx"> if (Options::logGC() == GCLogging::Verbose) {
</span><del>- size_t visitCount = m_collectorSlotVisitor->visitCount();
- visitCount += threadVisitCount();
- dataLogF("\nNumber of live Objects after GC %lu, took %.6f secs\n", static_cast<unsigned long>(visitCount), WTF::monotonicallyIncreasingTime() - gcStartTime);
</del><ins>+ dataLogF("\nNumber of live Objects after GC %lu, took %.6f secs\n", static_cast<unsigned long>(visitCount()), WTF::monotonicallyIncreasingTime() - gcStartTime);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> if (m_collectionScope == CollectionScope::Full)
</span><span class="lines">@@ -810,11 +820,11 @@
</span><span class="cx">
</span><span class="cx"> void Heap::endMarking()
</span><span class="cx"> {
</span><del>- m_collectorSlotVisitor->reset();
</del><ins>+ forEachSlotVisitor(
+ [&] (SlotVisitor& visitor) {
+ visitor.reset();
+ });
</ins><span class="cx">
</span><del>- for (auto& parallelVisitor : m_parallelSlotVisitors)
- parallelVisitor->reset();
-
</del><span class="cx"> assertSharedMarkStacksEmpty();
</span><span class="cx"> m_weakReferenceHarvesters.removeAll();
</span><span class="cx">
</span><span class="lines">@@ -1570,6 +1580,12 @@
</span><span class="cx">
</span><span class="cx"> void Heap::finalize()
</span><span class="cx"> {
</span><ins>+ MonotonicTime before;
+ if (Options::logGC()) {
+ before = MonotonicTime::now();
+ dataLog("[GC: finalize ");
+ }
+
</ins><span class="cx"> {
</span><span class="cx"> HelpingGCScope helpingGCScope(*this);
</span><span class="cx"> deleteUnmarkedCompiledCode();
</span><span class="lines">@@ -1579,6 +1595,11 @@
</span><span class="cx">
</span><span class="cx"> if (HasOwnPropertyCache* cache = vm()->hasOwnPropertyCache())
</span><span class="cx"> cache->clear();
</span><ins>+
+ if (Options::logGC()) {
+ MonotonicTime after = MonotonicTime::now();
+ dataLog((after - before).milliseconds(), "ms]\n");
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Heap::Ticket Heap::requestCollection(std::optional<CollectionScope> scope)
</span><span class="lines">@@ -1871,6 +1892,7 @@
</span><span class="cx"> if (m_edenActivityCallback)
</span><span class="cx"> m_edenActivityCallback->didAllocate(m_bytesAllocatedThisCycle + m_bytesAbandonedSinceLastFullCollect);
</span><span class="cx"> m_bytesAllocatedThisCycle += bytes;
</span><ins>+ performIncrement(bytes);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> bool Heap::isValidAllocation(size_t)
</span><span class="lines">@@ -1996,24 +2018,23 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-size_t Heap::threadVisitCount()
-{
- unsigned long result = 0;
- for (auto& parallelVisitor : m_parallelSlotVisitors)
- result += parallelVisitor->visitCount();
</del><ins>+size_t Heap::visitCount()
+{
+ size_t result = 0;
+ forEachSlotVisitor(
+ [&] (SlotVisitor& visitor) {
+ result += visitor.visitCount();
+ });
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> size_t Heap::bytesVisited()
</span><span class="cx"> {
</span><del>- return m_collectorSlotVisitor->bytesVisited() + threadBytesVisited();
-}
-
-size_t Heap::threadBytesVisited()
-{
</del><span class="cx"> size_t result = 0;
</span><del>- for (auto& parallelVisitor : m_parallelSlotVisitors)
- result += parallelVisitor->bytesVisited();
</del><ins>+ forEachSlotVisitor(
+ [&] (SlotVisitor& visitor) {
+ result += visitor.bytesVisited();
+ });
</ins><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -2374,6 +2395,7 @@
</span><span class="cx"> {
</span><span class="cx"> auto locker = holdLock(m_parallelSlotVisitorLock);
</span><span class="cx"> func(*m_collectorSlotVisitor);
</span><ins>+ func(*m_mutatorSlotVisitor);
</ins><span class="cx"> for (auto& slotVisitor : m_parallelSlotVisitors)
</span><span class="cx"> func(*slotVisitor);
</span><span class="cx"> }
</span><span class="lines">@@ -2383,5 +2405,43 @@
</span><span class="cx"> m_mutatorShouldBeFenced = value;
</span><span class="cx"> m_barrierThreshold = value ? tautologicalThreshold : blackThreshold;
</span><span class="cx"> }
</span><ins>+
+void Heap::performIncrement(size_t bytes)
+{
+ if (!m_objectSpace.isMarking())
+ return;
+
+ m_incrementBalance += bytes * Options::gcIncrementScale();
+
+ // Save ourselves from crazy. Since this is an optimization, it's OK to go back to any consistent
+ // state when the double goes wild.
+ if (std::isnan(m_incrementBalance) || std::isinf(m_incrementBalance))
+ m_incrementBalance = 0;
</ins><span class="cx">
</span><ins>+ if (m_incrementBalance < static_cast<double>(Options::gcIncrementBytes()))
+ return;
+
+ double targetBytes = m_incrementBalance;
+ if (targetBytes <= 0)
+ return;
+ targetBytes = std::min(targetBytes, Options::gcIncrementMaxBytes());
+
+ MonotonicTime before;
+ if (Options::logGC()) {
+ dataLog("[GC: increment t=", targetBytes / 1024, "kb ");
+ before = MonotonicTime::now();
+ }
+
+ SlotVisitor& slotVisitor = *m_mutatorSlotVisitor;
+ ParallelModeEnabler parallelModeEnabler(slotVisitor);
+ size_t bytesVisited = slotVisitor.performIncrementOfDraining(static_cast<size_t>(targetBytes));
+ // incrementBalance may go negative here because it'll remember how many bytes we overshot.
+ m_incrementBalance -= bytesVisited;
+
+ if (Options::logGC()) {
+ MonotonicTime after = MonotonicTime::now();
+ dataLog("p=", (after - before).milliseconds(), "ms b=", m_incrementBalance / 1024, "kb]\n");
+ }
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/Heap.h (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/Heap.h 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/Heap.h 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -325,6 +325,8 @@
</span><span class="cx"> void stopIfNecessary();
</span><span class="cx">
</span><span class="cx"> bool mayNeedToStop();
</span><ins>+
+ void performIncrement(size_t bytes);
</ins><span class="cx">
</span><span class="cx"> // This is a much stronger kind of stopping of the collector, and it may require waiting for a
</span><span class="cx"> // while. This is meant to be a legacy API for clients of collectAllGarbage that expect that there
</span><span class="lines">@@ -336,8 +338,6 @@
</span><span class="cx"> void preventCollection();
</span><span class="cx"> void allowCollection();
</span><span class="cx">
</span><del>- size_t bytesVisited();
-
</del><span class="cx"> uint64_t mutatorExecutionVersion() const { return m_mutatorExecutionVersion; }
</span><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE void addMarkingConstraint(std::unique_ptr<MarkingConstraint>);
</span><span class="lines">@@ -466,8 +466,8 @@
</span><span class="cx"> void decrementDeferralDepthAndGCIfNeeded();
</span><span class="cx"> JS_EXPORT_PRIVATE void decrementDeferralDepthAndGCIfNeededSlow();
</span><span class="cx">
</span><del>- size_t threadVisitCount();
- size_t threadBytesVisited();
</del><ins>+ size_t visitCount();
+ size_t bytesVisited();
</ins><span class="cx">
</span><span class="cx"> void forEachCodeBlockImpl(const ScopedLambda<bool(CodeBlock*)>&);
</span><span class="cx">
</span><span class="lines">@@ -499,6 +499,7 @@
</span><span class="cx"> bool m_shouldDoFullCollection;
</span><span class="cx"> size_t m_totalBytesVisited;
</span><span class="cx"> size_t m_totalBytesVisitedThisCycle;
</span><ins>+ double m_incrementBalance { 0 };
</ins><span class="cx">
</span><span class="cx"> std::optional<CollectionScope> m_collectionScope;
</span><span class="cx"> std::optional<CollectionScope> m_lastCollectionScope;
</span><span class="lines">@@ -517,6 +518,7 @@
</span><span class="cx"> MachineThreads m_machineThreads;
</span><span class="cx">
</span><span class="cx"> std::unique_ptr<SlotVisitor> m_collectorSlotVisitor;
</span><ins>+ std::unique_ptr<SlotVisitor> m_mutatorSlotVisitor;
</ins><span class="cx"> std::unique_ptr<MarkStackArray> m_mutatorMarkStack;
</span><span class="cx">
</span><span class="cx"> Lock m_raceMarkStackLock;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkStackcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkStack.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkStack.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/MarkStack.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009-2016 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009-2017 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -64,6 +64,20 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+size_t MarkStackArray::transferTo(MarkStackArray& other, size_t limit)
+{
+ size_t count = 0;
+ while (count < limit && !isEmpty()) {
+ refill();
+ while (count < limit && canRemoveLast()) {
+ other.append(removeLast());
+ count++;
+ }
+ }
+ RELEASE_ASSERT(count <= limit);
+ return count;
+}
+
</ins><span class="cx"> void MarkStackArray::donateSomeCellsTo(MarkStackArray& other)
</span><span class="cx"> {
</span><span class="cx"> // Try to donate about 1 / 2 of our cells. To reduce copying costs,
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapMarkStackh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/MarkStack.h (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/MarkStack.h 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/MarkStack.h 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -1,5 +1,5 @@
</span><span class="cx"> /*
</span><del>- * Copyright (C) 2009-2016 Apple Inc. All rights reserved.
</del><ins>+ * Copyright (C) 2009-2017 Apple Inc. All rights reserved.
</ins><span class="cx"> *
</span><span class="cx"> * Redistribution and use in source and binary forms, with or without
</span><span class="cx"> * modification, are permitted provided that the following conditions
</span><span class="lines">@@ -36,6 +36,7 @@
</span><span class="cx"> MarkStackArray();
</span><span class="cx">
</span><span class="cx"> void transferTo(MarkStackArray&);
</span><ins>+ size_t transferTo(MarkStackArray&, size_t limit); // Optimized for when `limit` is small.
</ins><span class="cx"> void donateSomeCellsTo(MarkStackArray&);
</span><span class="cx"> void stealSomeCellsFrom(MarkStackArray&, size_t idleThreadCount);
</span><span class="cx"> };
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -96,7 +96,7 @@
</span><span class="cx"> void SlotVisitor::didStartMarking()
</span><span class="cx"> {
</span><span class="cx"> if (heap()->collectionScope() == CollectionScope::Full)
</span><del>- ASSERT(m_opaqueRoots.isEmpty()); // Should have merged by now.
</del><ins>+ RELEASE_ASSERT(m_opaqueRoots.isEmpty()); // Should have merged by now.
</ins><span class="cx"> else
</span><span class="cx"> reset();
</span><span class="cx">
</span><span class="lines">@@ -117,8 +117,11 @@
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::clearMarkStacks()
</span><span class="cx"> {
</span><del>- m_collectorStack.clear();
- m_mutatorStack.clear();
</del><ins>+ forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ stack.clear();
+ return IterationStatus::Continue;
+ });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::append(ConservativeRoots& conservativeRoots)
</span><span class="lines">@@ -290,7 +293,7 @@
</span><span class="cx">
</span><span class="cx"> m_visitCount++;
</span><span class="cx"> m_bytesVisited += container.cellSize();
</span><del>-
</del><ins>+
</ins><span class="cx"> m_collectorStack.append(cell);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -325,7 +328,10 @@
</span><span class="cx"> container.noteMarked();
</span><span class="cx">
</span><span class="cx"> m_visitCount++;
</span><del>- m_bytesVisited += container.cellSize();
</del><ins>+
+ size_t cellSize = container.cellSize();
+ m_bytesVisited += cellSize;
+ m_nonCellVisitCount += cellSize;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> class SetCurrentCellScope {
</span><span class="lines">@@ -429,8 +435,11 @@
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::donateKnownParallel()
</span><span class="cx"> {
</span><del>- donateKnownParallel(m_collectorStack, *m_heap.m_sharedCollectorMarkStack);
- donateKnownParallel(m_mutatorStack, *m_heap.m_sharedMutatorMarkStack);
</del><ins>+ forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ donateKnownParallel(stack, correspondingGlobalStack(stack));
+ return IterationStatus::Continue;
+ });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::updateMutatorIsStopped(const AbstractLocker&)
</span><span class="lines">@@ -462,27 +471,28 @@
</span><span class="cx">
</span><span class="cx"> void SlotVisitor::drain(MonotonicTime timeout)
</span><span class="cx"> {
</span><del>- ASSERT(m_isInParallelMode);
</del><ins>+ RELEASE_ASSERT(m_isInParallelMode);
</ins><span class="cx">
</span><span class="cx"> auto locker = holdLock(m_rightToRun);
</span><span class="cx">
</span><span class="cx"> while (!hasElapsed(timeout)) {
</span><span class="cx"> updateMutatorIsStopped(locker);
</span><del>- if (!m_collectorStack.isEmpty()) {
- m_collectorStack.refill();
- m_isFirstVisit = true;
- for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); m_collectorStack.canRemoveLast() && countdown--;)
- visitChildren(m_collectorStack.removeLast());
- } else if (!m_mutatorStack.isEmpty()) {
- m_mutatorStack.refill();
- // We know for sure that we are visiting objects because of the barrier, not because of
- // marking. Marking will visit an object exactly once. The barrier will visit it
- // possibly many times, and always after it was already marked.
- m_isFirstVisit = false;
- for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); m_mutatorStack.canRemoveLast() && countdown--;)
- visitChildren(m_mutatorStack.removeLast());
- } else
</del><ins>+ IterationStatus status = forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ if (stack.isEmpty())
+ return IterationStatus::Continue;
+
+ stack.refill();
+
+ m_isFirstVisit = (&stack == &m_collectorStack);
+
+ for (unsigned countdown = Options::minimumNumberOfScansBetweenRebalance(); stack.canRemoveLast() && countdown--;)
+ visitChildren(stack.removeLast());
+ return IterationStatus::Done;
+ });
+ if (status == IterationStatus::Continue)
</ins><span class="cx"> break;
</span><ins>+
</ins><span class="cx"> m_rightToRun.safepoint();
</span><span class="cx"> donateKnownParallel();
</span><span class="cx"> }
</span><span class="lines">@@ -490,6 +500,67 @@
</span><span class="cx"> mergeIfNecessary();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+size_t SlotVisitor::performIncrementOfDraining(size_t bytesRequested)
+{
+ RELEASE_ASSERT(m_isInParallelMode);
+
+ size_t cellsRequested = bytesRequested / MarkedBlock::atomSize;
+ {
+ auto locker = holdLock(m_heap.m_markingMutex);
+ forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ cellsRequested -= correspondingGlobalStack(stack).transferTo(stack, cellsRequested);
+ return cellsRequested ? IterationStatus::Continue : IterationStatus::Done;
+ });
+ }
+
+ size_t cellBytesVisited = 0;
+ m_nonCellVisitCount = 0;
+
+ auto bytesVisited = [&] () -> size_t {
+ return cellBytesVisited + m_nonCellVisitCount;
+ };
+
+ auto isDone = [&] () -> bool {
+ return bytesVisited() >= bytesRequested;
+ };
+
+ {
+ auto locker = holdLock(m_rightToRun);
+
+ while (!isDone()) {
+ updateMutatorIsStopped(locker);
+ IterationStatus status = forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ if (stack.isEmpty() || isDone())
+ return IterationStatus::Continue;
+
+ stack.refill();
+
+ m_isFirstVisit = (&stack == &m_collectorStack);
+
+ unsigned countdown = Options::minimumNumberOfScansBetweenRebalance();
+ while (countdown && stack.canRemoveLast() && !isDone()) {
+ const JSCell* cell = stack.removeLast();
+ cellBytesVisited += cell->cellSize();
+ visitChildren(cell);
+ countdown--;
+ }
+ return IterationStatus::Done;
+ });
+ if (status == IterationStatus::Continue)
+ break;
+ m_rightToRun.safepoint();
+ donateKnownParallel();
+ }
+ }
+
+ donateAll();
+ mergeIfNecessary();
+
+ return bytesVisited();
+}
+
</ins><span class="cx"> bool SlotVisitor::didReachTermination()
</span><span class="cx"> {
</span><span class="cx"> LockHolder locker(m_heap.m_markingMutex);
</span><span class="lines">@@ -496,7 +567,7 @@
</span><span class="cx"> return didReachTermination(locker);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-bool SlotVisitor::didReachTermination(const LockHolder&)
</del><ins>+bool SlotVisitor::didReachTermination(const AbstractLocker&)
</ins><span class="cx"> {
</span><span class="cx"> return isEmpty()
</span><span class="cx"> && !m_heap.m_numberOfActiveParallelMarkers
</span><span class="lines">@@ -504,7 +575,7 @@
</span><span class="cx"> && m_heap.m_sharedMutatorMarkStack->isEmpty();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-bool SlotVisitor::hasWork(const LockHolder&)
</del><ins>+bool SlotVisitor::hasWork(const AbstractLocker&)
</ins><span class="cx"> {
</span><span class="cx"> return !m_heap.m_sharedCollectorMarkStack->isEmpty()
</span><span class="cx"> || !m_heap.m_sharedMutatorMarkStack->isEmpty();
</span><span class="lines">@@ -515,7 +586,7 @@
</span><span class="cx"> ASSERT(m_isInParallelMode);
</span><span class="cx">
</span><span class="cx"> ASSERT(Options::numberOfGCMarkers());
</span><del>-
</del><ins>+
</ins><span class="cx"> bool isActive = false;
</span><span class="cx"> while (true) {
</span><span class="cx"> {
</span><span class="lines">@@ -528,7 +599,7 @@
</span><span class="cx"> while (true) {
</span><span class="cx"> if (hasElapsed(timeout))
</span><span class="cx"> return SharedDrainResult::TimedOut;
</span><del>-
</del><ins>+
</ins><span class="cx"> if (didReachTermination(locker)) {
</span><span class="cx"> m_heap.m_markingConditionVariable.notifyAll();
</span><span class="cx"> return SharedDrainResult::Done;
</span><span class="lines">@@ -536,7 +607,7 @@
</span><span class="cx">
</span><span class="cx"> if (hasWork(locker))
</span><span class="cx"> break;
</span><del>-
</del><ins>+
</ins><span class="cx"> m_heap.m_markingConditionVariable.waitUntil(m_heap.m_markingMutex, timeout);
</span><span class="cx"> }
</span><span class="cx"> } else {
</span><span class="lines">@@ -548,21 +619,25 @@
</span><span class="cx"> if (didReachTermination(locker))
</span><span class="cx"> m_heap.m_markingConditionVariable.notifyAll();
</span><span class="cx">
</span><del>- m_heap.m_markingConditionVariable.waitUntil(
- m_heap.m_markingMutex, timeout,
- [&] {
- return hasWork(locker)
- || m_heap.m_parallelMarkersShouldExit;
- });
</del><ins>+ auto isReady = [&] () -> bool {
+ return hasWork(locker)
+ || m_heap.m_parallelMarkersShouldExit;
+ };
+
+ m_heap.m_markingConditionVariable.waitUntil(m_heap.m_markingMutex, timeout, isReady);
</ins><span class="cx">
</span><span class="cx"> if (m_heap.m_parallelMarkersShouldExit)
</span><span class="cx"> return SharedDrainResult::Done;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_collectorStack.stealSomeCellsFrom(
- *m_heap.m_sharedCollectorMarkStack, m_heap.m_numberOfWaitingParallelMarkers);
- m_mutatorStack.stealSomeCellsFrom(
- *m_heap.m_sharedMutatorMarkStack, m_heap.m_numberOfWaitingParallelMarkers);
</del><ins>+ forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ stack.stealSomeCellsFrom(
+ correspondingGlobalStack(stack),
+ m_heap.m_numberOfWaitingParallelMarkers);
+ return IterationStatus::Continue;
+ });
+
</ins><span class="cx"> m_heap.m_numberOfActiveParallelMarkers++;
</span><span class="cx"> m_heap.m_numberOfWaitingParallelMarkers--;
</span><span class="cx"> }
</span><span class="lines">@@ -584,8 +659,7 @@
</span><span class="cx">
</span><span class="cx"> ASSERT(Options::numberOfGCMarkers());
</span><span class="cx">
</span><del>- if (Options::numberOfGCMarkers() < 4
- || !m_heap.hasHeapAccess()
</del><ins>+ if (!m_heap.hasHeapAccess()
</ins><span class="cx"> || m_heap.collectorBelievesThatTheWorldIsStopped()) {
</span><span class="cx"> // This is an optimization over drainInParallel() when we have a concurrent mutator but
</span><span class="cx"> // otherwise it is not profitable.
</span><span class="lines">@@ -593,9 +667,7 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> LockHolder locker(m_heap.m_markingMutex);
</span><del>- m_collectorStack.transferTo(*m_heap.m_sharedCollectorMarkStack);
- m_mutatorStack.transferTo(*m_heap.m_sharedMutatorMarkStack);
- m_heap.m_markingConditionVariable.notifyAll();
</del><ins>+ donateAll(locker);
</ins><span class="cx">
</span><span class="cx"> for (;;) {
</span><span class="cx"> if (hasElapsed(timeout))
</span><span class="lines">@@ -610,6 +682,22 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void SlotVisitor::donateAll()
+{
+ donateAll(holdLock(m_heap.m_markingMutex));
+}
+
+void SlotVisitor::donateAll(const AbstractLocker&)
+{
+ forEachMarkStack(
+ [&] (MarkStackArray& stack) -> IterationStatus {
+ stack.transferTo(correspondingGlobalStack(stack));
+ return IterationStatus::Continue;
+ });
+
+ m_heap.m_markingConditionVariable.notifyAll();
+}
+
</ins><span class="cx"> void SlotVisitor::addOpaqueRoot(void* root)
</span><span class="cx"> {
</span><span class="cx"> if (!root)
</span><span class="lines">@@ -716,4 +804,12 @@
</span><span class="cx"> out.print("Collector: [", pointerListDump(collectorMarkStack()), "], Mutator: [", pointerListDump(mutatorMarkStack()), "]");
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+MarkStackArray& SlotVisitor::correspondingGlobalStack(MarkStackArray& stack)
+{
+ if (&stack == &m_collectorStack)
+ return *m_heap.m_sharedCollectorMarkStack;
+ RELEASE_ASSERT(&stack == &m_mutatorStack);
+ return *m_heap.m_sharedMutatorMarkStack;
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitor.h (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitor.h 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitor.h 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -27,6 +27,7 @@
</span><span class="cx">
</span><span class="cx"> #include "CellState.h"
</span><span class="cx"> #include "HandleTypes.h"
</span><ins>+#include "IterationStatus.h"
</ins><span class="cx"> #include "MarkStack.h"
</span><span class="cx"> #include "OpaqueRootSet.h"
</span><span class="cx"> #include "VisitRaceKey.h"
</span><span class="lines">@@ -117,6 +118,12 @@
</span><span class="cx">
</span><span class="cx"> SharedDrainResult drainInParallel(MonotonicTime timeout = MonotonicTime::infinity());
</span><span class="cx"> SharedDrainResult drainInParallelPassively(MonotonicTime timeout = MonotonicTime::infinity());
</span><ins>+
+ // Attempts to perform an increment of draining that involves only walking `bytes` worth of data. This
+ // is likely to accidentally walk more or less than that. It will usually mark more than bytes. It may
+ // mark less than bytes if we're reaching termination or if the global worklist is empty (which may in
+ // rare cases happen temporarily even if we're not reaching termination).
+ size_t performIncrementOfDraining(size_t bytes);
</ins><span class="cx">
</span><span class="cx"> JS_EXPORT_PRIVATE void mergeIfNecessary();
</span><span class="cx">
</span><span class="lines">@@ -159,6 +166,8 @@
</span><span class="cx">
</span><span class="cx"> void setIgnoreNewOpaqueRoots(bool value) { m_ignoreNewOpaqueRoots = value; }
</span><span class="cx">
</span><ins>+ void donateAll();
+
</ins><span class="cx"> private:
</span><span class="cx"> friend class ParallelModeEnabler;
</span><span class="cx">
</span><span class="lines">@@ -187,10 +196,17 @@
</span><span class="cx">
</span><span class="cx"> void donateKnownParallel();
</span><span class="cx"> void donateKnownParallel(MarkStackArray& from, MarkStackArray& to);
</span><del>-
- bool hasWork(const LockHolder&);
- bool didReachTermination(const LockHolder&);
</del><span class="cx">
</span><ins>+ void donateAll(const AbstractLocker&);
+
+ bool hasWork(const AbstractLocker&);
+ bool didReachTermination(const AbstractLocker&);
+
+ template<typename Func>
+ IterationStatus forEachMarkStack(const Func&);
+
+ MarkStackArray& correspondingGlobalStack(MarkStackArray&);
+
</ins><span class="cx"> MarkStackArray m_collectorStack;
</span><span class="cx"> MarkStackArray m_mutatorStack;
</span><span class="cx"> OpaqueRootSet m_opaqueRoots; // Handle-owning data structures not visible to the garbage collector.
</span><span class="lines">@@ -198,8 +214,9 @@
</span><span class="cx">
</span><span class="cx"> size_t m_bytesVisited;
</span><span class="cx"> size_t m_visitCount;
</span><ins>+ size_t m_nonCellVisitCount { 0 }; // Used for incremental draining, ignored otherwise.
</ins><span class="cx"> bool m_isInParallelMode;
</span><del>-
</del><ins>+
</ins><span class="cx"> HeapVersion m_markingVersion;
</span><span class="cx">
</span><span class="cx"> Heap& m_heap;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSlotVisitorInlinesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/SlotVisitorInlines.h 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -81,8 +81,10 @@
</span><span class="cx">
</span><span class="cx"> inline void SlotVisitor::reportExtraMemoryVisited(size_t size)
</span><span class="cx"> {
</span><del>- if (m_isFirstVisit)
</del><ins>+ if (m_isFirstVisit) {
</ins><span class="cx"> heap()->reportExtraMemoryVisited(size);
</span><ins>+ m_nonCellVisitCount += size;
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> #if ENABLE(RESOURCE_USAGE)
</span><span class="lines">@@ -108,4 +110,14 @@
</span><span class="cx"> return *m_heap.m_vm;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+template<typename Func>
+IterationStatus SlotVisitor::forEachMarkStack(const Func& func)
+{
+ if (func(m_collectorStack) == IterationStatus::Done)
+ return IterationStatus::Done;
+ if (func(m_mutatorStack) == IterationStatus::Done)
+ return IterationStatus::Done;
+ return IterationStatus::Continue;
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapSpaceTimeMutatorSchedulercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/SpaceTimeMutatorScheduler.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/SpaceTimeMutatorScheduler.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/SpaceTimeMutatorScheduler.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -141,7 +141,7 @@
</span><span class="cx"> ASSERT(Options::logGC());
</span><span class="cx"> Snapshot snapshot(*this);
</span><span class="cx"> dataLog(
</span><del>- "a=", format("%.0lf", bytesSinceBeginningOfCycle(snapshot) / 1024), " kb ",
</del><ins>+ "a=", format("%.0lf", bytesSinceBeginningOfCycle(snapshot) / 1024), "kb ",
</ins><span class="cx"> "hf=", format("%.3lf", headroomFullness(snapshot)), " ",
</span><span class="cx"> "mu=", format("%.3lf", mutatorUtilization(snapshot)), " ");
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreheapStochasticSpaceTimeMutatorSchedulercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/heap/StochasticSpaceTimeMutatorScheduler.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/heap/StochasticSpaceTimeMutatorScheduler.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/heap/StochasticSpaceTimeMutatorScheduler.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -163,7 +163,7 @@
</span><span class="cx"> ASSERT(Options::logGC());
</span><span class="cx"> Snapshot snapshot(*this);
</span><span class="cx"> dataLog(
</span><del>- "a=", format("%.0lf", bytesSinceBeginningOfCycle(snapshot) / 1024), " kb ",
</del><ins>+ "a=", format("%.0lf", bytesSinceBeginningOfCycle(snapshot) / 1024), "kb ",
</ins><span class="cx"> "hf=", format("%.3lf", headroomFullness(snapshot)), " ",
</span><span class="cx"> "mu=", format("%.3lf", mutatorUtilization(snapshot)), " ");
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejsccpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jsc.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jsc.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/jsc.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -1087,6 +1087,7 @@
</span><span class="cx"> static EncodedJSValue JSC_HOST_CALL functionDollarAgentGetReport(ExecState*);
</span><span class="cx"> static EncodedJSValue JSC_HOST_CALL functionDollarAgentLeaving(ExecState*);
</span><span class="cx"> static EncodedJSValue JSC_HOST_CALL functionWaitForReport(ExecState*);
</span><ins>+static EncodedJSValue JSC_HOST_CALL functionHeapCapacity(ExecState*);
</ins><span class="cx">
</span><span class="cx"> struct Script {
</span><span class="cx"> enum class StrictMode {
</span><span class="lines">@@ -1367,6 +1368,8 @@
</span><span class="cx"> addFunction(vm, agent, "leaving", functionDollarAgentLeaving, 0);
</span><span class="cx">
</span><span class="cx"> addFunction(vm, "waitForReport", functionWaitForReport, 0);
</span><ins>+
+ addFunction(vm, "heapCapacity", functionHeapCapacity, 0);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void addFunction(VM& vm, JSObject* object, const char* name, NativeFunction function, unsigned arguments)
</span><span class="lines">@@ -2639,6 +2642,12 @@
</span><span class="cx"> return JSValue::encode(jsString(&vm, string));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+EncodedJSValue JSC_HOST_CALL functionHeapCapacity(ExecState* exec)
+{
+ VM& vm = exec->vm();
+ return JSValue::encode(jsNumber(vm.heap.capacity()));
+}
+
</ins><span class="cx"> template<typename ValueType>
</span><span class="cx"> typename std::enable_if<!std::is_fundamental<ValueType>::value>::type addOption(VM&, JSObject*, Identifier, ValueType) { }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeOptionscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Options.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Options.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/runtime/Options.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -311,9 +311,9 @@
</span><span class="cx"> if (WTF::numberOfProcessorCores() < 4) {
</span><span class="cx"> Options::maximumMutatorUtilization() = 0.6;
</span><span class="cx"> Options::concurrentGCMaxHeadroom() = 1.4;
</span><del>- Options::concurrentGCPeriodMS() = 10;
- } else
- Options::useStochasticMutatorScheduler() = true;
</del><ins>+ Options::minimumGCPauseMS() = 1;
+ Options::gcIncrementScale() = 1;
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> static void recomputeDependentOptions()
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreruntimeOptionsh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/runtime/Options.h (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/runtime/Options.h 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/JavaScriptCore/runtime/Options.h 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -202,9 +202,12 @@
</span><span class="cx"> v(double, maximumMutatorUtilization, 0.7, Normal, nullptr) \
</span><span class="cx"> v(double, concurrentGCMaxHeadroom, 1.5, Normal, nullptr) \
</span><span class="cx"> v(double, concurrentGCPeriodMS, 2, Normal, nullptr) \
</span><del>- v(bool, useStochasticMutatorScheduler, false, Normal, nullptr) \
</del><ins>+ v(bool, useStochasticMutatorScheduler, true, Normal, nullptr) \
</ins><span class="cx"> v(double, minimumGCPauseMS, 0.3, Normal, nullptr) \
</span><span class="cx"> v(double, gcPauseScale, 0.3, Normal, nullptr) \
</span><ins>+ v(double, gcIncrementBytes, 10000, Normal, nullptr) \
+ v(double, gcIncrementMaxBytes, 100000, Normal, nullptr) \
+ v(double, gcIncrementScale, 0, Normal, nullptr) \
</ins><span class="cx"> v(bool, scribbleFreeCells, false, Normal, nullptr) \
</span><span class="cx"> v(double, sizeClassProgression, 1.4, Normal, nullptr) \
</span><span class="cx"> v(unsigned, largeAllocationCutoff, 100000, Normal, nullptr) \
</span></span></pre></div>
<a id="trunkSourceWTFChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/ChangeLog (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/ChangeLog 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/WTF/ChangeLog 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -1,3 +1,15 @@
</span><ins>+2017-01-31 Filip Pizlo <fpizlo@apple.com>
+
+ The mutator should be able to perform increments of GC work
+ https://bugs.webkit.org/show_bug.cgi?id=167528
+
+ Reviewed by Keith Miller and Geoffrey Garen.
+
+ We want dataLog to be locked even if you're not logging to a file!
+
+ * wtf/DataLog.cpp:
+ (WTF::initializeLogFileOnce):
+
</ins><span class="cx"> 2017-01-28 Wenson Hsieh <wenson_hsieh@apple.com>
</span><span class="cx">
</span><span class="cx"> Check USE(APPLE_INTERNAL_SDK) instead of specific headers when importing from WebKitAdditions
</span></span></pre></div>
<a id="trunkSourceWTFwtfDataLogcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WTF/wtf/DataLog.cpp (211447 => 211448)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WTF/wtf/DataLog.cpp 2017-01-31 22:22:22 UTC (rev 211447)
+++ trunk/Source/WTF/wtf/DataLog.cpp 2017-01-31 22:31:24 UTC (rev 211448)
</span><span class="lines">@@ -53,6 +53,7 @@
</span><span class="cx"> static PrintStream* s_file;
</span><span class="cx">
</span><span class="cx"> static uint64_t s_fileData[(sizeof(FilePrintStream) + 7) / 8];
</span><ins>+static uint64_t s_lockedFileData[(sizeof(LockedPrintStream) + 7) / 8];
</ins><span class="cx">
</span><span class="cx"> static void initializeLogFileOnce()
</span><span class="cx"> {
</span><span class="lines">@@ -119,21 +120,15 @@
</span><span class="cx"> }
</span><span class="cx"> #endif // DATA_LOG_TO_FILE
</span><span class="cx">
</span><del>- bool wrapWithLocked = true;
-
</del><span class="cx"> if (!file) {
</span><span class="cx"> // Use placement new; this makes it easier to use dataLog() to debug
</span><span class="cx"> // fastMalloc.
</span><span class="cx"> file = new (s_fileData) FilePrintStream(stderr, FilePrintStream::Borrow);
</span><del>- wrapWithLocked = false;
</del><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> setvbuf(file->file(), 0, _IONBF, 0); // Prefer unbuffered output, so that we get a full log upon crash or deadlock.
</span><span class="cx">
</span><del>- if (wrapWithLocked)
- s_file = new LockedPrintStream(std::unique_ptr<FilePrintStream>(file));
- else
- s_file = file;
</del><ins>+ s_file = new (s_lockedFileData) LockedPrintStream(std::unique_ptr<FilePrintStream>(file));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> static void initializeLogFile()
</span></span></pre>
</div>
</div>
</body>
</html>