<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[218788] trunk/Source/bmalloc</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/218788">218788</a></dd>
<dt>Author</dt> <dd>ggaren@apple.com</dd>
<dt>Date</dt> <dd>2017-06-24 13:14:33 -0700 (Sat, 24 Jun 2017)</dd>
</dl>

<h3>Log Message</h3>
<pre>bmalloc: Add a per-thread line cache
https://bugs.webkit.org/show_bug.cgi?id=173552

Reviewed by Darin Adler.

Previously, any thread could allocate out of any page with free lines.
Now, the first thread to free a line in a page owns that page's free
lines until the whole page becomes free.

This patch is a big speedup on multi-threaded benchmarks.
tree_churn --parallel gets 14% faster on a 2-core (4-hyper-core) MacBook
Air and 2.85X faster on 12-core (24-hyper-core) Mac Pro. Other parallel
benchmarks show significant but smaller speedups.

Thread affinity is a great predictor of object lifetime. The per-thread
line cache avoids the pathology of shuffling pages between threads,
turning predictable lifetimes into unpredictable lifetimes, increasing
fragmentation. On tree_churn --parallel, the per-thread line cache
increases free memory found per page scanned by 2.85X.

Free line scanning in fragmented pages is pretty expensive relative to
other allocate / initialize / free operations. According to Instruments,
on tree_churn --parallel, scanning is about 10X more expensive than
freeing. This explains why a 2.85X improvement in scanning efficiency
translates into a 2.85X overall speedup on tree_churn --parallel.

* bmalloc/Allocator.cpp:
(bmalloc::Allocator::refillAllocatorSlowCase): Pass through our line
cache so the Heap can fill it.

* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::scavenge): Scavenge our line cache.

(bmalloc::Deallocator::processObjectLog): Deleted.

* bmalloc/Deallocator.h:
(bmalloc::Deallocator::lineCache): Added a line cache.

* bmalloc/Heap.cpp:
(bmalloc::Heap::deallocateLineCache): Deallocation function for thread
destruction.

(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::deallocateSmallLine):
(bmalloc::Heap::allocateSmallBumpRangesByMetadata):
(bmalloc::Heap::allocateSmallBumpRangesByObject): Consult the new per-thread line
cache for allocation and deallocation.

* bmalloc/Heap.h:
(bmalloc::Heap::allocateSmallBumpRanges):
(bmalloc::Heap::derefSmallLine):

* bmalloc/List.h:
(bmalloc::List::remove): Remove has always been a logically static
operation. Declare it static now so that the Heap can remove a page from
a thread's line cache without holding a direct pointer to the cache.

* bmalloc/SmallPage.h:</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourcebmallocChangeLog">trunk/Source/bmalloc/ChangeLog</a></li>
<li><a href="#trunkSourcebmallocbmallocAllocatorcpp">trunk/Source/bmalloc/bmalloc/Allocator.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocDeallocatorcpp">trunk/Source/bmalloc/bmalloc/Deallocator.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocDeallocatorh">trunk/Source/bmalloc/bmalloc/Deallocator.h</a></li>
<li><a href="#trunkSourcebmallocbmallocHeapcpp">trunk/Source/bmalloc/bmalloc/Heap.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocHeaph">trunk/Source/bmalloc/bmalloc/Heap.h</a></li>
<li><a href="#trunkSourcebmallocbmallocListh">trunk/Source/bmalloc/bmalloc/List.h</a></li>
<li><a href="#trunkSourcebmallocbmallocSmallPageh">trunk/Source/bmalloc/bmalloc/SmallPage.h</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourcebmallocChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/ChangeLog (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/ChangeLog   2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/ChangeLog      2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -1,3 +1,64 @@
</span><ins>+2017-06-19  Geoffrey Garen  <ggaren@apple.com>
+
+        bmalloc: Add a per-thread line cache
+        https://bugs.webkit.org/show_bug.cgi?id=173552
+
+        Reviewed by Darin Adler.
+
+        Previously, any thread could allocate out of any page with free lines.
+        Now, the first thread to free a line in a page owns that page's free
+        lines until the whole page becomes free.
+
+        This patch is a big speedup on multi-threaded benchmarks.
+        tree_churn --parallel gets 14% faster on a 2-core (4-hyper-core) MacBook
+        Air and 2.85X faster on 12-core (24-hyper-core) Mac Pro. Other parallel
+        benchmarks show significant but smaller speedups.
+
+        Thread affinity is a great predictor of object lifetime. The per-thread
+        line cache avoids the pathology of shuffling pages between threads,
+        turning predictable lifetimes into unpredictable lifetimes, increasing
+        fragmentation. On tree_churn --parallel, the per-thread line cache
+        increases free memory found per page scanned by 2.85X.
+
+        Free line scanning in fragmented pages is pretty expensive relative to
+        other allocate / initialize / free operations. According to Instruments,
+        on tree_churn --parallel, scanning is about 10X more expensive than
+        freeing. This explains why a 2.85X improvement in scanning efficiency
+        translates into a 2.85X overall speedup on tree_churn --parallel.
+
+        * bmalloc/Allocator.cpp:
+        (bmalloc::Allocator::refillAllocatorSlowCase): Pass through our line
+        cache so the Heap can fill it.
+
+        * bmalloc/Deallocator.cpp:
+        (bmalloc::Deallocator::scavenge): Scavenge our line cache.
+
+        (bmalloc::Deallocator::processObjectLog): Deleted.
+
+        * bmalloc/Deallocator.h:
+        (bmalloc::Deallocator::lineCache): Added a line cache.
+
+        * bmalloc/Heap.cpp:
+        (bmalloc::Heap::deallocateLineCache): Deallocation function for thread
+        destruction.
+
+        (bmalloc::Heap::allocateSmallPage):
+        (bmalloc::Heap::deallocateSmallLine):
+        (bmalloc::Heap::allocateSmallBumpRangesByMetadata):
+        (bmalloc::Heap::allocateSmallBumpRangesByObject): Consult the new per-thread line
+        cache for allocation and deallocation.
+
+        * bmalloc/Heap.h:
+        (bmalloc::Heap::allocateSmallBumpRanges):
+        (bmalloc::Heap::derefSmallLine):
+
+        * bmalloc/List.h:
+        (bmalloc::List::remove): Remove has always been a logically static
+        operation. Declare it static now so that the Heap can remove a page from
+        a thread's line cache without holding a direct pointer to the cache.
+
+        * bmalloc/SmallPage.h:
+
</ins><span class="cx"> 2017-06-10  Dan Bernstein  <mitz@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Reverted r218056 because it made the IDE reindex constantly.
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocAllocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Allocator.cpp (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Allocator.cpp       2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/Allocator.cpp  2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -155,7 +155,8 @@
</span><span class="cx"> 
</span><span class="cx">     std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
</span><span class="cx">     m_deallocator.processObjectLog(lock);
</span><del>-    PerProcess<Heap>::getFastCase()->allocateSmallBumpRanges(lock, sizeClass, allocator, bumpRangeCache);
</del><ins>+    PerProcess<Heap>::getFastCase()->allocateSmallBumpRanges(
+        lock, sizeClass, allocator, bumpRangeCache, m_deallocator.lineCache(lock));
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> INLINE void Allocator::refillAllocator(BumpAllocator& allocator, size_t sizeClass)
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocDeallocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Deallocator.cpp (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Deallocator.cpp     2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/Deallocator.cpp        2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -59,7 +59,10 @@
</span><span class="cx">     if (m_debugHeap)
</span><span class="cx">         return;
</span><span class="cx"> 
</span><del>-    processObjectLog();
</del><ins>+    std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
+
+    processObjectLog(lock);
+    PerProcess<Heap>::getFastCase()->deallocateLineCache(lock, lineCache(lock));
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void Deallocator::processObjectLog(std::lock_guard<StaticMutex>& lock)
</span><span class="lines">@@ -67,17 +70,10 @@
</span><span class="cx">     Heap* heap = PerProcess<Heap>::getFastCase();
</span><span class="cx">     
</span><span class="cx">     for (Object object : m_objectLog)
</span><del>-        heap->derefSmallLine(lock, object);
-
</del><ins>+        heap->derefSmallLine(lock, object, lineCache(lock));
</ins><span class="cx">     m_objectLog.clear();
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void Deallocator::processObjectLog()
-{
-    std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
-    processObjectLog(lock);
-}
-
</del><span class="cx"> void Deallocator::deallocateSlowCase(void* object)
</span><span class="cx"> {
</span><span class="cx">     if (m_debugHeap)
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocDeallocatorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Deallocator.h (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Deallocator.h       2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/Deallocator.h  2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -27,6 +27,7 @@
</span><span class="cx"> #define Deallocator_h
</span><span class="cx"> 
</span><span class="cx"> #include "FixedVector.h"
</span><ins>+#include "SmallPage.h"
</ins><span class="cx"> #include <mutex>
</span><span class="cx"> 
</span><span class="cx"> namespace bmalloc {
</span><span class="lines">@@ -45,8 +46,9 @@
</span><span class="cx">     void deallocate(void*);
</span><span class="cx">     void scavenge();
</span><span class="cx">     
</span><del>-    void processObjectLog();
</del><span class="cx">     void processObjectLog(std::lock_guard<StaticMutex>&);
</span><ins>+    
+    LineCache& lineCache(std::lock_guard<StaticMutex>&) { return m_lineCache; }
</ins><span class="cx"> 
</span><span class="cx"> private:
</span><span class="cx">     bool deallocateFastCase(void*);
</span><span class="lines">@@ -53,6 +55,7 @@
</span><span class="cx">     void deallocateSlowCase(void*);
</span><span class="cx"> 
</span><span class="cx">     FixedVector<void*, deallocatorLogCapacity> m_objectLog;
</span><ins>+    LineCache m_lineCache; // The Heap removes items from this cache.
</ins><span class="cx">     DebugHeap* m_debugHeap;
</span><span class="cx"> };
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.cpp (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.cpp    2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/Heap.cpp       2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -191,6 +191,16 @@
</span><span class="cx">     m_scavenger.runSoon();
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void Heap::deallocateLineCache(std::lock_guard<StaticMutex>&, LineCache& lineCache)
+{
+    for (auto& list : lineCache) {
+        while (!list.isEmpty()) {
+            size_t sizeClass = &list - &lineCache[0];
+            m_lineCache[sizeClass].push(list.popFront());
+        }
+    }
+}
+
</ins><span class="cx"> void Heap::allocateSmallChunk(std::lock_guard<StaticMutex>& lock, size_t pageClass)
</span><span class="cx"> {
</span><span class="cx">     size_t pageSize = bmalloc::pageSize(pageClass);
</span><span class="lines">@@ -235,11 +245,14 @@
</span><span class="cx">     m_largeFree.add(LargeRange(chunk, size, physicalSize));
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-SmallPage* Heap::allocateSmallPage(std::lock_guard<StaticMutex>& lock, size_t sizeClass)
</del><ins>+SmallPage* Heap::allocateSmallPage(std::lock_guard<StaticMutex>& lock, size_t sizeClass, LineCache& lineCache)
</ins><span class="cx"> {
</span><del>-    if (!m_freeLines[sizeClass].isEmpty())
-        return m_freeLines[sizeClass].popFront();
</del><ins>+    if (!lineCache[sizeClass].isEmpty())
+        return lineCache[sizeClass].popFront();
</ins><span class="cx"> 
</span><ins>+    if (!m_lineCache[sizeClass].isEmpty())
+        return m_lineCache[sizeClass].popFront();
+
</ins><span class="cx">     m_isGrowing = true;
</span><span class="cx">     
</span><span class="cx">     SmallPage* page = [&]() {
</span><span class="lines">@@ -270,7 +283,7 @@
</span><span class="cx">     return page;
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void Heap::deallocateSmallLine(std::lock_guard<StaticMutex>& lock, Object object)
</del><ins>+void Heap::deallocateSmallLine(std::lock_guard<StaticMutex>& lock, Object object, LineCache& lineCache)
</ins><span class="cx"> {
</span><span class="cx">     BASSERT(!object.line()->refCount(lock));
</span><span class="cx">     SmallPage* page = object.page();
</span><span class="lines">@@ -278,7 +291,7 @@
</span><span class="cx"> 
</span><span class="cx">     if (!page->hasFreeLines(lock)) {
</span><span class="cx">         page->setHasFreeLines(lock, true);
</span><del>-        m_freeLines[page->sizeClass()].push(page);
</del><ins>+        lineCache[page->sizeClass()].push(page);
</ins><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     if (page->refCount(lock))
</span><span class="lines">@@ -287,7 +300,7 @@
</span><span class="cx">     size_t sizeClass = page->sizeClass();
</span><span class="cx">     size_t pageClass = m_pageClasses[sizeClass];
</span><span class="cx"> 
</span><del>-    m_freeLines[sizeClass].remove(page);
</del><ins>+    List<SmallPage>::remove(page); // 'page' may be in any thread's line cache.
</ins><span class="cx">     
</span><span class="cx">     Chunk* chunk = Chunk::get(page);
</span><span class="cx">     if (chunk->freePages().isEmpty())
</span><span class="lines">@@ -310,9 +323,10 @@
</span><span class="cx"> 
</span><span class="cx"> void Heap::allocateSmallBumpRangesByMetadata(
</span><span class="cx">     std::lock_guard<StaticMutex>& lock, size_t sizeClass,
</span><del>-    BumpAllocator& allocator, BumpRangeCache& rangeCache)
</del><ins>+    BumpAllocator& allocator, BumpRangeCache& rangeCache,
+    LineCache& lineCache)
</ins><span class="cx"> {
</span><del>-    SmallPage* page = allocateSmallPage(lock, sizeClass);
</del><ins>+    SmallPage* page = allocateSmallPage(lock, sizeClass, lineCache);
</ins><span class="cx">     SmallLine* lines = page->begin();
</span><span class="cx">     BASSERT(page->hasFreeLines(lock));
</span><span class="cx">     size_t smallLineCount = m_vmPageSizePhysical / smallLineSize;
</span><span class="lines">@@ -356,7 +370,7 @@
</span><span class="cx"> 
</span><span class="cx">         // In a fragmented page, some free ranges might not fit in the cache.
</span><span class="cx">         if (rangeCache.size() == rangeCache.capacity()) {
</span><del>-            m_freeLines[sizeClass].push(page);
</del><ins>+            lineCache[sizeClass].push(page);
</ins><span class="cx">             BASSERT(allocator.canAllocate());
</span><span class="cx">             return;
</span><span class="cx">         }
</span><span class="lines">@@ -371,10 +385,11 @@
</span><span class="cx"> 
</span><span class="cx"> void Heap::allocateSmallBumpRangesByObject(
</span><span class="cx">     std::lock_guard<StaticMutex>& lock, size_t sizeClass,
</span><del>-    BumpAllocator& allocator, BumpRangeCache& rangeCache)
</del><ins>+    BumpAllocator& allocator, BumpRangeCache& rangeCache,
+    LineCache& lineCache)
</ins><span class="cx"> {
</span><span class="cx">     size_t size = allocator.size();
</span><del>-    SmallPage* page = allocateSmallPage(lock, sizeClass);
</del><ins>+    SmallPage* page = allocateSmallPage(lock, sizeClass, lineCache);
</ins><span class="cx">     BASSERT(page->hasFreeLines(lock));
</span><span class="cx"> 
</span><span class="cx">     auto findSmallBumpRange = [&](Object& it, Object& end) {
</span><span class="lines">@@ -410,7 +425,7 @@
</span><span class="cx"> 
</span><span class="cx">         // In a fragmented page, some free ranges might not fit in the cache.
</span><span class="cx">         if (rangeCache.size() == rangeCache.capacity()) {
</span><del>-            m_freeLines[sizeClass].push(page);
</del><ins>+            lineCache[sizeClass].push(page);
</ins><span class="cx">             BASSERT(allocator.canAllocate());
</span><span class="cx">             return;
</span><span class="cx">         }
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.h (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.h      2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/Heap.h 2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -59,8 +59,10 @@
</span><span class="cx">     
</span><span class="cx">     DebugHeap* debugHeap() { return m_debugHeap; }
</span><span class="cx"> 
</span><del>-    void allocateSmallBumpRanges(std::lock_guard<StaticMutex>&, size_t sizeClass, BumpAllocator&, BumpRangeCache&);
-    void derefSmallLine(std::lock_guard<StaticMutex>&, Object);
</del><ins>+    void allocateSmallBumpRanges(std::lock_guard<StaticMutex>&, size_t sizeClass,
+        BumpAllocator&, BumpRangeCache&, LineCache&);
+    void derefSmallLine(std::lock_guard<StaticMutex>&, Object, LineCache&);
+    void deallocateLineCache(std::lock_guard<StaticMutex>&, LineCache&);
</ins><span class="cx"> 
</span><span class="cx">     void* allocateLarge(std::lock_guard<StaticMutex>&, size_t alignment, size_t);
</span><span class="cx">     void* tryAllocateLarge(std::lock_guard<StaticMutex>&, size_t alignment, size_t);
</span><span class="lines">@@ -91,12 +93,12 @@
</span><span class="cx">     void initializePageMetadata();
</span><span class="cx"> 
</span><span class="cx">     void allocateSmallBumpRangesByMetadata(std::lock_guard<StaticMutex>&,
</span><del>-        size_t sizeClass, BumpAllocator&, BumpRangeCache&);
</del><ins>+        size_t sizeClass, BumpAllocator&, BumpRangeCache&, LineCache&);
</ins><span class="cx">     void allocateSmallBumpRangesByObject(std::lock_guard<StaticMutex>&,
</span><del>-        size_t sizeClass, BumpAllocator&, BumpRangeCache&);
</del><ins>+        size_t sizeClass, BumpAllocator&, BumpRangeCache&, LineCache&);
</ins><span class="cx"> 
</span><del>-    SmallPage* allocateSmallPage(std::lock_guard<StaticMutex>&, size_t sizeClass);
-    void deallocateSmallLine(std::lock_guard<StaticMutex>&, Object);
</del><ins>+    SmallPage* allocateSmallPage(std::lock_guard<StaticMutex>&, size_t sizeClass, LineCache&);
+    void deallocateSmallLine(std::lock_guard<StaticMutex>&, Object, LineCache&);
</ins><span class="cx"> 
</span><span class="cx">     void allocateSmallChunk(std::lock_guard<StaticMutex>&, size_t pageClass);
</span><span class="cx">     void deallocateSmallChunk(Chunk*, size_t pageClass);
</span><span class="lines">@@ -116,7 +118,7 @@
</span><span class="cx">     Vector<LineMetadata> m_smallLineMetadata;
</span><span class="cx">     std::array<size_t, sizeClassCount> m_pageClasses;
</span><span class="cx"> 
</span><del>-    std::array<List<SmallPage>, sizeClassCount> m_freeLines;
</del><ins>+    LineCache m_lineCache;
</ins><span class="cx">     std::array<List<Chunk>, pageClassCount> m_freePages;
</span><span class="cx">     std::array<List<Chunk>, pageClassCount> m_chunkCache;
</span><span class="cx"> 
</span><span class="lines">@@ -143,18 +145,19 @@
</span><span class="cx"> 
</span><span class="cx"> inline void Heap::allocateSmallBumpRanges(
</span><span class="cx">     std::lock_guard<StaticMutex>& lock, size_t sizeClass,
</span><del>-    BumpAllocator& allocator, BumpRangeCache& rangeCache)
</del><ins>+    BumpAllocator& allocator, BumpRangeCache& rangeCache,
+    LineCache& lineCache)
</ins><span class="cx"> {
</span><span class="cx">     if (sizeClass < bmalloc::sizeClass(smallLineSize))
</span><del>-        return allocateSmallBumpRangesByMetadata(lock, sizeClass, allocator, rangeCache);
-    return allocateSmallBumpRangesByObject(lock, sizeClass, allocator, rangeCache);
</del><ins>+        return allocateSmallBumpRangesByMetadata(lock, sizeClass, allocator, rangeCache, lineCache);
+    return allocateSmallBumpRangesByObject(lock, sizeClass, allocator, rangeCache, lineCache);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-inline void Heap::derefSmallLine(std::lock_guard<StaticMutex>& lock, Object object)
</del><ins>+inline void Heap::derefSmallLine(std::lock_guard<StaticMutex>& lock, Object object, LineCache& lineCache)
</ins><span class="cx"> {
</span><span class="cx">     if (!object.line()->deref(lock))
</span><span class="cx">         return;
</span><del>-    deallocateSmallLine(lock, object);
</del><ins>+    deallocateSmallLine(lock, object, lineCache);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> } // namespace bmalloc
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocListh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/List.h (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/List.h      2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/List.h 2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -103,7 +103,7 @@
</span><span class="cx">         return static_cast<T*>(result);
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    void insertAfter(ListNode<T>* it, ListNode<T>* node)
</del><ins>+    static void insertAfter(ListNode<T>* it, ListNode<T>* node)
</ins><span class="cx">     {
</span><span class="cx">         ListNode<T>* prev = it;
</span><span class="cx">         ListNode<T>* next = it->next;
</span><span class="lines">@@ -115,7 +115,7 @@
</span><span class="cx">         prev->next = node;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    void remove(ListNode<T>* node)
</del><ins>+    static void remove(ListNode<T>* node)
</ins><span class="cx">     {
</span><span class="cx">         ListNode<T>* next = node->next;
</span><span class="cx">         ListNode<T>* prev = node->prev;
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocSmallPageh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/SmallPage.h (218787 => 218788)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/SmallPage.h 2017-06-24 15:14:40 UTC (rev 218787)
+++ trunk/Source/bmalloc/bmalloc/SmallPage.h    2017-06-24 20:14:33 UTC (rev 218788)
</span><span class="lines">@@ -34,6 +34,8 @@
</span><span class="cx"> 
</span><span class="cx"> namespace bmalloc {
</span><span class="cx"> 
</span><ins>+class SmallLine;
+
</ins><span class="cx"> class SmallPage : public ListNode<SmallPage> {
</span><span class="cx"> public:
</span><span class="cx">     void ref(std::lock_guard<StaticMutex>&);
</span><span class="lines">@@ -66,6 +68,8 @@
</span><span class="cx">     "Largest size class must fit in SmallPage metadata");
</span><span class="cx"> };
</span><span class="cx"> 
</span><ins>+using LineCache = std::array<List<SmallPage>, sizeClassCount>;
+
</ins><span class="cx"> inline void SmallPage::ref(std::lock_guard<StaticMutex>&)
</span><span class="cx"> {
</span><span class="cx">     BASSERT(!m_slide);
</span></span></pre>
</div>
</div>

</body>
</html>