<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[173881] trunk/Source/bmalloc</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/173881">173881</a></dd>
<dt>Author</dt> <dd>ggaren@apple.com</dd>
<dt>Date</dt> <dd>2014-09-23 11:24:02 -0700 (Tue, 23 Sep 2014)</dd>
</dl>
<h3>Log Message</h3>
<pre>bmalloc: Allocation should be more precise
https://bugs.webkit.org/show_bug.cgi?id=136993
Reviewed by Gavin Barraclough.
13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
This patch teaches the allocator to merge adjacent free lines into a
single allocatable range. This allows us to shrink the size of an
individual line without increasing fragmentation or the rate of allocator
slow paths.
We'll only take more slow paths when available memory is sparse, which
is exactly when it's worth it. When available memory is dense, we'll
take fewer slow paths.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::divideRoundingUp):
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator): Updated for interface changes.
(bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
Now that we merge lines, it's not convenient to scavenge by line.
(bmalloc::Allocator::allocateSmallBumpRange):
(bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
instead of individual lines.
(bmalloc::Allocator::allocateSlowCase):
(bmalloc::Allocator::allocateSmallLine): Deleted.
(bmalloc::Allocator::allocateMediumLine): Deleted.
(bmalloc::Allocator::allocateMedium): Deleted.
* bmalloc/Allocator.h:
(bmalloc::Allocator::allocateFastCase): Folded medium allocations
into the standard fast path with small allocations. Since a BumpAllocator
just allocates out of an arbitrary range, it doesn't need to distinguish
between small and medium lines.
* bmalloc/BumpAllocator.h:
(bmalloc::BumpAllocator::size):
(bmalloc::BumpAllocator::BumpAllocator):
(bmalloc::BumpAllocator::init):
(bmalloc::BumpAllocator::refill):
(bmalloc::BumpAllocator::line): Deleted. No need to track line information
anymore: the heap just gives us a pointer and a pre-computed number of
objects, and we allocate them.
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::processObjectLog): Updated for interface changes.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
detailing where all objects will lie in memory. After we merge two lines,
we might allocate an object that spans from one line to the next. This
metadata details which bits of memory overlap in that way, and how they
overlap.
(bmalloc::Heap::refillSmallBumpRangeCache):
(bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
and merge adjacent free lines into BumpRanges.
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::allocateMediumPage):
(bmalloc::Heap::deallocateSmallLine):
(bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
since we scan for free memory a page at a time.
(bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
(bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
fast path.
* bmalloc/Heap.h:
(bmalloc::Heap::derefSmallLine):
(bmalloc::Heap::derefMediumLine):
(bmalloc::Heap::deallocateSmallLine): Deleted.
(bmalloc::Heap::allocateSmallLine): Deleted.
(bmalloc::Heap::deallocateMediumLine): Deleted.
(bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
* bmalloc/Line.h:
(bmalloc::Line<Traits>::ref):
(bmalloc::Line<Traits>::deref):
(bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
anymore, since we only ever deref by 1 now.
* bmalloc/MediumAllocator.h:
(bmalloc::MediumAllocator::isNull): Deleted.
(bmalloc::MediumAllocator::MediumAllocator): Deleted.
(bmalloc::MediumAllocator::line): Deleted.
(bmalloc::MediumAllocator::allocate): Deleted.
(bmalloc::MediumAllocator::derefCount): Deleted.
(bmalloc::MediumAllocator::refill): Deleted.
(bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
been dead for a while, since it doesn't build anymore with this patch.
* bmalloc/Page.h:
(bmalloc::Page::sizeClass):
(bmalloc::Page::setSizeClass):
(bmalloc::Page::smallSizeClass): Deleted.
(bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
to sizeClass, since we use it for medium sizes too.
* bmalloc/Sizes.h:
(bmalloc::Sizes::sizeClass):
(bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
(bmalloc::Sizes::smallSizeClassFor): Deleted.
(bmalloc::Sizes::mediumSizeClassFor): Deleted.
* bmalloc/bmalloc.h:
(bmalloc::api::realloc): Now that we have precise objects sizes, realloc
can be a bit more precise. It also has to be, since we can't guarantee
that an object ends at the end of a line anymore.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourcebmallocChangeLog">trunk/Source/bmalloc/ChangeLog</a></li>
<li><a href="#trunkSourcebmallocbmallocAlgorithmh">trunk/Source/bmalloc/bmalloc/Algorithm.h</a></li>
<li><a href="#trunkSourcebmallocbmallocAllocatorcpp">trunk/Source/bmalloc/bmalloc/Allocator.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocAllocatorh">trunk/Source/bmalloc/bmalloc/Allocator.h</a></li>
<li><a href="#trunkSourcebmallocbmallocBumpAllocatorh">trunk/Source/bmalloc/bmalloc/BumpAllocator.h</a></li>
<li><a href="#trunkSourcebmallocbmallocDeallocatorcpp">trunk/Source/bmalloc/bmalloc/Deallocator.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocHeapcpp">trunk/Source/bmalloc/bmalloc/Heap.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocHeaph">trunk/Source/bmalloc/bmalloc/Heap.h</a></li>
<li><a href="#trunkSourcebmallocbmallocLineh">trunk/Source/bmalloc/bmalloc/Line.h</a></li>
<li><a href="#trunkSourcebmallocbmallocPageh">trunk/Source/bmalloc/bmalloc/Page.h</a></li>
<li><a href="#trunkSourcebmallocbmallocSizesh">trunk/Source/bmalloc/bmalloc/Sizes.h</a></li>
<li><a href="#trunkSourcebmallocbmallocbmalloch">trunk/Source/bmalloc/bmalloc/bmalloc.h</a></li>
<li><a href="#trunkSourcebmallocbmallocxcodeprojprojectpbxproj">trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkSourcebmallocbmallocBumpRangeh">trunk/Source/bmalloc/bmalloc/BumpRange.h</a></li>
<li><a href="#trunkSourcebmallocbmallocLineMetadatah">trunk/Source/bmalloc/bmalloc/LineMetadata.h</a></li>
</ul>
<h3>Removed Paths</h3>
<ul>
<li><a href="#trunkSourcebmallocbmallocMediumAllocatorh">trunk/Source/bmalloc/bmalloc/MediumAllocator.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourcebmallocChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/ChangeLog (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/ChangeLog 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/ChangeLog 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -1,3 +1,122 @@
</span><ins>+2014-09-23 Geoffrey Garen <ggaren@apple.com>
+
+ bmalloc: Allocation should be more precise
+ https://bugs.webkit.org/show_bug.cgi?id=136993
+
+ Reviewed by Gavin Barraclough.
+
+ 13% reduction in heap size on the MallocBench *_memory_warning benchmarks.
+
+ This patch teaches the allocator to merge adjacent free lines into a
+ single allocatable range. This allows us to shrink the size of an
+ individual line without increasing fragmentation or the rate of allocator
+ slow paths.
+
+ We'll only take more slow paths when available memory is sparse, which
+ is exactly when it's worth it. When available memory is dense, we'll
+ take fewer slow paths.
+
+ * bmalloc.xcodeproj/project.pbxproj:
+ * bmalloc/Algorithm.h:
+ (bmalloc::divideRoundingUp):
+
+ * bmalloc/Allocator.cpp:
+ (bmalloc::Allocator::Allocator): Updated for interface changes.
+
+ (bmalloc::Allocator::scavenge): Scavenge by object instead of by line.
+ Now that we merge lines, it's not convenient to scavenge by line.
+
+ (bmalloc::Allocator::allocateSmallBumpRange):
+ (bmalloc::Allocator::allocateMediumBumpRange): Allocate whole ranges
+ instead of individual lines.
+
+ (bmalloc::Allocator::allocateSlowCase):
+ (bmalloc::Allocator::allocateSmallLine): Deleted.
+ (bmalloc::Allocator::allocateMediumLine): Deleted.
+ (bmalloc::Allocator::allocateMedium): Deleted.
+ * bmalloc/Allocator.h:
+ (bmalloc::Allocator::allocateFastCase): Folded medium allocations
+ into the standard fast path with small allocations. Since a BumpAllocator
+ just allocates out of an arbitrary range, it doesn't need to distinguish
+ between small and medium lines.
+
+ * bmalloc/BumpAllocator.h:
+ (bmalloc::BumpAllocator::size):
+ (bmalloc::BumpAllocator::BumpAllocator):
+ (bmalloc::BumpAllocator::init):
+ (bmalloc::BumpAllocator::refill):
+ (bmalloc::BumpAllocator::line): Deleted. No need to track line information
+ anymore: the heap just gives us a pointer and a pre-computed number of
+ objects, and we allocate them.
+
+ * bmalloc/Deallocator.cpp:
+ (bmalloc::Deallocator::processObjectLog): Updated for interface changes.
+
+ * bmalloc/Heap.cpp:
+ (bmalloc::Heap::Heap):
+ (bmalloc::Heap::initializeLineMetadata): Pre-compute precise metadata
+ detailing where all objects will lie in memory. After we merge two lines,
+ we might allocate an object that spans from one line to the next. This
+ metadata details which bits of memory overlap in that way, and how they
+ overlap.
+
+ (bmalloc::Heap::refillSmallBumpRangeCache):
+ (bmalloc::Heap::refillMediumBumpRangeCache): Scan a whole page at a time,
+ and merge adjacent free lines into BumpRanges.
+
+ (bmalloc::Heap::allocateSmallPage):
+ (bmalloc::Heap::allocateMediumPage):
+ (bmalloc::Heap::deallocateSmallLine):
+ (bmalloc::Heap::deallocateMediumLine): Track pages rather than lines,
+ since we scan for free memory a page at a time.
+
+ (bmalloc::Heap::allocateSmallLineSlowCase): Deleted.
+ (bmalloc::Heap::allocateMediumLineSlowCase): Deleted. Folded into the
+ fast path.
+
+ * bmalloc/Heap.h:
+ (bmalloc::Heap::derefSmallLine):
+ (bmalloc::Heap::derefMediumLine):
+ (bmalloc::Heap::deallocateSmallLine): Deleted.
+ (bmalloc::Heap::allocateSmallLine): Deleted.
+ (bmalloc::Heap::deallocateMediumLine): Deleted.
+ (bmalloc::Heap::allocateMediumLine): Deleted. Updated for interface changes.
+
+ * bmalloc/Line.h:
+ (bmalloc::Line<Traits>::ref):
+ (bmalloc::Line<Traits>::deref):
+ (bmalloc::Line<Traits>::concurrentRef): Deleted. We don't pass a derefCount
+ anymore, since we only ever deref by 1 now.
+
+ * bmalloc/MediumAllocator.h:
+ (bmalloc::MediumAllocator::isNull): Deleted.
+ (bmalloc::MediumAllocator::MediumAllocator): Deleted.
+ (bmalloc::MediumAllocator::line): Deleted.
+ (bmalloc::MediumAllocator::allocate): Deleted.
+ (bmalloc::MediumAllocator::derefCount): Deleted.
+ (bmalloc::MediumAllocator::refill): Deleted.
+ (bmalloc::MediumAllocator::clear): Deleted. Deleted some code that's
+ been dead for a while, since it doesn't build anymore with this patch.
+
+ * bmalloc/Page.h:
+ (bmalloc::Page::sizeClass):
+ (bmalloc::Page::setSizeClass):
+ (bmalloc::Page::smallSizeClass): Deleted.
+ (bmalloc::Page::setSmallSizeClass): Deleted. Renamed setSmallSizeClass
+ to sizeClass, since we use it for medium sizes too.
+
+ * bmalloc/Sizes.h:
+ (bmalloc::Sizes::sizeClass):
+ (bmalloc::Sizes::objectSize): Shrank line sizes to save memory.
+
+ (bmalloc::Sizes::smallSizeClassFor): Deleted.
+ (bmalloc::Sizes::mediumSizeClassFor): Deleted.
+
+ * bmalloc/bmalloc.h:
+ (bmalloc::api::realloc): Now that we have precise objects sizes, realloc
+ can be a bit more precise. It also has to be, since we can't guarantee
+ that an object ends at the end of a line anymore.
+
</ins><span class="cx"> 2014-09-19 Daniel Bates <dabates@apple.com>
</span><span class="cx">
</span><span class="cx"> Always assume internal SDK when building configuration Production
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocAlgorithmh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Algorithm.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Algorithm.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Algorithm.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -74,6 +74,15 @@
</span><span class="cx"> return reinterpret_cast<T>(mask(reinterpret_cast<uintptr_t>(x), ~(divisor - 1ul)));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+template<typename T> void divideRoundingUp(T numerator, T denominator, T& quotient, T& remainder)
+{
+ // We expect the compiler to emit a single divide instruction to extract both the quotient and the remainder.
+ quotient = numerator / denominator;
+ remainder = numerator % denominator;
+ if (remainder)
+ quotient += 1;
+}
+
</ins><span class="cx"> // Version of sizeof that returns 0 for empty classes.
</span><span class="cx">
</span><span class="cx"> template<typename T> inline constexpr size_t sizeOf()
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocAllocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Allocator.cpp (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Allocator.cpp 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Allocator.cpp 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -38,11 +38,11 @@
</span><span class="cx"> Allocator::Allocator(Deallocator& deallocator)
</span><span class="cx"> : m_deallocator(deallocator)
</span><span class="cx"> {
</span><del>- for (unsigned short i = alignment; i <= smallMax; i += alignment)
- m_smallAllocators[smallSizeClassFor(i)].init<SmallLine>(i);
</del><ins>+ for (unsigned short size = alignment; size <= smallMax; size += alignment)
+ m_bumpAllocators[sizeClass(size)].init(size);
</ins><span class="cx">
</span><del>- for (unsigned short i = smallMax + alignment; i <= mediumMax; i += alignment)
- m_mediumAllocators[mediumSizeClassFor(i)].init<MediumLine>(i);
</del><ins>+ for (unsigned short size = smallMax + alignment; size <= mediumMax; size += alignment)
+ m_bumpAllocators[sizeClass(size)].init(size);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> Allocator::~Allocator()
</span><span class="lines">@@ -52,54 +52,59 @@
</span><span class="cx">
</span><span class="cx"> void Allocator::scavenge()
</span><span class="cx"> {
</span><del>- for (auto& allocator : m_smallAllocators) {
</del><ins>+ for (unsigned short i = alignment; i <= smallMax; i += alignment) {
+ BumpAllocator& allocator = m_bumpAllocators[sizeClass(i)];
+ SmallBumpRangeCache& bumpRangeCache = m_smallBumpRangeCaches[sizeClass(i)];
+
</ins><span class="cx"> while (allocator.canAllocate())
</span><span class="cx"> m_deallocator.deallocate(allocator.allocate());
</span><ins>+
+ while (bumpRangeCache.size()) {
+ allocator.refill(bumpRangeCache.pop());
+ while (allocator.canAllocate())
+ m_deallocator.deallocate(allocator.allocate());
+ }
+
</ins><span class="cx"> allocator.clear();
</span><span class="cx"> }
</span><span class="cx">
</span><del>- for (auto& allocator : m_mediumAllocators) {
</del><ins>+ for (unsigned short i = smallMax + alignment; i <= mediumMax; i += alignment) {
+ BumpAllocator& allocator = m_bumpAllocators[sizeClass(i)];
+ MediumBumpRangeCache& bumpRangeCache = m_mediumBumpRangeCaches[sizeClass(i)];
+
</ins><span class="cx"> while (allocator.canAllocate())
</span><span class="cx"> m_deallocator.deallocate(allocator.allocate());
</span><ins>+
+ while (bumpRangeCache.size()) {
+ allocator.refill(bumpRangeCache.pop());
+ while (allocator.canAllocate())
+ m_deallocator.deallocate(allocator.allocate());
+ }
+
</ins><span class="cx"> allocator.clear();
</span><span class="cx"> }
</span><del>-
- std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
- Heap* heap = PerProcess<Heap>::getFastCase();
-
- for (auto& smallLineCache : m_smallLineCaches) {
- while (smallLineCache.size())
- heap->deallocateSmallLine(lock, smallLineCache.pop());
- }
- while (m_mediumLineCache.size())
- heap->deallocateMediumLine(lock, m_mediumLineCache.pop());
</del><span class="cx"> }
</span><span class="cx">
</span><del>-SmallLine* Allocator::allocateSmallLine(size_t smallSizeClass)
</del><ins>+BumpRange Allocator::allocateSmallBumpRange(size_t sizeClass)
</ins><span class="cx"> {
</span><del>- SmallLineCache& smallLineCache = m_smallLineCaches[smallSizeClass];
- if (!smallLineCache.size()) {
</del><ins>+ SmallBumpRangeCache& bumpRangeCache = m_smallBumpRangeCaches[sizeClass];
+ if (!bumpRangeCache.size()) {
</ins><span class="cx"> std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
</span><del>- Heap* heap = PerProcess<Heap>::getFastCase();
-
- while (smallLineCache.size() != smallLineCache.capacity())
- smallLineCache.push(heap->allocateSmallLine(lock, smallSizeClass));
</del><ins>+ PerProcess<Heap>::getFastCase()->refillSmallBumpRangeCache(lock, sizeClass, bumpRangeCache);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- return smallLineCache.pop();
</del><ins>+ return bumpRangeCache.pop();
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-MediumLine* Allocator::allocateMediumLine()
</del><ins>+BumpRange Allocator::allocateMediumBumpRange(size_t sizeClass)
</ins><span class="cx"> {
</span><del>- if (!m_mediumLineCache.size()) {
</del><ins>+ MediumBumpRangeCache& bumpRangeCache = m_mediumBumpRangeCaches[sizeClass];
+ if (!bumpRangeCache.size()) {
</ins><span class="cx"> std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
</span><del>- Heap* heap = PerProcess<Heap>::getFastCase();
-
- while (m_mediumLineCache.size() != m_mediumLineCache.capacity())
- m_mediumLineCache.push(heap->allocateMediumLine(lock));
</del><ins>+ PerProcess<Heap>::getFastCase()->refillMediumBumpRangeCache(lock, sizeClass, bumpRangeCache);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- return m_mediumLineCache.pop();
</del><ins>+ return bumpRangeCache.pop();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void* Allocator::allocateLarge(size_t size)
</span><span class="lines">@@ -116,30 +121,27 @@
</span><span class="cx"> return PerProcess<Heap>::getFastCase()->allocateXLarge(lock, size);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void* Allocator::allocateMedium(size_t size)
-{
- BumpAllocator& allocator = m_mediumAllocators[mediumSizeClassFor(size)];
-
- if (!allocator.canAllocate())
- allocator.refill(allocateMediumLine());
- return allocator.allocate();
-}
-
</del><span class="cx"> void* Allocator::allocateSlowCase(size_t size)
</span><span class="cx"> {
</span><span class="cx"> IF_DEBUG(
</span><span class="cx"> void* dummy;
</span><span class="cx"> BASSERT(!allocateFastCase(size, dummy));
</span><span class="cx"> )
</span><del>- if (size <= smallMax) {
- size_t smallSizeClass = smallSizeClassFor(size);
- BumpAllocator& allocator = m_smallAllocators[smallSizeClass];
- allocator.refill(allocateSmallLine(smallSizeClass));
- return allocator.allocate();
</del><ins>+
+ if (size <= mediumMax) {
+ size_t sizeClass = bmalloc::sizeClass(size);
+ BumpAllocator& allocator = m_bumpAllocators[sizeClass];
+
+ if (allocator.size() <= smallMax) {
+ allocator.refill(allocateSmallBumpRange(sizeClass));
+ return allocator.allocate();
+ }
+
+ if (allocator.size() <= mediumMax) {
+ allocator.refill(allocateMediumBumpRange(sizeClass));
+ return allocator.allocate();
+ }
</ins><span class="cx"> }
</span><del>-
- if (size <= mediumMax)
- return allocateMedium(size);
</del><span class="cx">
</span><span class="cx"> if (size <= largeMax)
</span><span class="cx"> return allocateLarge(size);
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocAllocatorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Allocator.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Allocator.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Allocator.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -28,7 +28,7 @@
</span><span class="cx">
</span><span class="cx"> #include "BumpAllocator.h"
</span><span class="cx"> #include "FixedVector.h"
</span><del>-#include "MediumAllocator.h"
</del><ins>+#include "Heap.h"
</ins><span class="cx"> #include "Sizes.h"
</span><span class="cx"> #include "SmallLine.h"
</span><span class="cx"> #include <array>
</span><span class="lines">@@ -51,33 +51,29 @@
</span><span class="cx"> void scavenge();
</span><span class="cx">
</span><span class="cx"> private:
</span><del>- typedef FixedVector<SmallLine*, smallLineCacheCapacity> SmallLineCache;
- typedef FixedVector<MediumLine*, mediumLineCacheCapacity> MediumLineCache;
-
</del><span class="cx"> void* allocateFastCase(BumpAllocator&);
</span><span class="cx">
</span><span class="cx"> void* allocateMedium(size_t);
</span><span class="cx"> void* allocateLarge(size_t);
</span><span class="cx"> void* allocateXLarge(size_t);
</span><span class="cx">
</span><del>- SmallLine* allocateSmallLine(size_t smallSizeClass);
- MediumLine* allocateMediumLine();
</del><ins>+ BumpRange allocateSmallBumpRange(size_t sizeClass);
+ BumpRange allocateMediumBumpRange(size_t sizeClass);
</ins><span class="cx">
</span><span class="cx"> Deallocator& m_deallocator;
</span><span class="cx">
</span><del>- std::array<BumpAllocator, smallMax / alignment> m_smallAllocators;
- std::array<BumpAllocator, mediumMax / alignment> m_mediumAllocators;
</del><ins>+ std::array<BumpAllocator, mediumMax / alignment> m_bumpAllocators;
</ins><span class="cx">
</span><del>- std::array<SmallLineCache, smallMax / alignment> m_smallLineCaches;
- MediumLineCache m_mediumLineCache;
</del><ins>+ std::array<SmallBumpRangeCache, smallMax / alignment> m_smallBumpRangeCaches;
+ std::array<MediumBumpRangeCache, mediumMax / alignment> m_mediumBumpRangeCaches;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> inline bool Allocator::allocateFastCase(size_t size, void*& object)
</span><span class="cx"> {
</span><del>- if (size > smallMax)
</del><ins>+ if (size > mediumMax)
</ins><span class="cx"> return false;
</span><span class="cx">
</span><del>- BumpAllocator& allocator = m_smallAllocators[smallSizeClassFor(size)];
</del><ins>+ BumpAllocator& allocator = m_bumpAllocators[sizeClass(size)];
</ins><span class="cx"> if (!allocator.canAllocate())
</span><span class="cx"> return false;
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocBumpAllocatorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/BumpAllocator.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/BumpAllocator.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/BumpAllocator.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -27,6 +27,7 @@
</span><span class="cx"> #define BumpAllocator_h
</span><span class="cx">
</span><span class="cx"> #include "BAssert.h"
</span><ins>+#include "BumpRange.h"
</ins><span class="cx"> #include "ObjectType.h"
</span><span class="cx">
</span><span class="cx"> namespace bmalloc {
</span><span class="lines">@@ -36,51 +37,40 @@
</span><span class="cx"> class BumpAllocator {
</span><span class="cx"> public:
</span><span class="cx"> BumpAllocator();
</span><del>- template<typename Line> void init(size_t);
</del><ins>+ void init(size_t);
</ins><span class="cx">
</span><ins>+ size_t size() { return m_size; }
+
</ins><span class="cx"> bool isNull() { return !m_ptr; }
</span><del>- template<typename Line> Line* line();
</del><ins>+ void clear();
</ins><span class="cx">
</span><span class="cx"> bool canAllocate() { return !!m_remaining; }
</span><span class="cx"> void* allocate();
</span><span class="cx">
</span><del>- template<typename Line> void refill(Line*);
- void clear();
</del><ins>+ void refill(const BumpRange&);
</ins><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> void validate(void*);
</span><span class="cx">
</span><span class="cx"> char* m_ptr;
</span><span class="cx"> unsigned short m_size;
</span><del>- unsigned char m_remaining;
- unsigned char m_maxObjectCount;
</del><ins>+ unsigned short m_remaining;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> inline BumpAllocator::BumpAllocator()
</span><span class="cx"> : m_ptr()
</span><span class="cx"> , m_size()
</span><span class="cx"> , m_remaining()
</span><del>- , m_maxObjectCount()
</del><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<typename Line>
</del><span class="cx"> inline void BumpAllocator::init(size_t size)
</span><span class="cx"> {
</span><del>- BASSERT(size >= Line::minimumObjectSize);
-
</del><span class="cx"> m_ptr = nullptr;
</span><span class="cx"> m_size = size;
</span><span class="cx"> m_remaining = 0;
</span><del>- m_maxObjectCount = Line::lineSize / size;
</del><span class="cx"> }
</span><span class="cx">
</span><del>-template<typename Line>
-inline Line* BumpAllocator::line()
-{
- return Line::get(canAllocate() ? m_ptr : m_ptr - 1);
-}
-
</del><span class="cx"> inline void BumpAllocator::validate(void* ptr)
</span><span class="cx"> {
</span><span class="cx"> UNUSED(ptr);
</span><span class="lines">@@ -104,13 +94,11 @@
</span><span class="cx"> return result;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-template<typename Line>
-inline void BumpAllocator::refill(Line* line)
</del><ins>+inline void BumpAllocator::refill(const BumpRange& bumpRange)
</ins><span class="cx"> {
</span><span class="cx"> BASSERT(!canAllocate());
</span><del>- line->concurrentRef(m_maxObjectCount);
- m_ptr = line->begin();
- m_remaining = m_maxObjectCount;
</del><ins>+ m_ptr = bumpRange.begin;
+ m_remaining = bumpRange.objectCount;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void BumpAllocator::clear()
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocBumpRangeh"></a>
<div class="addfile"><h4>Added: trunk/Source/bmalloc/bmalloc/BumpRange.h (0 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/BumpRange.h (rev 0)
+++ trunk/Source/bmalloc/bmalloc/BumpRange.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -0,0 +1,45 @@
</span><ins>+/*
+ * Copyright (C) 2014 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef BumpRange_h
+#define BumpRange_h
+
+#include "FixedVector.h"
+#include "Range.h"
+#include "Sizes.h"
+
+namespace bmalloc {
+
+struct BumpRange {
+ char* begin;
+ unsigned short objectCount;
+};
+
+typedef FixedVector<BumpRange, smallRangeCacheCapacity> SmallBumpRangeCache;
+typedef FixedVector<BumpRange, mediumRangeCacheCapacity> MediumBumpRangeCache;
+
+} // namespace bmalloc
+
+#endif // BumpRange_h
</ins></span></pre></div>
<a id="trunkSourcebmallocbmallocDeallocatorcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Deallocator.cpp (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Deallocator.cpp 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Deallocator.cpp 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -72,15 +72,11 @@
</span><span class="cx"> for (auto object : m_objectLog) {
</span><span class="cx"> if (isSmall(object)) {
</span><span class="cx"> SmallLine* line = SmallLine::get(object);
</span><del>- if (!line->deref(lock))
- continue;
- heap->deallocateSmallLine(lock, line);
</del><ins>+ heap->derefSmallLine(lock, line);
</ins><span class="cx"> } else {
</span><del>- BASSERT(isSmallOrMedium(object));
</del><ins>+ BASSERT(isMedium(object));
</ins><span class="cx"> MediumLine* line = MediumLine::get(object);
</span><del>- if (!line->deref(lock))
- continue;
- heap->deallocateMediumLine(lock, line);
</del><ins>+ heap->derefMediumLine(lock, line);
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.cpp (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.cpp 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Heap.cpp 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -50,8 +50,44 @@
</span><span class="cx"> : m_isAllocatingPages(false)
</span><span class="cx"> , m_scavenger(*this, &Heap::concurrentScavenge)
</span><span class="cx"> {
</span><ins>+ initializeLineMetadata();
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void Heap::initializeLineMetadata()
+{
+ for (unsigned short size = alignment; size <= smallMax; size += alignment) {
+ unsigned short startOffset = 0;
+ for (size_t lineNumber = 0; lineNumber < SmallPage::lineCount - 1; ++lineNumber) {
+ unsigned short objectCount;
+ unsigned short remainder;
+ divideRoundingUp(static_cast<unsigned short>(SmallPage::lineSize - startOffset), size, objectCount, remainder);
+ BASSERT(objectCount);
+ m_smallLineMetadata[sizeClass(size)][lineNumber] = { startOffset, objectCount };
+ startOffset = remainder ? size - remainder : 0;
+ }
+
+ // The last line in the page rounds down instead of up because it's not allowed to overlap into its neighbor.
+ unsigned short objectCount = static_cast<unsigned short>((SmallPage::lineSize - startOffset) / size);
+ m_smallLineMetadata[sizeClass(size)][SmallPage::lineCount - 1] = { startOffset, objectCount };
+ }
+
+ for (unsigned short size = smallMax + alignment; size <= mediumMax; size += alignment) {
+ unsigned short startOffset = 0;
+ for (size_t lineNumber = 0; lineNumber < MediumPage::lineCount - 1; ++lineNumber) {
+ unsigned short objectCount;
+ unsigned short remainder;
+ divideRoundingUp(static_cast<unsigned short>(MediumPage::lineSize - startOffset), size, objectCount, remainder);
+ BASSERT(objectCount);
+ m_mediumLineMetadata[sizeClass(size)][lineNumber] = { startOffset, objectCount };
+ startOffset = remainder ? size - remainder : 0;
+ }
+
+ // The last line in the page rounds down instead of up because it's not allowed to overlap into its neighbor.
+ unsigned short objectCount = static_cast<unsigned short>((MediumPage::lineSize - startOffset) / size);
+ m_mediumLineMetadata[sizeClass(size)][MediumPage::lineCount - 1] = { startOffset, objectCount };
+ }
+}
+
</ins><span class="cx"> void Heap::concurrentScavenge()
</span><span class="cx"> {
</span><span class="cx"> std::unique_lock<StaticMutex> lock(PerProcess<Heap>::mutex());
</span><span class="lines">@@ -116,11 +152,93 @@
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx">
</span><del>-SmallLine* Heap::allocateSmallLineSlowCase(std::lock_guard<StaticMutex>& lock, size_t smallSizeClass)
</del><ins>+void Heap::refillSmallBumpRangeCache(std::lock_guard<StaticMutex>& lock, size_t sizeClass, SmallBumpRangeCache& rangeCache)
</ins><span class="cx"> {
</span><ins>+ BASSERT(!rangeCache.size());
+ SmallPage* page = allocateSmallPage(lock, sizeClass);
+ SmallLine* lines = page->begin();
+
+ // Due to overlap from the previous line, the last line in the page may not be able to fit any objects.
+ size_t end = SmallPage::lineCount;
+ if (!m_smallLineMetadata[sizeClass][SmallPage::lineCount - 1].objectCount)
+ --end;
+
+ // Find a free line.
+ for (size_t lineNumber = 0; lineNumber < end; ++lineNumber) {
+ if (lines[lineNumber].refCount(lock))
+ continue;
+
+ LineMetadata& lineMetadata = m_smallLineMetadata[sizeClass][lineNumber];
+ char* begin = lines[lineNumber].begin() + lineMetadata.startOffset;
+ unsigned short objectCount = lineMetadata.objectCount;
+ lines[lineNumber].ref(lock, lineMetadata.objectCount);
+ page->ref(lock);
+
+ // Merge with subsequent free lines.
+ while (++lineNumber < end) {
+ if (lines[lineNumber].refCount(lock))
+ break;
+
+ LineMetadata& lineMetadata = m_smallLineMetadata[sizeClass][lineNumber];
+ objectCount += lineMetadata.objectCount;
+ lines[lineNumber].ref(lock, lineMetadata.objectCount);
+ page->ref(lock);
+ }
+
+ rangeCache.push({ begin, objectCount });
+ }
+}
+
+void Heap::refillMediumBumpRangeCache(std::lock_guard<StaticMutex>& lock, size_t sizeClass, MediumBumpRangeCache& rangeCache)
+{
+ MediumPage* page = allocateMediumPage(lock, sizeClass);
+ BASSERT(!rangeCache.size());
+ MediumLine* lines = page->begin();
+
+ // Due to overlap from the previous line, the last line in the page may not be able to fit any objects.
+ size_t end = MediumPage::lineCount;
+ if (!m_mediumLineMetadata[sizeClass][MediumPage::lineCount - 1].objectCount)
+ --end;
+
+ // Find a free line.
+ for (size_t lineNumber = 0; lineNumber < end; ++lineNumber) {
+ if (lines[lineNumber].refCount(lock))
+ continue;
+
+ LineMetadata& lineMetadata = m_mediumLineMetadata[sizeClass][lineNumber];
+ char* begin = lines[lineNumber].begin() + lineMetadata.startOffset;
+ unsigned short objectCount = lineMetadata.objectCount;
+ lines[lineNumber].ref(lock, lineMetadata.objectCount);
+ page->ref(lock);
+
+ // Merge with subsequent free lines.
+ while (++lineNumber < end) {
+ if (lines[lineNumber].refCount(lock))
+ break;
+
+ LineMetadata& lineMetadata = m_mediumLineMetadata[sizeClass][lineNumber];
+ objectCount += lineMetadata.objectCount;
+ lines[lineNumber].ref(lock, lineMetadata.objectCount);
+ page->ref(lock);
+ }
+
+ rangeCache.push({ begin, objectCount });
+ }
+}
+
+SmallPage* Heap::allocateSmallPage(std::lock_guard<StaticMutex>& lock, size_t sizeClass)
+{
+ Vector<SmallPage*>& smallPagesWithFreeLines = m_smallPagesWithFreeLines[sizeClass];
+ while (smallPagesWithFreeLines.size()) {
+ SmallPage* page = smallPagesWithFreeLines.pop();
+ if (!page->refCount(lock) || page->sizeClass() != sizeClass) // Page was promoted to the pages list.
+ continue;
+ return page;
+ }
+
</ins><span class="cx"> m_isAllocatingPages = true;
</span><span class="cx">
</span><del>- SmallPage* page = [this]() {
</del><ins>+ SmallPage* page = [this, sizeClass]() {
</ins><span class="cx"> if (m_smallPages.size())
</span><span class="cx"> return m_smallPages.pop();
</span><span class="cx">
</span><span class="lines">@@ -129,22 +247,23 @@
</span><span class="cx"> return page;
</span><span class="cx"> }();
</span><span class="cx">
</span><del>- SmallLine* line = page->begin();
- Vector<SmallLine*>& smallLines = m_smallLines[smallSizeClass];
- for (auto it = line + 1; it != page->end(); ++it)
- smallLines.push(it);
-
- BASSERT(!line->refCount(lock));
- page->setSmallSizeClass(smallSizeClass);
- page->ref(lock);
- return line;
</del><ins>+ page->setSizeClass(sizeClass);
+ return page;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-MediumLine* Heap::allocateMediumLineSlowCase(std::lock_guard<StaticMutex>& lock)
</del><ins>+MediumPage* Heap::allocateMediumPage(std::lock_guard<StaticMutex>& lock, size_t sizeClass)
</ins><span class="cx"> {
</span><ins>+ Vector<MediumPage*>& mediumPagesWithFreeLines = m_mediumPagesWithFreeLines[sizeClass];
+ while (mediumPagesWithFreeLines.size()) {
+ MediumPage* page = mediumPagesWithFreeLines.pop();
+ if (!page->refCount(lock) || page->sizeClass() != sizeClass) // Page was promoted to the pages list.
+ continue;
+ return page;
+ }
+
</ins><span class="cx"> m_isAllocatingPages = true;
</span><span class="cx">
</span><del>- MediumPage* page = [this]() {
</del><ins>+ MediumPage* page = [this, sizeClass]() {
</ins><span class="cx"> if (m_mediumPages.size())
</span><span class="cx"> return m_mediumPages.pop();
</span><span class="cx">
</span><span class="lines">@@ -153,14 +272,54 @@
</span><span class="cx"> return page;
</span><span class="cx"> }();
</span><span class="cx">
</span><del>- MediumLine* line = page->begin();
- for (auto it = line + 1; it != page->end(); ++it)
- m_mediumLines.push(it);
</del><ins>+ page->setSizeClass(sizeClass);
+ return page;
+}
</ins><span class="cx">
</span><del>- page->ref(lock);
- return line;
</del><ins>+void Heap::deallocateSmallLine(std::lock_guard<StaticMutex>& lock, SmallLine* line)
+{
+ BASSERT(!line->refCount(lock));
+ SmallPage* page = SmallPage::get(line);
+ size_t refCount = page->refCount(lock);
+ page->deref(lock);
+
+ switch (refCount) {
+ case SmallPage::lineCount: {
+ // First free line in the page.
+ m_smallPagesWithFreeLines[page->sizeClass()].push(page);
+ break;
+ }
+ case 1: {
+ // Last free line in the page.
+ m_smallPages.push(page);
+ m_scavenger.run();
+ break;
+ }
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+void Heap::deallocateMediumLine(std::lock_guard<StaticMutex>& lock, MediumLine* line)
+{
+ BASSERT(!line->refCount(lock));
+ MediumPage* page = MediumPage::get(line);
+ size_t refCount = page->refCount(lock);
+ page->deref(lock);
+
+ switch (refCount) {
+ case MediumPage::lineCount: {
+ // First free line in the page.
+ m_mediumPagesWithFreeLines[page->sizeClass()].push(page);
+ break;
+ }
+ case 1: {
+ // Last free line in the page.
+ m_mediumPages.push(page);
+ m_scavenger.run();
+ break;
+ }
+ }
+}
+
</ins><span class="cx"> void* Heap::allocateXLarge(std::lock_guard<StaticMutex>&, size_t size)
</span><span class="cx"> {
</span><span class="cx"> XLargeChunk* chunk = XLargeChunk::create(size);
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Heap.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -26,16 +26,17 @@
</span><span class="cx"> #ifndef Heap_h
</span><span class="cx"> #define Heap_h
</span><span class="cx">
</span><del>-#include "FixedVector.h"
-#include "VMHeap.h"
-#include "MediumLine.h"
-#include "Mutex.h"
-#include "SmallPage.h"
</del><ins>+#include "BumpRange.h"
+#include "LineMetadata.h"
</ins><span class="cx"> #include "MediumChunk.h"
</span><ins>+#include "MediumLine.h"
</ins><span class="cx"> #include "MediumPage.h"
</span><ins>+#include "Mutex.h"
</ins><span class="cx"> #include "SegregatedFreeList.h"
</span><span class="cx"> #include "SmallChunk.h"
</span><span class="cx"> #include "SmallLine.h"
</span><ins>+#include "SmallPage.h"
+#include "VMHeap.h"
</ins><span class="cx"> #include "Vector.h"
</span><span class="cx"> #include <array>
</span><span class="cx"> #include <mutex>
</span><span class="lines">@@ -49,12 +50,12 @@
</span><span class="cx"> public:
</span><span class="cx"> Heap(std::lock_guard<StaticMutex>&);
</span><span class="cx">
</span><del>- SmallLine* allocateSmallLine(std::lock_guard<StaticMutex>&, size_t smallSizeClass);
- void deallocateSmallLine(std::lock_guard<StaticMutex>&, SmallLine*);
</del><ins>+ void refillSmallBumpRangeCache(std::lock_guard<StaticMutex>&, size_t sizeClass, SmallBumpRangeCache&);
+ void derefSmallLine(std::lock_guard<StaticMutex>&, SmallLine*);
</ins><span class="cx">
</span><del>- MediumLine* allocateMediumLine(std::lock_guard<StaticMutex>&);
- void deallocateMediumLine(std::lock_guard<StaticMutex>&, MediumLine*);
-
</del><ins>+ void refillMediumBumpRangeCache(std::lock_guard<StaticMutex>&, size_t sizeClass, MediumBumpRangeCache&);
+ void derefMediumLine(std::lock_guard<StaticMutex>&, MediumLine*);
+
</ins><span class="cx"> void* allocateLarge(std::lock_guard<StaticMutex>&, size_t);
</span><span class="cx"> void deallocateLarge(std::lock_guard<StaticMutex>&, void*);
</span><span class="cx">
</span><span class="lines">@@ -68,10 +69,15 @@
</span><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> ~Heap() = delete;
</span><ins>+
+ void initializeLineMetadata();
</ins><span class="cx">
</span><del>- SmallLine* allocateSmallLineSlowCase(std::lock_guard<StaticMutex>&, size_t smallSizeClass);
- MediumLine* allocateMediumLineSlowCase(std::lock_guard<StaticMutex>&);
</del><ins>+ SmallPage* allocateSmallPage(std::lock_guard<StaticMutex>&, size_t sizeClass);
+ MediumPage* allocateMediumPage(std::lock_guard<StaticMutex>&, size_t sizeClass);
</ins><span class="cx">
</span><ins>+ void deallocateSmallLine(std::lock_guard<StaticMutex>&, SmallLine*);
+ void deallocateMediumLine(std::lock_guard<StaticMutex>&, MediumLine*);
+
</ins><span class="cx"> void* allocateLarge(Range, size_t);
</span><span class="cx"> Range allocateLargeChunk();
</span><span class="cx">
</span><span class="lines">@@ -85,9 +91,12 @@
</span><span class="cx"> void scavengeMediumPages(std::unique_lock<StaticMutex>&, std::chrono::milliseconds);
</span><span class="cx"> void scavengeLargeRanges(std::unique_lock<StaticMutex>&, std::chrono::milliseconds);
</span><span class="cx">
</span><del>- std::array<Vector<SmallLine*>, smallMax / alignment> m_smallLines;
- Vector<MediumLine*> m_mediumLines;
</del><ins>+ std::array<std::array<LineMetadata, SmallPage::lineCount>, smallMax / alignment> m_smallLineMetadata;
+ std::array<std::array<LineMetadata, MediumPage::lineCount>, mediumMax / alignment> m_mediumLineMetadata;
</ins><span class="cx">
</span><ins>+ std::array<Vector<SmallPage*>, smallMax / alignment> m_smallPagesWithFreeLines;
+ std::array<Vector<MediumPage*>, mediumMax / alignment> m_mediumPagesWithFreeLines;
+
</ins><span class="cx"> Vector<SmallPage*> m_smallPages;
</span><span class="cx"> Vector<MediumPage*> m_mediumPages;
</span><span class="cx">
</span><span class="lines">@@ -99,61 +108,20 @@
</span><span class="cx"> AsyncTask<Heap, decltype(&Heap::concurrentScavenge)> m_scavenger;
</span><span class="cx"> };
</span><span class="cx">
</span><del>-inline void Heap::deallocateSmallLine(std::lock_guard<StaticMutex>& lock, SmallLine* line)
</del><ins>+inline void Heap::derefSmallLine(std::lock_guard<StaticMutex>& lock, SmallLine* line)
</ins><span class="cx"> {
</span><del>- BASSERT(!line->refCount(lock));
- SmallPage* page = SmallPage::get(line);
- if (page->deref(lock)) {
- m_smallPages.push(page);
- m_scavenger.run();
</del><ins>+ if (!line->deref(lock))
</ins><span class="cx"> return;
</span><del>- }
- m_smallLines[page->smallSizeClass()].push(line);
</del><ins>+ deallocateSmallLine(lock, line);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline SmallLine* Heap::allocateSmallLine(std::lock_guard<StaticMutex>& lock, size_t smallSizeClass)
</del><ins>+inline void Heap::derefMediumLine(std::lock_guard<StaticMutex>& lock, MediumLine* line)
</ins><span class="cx"> {
</span><del>- Vector<SmallLine*>& smallLines = m_smallLines[smallSizeClass];
- while (smallLines.size()) {
- SmallLine* line = smallLines.pop();
- SmallPage* page = SmallPage::get(line);
- if (!page->refCount(lock) || page->smallSizeClass() != smallSizeClass) // The line was promoted to the small pages list.
- continue;
- BASSERT(!line->refCount(lock));
- page->ref(lock);
- return line;
- }
-
- return allocateSmallLineSlowCase(lock, smallSizeClass);
-}
-
-inline void Heap::deallocateMediumLine(std::lock_guard<StaticMutex>& lock, MediumLine* line)
-{
- BASSERT(!line->refCount(lock));
- MediumPage* page = MediumPage::get(line);
- if (page->deref(lock)) {
- m_mediumPages.push(page);
- m_scavenger.run();
</del><ins>+ if (!line->deref(lock))
</ins><span class="cx"> return;
</span><del>- }
- m_mediumLines.push(line);
</del><ins>+ deallocateMediumLine(lock, line);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-inline MediumLine* Heap::allocateMediumLine(std::lock_guard<StaticMutex>& lock)
-{
- while (m_mediumLines.size()) {
- MediumLine* line = m_mediumLines.pop();
- MediumPage* page = MediumPage::get(line);
- if (!page->refCount(lock)) // The line was promoted to the medium pages list.
- continue;
- BASSERT(!line->refCount(lock));
- page->ref(lock);
- return line;
- }
-
- return allocateMediumLineSlowCase(lock);
-}
-
</del><span class="cx"> } // namespace bmalloc
</span><span class="cx">
</span><span class="cx"> #endif // Heap_h
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocLineh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Line.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Line.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Line.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -45,8 +45,8 @@
</span><span class="cx">
</span><span class="cx"> static Line* get(void*);
</span><span class="cx">
</span><del>- void concurrentRef(unsigned char = 1);
- bool deref(std::lock_guard<StaticMutex>&, unsigned char = 1);
</del><ins>+ void ref(std::lock_guard<StaticMutex>&, unsigned char);
+ bool deref(std::lock_guard<StaticMutex>&);
</ins><span class="cx"> unsigned refCount(std::lock_guard<StaticMutex>&) { return m_refCount; }
</span><span class="cx">
</span><span class="cx"> char* begin();
</span><span class="lines">@@ -81,18 +81,18 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<class Traits>
</span><del>-inline void Line<Traits>::concurrentRef(unsigned char count)
</del><ins>+inline void Line<Traits>::ref(std::lock_guard<StaticMutex>&, unsigned char refCount)
</ins><span class="cx"> {
</span><del>- BASSERT(!m_refCount); // Up-ref from zero can be lock-free because there are no other clients.
- BASSERT(count <= maxRefCount);
- m_refCount = count;
</del><ins>+ BASSERT(!m_refCount);
+ BASSERT(refCount <= maxRefCount);
+ m_refCount = refCount;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<class Traits>
</span><del>-inline bool Line<Traits>::deref(std::lock_guard<StaticMutex>&, unsigned char count)
</del><ins>+inline bool Line<Traits>::deref(std::lock_guard<StaticMutex>&)
</ins><span class="cx"> {
</span><del>- BASSERT(count <= m_refCount);
- m_refCount -= count;
</del><ins>+ BASSERT(m_refCount);
+ --m_refCount;
</ins><span class="cx"> return !m_refCount;
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocLineMetadatah"></a>
<div class="addfile"><h4>Added: trunk/Source/bmalloc/bmalloc/LineMetadata.h (0 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/LineMetadata.h (rev 0)
+++ trunk/Source/bmalloc/bmalloc/LineMetadata.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -0,0 +1,38 @@
</span><ins>+/*
+ * Copyright (C) 2014 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef LineMetadata_h
+#define LineMetadata_h
+
+namespace bmalloc {
+
+struct LineMetadata {
+ unsigned short startOffset;
+ unsigned short objectCount;
+};
+
+} // namespace bmalloc
+
+#endif // LineMetadata_h
</ins></span></pre></div>
<a id="trunkSourcebmallocbmallocMediumAllocatorh"></a>
<div class="delfile"><h4>Deleted: trunk/Source/bmalloc/bmalloc/MediumAllocator.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/MediumAllocator.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/MediumAllocator.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -1,115 +0,0 @@
</span><del>-/*
- * Copyright (C) 2014 Apple Inc. All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
- * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
- * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
- * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#ifndef MediumAllocator_h
-#define MediumAllocator_h
-
-#include "BAssert.h"
-#include "MediumChunk.h"
-#include "MediumLine.h"
-
-namespace bmalloc {
-
-// Helper object for allocating medium objects.
-
-class MediumAllocator {
-public:
- MediumAllocator();
-
- bool isNull() { return !m_end; }
- MediumLine* line();
-
- void* allocate(size_t);
- bool allocate(size_t, void*&);
-
- unsigned char derefCount();
-
- void refill(MediumLine*);
- void clear();
-
-private:
- char* m_end;
- size_t m_remaining;
- size_t m_objectCount;
-};
-
-inline MediumAllocator::MediumAllocator()
- : m_end()
- , m_remaining()
- , m_objectCount()
-{
-}
-
-inline MediumLine* MediumAllocator::line()
-{
- return MediumLine::get(m_end - 1);
-}
-
-inline void* MediumAllocator::allocate(size_t size)
-{
- BASSERT(size <= m_remaining);
- BASSERT(size == roundUpToMultipleOf<alignment>(size));
- BASSERT(size >= MediumLine::minimumObjectSize);
-
- m_remaining -= size;
- void* object = m_end - m_remaining - size;
- BASSERT(isSmallOrMedium(object) && !isSmall(object));
-
- ++m_objectCount;
- return object;
-}
-
-inline bool MediumAllocator::allocate(size_t size, void*& object)
-{
- if (size > m_remaining)
- return false;
-
- object = allocate(size);
- return true;
-}
-
-inline unsigned char MediumAllocator::derefCount()
-{
- return MediumLine::maxRefCount - m_objectCount;
-}
-
-inline void MediumAllocator::refill(MediumLine* line)
-{
- line->concurrentRef(MediumLine::maxRefCount);
- m_end = line->end();
- m_remaining = mediumLineSize;
- m_objectCount = 0;
-}
-
-inline void MediumAllocator::clear()
-{
- m_end = nullptr;
- m_remaining = 0;
- m_objectCount = 0;
-}
-
-} // namespace bmalloc
-
-#endif // MediumAllocator_h
</del></span></pre></div>
<a id="trunkSourcebmallocbmallocPageh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Page.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Page.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Page.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -39,9 +39,10 @@
</span><span class="cx"> typedef typename Traits::Chunk Chunk;
</span><span class="cx"> typedef typename Traits::Line Line;
</span><span class="cx"> static const size_t lineSize = Traits::lineSize;
</span><ins>+ static const size_t lineCount = vmPageSize / lineSize;
</ins><span class="cx">
</span><span class="cx"> static const unsigned char maxRefCount = std::numeric_limits<unsigned char>::max();
</span><del>- static_assert(vmPageSize / lineSize < maxRefCount, "maximum line count must fit in Page");
</del><ins>+ static_assert(lineCount < maxRefCount, "maximum line count must fit in Page");
</ins><span class="cx">
</span><span class="cx"> static Page* get(Line*);
</span><span class="cx">
</span><span class="lines">@@ -49,15 +50,15 @@
</span><span class="cx"> bool deref(std::lock_guard<StaticMutex>&);
</span><span class="cx"> unsigned refCount(std::lock_guard<StaticMutex>&) { return m_refCount; }
</span><span class="cx">
</span><del>- size_t smallSizeClass() { return m_smallSizeClass; }
- void setSmallSizeClass(size_t smallSizeClass) { m_smallSizeClass = smallSizeClass; }
</del><ins>+ size_t sizeClass() { return m_sizeClass; }
+ void setSizeClass(size_t sizeClass) { m_sizeClass = sizeClass; }
</ins><span class="cx">
</span><span class="cx"> Line* begin();
</span><span class="cx"> Line* end();
</span><span class="cx">
</span><span class="cx"> private:
</span><span class="cx"> unsigned char m_refCount;
</span><del>- unsigned char m_smallSizeClass;
</del><ins>+ unsigned char m_sizeClass;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> template<typename Traits>
</span><span class="lines">@@ -89,14 +90,14 @@
</span><span class="cx"> {
</span><span class="cx"> Chunk* chunk = Chunk::get(this);
</span><span class="cx"> size_t pageNumber = this - chunk->pages();
</span><del>- size_t lineNumber = pageNumber * vmPageSize / lineSize;
</del><ins>+ size_t lineNumber = pageNumber * lineCount;
</ins><span class="cx"> return &chunk->lines()[lineNumber];
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename Traits>
</span><span class="cx"> inline auto Page<Traits>::end() -> Line*
</span><span class="cx"> {
</span><del>- return begin() + vmPageSize / lineSize;
</del><ins>+ return begin() + lineCount;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> } // namespace bmalloc
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocSizesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Sizes.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Sizes.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/Sizes.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -56,7 +56,7 @@
</span><span class="cx"> static const size_t superChunkSize = 32 * MB;
</span><span class="cx">
</span><span class="cx"> static const size_t smallMax = 256;
</span><del>- static const size_t smallLineSize = 512;
</del><ins>+ static const size_t smallLineSize = 256;
</ins><span class="cx"> static const size_t smallLineMask = ~(smallLineSize - 1ul);
</span><span class="cx">
</span><span class="cx"> static const size_t smallChunkSize = superChunkSize / 4;
</span><span class="lines">@@ -64,7 +64,7 @@
</span><span class="cx"> static const size_t smallChunkMask = ~(smallChunkSize - 1ul);
</span><span class="cx">
</span><span class="cx"> static const size_t mediumMax = 1024;
</span><del>- static const size_t mediumLineSize = 2048;
</del><ins>+ static const size_t mediumLineSize = 1024;
</ins><span class="cx"> static const size_t mediumLineMask = ~(mediumLineSize - 1ul);
</span><span class="cx">
</span><span class="cx"> static const size_t mediumChunkSize = superChunkSize / 4;
</span><span class="lines">@@ -92,21 +92,20 @@
</span><span class="cx">
</span><span class="cx"> static const size_t deallocatorLogCapacity = 256;
</span><span class="cx">
</span><del>- static const size_t smallLineCacheCapacity = vmPageSize / smallLineSize;
- static const size_t mediumLineCacheCapacity = vmPageSize / mediumLineSize;
</del><ins>+ static const size_t smallRangeCacheCapacity = vmPageSize / smallLineSize;
+ static const size_t mediumRangeCacheCapacity = vmPageSize / mediumLineSize;
</ins><span class="cx">
</span><span class="cx"> static const std::chrono::milliseconds scavengeSleepDuration = std::chrono::milliseconds(512);
</span><span class="cx">
</span><del>- inline size_t smallSizeClassFor(size_t size)
</del><ins>+ inline size_t sizeClass(size_t size)
</ins><span class="cx"> {
</span><del>- static const size_t smallSizeClassMask = (smallMax / alignment) - 1;
- return mask((size - 1ul) / alignment, smallSizeClassMask);
</del><ins>+ static const size_t sizeClassMask = (mediumMax / alignment) - 1;
+ return mask((size - 1ul) / alignment, sizeClassMask);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- inline size_t mediumSizeClassFor(size_t size)
</del><ins>+ inline size_t objectSize(size_t sizeClass)
</ins><span class="cx"> {
</span><del>- static const size_t mediumSizeClassMask = (mediumMax / alignment) - 1;
- return mask((size - 1ul) / alignment, mediumSizeClassMask);
</del><ins>+ return (sizeClass + 1) * alignment;
</ins><span class="cx"> }
</span><span class="cx"> };
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocbmalloch"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/bmalloc.h (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/bmalloc.h 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc/bmalloc.h 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -52,15 +52,13 @@
</span><span class="cx"> size_t oldSize = 0;
</span><span class="cx"> switch(objectType(object)) {
</span><span class="cx"> case Small: {
</span><del>- // We don't have an exact size, but we can calculate a maximum.
- void* end = roundUpToMultipleOf<smallLineSize>(static_cast<char*>(object) + 1);
- oldSize = static_cast<char*>(end) - static_cast<char*>(object);
</del><ins>+ SmallPage* page = SmallPage::get(SmallLine::get(object));
+ oldSize = objectSize(page->sizeClass());
</ins><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> case Medium: {
</span><del>- // We don't have an exact size, but we can calculate a maximum.
- void* end = roundUpToMultipleOf<mediumLineSize>(static_cast<char*>(object) + 1);
- oldSize = static_cast<char*>(end) - static_cast<char*>(object);
</del><ins>+ MediumPage* page = MediumPage::get(MediumLine::get(object));
+ oldSize = objectSize(page->sizeClass());
</ins><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx"> case Large: {
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocxcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj (173880 => 173881)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj 2014-09-23 18:09:02 UTC (rev 173880)
+++ trunk/Source/bmalloc/bmalloc.xcodeproj/project.pbxproj 2014-09-23 18:24:02 UTC (rev 173881)
</span><span class="lines">@@ -11,6 +11,8 @@
</span><span class="cx"> 1400274A18F89C2300115C97 /* VMHeap.h in Headers */ = {isa = PBXBuildFile; fileRef = 144F7BFC18BFC517003537F3 /* VMHeap.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 1400274B18F89C3D00115C97 /* BoundaryTagInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 14105E7B18DBD7AF003A106E /* BoundaryTagInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 1400274C18F89C3D00115C97 /* SegregatedFreeList.h in Headers */ = {isa = PBXBuildFile; fileRef = 146BEE1E18C841C50002D5A2 /* SegregatedFreeList.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><ins>+ 140FA00319CE429C00FFD3C8 /* BumpRange.h in Headers */ = {isa = PBXBuildFile; fileRef = 140FA00219CE429C00FFD3C8 /* BumpRange.h */; settings = {ATTRIBUTES = (Private, ); }; };
+ 140FA00519CE4B6800FFD3C8 /* LineMetadata.h in Headers */ = {isa = PBXBuildFile; fileRef = 140FA00419CE4B6800FFD3C8 /* LineMetadata.h */; settings = {ATTRIBUTES = (Private, ); }; };
</ins><span class="cx"> 143CB81C19022BC900B16A45 /* StaticMutex.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 143CB81A19022BC900B16A45 /* StaticMutex.cpp */; };
</span><span class="cx"> 143CB81D19022BC900B16A45 /* StaticMutex.h in Headers */ = {isa = PBXBuildFile; fileRef = 143CB81B19022BC900B16A45 /* StaticMutex.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 1448C30018F3754600502839 /* mbmalloc.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1448C2FF18F3754300502839 /* mbmalloc.cpp */; };
</span><span class="lines">@@ -26,7 +28,6 @@
</span><span class="cx"> 14DD789818F48D4A00950702 /* Allocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 145F6856179DC8CA00D65598 /* Allocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 14DD789918F48D4A00950702 /* Cache.h in Headers */ = {isa = PBXBuildFile; fileRef = 144469E517A46BFE00F9EA1D /* Cache.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 14DD789A18F48D4A00950702 /* Deallocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 145F685A179DC90200D65598 /* Deallocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><del>- 14DD789B18F48D4A00950702 /* MediumAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E47018A0661700546D68 /* MediumAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
</del><span class="cx"> 14DD789C18F48D4A00950702 /* BumpAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 1413E462189DE1CD00546D68 /* BumpAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 14DD78B418F48D6B00950702 /* Chunk.h in Headers */ = {isa = PBXBuildFile; fileRef = 147AAA9418CE5CA6002201E4 /* Chunk.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> 14DD78B518F48D6B00950702 /* Line.h in Headers */ = {isa = PBXBuildFile; fileRef = 14DA32071885F9E6007269E0 /* Line.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="lines">@@ -72,12 +73,13 @@
</span><span class="cx"> /* End PBXContainerItemProxy section */
</span><span class="cx">
</span><span class="cx"> /* Begin PBXFileReference section */
</span><ins>+ 140FA00219CE429C00FFD3C8 /* BumpRange.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BumpRange.h; path = bmalloc/BumpRange.h; sourceTree = "<group>"; };
+ 140FA00419CE4B6800FFD3C8 /* LineMetadata.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = LineMetadata.h; path = bmalloc/LineMetadata.h; sourceTree = "<group>"; };
</ins><span class="cx"> 14105E7B18DBD7AF003A106E /* BoundaryTagInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BoundaryTagInlines.h; path = bmalloc/BoundaryTagInlines.h; sourceTree = "<group>"; };
</span><span class="cx"> 14105E8318E14374003A106E /* ObjectType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ObjectType.cpp; path = bmalloc/ObjectType.cpp; sourceTree = "<group>"; };
</span><span class="cx"> 1413E460189DCE1E00546D68 /* Inline.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Inline.h; path = bmalloc/Inline.h; sourceTree = "<group>"; };
</span><span class="cx"> 1413E462189DE1CD00546D68 /* BumpAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; lineEnding = 0; name = BumpAllocator.h; path = bmalloc/BumpAllocator.h; sourceTree = "<group>"; };
</span><span class="cx"> 1413E468189EEDE400546D68 /* BAssert.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BAssert.h; path = bmalloc/BAssert.h; sourceTree = "<group>"; };
</span><del>- 1413E47018A0661700546D68 /* MediumAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = MediumAllocator.h; path = bmalloc/MediumAllocator.h; sourceTree = "<group>"; };
</del><span class="cx"> 1417F64518B54A700076FA3F /* BeginTag.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = BeginTag.h; path = bmalloc/BeginTag.h; sourceTree = "<group>"; };
</span><span class="cx"> 1417F64618B54A700076FA3F /* EndTag.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = EndTag.h; path = bmalloc/EndTag.h; sourceTree = "<group>"; };
</span><span class="cx"> 1417F64F18B7280C0076FA3F /* Syscall.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Syscall.h; path = bmalloc/Syscall.h; sourceTree = "<group>"; };
</span><span class="lines">@@ -235,7 +237,6 @@
</span><span class="cx"> 144469E517A46BFE00F9EA1D /* Cache.h */,
</span><span class="cx"> 145F6859179DC90200D65598 /* Deallocator.cpp */,
</span><span class="cx"> 145F685A179DC90200D65598 /* Deallocator.h */,
</span><del>- 1413E47018A0661700546D68 /* MediumAllocator.h */,
</del><span class="cx"> 1413E462189DE1CD00546D68 /* BumpAllocator.h */,
</span><span class="cx"> );
</span><span class="cx"> name = cache;
</span><span class="lines">@@ -244,8 +245,10 @@
</span><span class="cx"> 14D9DB4E17F2866E00EAAB79 /* heap */ = {
</span><span class="cx"> isa = PBXGroup;
</span><span class="cx"> children = (
</span><ins>+ 140FA00219CE429C00FFD3C8 /* BumpRange.h */,
</ins><span class="cx"> 14DA320E18875D9F007269E0 /* Heap.cpp */,
</span><span class="cx"> 14DA320C18875B09007269E0 /* Heap.h */,
</span><ins>+ 140FA00419CE4B6800FFD3C8 /* LineMetadata.h */,
</ins><span class="cx"> 14105E8318E14374003A106E /* ObjectType.cpp */,
</span><span class="cx"> 1485656018A43DBA00ED6942 /* ObjectType.h */,
</span><span class="cx"> 145F6874179DF84100D65598 /* Sizes.h */,
</span><span class="lines">@@ -296,6 +299,7 @@
</span><span class="cx"> 14C919C918FCC59F0028DB43 /* BPlatform.h in Headers */,
</span><span class="cx"> 14DD788C18F48CAE00950702 /* LargeChunk.h in Headers */,
</span><span class="cx"> 14DD789218F48CFC00950702 /* EndTag.h in Headers */,
</span><ins>+ 140FA00519CE4B6800FFD3C8 /* LineMetadata.h in Headers */,
</ins><span class="cx"> 14DD78CC18F48D7500950702 /* PerThread.h in Headers */,
</span><span class="cx"> 14DD78B418F48D6B00950702 /* Chunk.h in Headers */,
</span><span class="cx"> 14DD78CA18F48D7500950702 /* Mutex.h in Headers */,
</span><span class="lines">@@ -309,7 +313,6 @@
</span><span class="cx"> 14DD78CD18F48D7500950702 /* Range.h in Headers */,
</span><span class="cx"> 14DD789C18F48D4A00950702 /* BumpAllocator.h in Headers */,
</span><span class="cx"> 14DD789918F48D4A00950702 /* Cache.h in Headers */,
</span><del>- 14DD789B18F48D4A00950702 /* MediumAllocator.h in Headers */,
</del><span class="cx"> 14DD78BE18F48D6B00950702 /* SmallTraits.h in Headers */,
</span><span class="cx"> 14DD789018F48CEB00950702 /* Sizes.h in Headers */,
</span><span class="cx"> 14DD78C718F48D7500950702 /* BAssert.h in Headers */,
</span><span class="lines">@@ -322,6 +325,7 @@
</span><span class="cx"> 1400274A18F89C2300115C97 /* VMHeap.h in Headers */,
</span><span class="cx"> 1400274918F89C1300115C97 /* Heap.h in Headers */,
</span><span class="cx"> 14DD78B818F48D6B00950702 /* MediumPage.h in Headers */,
</span><ins>+ 140FA00319CE429C00FFD3C8 /* BumpRange.h in Headers */,
</ins><span class="cx"> 14DD78C518F48D7500950702 /* Algorithm.h in Headers */,
</span><span class="cx"> 14DD78BD18F48D6B00950702 /* SmallPage.h in Headers */,
</span><span class="cx"> 1400274B18F89C3D00115C97 /* BoundaryTagInlines.h in Headers */,
</span></span></pre>
</div>
</div>
</body>
</html>