<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[198821] trunk/Source/bmalloc</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/198821">198821</a></dd>
<dt>Author</dt> <dd>ggaren@apple.com</dd>
<dt>Date</dt> <dd>2016-03-29 19:22:47 -0700 (Tue, 29 Mar 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>bmalloc: page size should be configurable at runtime
https://bugs.webkit.org/show_bug.cgi?id=155993
Reviewed by Andreas Kling.
This is a memory win on 32bit iOS devices, since their page sizes are
4kB and not 16kB.
It's also a step toward supporting 64bit iOS devices that have a
16kB/4kB virtual/physical page size split.
* bmalloc/Chunk.h: Align to largeAlignment since 2 * smallMax isn't
required by the boundary tag allocator.
(bmalloc::Chunk::page): Account for the slide when accessing a page.
Each SmallPage hashes 4kB of memory. When we want to allocate a region
of memory larger than 4kB, we store our metadata in the first SmallPage
in the region and we assign a slide to the remaining SmallPages, so
they forward to that first SmallPage when accessed.
NOTE: We could use a less flexible technique that just hashed by
vmPageSize() instead of 4kB at runtime, with no slide, but I think we'll
be able to use this slide technique to make even more page sizes
dynamically at runtime, which should save some memory and simplify
the allocator.
(bmalloc::SmallPage::begin): It's invalid to access a SmallPage with
a slide, since such SmallPages do not contain meaningful data.
(bmalloc::SmallPage::end): Account for smallPageCount when computing
the size of a page.
(bmalloc::Chunk::pageBegin): Deleted.
(bmalloc::Chunk::pageEnd): Deleted.
(bmalloc::Object::pageBegin): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap): Cache vmPageSize because computing it might require
a syscall.
(bmalloc::Heap::initializeLineMetadata): Line metadata is a vector instead
of a 2D array because we don't know how much metadata we'll need until
we know the page size.
(bmalloc::Heap::scavengeSmallPage): Be sure to revert the slide when
deallocating a page. Otherwise, the next attempt to allocate the page
will slide when initializing it, sliding to nowhere.
(bmalloc::Heap::allocateSmallBumpRanges): Account for vector change to
line metadata.
(bmalloc::Heap::allocateSmallPage): Initialize slide and smallPageCount
since they aren't constant anymore.
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateXLarge):
(bmalloc::Heap::shrinkXLarge): Adopt dynamic page size.
* bmalloc/Heap.h:
* bmalloc/Sizes.h: smallPageSize is no longer equal to the VM page
size -- it's just the smallest VM page size we're interested in supporting.
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::slide):
(bmalloc::SmallPage::setSlide):
(bmalloc::SmallPage::smallPageCount):
(bmalloc::SmallPage::setSmallPageCount):
(bmalloc::SmallPage::ref):
(bmalloc::SmallPage::deref): Support slide and small page count as
dynamic values. This doesn't increase metadata size since sizeof(SmallPage)
rounds up to alignment anyway.
* bmalloc/VMAllocate.h:
(bmalloc::vmPageSize):
(bmalloc::vmPageShift):
(bmalloc::vmSize):
(bmalloc::vmValidate):
(bmalloc::tryVMAllocate):
(bmalloc::vmDeallocatePhysicalPagesSloppy):
(bmalloc::vmAllocatePhysicalPagesSloppy): Treat page size as a variable.
* bmalloc/Vector.h:
(bmalloc::Vector::initialCapacity):
(bmalloc::Vector<T>::insert):
(bmalloc::Vector<T>::grow):
(bmalloc::Vector<T>::shrink):
(bmalloc::Vector<T>::shrinkCapacity):
(bmalloc::Vector<T>::growCapacity): Treat page size as a variable.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourcebmallocChangeLog">trunk/Source/bmalloc/ChangeLog</a></li>
<li><a href="#trunkSourcebmallocbmallocChunkh">trunk/Source/bmalloc/bmalloc/Chunk.h</a></li>
<li><a href="#trunkSourcebmallocbmallocHeapcpp">trunk/Source/bmalloc/bmalloc/Heap.cpp</a></li>
<li><a href="#trunkSourcebmallocbmallocHeaph">trunk/Source/bmalloc/bmalloc/Heap.h</a></li>
<li><a href="#trunkSourcebmallocbmallocSizesh">trunk/Source/bmalloc/bmalloc/Sizes.h</a></li>
<li><a href="#trunkSourcebmallocbmallocSmallPageh">trunk/Source/bmalloc/bmalloc/SmallPage.h</a></li>
<li><a href="#trunkSourcebmallocbmallocVMAllocateh">trunk/Source/bmalloc/bmalloc/VMAllocate.h</a></li>
<li><a href="#trunkSourcebmallocbmallocVectorh">trunk/Source/bmalloc/bmalloc/Vector.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourcebmallocChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/ChangeLog (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/ChangeLog 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/ChangeLog 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -1,3 +1,96 @@
</span><ins>+2016-03-29 Geoffrey Garen <ggaren@apple.com>
+
+ bmalloc: page size should be configurable at runtime
+ https://bugs.webkit.org/show_bug.cgi?id=155993
+
+ Reviewed by Andreas Kling.
+
+ This is a memory win on 32bit iOS devices, since their page sizes are
+ 4kB and not 16kB.
+
+ It's also a step toward supporting 64bit iOS devices that have a
+ 16kB/4kB virtual/physical page size split.
+
+ * bmalloc/Chunk.h: Align to largeAlignment since 2 * smallMax isn't
+ required by the boundary tag allocator.
+
+ (bmalloc::Chunk::page): Account for the slide when accessing a page.
+ Each SmallPage hashes 4kB of memory. When we want to allocate a region
+ of memory larger than 4kB, we store our metadata in the first SmallPage
+ in the region and we assign a slide to the remaining SmallPages, so
+ they forward to that first SmallPage when accessed.
+
+ NOTE: We could use a less flexible technique that just hashed by
+ vmPageSize() instead of 4kB at runtime, with no slide, but I think we'll
+ be able to use this slide technique to make even more page sizes
+ dynamically at runtime, which should save some memory and simplify
+ the allocator.
+
+ (bmalloc::SmallPage::begin): It's invalid to access a SmallPage with
+ a slide, since such SmallPages do not contain meaningful data.
+
+ (bmalloc::SmallPage::end): Account for smallPageCount when computing
+ the size of a page.
+
+ (bmalloc::Chunk::pageBegin): Deleted.
+ (bmalloc::Chunk::pageEnd): Deleted.
+ (bmalloc::Object::pageBegin): Deleted.
+
+ * bmalloc/Heap.cpp:
+ (bmalloc::Heap::Heap): Cache vmPageSize because computing it might require
+ a syscall.
+
+ (bmalloc::Heap::initializeLineMetadata): Line metadata is a vector instead
+ of a 2D array because we don't know how much metadata we'll need until
+ we know the page size.
+
+ (bmalloc::Heap::scavengeSmallPage): Be sure to revert the slide when
+ deallocating a page. Otherwise, the next attempt to allocate the page
+ will slide when initializing it, sliding to nowhere.
+
+ (bmalloc::Heap::allocateSmallBumpRanges): Account for vector change to
+ line metadata.
+
+ (bmalloc::Heap::allocateSmallPage): Initialize slide and smallPageCount
+ since they aren't constant anymore.
+
+ (bmalloc::Heap::allocateLarge):
+ (bmalloc::Heap::splitAndAllocate):
+ (bmalloc::Heap::tryAllocateXLarge):
+ (bmalloc::Heap::shrinkXLarge): Adopt dynamic page size.
+
+ * bmalloc/Heap.h:
+
+ * bmalloc/Sizes.h: smallPageSize is no longer equal to the VM page
+ size -- it's just the smallest VM page size we're interested in supporting.
+
+ * bmalloc/SmallPage.h:
+ (bmalloc::SmallPage::slide):
+ (bmalloc::SmallPage::setSlide):
+ (bmalloc::SmallPage::smallPageCount):
+ (bmalloc::SmallPage::setSmallPageCount):
+ (bmalloc::SmallPage::ref):
+ (bmalloc::SmallPage::deref): Support slide and small page count as
+ dynamic values. This doesn't increase metadata size since sizeof(SmallPage)
+ rounds up to alignment anyway.
+
+ * bmalloc/VMAllocate.h:
+ (bmalloc::vmPageSize):
+ (bmalloc::vmPageShift):
+ (bmalloc::vmSize):
+ (bmalloc::vmValidate):
+ (bmalloc::tryVMAllocate):
+ (bmalloc::vmDeallocatePhysicalPagesSloppy):
+ (bmalloc::vmAllocatePhysicalPagesSloppy): Treat page size as a variable.
+
+ * bmalloc/Vector.h:
+ (bmalloc::Vector::initialCapacity):
+ (bmalloc::Vector<T>::insert):
+ (bmalloc::Vector<T>::grow):
+ (bmalloc::Vector<T>::shrink):
+ (bmalloc::Vector<T>::shrinkCapacity):
+ (bmalloc::Vector<T>::growCapacity): Treat page size as a variable.
+
</ins><span class="cx"> 2016-03-29 David Kilzer <ddkilzer@apple.com>
</span><span class="cx">
</span><span class="cx"> bmalloc: add logging for mmap() failures
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocChunkh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Chunk.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Chunk.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/Chunk.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -53,9 +53,6 @@
</span><span class="cx"> SmallPage* page(size_t offset);
</span><span class="cx"> SmallLine* line(size_t offset);
</span><span class="cx">
</span><del>- SmallPage* pageBegin() { return Object(m_memory).page(); }
- SmallPage* pageEnd() { return m_pages.end(); }
-
</del><span class="cx"> SmallLine* lines() { return m_lines.begin(); }
</span><span class="cx"> SmallPage* pages() { return m_pages.begin(); }
</span><span class="cx">
</span><span class="lines">@@ -81,15 +78,12 @@
</span><span class="cx"> // We use the X's for boundary tags and the O's for edge sentinels.
</span><span class="cx">
</span><span class="cx"> std::array<SmallLine, chunkSize / smallLineSize> m_lines;
</span><del>- std::array<SmallPage, chunkSize / vmPageSize> m_pages;
</del><ins>+ std::array<SmallPage, chunkSize / smallPageSize> m_pages;
</ins><span class="cx"> std::array<BoundaryTag, boundaryTagCount> m_boundaryTags;
</span><del>- char m_memory[] __attribute__((aligned(2 * smallMax + 0)));
</del><ins>+ char m_memory[] __attribute__((aligned(largeAlignment + 0)));
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> static_assert(sizeof(Chunk) + largeMax <= chunkSize, "largeMax is too big");
</span><del>-static_assert(
- sizeof(Chunk) % vmPageSize + 2 * smallMax <= vmPageSize,
- "the first page of object memory in a small chunk must be able to allocate smallMax");
</del><span class="cx">
</span><span class="cx"> inline Chunk::Chunk(std::lock_guard<StaticMutex>& lock)
</span><span class="cx"> {
</span><span class="lines">@@ -165,8 +159,9 @@
</span><span class="cx">
</span><span class="cx"> inline SmallPage* Chunk::page(size_t offset)
</span><span class="cx"> {
</span><del>- size_t pageNumber = offset / vmPageSize;
- return &m_pages[pageNumber];
</del><ins>+ size_t pageNumber = offset / smallPageSize;
+ SmallPage* page = &m_pages[pageNumber];
+ return page - page->slide();
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline SmallLine* Chunk::line(size_t offset)
</span><span class="lines">@@ -190,15 +185,17 @@
</span><span class="cx">
</span><span class="cx"> inline SmallLine* SmallPage::begin()
</span><span class="cx"> {
</span><ins>+ BASSERT(!m_slide);
</ins><span class="cx"> Chunk* chunk = Chunk::get(this);
</span><span class="cx"> size_t pageNumber = this - chunk->pages();
</span><del>- size_t lineNumber = pageNumber * smallLineCount;
</del><ins>+ size_t lineNumber = pageNumber * smallPageLineCount;
</ins><span class="cx"> return &chunk->lines()[lineNumber];
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline SmallLine* SmallPage::end()
</span><span class="cx"> {
</span><del>- return begin() + smallLineCount;
</del><ins>+ BASSERT(!m_slide);
+ return begin() + m_smallPageCount * smallPageLineCount;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline Object::Object(void* object)
</span><span class="lines">@@ -219,11 +216,6 @@
</span><span class="cx"> return m_chunk->object(m_offset);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-inline void* Object::pageBegin()
-{
- return m_chunk->object(roundDownToMultipleOf(vmPageSize, m_offset));
-}
-
</del><span class="cx"> inline SmallLine* Object::line()
</span><span class="cx"> {
</span><span class="cx"> return m_chunk->line(m_offset);
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeapcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.cpp (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.cpp 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/Heap.cpp 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -35,7 +35,8 @@
</span><span class="cx"> namespace bmalloc {
</span><span class="cx">
</span><span class="cx"> Heap::Heap(std::lock_guard<StaticMutex>&)
</span><del>- : m_largeObjects(VMState::HasPhysical::True)
</del><ins>+ : m_vmPageSize(vmPageSize())
+ , m_largeObjects(VMState::HasPhysical::True)
</ins><span class="cx"> , m_isAllocatingPages(false)
</span><span class="cx"> , m_scavenger(*this, &Heap::concurrentScavenge)
</span><span class="cx"> {
</span><span class="lines">@@ -46,13 +47,16 @@
</span><span class="cx"> {
</span><span class="cx"> // We assume that m_smallLineMetadata is zero-filled.
</span><span class="cx">
</span><ins>+ size_t smallLineCount = m_vmPageSize / smallLineSize;
+ m_smallLineMetadata.grow(sizeClassCount * smallLineCount);
+
</ins><span class="cx"> for (size_t sizeClass = 0; sizeClass < sizeClassCount; ++sizeClass) {
</span><span class="cx"> size_t size = objectSize(sizeClass);
</span><del>- auto& metadata = m_smallLineMetadata[sizeClass];
</del><ins>+ LineMetadata* pageMetadata = &m_smallLineMetadata[sizeClass * smallLineCount];
</ins><span class="cx">
</span><span class="cx"> size_t object = 0;
</span><span class="cx"> size_t line = 0;
</span><del>- while (object < vmPageSize) {
</del><ins>+ while (object < m_vmPageSize) {
</ins><span class="cx"> line = object / smallLineSize;
</span><span class="cx"> size_t leftover = object % smallLineSize;
</span><span class="cx">
</span><span class="lines">@@ -60,15 +64,15 @@
</span><span class="cx"> size_t remainder;
</span><span class="cx"> divideRoundingUp(smallLineSize - leftover, size, objectCount, remainder);
</span><span class="cx">
</span><del>- metadata[line] = { static_cast<unsigned short>(leftover), static_cast<unsigned short>(objectCount) };
</del><ins>+ pageMetadata[line] = { static_cast<unsigned short>(leftover), static_cast<unsigned short>(objectCount) };
</ins><span class="cx">
</span><span class="cx"> object += objectCount * size;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // Don't allow the last object in a page to escape the page.
</span><del>- if (object > vmPageSize) {
- BASSERT(metadata[line].objectCount);
- --metadata[line].objectCount;
</del><ins>+ if (object > m_vmPageSize) {
+ BASSERT(pageMetadata[line].objectCount);
+ --pageMetadata[line].objectCount;
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx"> }
</span><span class="lines">@@ -100,7 +104,12 @@
</span><span class="cx"> {
</span><span class="cx"> SmallPage* page = m_smallPages.pop();
</span><span class="cx">
</span><del>- // Transform small object page back into a large object.
</del><ins>+ // Revert the slide() value on intermediate SmallPages so they hash to
+ // themselves again.
+ for (size_t i = 1; i < page->smallPageCount(); ++i)
+ page[i].setSlide(0);
+
+ // Revert our small object page back to large object.
</ins><span class="cx"> page->setObjectType(ObjectType::Large);
</span><span class="cx">
</span><span class="cx"> LargeObject largeObject(page->begin()->begin());
</span><span class="lines">@@ -143,13 +152,15 @@
</span><span class="cx"> SmallPage* page = allocateSmallPage(lock, sizeClass);
</span><span class="cx"> SmallLine* lines = page->begin();
</span><span class="cx"> BASSERT(page->hasFreeLines(lock));
</span><ins>+ size_t smallLineCount = m_vmPageSize / smallLineSize;
+ LineMetadata* pageMetadata = &m_smallLineMetadata[sizeClass * smallLineCount];
</ins><span class="cx">
</span><span class="cx"> // Find a free line.
</span><span class="cx"> for (size_t lineNumber = 0; lineNumber < smallLineCount; ++lineNumber) {
</span><span class="cx"> if (lines[lineNumber].refCount(lock))
</span><span class="cx"> continue;
</span><span class="cx">
</span><del>- LineMetadata& lineMetadata = m_smallLineMetadata[sizeClass][lineNumber];
</del><ins>+ LineMetadata& lineMetadata = pageMetadata[lineNumber];
</ins><span class="cx"> if (!lineMetadata.objectCount)
</span><span class="cx"> continue;
</span><span class="cx">
</span><span class="lines">@@ -170,7 +181,7 @@
</span><span class="cx"> if (lines[lineNumber].refCount(lock))
</span><span class="cx"> break;
</span><span class="cx">
</span><del>- LineMetadata& lineMetadata = m_smallLineMetadata[sizeClass][lineNumber];
</del><ins>+ LineMetadata& lineMetadata = pageMetadata[lineNumber];
</ins><span class="cx"> if (!lineMetadata.objectCount)
</span><span class="cx"> continue;
</span><span class="cx">
</span><span class="lines">@@ -200,16 +211,24 @@
</span><span class="cx"> return page;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- size_t unalignedSize = largeMin + vmPageSize - largeAlignment + vmPageSize;
- LargeObject largeObject = allocateLarge(lock, vmPageSize, vmPageSize, unalignedSize);
-
</del><ins>+ size_t unalignedSize = largeMin + m_vmPageSize - largeAlignment + m_vmPageSize;
+ LargeObject largeObject = allocateLarge(lock, m_vmPageSize, m_vmPageSize, unalignedSize);
+
</ins><span class="cx"> // Transform our large object into a small object page. We deref here
</span><del>- // because our small objects will keep their own refcounts on the line.
</del><ins>+ // because our small objects will keep their own line refcounts.
</ins><span class="cx"> Object object(largeObject.begin());
</span><span class="cx"> object.line()->deref(lock);
</span><span class="cx"> object.page()->setObjectType(ObjectType::Small);
</span><span class="cx">
</span><del>- object.page()->setSizeClass(sizeClass);
</del><ins>+ SmallPage* page = object.page();
+ page->setSizeClass(sizeClass);
+ page->setSmallPageCount(m_vmPageSize / smallPageSize);
+
+ // Set a slide() value on intermediate SmallPages so they hash to their
+ // vmPageSize-sized page.
+ for (size_t i = 1; i < page->smallPageCount(); ++i)
+ page[i].setSlide(i);
+
</ins><span class="cx"> return object.page();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -307,7 +326,7 @@
</span><span class="cx"> BASSERT(size >= largeMin);
</span><span class="cx"> BASSERT(size == roundUpToMultipleOf<largeAlignment>(size));
</span><span class="cx">
</span><del>- if (size <= vmPageSize)
</del><ins>+ if (size <= m_vmPageSize)
</ins><span class="cx"> scavengeSmallPages(lock);
</span><span class="cx">
</span><span class="cx"> LargeObject largeObject = m_largeObjects.take(size);
</span><span class="lines">@@ -338,7 +357,7 @@
</span><span class="cx"> BASSERT(alignment >= largeAlignment);
</span><span class="cx"> BASSERT(isPowerOfTwo(alignment));
</span><span class="cx">
</span><del>- if (size <= vmPageSize)
</del><ins>+ if (size <= m_vmPageSize)
</ins><span class="cx"> scavengeSmallPages(lock);
</span><span class="cx">
</span><span class="cx"> LargeObject largeObject = m_largeObjects.take(alignment, size, unalignedSize);
</span><span class="lines">@@ -412,7 +431,7 @@
</span><span class="cx"> // in the allocated list. This is an important optimization because it
</span><span class="cx"> // keeps the free list short, speeding up allocation and merging.
</span><span class="cx">
</span><del>- std::pair<XLargeRange, XLargeRange> allocated = range.split(roundUpToMultipleOf<vmPageSize>(size));
</del><ins>+ std::pair<XLargeRange, XLargeRange> allocated = range.split(roundUpToMultipleOf(m_vmPageSize, size));
</ins><span class="cx"> if (allocated.first.vmState().hasVirtual()) {
</span><span class="cx"> vmAllocatePhysicalPagesSloppy(allocated.first.begin(), allocated.first.size());
</span><span class="cx"> allocated.first.setVMState(VMState::Physical);
</span><span class="lines">@@ -429,7 +448,7 @@
</span><span class="cx">
</span><span class="cx"> m_isAllocatingPages = true;
</span><span class="cx">
</span><del>- size = std::max(vmPageSize, size);
</del><ins>+ size = std::max(m_vmPageSize, size);
</ins><span class="cx"> alignment = roundUpToMultipleOf<xLargeAlignment>(alignment);
</span><span class="cx">
</span><span class="cx"> XLargeRange range = m_xLargeMap.takeFree(alignment, size);
</span><span class="lines">@@ -456,7 +475,7 @@
</span><span class="cx"> {
</span><span class="cx"> BASSERT(object.size() > newSize);
</span><span class="cx">
</span><del>- if (object.size() - newSize < vmPageSize)
</del><ins>+ if (object.size() - newSize < m_vmPageSize)
</ins><span class="cx"> return;
</span><span class="cx">
</span><span class="cx"> XLargeRange range = m_xLargeMap.takeAllocated(object.begin());
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocHeaph"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Heap.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Heap.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/Heap.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -93,8 +93,10 @@
</span><span class="cx"> void scavengeLargeObjects(std::unique_lock<StaticMutex>&, std::chrono::milliseconds);
</span><span class="cx"> void scavengeXLargeObjects(std::unique_lock<StaticMutex>&, std::chrono::milliseconds);
</span><span class="cx">
</span><del>- std::array<std::array<LineMetadata, smallLineCount>, sizeClassCount> m_smallLineMetadata;
</del><ins>+ size_t m_vmPageSize;
</ins><span class="cx">
</span><ins>+ Vector<LineMetadata> m_smallLineMetadata;
+
</ins><span class="cx"> std::array<List<SmallPage>, sizeClassCount> m_smallPagesWithFreeLines;
</span><span class="cx">
</span><span class="cx"> List<SmallPage> m_smallPages;
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocSizesh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Sizes.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Sizes.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/Sizes.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -46,14 +46,9 @@
</span><span class="cx"> static const size_t alignment = 8;
</span><span class="cx"> static const size_t alignmentMask = alignment - 1ul;
</span><span class="cx">
</span><del>-#if BPLATFORM(IOS)
- static const size_t vmPageSize = 16 * kB;
-#else
- static const size_t vmPageSize = 4 * kB;
-#endif
-
</del><span class="cx"> static const size_t smallLineSize = 256;
</span><del>- static const size_t smallLineCount = vmPageSize / smallLineSize;
</del><ins>+ static const size_t smallPageSize = 4 * kB;
+ static const size_t smallPageLineCount = smallPageSize / smallLineSize;
</ins><span class="cx">
</span><span class="cx"> static const size_t smallMax = 1 * kB;
</span><span class="cx"> static const size_t maskSizeClassMax = 512;
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocSmallPageh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/SmallPage.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/SmallPage.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/SmallPage.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -58,10 +58,18 @@
</span><span class="cx"> SmallLine* begin();
</span><span class="cx"> SmallLine* end();
</span><span class="cx">
</span><ins>+ unsigned char slide() const { return m_slide; }
+ void setSlide(unsigned char slide) { m_slide = slide; }
+
+ unsigned char smallPageCount() const { return m_smallPageCount; }
+ void setSmallPageCount(unsigned char smallPageCount) { m_smallPageCount = smallPageCount; }
+
</ins><span class="cx"> private:
</span><span class="cx"> unsigned char m_hasFreeLines: 1;
</span><span class="cx"> unsigned char m_refCount: 7;
</span><span class="cx"> unsigned char m_sizeClass;
</span><ins>+ unsigned char m_smallPageCount;
+ unsigned char m_slide;
</ins><span class="cx"> ObjectType m_objectType;
</span><span class="cx">
</span><span class="cx"> static_assert(
</span><span class="lines">@@ -71,12 +79,14 @@
</span><span class="cx">
</span><span class="cx"> inline void SmallPage::ref(std::lock_guard<StaticMutex>&)
</span><span class="cx"> {
</span><ins>+ BASSERT(!m_slide);
</ins><span class="cx"> ++m_refCount;
</span><span class="cx"> BASSERT(m_refCount);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline bool SmallPage::deref(std::lock_guard<StaticMutex>&)
</span><span class="cx"> {
</span><ins>+ BASSERT(!m_slide);
</ins><span class="cx"> BASSERT(m_refCount);
</span><span class="cx"> --m_refCount;
</span><span class="cx"> return !m_refCount;
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocVMAllocateh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/VMAllocate.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/VMAllocate.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/VMAllocate.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -47,31 +47,35 @@
</span><span class="cx"> #define BMALLOC_VM_TAG -1
</span><span class="cx"> #endif
</span><span class="cx">
</span><ins>+inline size_t vmPageSize()
+{
+ return sysconf(_SC_PAGESIZE);
+}
+
+inline size_t vmPageShift()
+{
+ return log2(vmPageSize());
+}
+
</ins><span class="cx"> inline size_t vmSize(size_t size)
</span><span class="cx"> {
</span><del>- return roundUpToMultipleOf<vmPageSize>(size);
</del><ins>+ return roundUpToMultipleOf(vmPageSize(), size);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void vmValidate(size_t vmSize)
</span><span class="cx"> {
</span><del>- // We use getpagesize() here instead of vmPageSize because vmPageSize is
- // allowed to be larger than the OS's true page size.
-
</del><span class="cx"> UNUSED(vmSize);
</span><span class="cx"> BASSERT(vmSize);
</span><del>- BASSERT(vmSize == roundUpToMultipleOf(static_cast<size_t>(getpagesize()), vmSize));
</del><ins>+ BASSERT(vmSize == roundUpToMultipleOf(vmPageSize(), vmSize));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void vmValidate(void* p, size_t vmSize)
</span><span class="cx"> {
</span><del>- // We use getpagesize() here instead of vmPageSize because vmPageSize is
- // allowed to be larger than the OS's true page size.
-
</del><span class="cx"> vmValidate(vmSize);
</span><span class="cx">
</span><span class="cx"> UNUSED(p);
</span><span class="cx"> BASSERT(p);
</span><del>- BASSERT(p == mask(p, ~(getpagesize() - 1)));
</del><ins>+ BASSERT(p == mask(p, ~(vmPageSize() - 1)));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> inline void* tryVMAllocate(size_t vmSize)
</span><span class="lines">@@ -106,10 +110,7 @@
</span><span class="cx"> vmValidate(vmSize);
</span><span class="cx"> vmValidate(vmAlignment);
</span><span class="cx">
</span><del>- // We use getpagesize() here instead of vmPageSize because vmPageSize is
- // allowed to be larger than the OS's true page size.
-
- size_t mappedSize = vmAlignment - getpagesize() + vmSize;
</del><ins>+ size_t mappedSize = vmAlignment - vmPageSize() + vmSize;
</ins><span class="cx"> char* mapped = static_cast<char*>(tryVMAllocate(mappedSize));
</span><span class="cx"> if (!mapped)
</span><span class="cx"> return nullptr;
</span><span class="lines">@@ -159,8 +160,8 @@
</span><span class="cx"> // Trims requests that are un-page-aligned.
</span><span class="cx"> inline void vmDeallocatePhysicalPagesSloppy(void* p, size_t size)
</span><span class="cx"> {
</span><del>- char* begin = roundUpToMultipleOf<vmPageSize>(static_cast<char*>(p));
- char* end = roundDownToMultipleOf<vmPageSize>(static_cast<char*>(p) + size);
</del><ins>+ char* begin = roundUpToMultipleOf(vmPageSize(), static_cast<char*>(p));
+ char* end = roundDownToMultipleOf(vmPageSize(), static_cast<char*>(p) + size);
</ins><span class="cx">
</span><span class="cx"> if (begin >= end)
</span><span class="cx"> return;
</span><span class="lines">@@ -171,8 +172,8 @@
</span><span class="cx"> // Expands requests that are un-page-aligned.
</span><span class="cx"> inline void vmAllocatePhysicalPagesSloppy(void* p, size_t size)
</span><span class="cx"> {
</span><del>- char* begin = roundDownToMultipleOf<vmPageSize>(static_cast<char*>(p));
- char* end = roundUpToMultipleOf<vmPageSize>(static_cast<char*>(p) + size);
</del><ins>+ char* begin = roundDownToMultipleOf(vmPageSize(), static_cast<char*>(p));
+ char* end = roundUpToMultipleOf(vmPageSize(), static_cast<char*>(p) + size);
</ins><span class="cx">
</span><span class="cx"> if (begin >= end)
</span><span class="cx"> return;
</span></span></pre></div>
<a id="trunkSourcebmallocbmallocVectorh"></a>
<div class="modfile"><h4>Modified: trunk/Source/bmalloc/bmalloc/Vector.h (198820 => 198821)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/bmalloc/bmalloc/Vector.h 2016-03-30 02:18:56 UTC (rev 198820)
+++ trunk/Source/bmalloc/bmalloc/Vector.h 2016-03-30 02:22:47 UTC (rev 198821)
</span><span class="lines">@@ -66,6 +66,7 @@
</span><span class="cx">
</span><span class="cx"> void insert(iterator, const T&);
</span><span class="cx">
</span><ins>+ void grow(size_t);
</ins><span class="cx"> void shrink(size_t);
</span><span class="cx">
</span><span class="cx"> void shrinkToFit();
</span><span class="lines">@@ -73,7 +74,7 @@
</span><span class="cx"> private:
</span><span class="cx"> static const size_t growFactor = 2;
</span><span class="cx"> static const size_t shrinkFactor = 4;
</span><del>- static const size_t initialCapacity = vmPageSize / sizeof(T);
</del><ins>+ static size_t initialCapacity() { return vmPageSize() / sizeof(T); }
</ins><span class="cx">
</span><span class="cx"> void growCapacity();
</span><span class="cx"> void shrinkCapacity();
</span><span class="lines">@@ -146,11 +147,19 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename T>
</span><ins>+inline void Vector<T>::grow(size_t size)
+{
+ BASSERT(size >= m_size);
+ while (m_size < size)
+ push(T());
+}
+
+template<typename T>
</ins><span class="cx"> inline void Vector<T>::shrink(size_t size)
</span><span class="cx"> {
</span><span class="cx"> BASSERT(size <= m_size);
</span><span class="cx"> m_size = size;
</span><del>- if (m_capacity > initialCapacity && m_size < m_capacity / shrinkFactor)
</del><ins>+ if (m_size < m_capacity / shrinkFactor && m_capacity > initialCapacity())
</ins><span class="cx"> shrinkCapacity();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -171,14 +180,14 @@
</span><span class="cx"> template<typename T>
</span><span class="cx"> NO_INLINE void Vector<T>::shrinkCapacity()
</span><span class="cx"> {
</span><del>- size_t newCapacity = max(initialCapacity, m_capacity / shrinkFactor);
</del><ins>+ size_t newCapacity = max(initialCapacity(), m_capacity / shrinkFactor);
</ins><span class="cx"> reallocateBuffer(newCapacity);
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<typename T>
</span><span class="cx"> NO_INLINE void Vector<T>::growCapacity()
</span><span class="cx"> {
</span><del>- size_t newCapacity = max(initialCapacity, m_size * growFactor);
</del><ins>+ size_t newCapacity = max(initialCapacity(), m_size * growFactor);
</ins><span class="cx"> reallocateBuffer(newCapacity);
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre>
</div>
</div>
</body>
</html>