<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[182803] trunk/Source/WebKit2</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/182803">182803</a></dd>
<dt>Author</dt> <dd>antti@apple.com</dd>
<dt>Date</dt> <dd>2015-04-14 11:40:27 -0700 (Tue, 14 Apr 2015)</dd>
</dl>
<h3>Log Message</h3>
<pre>Network Cache: Deduplicate body data
https://bugs.webkit.org/show_bug.cgi?id=143652
Reviewed by Darin Adler.
It is common to have cache entries with identical body data. This happens when the same resource is loaded from
a different URL (https vs http, slash vs no-slash at end, etc.). It also happens when the same URL is
referenced from different cache partitions.
We can improve disk space efficiency and use less memory by sharing identical body data between cache entries.
This patch splits the body data out from the record file. The new record file contains meta data and response
headers only. Body data is stored using the new BlobStorage interface. Files are deduplicated by computing
SHA1 hash over the data and looking for an existing blob with the same hash. If found the existing entry
is reused by creating a hard link to it.
The new disk structure looks like this:
WebKitCache/
Version 3/
Blobs/
0A3C9A970ADA27FAE9BD7BC630BAD0B929C293C0
0A6B8060BA77DF92C82A2FD7AF58F79524D8F34C
...
Records/
apple.com/
0B8645B04E7EC78C178B7460052601C2
0B8645B04E7EC78C178B7460052601C2-body
0CB1A3638D1C5A09C5E3283A74FA040B
0CB1A3638D1C5A09C5E3283A74FA040B-body
...
Each record file has an associated -body which is a hard link to a file in the Blobs directory.
The patch increases effective capacity by 10-20% with a typical cache. It also saves memory especially when identical
resources are used in multiple tabs.
Currently all >0 sized resources are stored as shared blobs. In future small resources should be integrated into record
files and blobs used for larger files only.
* NetworkProcess/cache/NetworkCache.cpp:
(WebKit::NetworkCache::Cache::store):
(WebKit::NetworkCache::Cache::update):
Adopt the new storage interface.
(WebKit::NetworkCache::Cache::dumpContentsToFile):
* NetworkProcess/cache/NetworkCacheBlobStorage.cpp: Added.
(WebKit::NetworkCache::BlobStorage::BlobStorage):
(WebKit::NetworkCache::BlobStorage::synchronize):
Compute size and delete unused files from the Blobs directory (link count == 1).
(WebKit::NetworkCache::BlobStorage::blobPath):
(WebKit::NetworkCache::BlobStorage::add):
(WebKit::NetworkCache::BlobStorage::get):
Interface for storing and retrieving data blobs. Blobs are deduplicated on add.
(WebKit::NetworkCache::BlobStorage::remove):
Removes the link but doesn't remove the blob even if there are no other clients. That happens on next synchronize().
(WebKit::NetworkCache::BlobStorage::shareCount):
Checks the link count to get the number of clients.
* NetworkProcess/cache/NetworkCacheBlobStorage.h: Added.
(WebKit::NetworkCache::BlobStorage::approximateSize):
* NetworkProcess/cache/NetworkCacheCoders.cpp:
(WebKit::NetworkCache::Coder<SHA1::Digest>::encode):
(WebKit::NetworkCache::Coder<SHA1::Digest>::decode):
* NetworkProcess/cache/NetworkCacheCoders.h:
* NetworkProcess/cache/NetworkCacheData.h:
(WebKit::NetworkCache::Data::isEmpty):
* NetworkProcess/cache/NetworkCacheDataCocoa.mm:
(WebKit::NetworkCache::Data::empty):
(WebKit::NetworkCache::Data::fromMap):
(WebKit::NetworkCache::mapFile):
(WebKit::NetworkCache::computeSHA1):
(WebKit::NetworkCache::bytesEqual):
Add some helpers.
* NetworkProcess/cache/NetworkCacheEntry.cpp:
(WebKit::NetworkCache::Entry::asJSON):
* NetworkProcess/cache/NetworkCacheIOChannelCocoa.mm:
(WebKit::NetworkCache::IOChannel::IOChannel):
* NetworkProcess/cache/NetworkCacheStorage.cpp:
(WebKit::NetworkCache::makeRecordDirectoryPath):
(WebKit::NetworkCache::makeBlobDirectoryPath):
(WebKit::NetworkCache::Storage::Storage):
(WebKit::NetworkCache::Storage::approximateSize):
(WebKit::NetworkCache::Storage::synchronize):
(WebKit::NetworkCache::partitionPathForKey):
(WebKit::NetworkCache::recordPathForKey):
(WebKit::NetworkCache::bodyPath):
(WebKit::NetworkCache::decodeRecordMetaData):
(WebKit::NetworkCache::decodeRecordHeader):
(WebKit::NetworkCache::createRecord):
(WebKit::NetworkCache::encodeRecordMetaData):
(WebKit::NetworkCache::encodeRecordHeader):
(WebKit::NetworkCache::Storage::remove):
(WebKit::NetworkCache::Storage::updateFileModificationTime):
(WebKit::NetworkCache::Storage::dispatchReadOperation):
Read both the blob and the record entry.
(WebKit::NetworkCache::Storage::finishReadOperation):
Factor to a function.
(WebKit::NetworkCache::Storage::store):
(WebKit::NetworkCache::Storage::traverse):
(WebKit::NetworkCache::Storage::dispatchPendingWriteOperations):
(WebKit::NetworkCache::Storage::dispatchWriteOperation):
We don't need separate full write and header write paths anymore. Everything is treated
as a full write and deduplication stops us writing the body again.
This simplifies the code and data structures.
(WebKit::NetworkCache::Storage::finishWriteOperation):
Factor to a function.
(WebKit::NetworkCache::Storage::clear):
(WebKit::NetworkCache::deletionProbability):
Take the sharing count into account when computing deletion probability.
It is less useful to delete a record that shares its body with others as data won't get deleted.
(WebKit::NetworkCache::Storage::shrinkIfNeeded):
(WebKit::NetworkCache::Storage::shrink):
(WebKit::NetworkCache::Storage::deleteOldVersions):
(WebKit::NetworkCache::directoryPathForKey): Deleted.
(WebKit::NetworkCache::filePathForKey): Deleted.
(WebKit::NetworkCache::openFileForKey): Deleted.
(WebKit::NetworkCache::decodeRecord): Deleted.
(WebKit::NetworkCache::Storage::update): Deleted.
No need for separate update interface anymore. Regular store() avoids unnecessary body write.
(WebKit::NetworkCache::Storage::dispatchFullWriteOperation): Deleted.
(WebKit::NetworkCache::Storage::dispatchHeaderWriteOperation): Deleted.
* NetworkProcess/cache/NetworkCacheStorage.h:
* WebKit2.xcodeproj/project.pbxproj:</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceWebKit2ChangeLog">trunk/Source/WebKit2/ChangeLog</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCachecpp">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCache.cpp</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheCoderscpp">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.cpp</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheCodersh">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.h</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheDatah">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheData.h</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheDataCocoamm">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheDataCocoa.mm</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheEntrycpp">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheEntry.cpp</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheStoragecpp">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.cpp</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheStorageh">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h</a></li>
<li><a href="#trunkSourceWebKit2WebKit2xcodeprojprojectpbxproj">trunk/Source/WebKit2/WebKit2.xcodeproj/project.pbxproj</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheBlobStoragecpp">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.cpp</a></li>
<li><a href="#trunkSourceWebKit2NetworkProcesscacheNetworkCacheBlobStorageh">trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.h</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceWebKit2ChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/ChangeLog (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/ChangeLog 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/ChangeLog 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -1,3 +1,153 @@
</span><ins>+2015-04-14 Antti Koivisto <antti@apple.com>
+
+ Network Cache: Deduplicate body data
+ https://bugs.webkit.org/show_bug.cgi?id=143652
+
+ Reviewed by Darin Adler.
+
+ It is common to have cache entries with identical body data. This happens when the same resource is loaded from
+ a different URL (https vs http, slash vs no-slash at end, etc.). It also happens when the same URL is
+ referenced from different cache partitions.
+
+ We can improve disk space efficiency and use less memory by sharing identical body data between cache entries.
+
+ This patch splits the body data out from the record file. The new record file contains meta data and response
+ headers only. Body data is stored using the new BlobStorage interface. Files are deduplicated by computing
+ SHA1 hash over the data and looking for an existing blob with the same hash. If found the existing entry
+ is reused by creating a hard link to it.
+
+ The new disk structure looks like this:
+
+ WebKitCache/
+ Version 3/
+ Blobs/
+ 0A3C9A970ADA27FAE9BD7BC630BAD0B929C293C0
+ 0A6B8060BA77DF92C82A2FD7AF58F79524D8F34C
+ ...
+ Records/
+ apple.com/
+ 0B8645B04E7EC78C178B7460052601C2
+ 0B8645B04E7EC78C178B7460052601C2-body
+ 0CB1A3638D1C5A09C5E3283A74FA040B
+ 0CB1A3638D1C5A09C5E3283A74FA040B-body
+ ...
+
+ Each record file has an associated -body which is a hard link to a file in the Blobs directory.
+
+ The patch increases effective capacity by 10-20% with a typical cache. It also saves memory especially when identical
+ resources are used in multiple tabs.
+
+ Currently all >0 sized resources are stored as shared blobs. In future small resources should be integrated into record
+ files and blobs used for larger files only.
+
+ * NetworkProcess/cache/NetworkCache.cpp:
+ (WebKit::NetworkCache::Cache::store):
+ (WebKit::NetworkCache::Cache::update):
+
+ Adopt the new storage interface.
+
+ (WebKit::NetworkCache::Cache::dumpContentsToFile):
+ * NetworkProcess/cache/NetworkCacheBlobStorage.cpp: Added.
+ (WebKit::NetworkCache::BlobStorage::BlobStorage):
+ (WebKit::NetworkCache::BlobStorage::synchronize):
+
+ Compute size and delete unused files from the Blobs directory (link count == 1).
+
+ (WebKit::NetworkCache::BlobStorage::blobPath):
+ (WebKit::NetworkCache::BlobStorage::add):
+ (WebKit::NetworkCache::BlobStorage::get):
+
+ Interface for storing and retrieving data blobs. Blobs are deduplicated on add.
+
+ (WebKit::NetworkCache::BlobStorage::remove):
+
+ Removes the link but doesn't remove the blob even if there are no other clients. That happens on next synchronize().
+
+ (WebKit::NetworkCache::BlobStorage::shareCount):
+
+ Checks the link count to get the number of clients.
+
+ * NetworkProcess/cache/NetworkCacheBlobStorage.h: Added.
+ (WebKit::NetworkCache::BlobStorage::approximateSize):
+ * NetworkProcess/cache/NetworkCacheCoders.cpp:
+ (WebKit::NetworkCache::Coder<SHA1::Digest>::encode):
+ (WebKit::NetworkCache::Coder<SHA1::Digest>::decode):
+ * NetworkProcess/cache/NetworkCacheCoders.h:
+ * NetworkProcess/cache/NetworkCacheData.h:
+ (WebKit::NetworkCache::Data::isEmpty):
+ * NetworkProcess/cache/NetworkCacheDataCocoa.mm:
+ (WebKit::NetworkCache::Data::empty):
+ (WebKit::NetworkCache::Data::fromMap):
+ (WebKit::NetworkCache::mapFile):
+ (WebKit::NetworkCache::computeSHA1):
+ (WebKit::NetworkCache::bytesEqual):
+
+ Add some helpers.
+
+ * NetworkProcess/cache/NetworkCacheEntry.cpp:
+ (WebKit::NetworkCache::Entry::asJSON):
+ * NetworkProcess/cache/NetworkCacheIOChannelCocoa.mm:
+ (WebKit::NetworkCache::IOChannel::IOChannel):
+ * NetworkProcess/cache/NetworkCacheStorage.cpp:
+ (WebKit::NetworkCache::makeRecordDirectoryPath):
+ (WebKit::NetworkCache::makeBlobDirectoryPath):
+ (WebKit::NetworkCache::Storage::Storage):
+ (WebKit::NetworkCache::Storage::approximateSize):
+ (WebKit::NetworkCache::Storage::synchronize):
+ (WebKit::NetworkCache::partitionPathForKey):
+ (WebKit::NetworkCache::recordPathForKey):
+ (WebKit::NetworkCache::bodyPath):
+ (WebKit::NetworkCache::decodeRecordMetaData):
+ (WebKit::NetworkCache::decodeRecordHeader):
+ (WebKit::NetworkCache::createRecord):
+ (WebKit::NetworkCache::encodeRecordMetaData):
+ (WebKit::NetworkCache::encodeRecordHeader):
+ (WebKit::NetworkCache::Storage::remove):
+ (WebKit::NetworkCache::Storage::updateFileModificationTime):
+ (WebKit::NetworkCache::Storage::dispatchReadOperation):
+
+ Read both the blob and the record entry.
+
+ (WebKit::NetworkCache::Storage::finishReadOperation):
+
+ Factor to a function.
+
+ (WebKit::NetworkCache::Storage::store):
+ (WebKit::NetworkCache::Storage::traverse):
+ (WebKit::NetworkCache::Storage::dispatchPendingWriteOperations):
+ (WebKit::NetworkCache::Storage::dispatchWriteOperation):
+
+ We don't need separate full write and header write paths anymore. Everything is treated
+ as a full write and deduplication stops us writing the body again.
+
+ This simplifies the code and data structures.
+
+ (WebKit::NetworkCache::Storage::finishWriteOperation):
+
+ Factor to a function.
+
+ (WebKit::NetworkCache::Storage::clear):
+ (WebKit::NetworkCache::deletionProbability):
+
+ Take the sharing count into account when computing deletion probability.
+ It is less useful to delete a record that shares its body with others as data won't get deleted.
+
+ (WebKit::NetworkCache::Storage::shrinkIfNeeded):
+ (WebKit::NetworkCache::Storage::shrink):
+ (WebKit::NetworkCache::Storage::deleteOldVersions):
+ (WebKit::NetworkCache::directoryPathForKey): Deleted.
+ (WebKit::NetworkCache::filePathForKey): Deleted.
+ (WebKit::NetworkCache::openFileForKey): Deleted.
+ (WebKit::NetworkCache::decodeRecord): Deleted.
+ (WebKit::NetworkCache::Storage::update): Deleted.
+
+ No need for separate update interface anymore. Regular store() avoids unnecessary body write.
+
+ (WebKit::NetworkCache::Storage::dispatchFullWriteOperation): Deleted.
+ (WebKit::NetworkCache::Storage::dispatchHeaderWriteOperation): Deleted.
+ * NetworkProcess/cache/NetworkCacheStorage.h:
+ * WebKit2.xcodeproj/project.pbxproj:
+
</ins><span class="cx"> 2015-04-14 Chris Dumez <cdumez@apple.com>
</span><span class="cx">
</span><span class="cx"> REGRESSION(r182603): [GTK] More than 500 crashes on the layout tests with the debug build.
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCachecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCache.cpp (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCache.cpp 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCache.cpp 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -370,7 +370,7 @@
</span><span class="cx">
</span><span class="cx"> auto record = cacheEntry.encodeAsStorageRecord();
</span><span class="cx">
</span><del>- m_storage->store(record, [completionHandler](bool success, const Data& bodyData) {
</del><ins>+ m_storage->store(record, [completionHandler](const Data& bodyData) {
</ins><span class="cx"> MappedBody mappedBody;
</span><span class="cx"> #if ENABLE(SHAREABLE_RESOURCE)
</span><span class="cx"> if (bodyData.isMap()) {
</span><span class="lines">@@ -381,7 +381,7 @@
</span><span class="cx"> }
</span><span class="cx"> #endif
</span><span class="cx"> completionHandler(mappedBody);
</span><del>- LOG(NetworkCache, "(NetworkProcess) store success=%d", success);
</del><ins>+ LOG(NetworkCache, "(NetworkProcess) stored");
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -396,9 +396,7 @@
</span><span class="cx">
</span><span class="cx"> auto updateRecord = updateEntry.encodeAsStorageRecord();
</span><span class="cx">
</span><del>- m_storage->update(updateRecord, existingEntry.sourceStorageRecord(), [](bool success, const Data&) {
- LOG(NetworkCache, "(NetworkProcess) updated, success=%d", success);
- });
</del><ins>+ m_storage->store(updateRecord, { });
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Cache::remove(const Key& key)
</span><span class="lines">@@ -447,7 +445,8 @@
</span><span class="cx"> size_t bodySize { 0 };
</span><span class="cx"> };
</span><span class="cx"> Totals totals;
</span><del>- m_storage->traverse(Storage::TraverseFlag::ComputeWorth, [fd, totals](const Storage::Record* record, const Storage::RecordInfo& info) mutable {
</del><ins>+ auto flags = Storage::TraverseFlag::ComputeWorth | Storage::TraverseFlag::ShareCount;
+ m_storage->traverse(flags, [fd, totals](const Storage::Record* record, const Storage::RecordInfo& info) mutable {
</ins><span class="cx"> if (!record) {
</span><span class="cx"> StringBuilder epilogue;
</span><span class="cx"> epilogue.appendLiteral("{}\n],\n");
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheBlobStoragecpp"></a>
<div class="addfile"><h4>Added: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.cpp (0 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.cpp (rev 0)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.cpp 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -0,0 +1,171 @@
</span><ins>+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "config.h"
+#include "NetworkCacheBlobStorage.h"
+
+#if ENABLE(NETWORK_CACHE)
+
+#include "Logging.h"
+#include "NetworkCacheFileSystemPosix.h"
+#include <WebCore/FileSystem.h>
+#include <sys/mman.h>
+#include <wtf/RunLoop.h>
+#include <wtf/SHA1.h>
+#include <wtf/text/StringBuilder.h>
+
+namespace WebKit {
+namespace NetworkCache {
+
+BlobStorage::BlobStorage(const String& blobDirectoryPath)
+ : m_blobDirectoryPath(blobDirectoryPath)
+{
+}
+
+String BlobStorage::blobDirectoryPath() const
+{
+ return m_blobDirectoryPath.isolatedCopy();
+}
+
+void BlobStorage::synchronize()
+{
+ ASSERT(!RunLoop::isMain());
+
+ WebCore::makeAllDirectories(blobDirectoryPath());
+
+ m_approximateSize = 0;
+ auto blobDirectory = blobDirectoryPath();
+ traverseDirectory(blobDirectory, DT_REG, [this, &blobDirectory](const String& name) {
+ auto path = WebCore::pathByAppendingComponent(blobDirectory, name);
+ auto filePath = WebCore::fileSystemRepresentation(path);
+ struct stat stat;
+ ::stat(filePath.data(), &stat);
+ // No clients left for this blob.
+ if (stat.st_nlink == 1)
+ unlink(filePath.data());
+ else
+ m_approximateSize += stat.st_size;
+ });
+
+ LOG(NetworkCacheStorage, "(NetworkProcess) blob synchronization completed approximateSize=%zu", approximateSize());
+}
+
+String BlobStorage::blobPathForHash(const SHA1::Digest& hash) const
+{
+ auto hashAsString = SHA1::hexDigest(hash);
+ return WebCore::pathByAppendingComponent(blobDirectoryPath(), String::fromUTF8(hashAsString));
+}
+
+BlobStorage::Blob BlobStorage::add(const String& path, const Data& data)
+{
+ ASSERT(!RunLoop::isMain());
+
+ auto hash = computeSHA1(data);
+ if (data.isEmpty())
+ return { data, hash };
+
+ auto blobPath = WebCore::fileSystemRepresentation(blobPathForHash(hash));
+ auto linkPath = WebCore::fileSystemRepresentation(path);
+ unlink(linkPath.data());
+
+ bool blobExists = access(blobPath.data(), F_OK) != -1;
+ if (blobExists) {
+ auto existingData = mapFile(blobPath.data());
+ if (bytesEqual(existingData, data)) {
+ link(blobPath.data(), linkPath.data());
+ return { existingData, hash };
+ }
+ unlink(blobPath.data());
+ }
+
+ int fd = open(blobPath.data(), O_CREAT | O_EXCL | O_RDWR , S_IRUSR | S_IWUSR);
+ if (fd < 0)
+ return { };
+
+ size_t size = data.size();
+ if (ftruncate(fd, size) < 0) {
+ close(fd);
+ return { };
+ }
+
+ void* map = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+ close(fd);
+
+ if (map == MAP_FAILED)
+ return { };
+
+ uint8_t* mapData = static_cast<uint8_t*>(map);
+ data.apply([&mapData](const uint8_t* bytes, size_t bytesSize) {
+ memcpy(mapData, bytes, bytesSize);
+ mapData += bytesSize;
+ return true;
+ });
+
+ // Drop the write permission.
+ mprotect(map, size, PROT_READ);
+
+ auto mappedData = Data::adoptMap(map, size);
+
+ link(blobPath.data(), linkPath.data());
+
+ m_approximateSize += size;
+
+ return { mappedData, hash };
+}
+
+BlobStorage::Blob BlobStorage::get(const String& path)
+{
+ ASSERT(!RunLoop::isMain());
+
+ auto linkPath = WebCore::fileSystemRepresentation(path);
+ auto data = mapFile(linkPath.data());
+
+ return { data, computeSHA1(data) };
+}
+
+void BlobStorage::remove(const String& path)
+{
+ ASSERT(!RunLoop::isMain());
+
+ auto linkPath = WebCore::fileSystemRepresentation(path);
+ unlink(linkPath.data());
+}
+
+unsigned BlobStorage::shareCount(const String& path)
+{
+ ASSERT(!RunLoop::isMain());
+
+ auto linkPath = WebCore::fileSystemRepresentation(path);
+ struct stat stat;
+ if (::stat(linkPath.data(), &stat) < 0)
+ return 0;
+ // Link count is 2 in the single client case (the blob file and a link).
+ return stat.st_nlink - 1;
+}
+
+}
+}
+
+#endif
</ins></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheBlobStorageh"></a>
<div class="addfile"><h4>Added: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.h (0 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.h (rev 0)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheBlobStorage.h 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -0,0 +1,74 @@
</span><ins>+/*
+ * Copyright (C) 2015 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef NetworkCacheBlobStorage_h
+#define NetworkCacheBlobStorage_h
+
+#if ENABLE(NETWORK_CACHE)
+
+#include "NetworkCacheData.h"
+#include "NetworkCacheKey.h"
+#include <wtf/SHA1.h>
+
+namespace WebKit {
+namespace NetworkCache {
+
+// BlobStorage deduplicates the data using SHA1 hash computed over the blob bytes.
+class BlobStorage {
+ WTF_MAKE_NONCOPYABLE(BlobStorage);
+public:
+ BlobStorage(const String& blobDirectoryPath);
+
+ struct Blob {
+ Data data;
+ SHA1::Digest hash;
+ };
+ // These are all synchronous and should not be used from the main thread.
+ Blob add(const String& path, const Data&);
+ Blob get(const String& path);
+
+ // Blob won't be removed until synchronization.
+ void remove(const String& path);
+
+ unsigned shareCount(const String& path);
+
+ size_t approximateSize() const { return m_approximateSize; }
+
+ void synchronize();
+
+private:
+ String blobDirectoryPath() const;
+ String blobPathForHash(const SHA1::Digest&) const;
+
+ const String m_blobDirectoryPath;
+
+ std::atomic<size_t> m_approximateSize { 0 };
+};
+
+}
+}
+
+#endif
+#endif
</ins></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheCoderscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.cpp (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.cpp 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.cpp 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -184,7 +184,17 @@
</span><span class="cx"> return decoder.decodeFixedLengthData(digest.data(), sizeof(digest));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void Coder<SHA1::Digest>::encode(Encoder& encoder, const SHA1::Digest& digest)
+{
+ encoder.encodeFixedLengthData(digest.data(), sizeof(digest));
</ins><span class="cx"> }
</span><ins>+
+bool Coder<SHA1::Digest>::decode(Decoder& decoder, SHA1::Digest& digest)
+{
+ return decoder.decodeFixedLengthData(digest.data(), sizeof(digest));
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+}
+}
+
</ins><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheCodersh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.h (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.h 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheCoders.h 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -36,6 +36,7 @@
</span><span class="cx"> #include <wtf/HashMap.h>
</span><span class="cx"> #include <wtf/HashSet.h>
</span><span class="cx"> #include <wtf/MD5.h>
</span><ins>+#include <wtf/SHA1.h>
</ins><span class="cx"> #include <wtf/Vector.h>
</span><span class="cx">
</span><span class="cx"> namespace WebKit {
</span><span class="lines">@@ -262,6 +263,11 @@
</span><span class="cx"> static bool decode(Decoder&, MD5::Digest&);
</span><span class="cx"> };
</span><span class="cx">
</span><ins>+template<> struct Coder<SHA1::Digest> {
+ static void encode(Encoder&, const SHA1::Digest&);
+ static bool decode(Decoder&, SHA1::Digest&);
+};
+
</ins><span class="cx"> }
</span><span class="cx"> }
</span><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheDatah"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheData.h (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheData.h 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheData.h 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx">
</span><span class="cx"> #include <functional>
</span><span class="cx"> #include <wtf/FunctionDispatcher.h>
</span><ins>+#include <wtf/SHA1.h>
</ins><span class="cx"> #include <wtf/ThreadSafeRefCounted.h>
</span><span class="cx"> #include <wtf/text/WTFString.h>
</span><span class="cx">
</span><span class="lines">@@ -98,11 +99,15 @@
</span><span class="cx"> Data() { }
</span><span class="cx"> Data(const uint8_t*, size_t);
</span><span class="cx">
</span><ins>+ static Data empty();
+ static Data adoptMap(void* map, size_t);
+
</ins><span class="cx"> enum class Backing { Buffer, Map };
</span><span class="cx"> #if PLATFORM(COCOA)
</span><span class="cx"> Data(DispatchPtr<dispatch_data_t>, Backing = Backing::Buffer);
</span><span class="cx"> #endif
</span><span class="cx"> bool isNull() const;
</span><ins>+ bool isEmpty() const { return !m_size; }
</ins><span class="cx">
</span><span class="cx"> const uint8_t* data() const;
</span><span class="cx"> size_t size() const { return m_size; }
</span><span class="lines">@@ -125,7 +130,10 @@
</span><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> Data concatenate(const Data&, const Data&);
</span><ins>+bool bytesEqual(const Data&, const Data&);
</ins><span class="cx"> Data mapFile(int fd, size_t offset, size_t);
</span><ins>+Data mapFile(const char* path);
+SHA1::Digest computeSHA1(const Data&);
</ins><span class="cx">
</span><span class="cx"> }
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheDataCocoamm"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheDataCocoa.mm (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheDataCocoa.mm 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheDataCocoa.mm 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx">
</span><span class="cx"> #include <dispatch/dispatch.h>
</span><span class="cx"> #include <sys/mman.h>
</span><ins>+#include <sys/stat.h>
</ins><span class="cx">
</span><span class="cx"> namespace WebKit {
</span><span class="cx"> namespace NetworkCache {
</span><span class="lines">@@ -47,6 +48,11 @@
</span><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+Data Data::empty()
+{
+ return { DispatchPtr<dispatch_data_t>(dispatch_data_empty) };
+}
+
</ins><span class="cx"> const uint8_t* Data::data() const
</span><span class="cx"> {
</span><span class="cx"> if (!m_data && m_dispatchData) {
</span><span class="lines">@@ -87,18 +93,65 @@
</span><span class="cx"> return { adoptDispatch(dispatch_data_create_concat(a.dispatchData(), b.dispatchData())) };
</span><span class="cx"> }
</span><span class="cx">
</span><del>-Data mapFile(int fd, size_t offset, size_t size)
</del><ins>+Data Data::adoptMap(void* map, size_t size)
</ins><span class="cx"> {
</span><del>- void* map = mmap(nullptr, size, PROT_READ, MAP_PRIVATE, fd, offset);
- if (map == MAP_FAILED)
- return { };
</del><ins>+ ASSERT(map && map != MAP_FAILED);
</ins><span class="cx"> auto bodyMap = adoptDispatch(dispatch_data_create(map, size, dispatch_get_main_queue(), [map, size] {
</span><span class="cx"> munmap(map, size);
</span><span class="cx"> }));
</span><span class="cx"> return { bodyMap, Data::Backing::Map };
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+Data mapFile(int fd, size_t offset, size_t size)
+{
+ if (!size)
+ return Data::empty();
+
+ void* map = mmap(nullptr, size, PROT_READ, MAP_PRIVATE, fd, offset);
+ if (map == MAP_FAILED)
+ return { };
+ return Data::adoptMap(map, size);
</ins><span class="cx"> }
</span><ins>+
+Data mapFile(const char* path)
+{
+ int fd = open(path, O_RDONLY, 0);
+ if (fd < 0)
+ return { };
+ struct stat stat;
+ if (fstat(fd, &stat) < 0) {
+ close(fd);
+ return { };
+ }
+ size_t size = stat.st_size;
+ auto data = mapFile(fd, 0, size);
+ close(fd);
+
+ return data;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+SHA1::Digest computeSHA1(const Data& data)
+{
+ SHA1 sha1;
+ data.apply([&sha1](const uint8_t* data, size_t size) {
+ sha1.addBytes(data, size);
+ return true;
+ });
+ SHA1::Digest digest;
+ sha1.computeHash(digest);
+ return digest;
+}
+
+bool bytesEqual(const Data& a, const Data& b)
+{
+ if (a.isNull() || b.isNull())
+ return false;
+ if (a.size() != b.size())
+ return false;
+ return !memcmp(a.data(), b.data(), a.size());
+}
+
+}
+}
+
</ins><span class="cx"> #endif
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheEntrycpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheEntry.cpp (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheEntry.cpp 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheEntry.cpp 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -165,6 +165,12 @@
</span><span class="cx"> json.appendLiteral("\"URL\": ");
</span><span class="cx"> JSC::appendQuotedJSONStringToBuilder(json, m_response.url().string());
</span><span class="cx"> json.appendLiteral(",\n");
</span><ins>+ json.appendLiteral("\"bodyHash\": ");
+ JSC::appendQuotedJSONStringToBuilder(json, info.bodyHash);
+ json.appendLiteral(",\n");
+ json.appendLiteral("\"bodyShareCount\": ");
+ json.appendNumber(info.bodyShareCount);
+ json.appendLiteral(",\n");
</ins><span class="cx"> json.appendLiteral("\"headers\": {\n");
</span><span class="cx"> bool firstHeader = true;
</span><span class="cx"> for (auto& header : m_response.httpHeaderFields()) {
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheStoragecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.cpp (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.cpp 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.cpp 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -43,6 +43,9 @@
</span><span class="cx">
</span><span class="cx"> static const char networkCacheSubdirectory[] = "WebKitCache";
</span><span class="cx"> static const char versionDirectoryPrefix[] = "Version ";
</span><ins>+static const char recordsDirectoryName[] = "Records";
+static const char blobsDirectoryName[] = "Blobs";
+static const char bodyPostfix[] = "-body";
</ins><span class="cx">
</span><span class="cx"> static double computeRecordWorth(FileTimes);
</span><span class="cx">
</span><span class="lines">@@ -62,17 +65,33 @@
</span><span class="cx"> return WebCore::pathByAppendingComponent(baseDirectoryPath, versionSubdirectory);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+static String makeRecordDirectoryPath(const String& baseDirectoryPath)
+{
+ return WebCore::pathByAppendingComponent(makeVersionedDirectoryPath(baseDirectoryPath), recordsDirectoryName);
+}
+
+static String makeBlobDirectoryPath(const String& baseDirectoryPath)
+{
+ return WebCore::pathByAppendingComponent(makeVersionedDirectoryPath(baseDirectoryPath), blobsDirectoryName);
+}
+
</ins><span class="cx"> Storage::Storage(const String& baseDirectoryPath)
</span><span class="cx"> : m_baseDirectoryPath(baseDirectoryPath)
</span><del>- , m_directoryPath(makeVersionedDirectoryPath(baseDirectoryPath))
</del><ins>+ , m_directoryPath(makeRecordDirectoryPath(baseDirectoryPath))
</ins><span class="cx"> , m_ioQueue(WorkQueue::create("com.apple.WebKit.Cache.Storage", WorkQueue::Type::Concurrent))
</span><span class="cx"> , m_backgroundIOQueue(WorkQueue::create("com.apple.WebKit.Cache.Storage.background", WorkQueue::Type::Concurrent, WorkQueue::QOS::Background))
</span><span class="cx"> , m_serialBackgroundIOQueue(WorkQueue::create("com.apple.WebKit.Cache.Storage.serialBackground", WorkQueue::Type::Serial, WorkQueue::QOS::Background))
</span><ins>+ , m_blobStorage(makeBlobDirectoryPath(baseDirectoryPath))
</ins><span class="cx"> {
</span><span class="cx"> deleteOldVersions();
</span><span class="cx"> synchronize();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+size_t Storage::approximateSize() const
+{
+ return m_approximateSize + m_blobStorage.approximateSize();
+}
+
</ins><span class="cx"> void Storage::synchronize()
</span><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="lines">@@ -117,7 +136,9 @@
</span><span class="cx"> m_synchronizationInProgress = false;
</span><span class="cx"> });
</span><span class="cx">
</span><del>- LOG(NetworkCacheStorage, "(NetworkProcess) cache synchronization completed approximateSize=%zu count=%d", size, count);
</del><ins>+ m_blobStorage.synchronize();
+
+ LOG(NetworkCacheStorage, "(NetworkProcess) cache synchronization completed size=%zu count=%d", size, count);
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -139,7 +160,7 @@
</span><span class="cx"> return !m_contentsFilter || m_contentsFilter->mayContain(key.hash());
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static String directoryPathForKey(const Key& key, const String& cachePath)
</del><ins>+static String partitionPathForKey(const Key& key, const String& cachePath)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(!key.partition().isEmpty());
</span><span class="cx"> return WebCore::pathByAppendingComponent(cachePath, key.partition());
</span><span class="lines">@@ -150,20 +171,21 @@
</span><span class="cx"> return key.hashAsString();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static String filePathForKey(const Key& key, const String& cachePath)
</del><ins>+static String recordPathForKey(const Key& key, const String& cachePath)
</ins><span class="cx"> {
</span><del>- return WebCore::pathByAppendingComponent(directoryPathForKey(key, cachePath), fileNameForKey(key));
</del><ins>+ return WebCore::pathByAppendingComponent(partitionPathForKey(key, cachePath), fileNameForKey(key));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-static Ref<IOChannel> openFileForKey(const Key& key, IOChannel::Type type, const String& cachePath)
</del><ins>+static String bodyPathForRecordPath(const String& recordPath)
</ins><span class="cx"> {
</span><del>- auto directoryPath = directoryPathForKey(key, cachePath);
- auto filePath = WebCore::pathByAppendingComponent(directoryPath, fileNameForKey(key));
- if (type == IOChannel::Type::Create)
- WebCore::makeAllDirectories(directoryPath);
- return IOChannel::open(filePath, type);
</del><ins>+ return recordPath + bodyPostfix;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+static String bodyPathForKey(const Key& key, const String& cachePath)
+{
+ return bodyPathForRecordPath(recordPathForKey(key, cachePath));
+}
+
</ins><span class="cx"> static unsigned hashData(const Data& data)
</span><span class="cx"> {
</span><span class="cx"> StringHasher hasher;
</span><span class="lines">@@ -188,8 +210,7 @@
</span><span class="cx"> unsigned headerChecksum;
</span><span class="cx"> uint64_t headerOffset;
</span><span class="cx"> uint64_t headerSize;
</span><del>- unsigned bodyChecksum;
- uint64_t bodyOffset;
</del><ins>+ SHA1::Digest bodyHash;
</ins><span class="cx"> uint64_t bodySize;
</span><span class="cx"> };
</span><span class="cx">
</span><span class="lines">@@ -208,14 +229,13 @@
</span><span class="cx"> return false;
</span><span class="cx"> if (!decoder.decode(metaData.headerSize))
</span><span class="cx"> return false;
</span><del>- if (!decoder.decode(metaData.bodyChecksum))
</del><ins>+ if (!decoder.decode(metaData.bodyHash))
</ins><span class="cx"> return false;
</span><span class="cx"> if (!decoder.decode(metaData.bodySize))
</span><span class="cx"> return false;
</span><span class="cx"> if (!decoder.verifyChecksum())
</span><span class="cx"> return false;
</span><span class="cx"> metaData.headerOffset = decoder.currentOffset();
</span><del>- metaData.bodyOffset = WTF::roundUpToMultipleOf(pageSize(), metaData.headerOffset + metaData.headerSize);
</del><span class="cx"> success = true;
</span><span class="cx"> return false;
</span><span class="cx"> });
</span><span class="lines">@@ -233,10 +253,6 @@
</span><span class="cx"> LOG(NetworkCacheStorage, "(NetworkProcess) version mismatch");
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><del>- if (metaData.headerOffset + metaData.headerSize > metaData.bodyOffset) {
- LOG(NetworkCacheStorage, "(NetworkProcess) body offset mismatch");
- return false;
- }
</del><span class="cx">
</span><span class="cx"> auto headerData = fileData.subrange(metaData.headerOffset, metaData.headerSize);
</span><span class="cx"> if (metaData.headerChecksum != hashData(headerData)) {
</span><span class="lines">@@ -247,11 +263,11 @@
</span><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static std::unique_ptr<Storage::Record> decodeRecord(const Data& fileData, int fd, const Key& key)
</del><ins>+static std::unique_ptr<Storage::Record> createRecord(const Data& recordData, const BlobStorage::Blob& bodyBlob, const Key& key)
</ins><span class="cx"> {
</span><span class="cx"> RecordMetaData metaData;
</span><span class="cx"> Data headerData;
</span><del>- if (!decodeRecordHeader(fileData, metaData, headerData))
</del><ins>+ if (!decodeRecordHeader(recordData, metaData, headerData))
</ins><span class="cx"> return nullptr;
</span><span class="cx">
</span><span class="cx"> if (metaData.key != key)
</span><span class="lines">@@ -261,29 +277,16 @@
</span><span class="cx"> auto timeStamp = std::chrono::system_clock::time_point(metaData.epochRelativeTimeStamp);
</span><span class="cx"> if (timeStamp > std::chrono::system_clock::now())
</span><span class="cx"> return nullptr;
</span><ins>+ if (metaData.bodySize != bodyBlob.data.size())
+ return nullptr;
+ if (metaData.bodyHash != bodyBlob.hash)
+ return nullptr;
</ins><span class="cx">
</span><del>- Data bodyData;
- if (metaData.bodySize) {
- if (metaData.bodyOffset + metaData.bodySize != fileData.size())
- return nullptr;
-
- bodyData = mapFile(fd, metaData.bodyOffset, metaData.bodySize);
- if (bodyData.isNull()) {
- LOG(NetworkCacheStorage, "(NetworkProcess) map failed");
- return nullptr;
- }
-
- if (metaData.bodyChecksum != hashData(bodyData)) {
- LOG(NetworkCacheStorage, "(NetworkProcess) data checksum mismatch");
- return nullptr;
- }
- }
-
</del><span class="cx"> return std::make_unique<Storage::Record>(Storage::Record {
</span><span class="cx"> metaData.key,
</span><span class="cx"> timeStamp,
</span><span class="cx"> headerData,
</span><del>- bodyData
</del><ins>+ bodyBlob.data
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -296,7 +299,7 @@
</span><span class="cx"> encoder << metaData.epochRelativeTimeStamp;
</span><span class="cx"> encoder << metaData.headerChecksum;
</span><span class="cx"> encoder << metaData.headerSize;
</span><del>- encoder << metaData.bodyChecksum;
</del><ins>+ encoder << metaData.bodyHash;
</ins><span class="cx"> encoder << metaData.bodySize;
</span><span class="cx">
</span><span class="cx"> encoder.encodeChecksum();
</span><span class="lines">@@ -304,25 +307,18 @@
</span><span class="cx"> return Data(encoder.buffer(), encoder.bufferSize());
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static Data encodeRecordHeader(const Storage::Record& record)
</del><ins>+static Data encodeRecordHeader(const Storage::Record& record, SHA1::Digest bodyHash)
</ins><span class="cx"> {
</span><span class="cx"> RecordMetaData metaData(record.key);
</span><span class="cx"> metaData.epochRelativeTimeStamp = std::chrono::duration_cast<std::chrono::milliseconds>(record.timeStamp.time_since_epoch());
</span><span class="cx"> metaData.headerChecksum = hashData(record.header);
</span><span class="cx"> metaData.headerSize = record.header.size();
</span><del>- metaData.bodyChecksum = hashData(record.body);
</del><ins>+ metaData.bodyHash = bodyHash;
</ins><span class="cx"> metaData.bodySize = record.body.size();
</span><span class="cx">
</span><span class="cx"> auto encodedMetaData = encodeRecordMetaData(metaData);
</span><span class="cx"> auto headerData = concatenate(encodedMetaData, record.header);
</span><del>- if (!record.body.size())
- return { headerData };
-
- size_t dataOffset = WTF::roundUpToMultipleOf(pageSize(), headerData.size());
- Vector<uint8_t, 4096> filler(dataOffset - headerData.size(), 0);
- Data alignmentData(filler.data(), filler.size());
-
- return concatenate(headerData, alignmentData);
</del><ins>+ return { headerData };
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Storage::remove(const Key& key)
</span><span class="lines">@@ -333,15 +329,17 @@
</span><span class="cx"> // For simplicity we also don't reduce m_approximateSize on removals.
</span><span class="cx"> // The next synchronization will update everything.
</span><span class="cx">
</span><del>- StringCapture filePathCapture(filePathForKey(key, m_directoryPath));
- serialBackgroundIOQueue().dispatch([this, filePathCapture] {
- WebCore::deleteFile(filePathCapture.string());
</del><ins>+ StringCapture recordPathCapture(recordPathForKey(key, m_directoryPath));
+ StringCapture bodyPathCapture(bodyPathForKey(key, m_directoryPath));
+ serialBackgroundIOQueue().dispatch([this, recordPathCapture, bodyPathCapture] {
+ WebCore::deleteFile(recordPathCapture.string());
+ m_blobStorage.remove(bodyPathCapture.string());
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Storage::updateFileModificationTime(IOChannel& channel)
</del><ins>+void Storage::updateFileModificationTime(const String& path)
</ins><span class="cx"> {
</span><del>- StringCapture filePathCapture(channel.path());
</del><ins>+ StringCapture filePathCapture(path);
</ins><span class="cx"> serialBackgroundIOQueue().dispatch([filePathCapture] {
</span><span class="cx"> updateFileModificationTimeIfNeeded(filePathCapture.string());
</span><span class="cx"> });
</span><span class="lines">@@ -354,29 +352,35 @@
</span><span class="cx">
</span><span class="cx"> StringCapture cachePathCapture(m_directoryPath);
</span><span class="cx"> ioQueue().dispatch([this, &read, cachePathCapture] {
</span><del>- RefPtr<IOChannel> channel = openFileForKey(read.key, IOChannel::Type::Read, cachePathCapture.string());
- channel->read(0, std::numeric_limits<size_t>::max(), [this, channel, &read](Data& fileData, int error) {
- if (error) {
- remove(read.key);
- read.completionHandler(nullptr);
- } else {
- auto record = decodeRecord(fileData, channel->fileDescriptor(), read.key);
- bool success = read.completionHandler(WTF::move(record));
- if (success)
- updateFileModificationTime(*channel);
- else
- remove(read.key);
- }
</del><ins>+ auto recordPath = recordPathForKey(read.key, cachePathCapture.string());
+ auto bodyPath = bodyPathForKey(read.key, cachePathCapture.string());
+ // FIXME: Body and header retrieves can be done in parallel.
+ auto bodyBlob = m_blobStorage.get(bodyPath);
</ins><span class="cx">
</span><del>- ASSERT(m_activeReadOperations.contains(&read));
- m_activeReadOperations.remove(&read);
- dispatchPendingReadOperations();
-
- LOG(NetworkCacheStorage, "(NetworkProcess) read complete error=%d", error);
</del><ins>+ RefPtr<IOChannel> channel = IOChannel::open(recordPath, IOChannel::Type::Read);
+ channel->read(0, std::numeric_limits<size_t>::max(), [this, &read, bodyBlob](Data& fileData, int error) {
+ auto record = error ? nullptr : createRecord(fileData, bodyBlob, read.key);
+ finishReadOperation(read, WTF::move(record));
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void Storage::finishReadOperation(const ReadOperation& read, std::unique_ptr<Record> record)
+{
+ ASSERT(RunLoop::isMain());
+
+ bool success = read.completionHandler(WTF::move(record));
+ if (success)
+ updateFileModificationTime(recordPathForKey(read.key, m_directoryPath));
+ else
+ remove(read.key);
+ ASSERT(m_activeReadOperations.contains(&read));
+ m_activeReadOperations.remove(&read);
+ dispatchPendingReadOperations();
+
+ LOG(NetworkCacheStorage, "(NetworkProcess) read complete success=%d", success);
+}
+
</ins><span class="cx"> void Storage::dispatchPendingReadOperations()
</span><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="lines">@@ -413,6 +417,83 @@
</span><span class="cx"> return false;
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void Storage::dispatchPendingWriteOperations()
+{
+ ASSERT(RunLoop::isMain());
+
+ const int maximumActiveWriteOperationCount { 3 };
+
+ while (!m_pendingWriteOperations.isEmpty()) {
+ if (m_activeWriteOperations.size() >= maximumActiveWriteOperationCount) {
+ LOG(NetworkCacheStorage, "(NetworkProcess) limiting parallel writes");
+ return;
+ }
+ auto writeOperation = m_pendingWriteOperations.takeFirst();
+ auto& write = *writeOperation;
+ m_activeWriteOperations.add(WTF::move(writeOperation));
+
+ dispatchWriteOperation(write);
+ }
+}
+
+void Storage::dispatchWriteOperation(const WriteOperation& write)
+{
+ ASSERT(RunLoop::isMain());
+ ASSERT(m_activeWriteOperations.contains(&write));
+
+ // This was added already when starting the store but filter might have been wiped.
+ addToContentsFilter(write.record.key);
+
+ StringCapture cachePathCapture(m_directoryPath);
+ backgroundIOQueue().dispatch([this, &write, cachePathCapture] {
+ auto partitionPath = partitionPathForKey(write.record.key, cachePathCapture.string());
+ auto recordPath = recordPathForKey(write.record.key, cachePathCapture.string());
+ auto bodyPath = bodyPathForKey(write.record.key, cachePathCapture.string());
+
+ WebCore::makeAllDirectories(partitionPath);
+
+ // Store the body.
+ auto blob = m_blobStorage.add(bodyPath, write.record.body);
+ if (blob.data.isNull()) {
+ RunLoop::main().dispatch([this, &write] {
+ finishWriteOperation(write);
+ });
+ return;
+ }
+
+ // Tell the client we now have a disk-backed map for this data.
+ size_t minimumMapSize = pageSize();
+ if (blob.data.size() >= minimumMapSize && blob.data.isMap() && write.mappedBodyHandler) {
+ auto& mappedBodyHandler = write.mappedBodyHandler;
+ RunLoop::main().dispatch([blob, mappedBodyHandler] {
+ mappedBodyHandler(blob.data);
+ });
+ }
+
+ // Store the header and meta data.
+ auto encodedHeader = encodeRecordHeader(write.record, blob.hash);
+ auto channel = IOChannel::open(recordPath, IOChannel::Type::Create);
+ int fd = channel->fileDescriptor();
+ size_t headerSize = encodedHeader.size();
+ channel->write(0, encodedHeader, [this, &write, headerSize, fd](int error) {
+ // On error the entry still stays in the contents filter until next synchronization.
+ m_approximateSize += headerSize;
+ finishWriteOperation(write);
+
+ LOG(NetworkCacheStorage, "(NetworkProcess) write complete error=%d", error);
+ });
+ });
+}
+
+void Storage::finishWriteOperation(const WriteOperation& write)
+{
+ ASSERT(m_activeWriteOperations.contains(&write));
+ m_activeWriteOperations.remove(&write);
+ dispatchPendingWriteOperations();
+
+ shrinkIfNeeded();
+}
+
</ins><span class="cx"> void Storage::retrieve(const Key& key, unsigned priority, RetrieveCompletionHandler&& completionHandler)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="lines">@@ -438,17 +519,15 @@
</span><span class="cx"> dispatchPendingReadOperations();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Storage::store(const Record& record, StoreCompletionHandler&& completionHandler)
</del><ins>+void Storage::store(const Record& record, MappedBodyHandler&& mappedBodyHandler)
</ins><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="cx"> ASSERT(!record.key.isNull());
</span><span class="cx">
</span><del>- if (!m_capacity) {
- completionHandler(false, { });
</del><ins>+ if (!m_capacity)
</ins><span class="cx"> return;
</span><del>- }
</del><span class="cx">
</span><del>- m_pendingWriteOperations.append(new WriteOperation { record, { }, WTF::move(completionHandler) });
</del><ins>+ m_pendingWriteOperations.append(new WriteOperation { record, WTF::move(mappedBodyHandler) });
</ins><span class="cx">
</span><span class="cx"> // Add key to the filter already here as we do lookups from the pending operations too.
</span><span class="cx"> addToContentsFilter(record.key);
</span><span class="lines">@@ -456,43 +535,29 @@
</span><span class="cx"> dispatchPendingWriteOperations();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Storage::update(const Record& updateRecord, const Record& existingRecord, StoreCompletionHandler&& completionHandler)
-{
- ASSERT(RunLoop::isMain());
- ASSERT(!existingRecord.key.isNull());
- ASSERT(existingRecord.key == updateRecord.key);
-
- if (!m_capacity) {
- completionHandler(false, { });
- return;
- }
-
- m_pendingWriteOperations.append(new WriteOperation { updateRecord, existingRecord, WTF::move(completionHandler) });
-
- dispatchPendingWriteOperations();
-}
-
</del><span class="cx"> void Storage::traverse(TraverseFlags flags, std::function<void (const Record*, const RecordInfo&)>&& traverseHandler)
</span><span class="cx"> {
</span><span class="cx"> StringCapture cachePathCapture(m_directoryPath);
</span><span class="cx"> ioQueue().dispatch([this, flags, cachePathCapture, traverseHandler] {
</span><span class="cx"> String cachePath = cachePathCapture.string();
</span><span class="cx"> traverseCacheFiles(cachePath, [this, flags, &traverseHandler](const String& fileName, const String& partitionPath) {
</span><del>- auto filePath = WebCore::pathByAppendingComponent(partitionPath, fileName);
</del><ins>+ auto recordPath = WebCore::pathByAppendingComponent(partitionPath, fileName);
</ins><span class="cx">
</span><span class="cx"> RecordInfo info;
</span><span class="cx"> if (flags & TraverseFlag::ComputeWorth)
</span><del>- info.worth = computeRecordWorth(fileTimes(filePath));
</del><ins>+ info.worth = computeRecordWorth(fileTimes(recordPath));
+ if (flags & TraverseFlag::ShareCount)
+ info.bodyShareCount = m_blobStorage.shareCount(bodyPathForRecordPath(recordPath));
</ins><span class="cx">
</span><del>- auto channel = IOChannel::open(filePath, IOChannel::Type::Read);
- const size_t headerReadSize = 16 << 10;
</del><ins>+ auto channel = IOChannel::open(recordPath, IOChannel::Type::Read);
</ins><span class="cx"> // FIXME: Traversal is slower than it should be due to lack of parallelism.
</span><del>- channel->readSync(0, headerReadSize, [this, &traverseHandler, &info](Data& fileData, int) {
</del><ins>+ channel->readSync(0, std::numeric_limits<size_t>::max(), [this, &traverseHandler, &info](Data& fileData, int) {
</ins><span class="cx"> RecordMetaData metaData;
</span><span class="cx"> Data headerData;
</span><span class="cx"> if (decodeRecordHeader(fileData, metaData, headerData)) {
</span><span class="cx"> Record record { metaData.key, std::chrono::system_clock::time_point(metaData.epochRelativeTimeStamp), headerData, { } };
</span><span class="cx"> info.bodySize = metaData.bodySize;
</span><ins>+ info.bodyHash = String::fromUTF8(SHA1::hexDigest(metaData.bodyHash));
</ins><span class="cx"> traverseHandler(&record, info);
</span><span class="cx"> }
</span><span class="cx"> });
</span><span class="lines">@@ -503,107 +568,6 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void Storage::dispatchPendingWriteOperations()
-{
- ASSERT(RunLoop::isMain());
-
- const int maximumActiveWriteOperationCount { 3 };
-
- while (!m_pendingWriteOperations.isEmpty()) {
- if (m_activeWriteOperations.size() >= maximumActiveWriteOperationCount) {
- LOG(NetworkCacheStorage, "(NetworkProcess) limiting parallel writes");
- return;
- }
- auto writeOperation = m_pendingWriteOperations.takeFirst();
- auto& write = *writeOperation;
- m_activeWriteOperations.add(WTF::move(writeOperation));
-
- if (write.existingRecord && mayContain(write.record.key)) {
- dispatchHeaderWriteOperation(write);
- continue;
- }
- dispatchFullWriteOperation(write);
- }
-}
-
-void Storage::dispatchFullWriteOperation(const WriteOperation& write)
-{
- ASSERT(RunLoop::isMain());
- ASSERT(m_activeWriteOperations.contains(&write));
-
- // This was added already when starting the store but filter might have been wiped.
- addToContentsFilter(write.record.key);
-
- StringCapture cachePathCapture(m_directoryPath);
- backgroundIOQueue().dispatch([this, &write, cachePathCapture] {
- auto encodedHeader = encodeRecordHeader(write.record);
- auto headerAndBodyData = concatenate(encodedHeader, write.record.body);
-
- auto channel = openFileForKey(write.record.key, IOChannel::Type::Create, cachePathCapture.string());
- int fd = channel->fileDescriptor();
- size_t bodyOffset = encodedHeader.size();
-
- channel->write(0, headerAndBodyData, [this, &write, bodyOffset, fd](int error) {
- size_t bodySize = write.record.body.size();
- size_t totalSize = bodyOffset + bodySize;
-
- // On error the entry still stays in the contents filter until next synchronization.
- m_approximateSize += totalSize;
-
- bool shouldMapBody = !error && bodySize >= pageSize();
- auto bodyMap = shouldMapBody ? mapFile(fd, bodyOffset, bodySize) : Data();
-
- write.completionHandler(!error, bodyMap);
-
- ASSERT(m_activeWriteOperations.contains(&write));
- m_activeWriteOperations.remove(&write);
- dispatchPendingWriteOperations();
-
- LOG(NetworkCacheStorage, "(NetworkProcess) write complete error=%d", error);
- });
- });
-
- shrinkIfNeeded();
-}
-
-void Storage::dispatchHeaderWriteOperation(const WriteOperation& write)
-{
- ASSERT(RunLoop::isMain());
- ASSERT(write.existingRecord);
- ASSERT(m_activeWriteOperations.contains(&write));
- ASSERT(mayContain(write.record.key));
-
- // Try to update the header of an existing entry.
- StringCapture cachePathCapture(m_directoryPath);
- backgroundIOQueue().dispatch([this, &write, cachePathCapture] {
- auto headerData = encodeRecordHeader(write.record);
- auto existingHeaderData = encodeRecordHeader(write.existingRecord.value());
-
- bool pageRoundedHeaderSizeChanged = headerData.size() != existingHeaderData.size();
- if (pageRoundedHeaderSizeChanged) {
- LOG(NetworkCacheStorage, "(NetworkProcess) page-rounded header size changed, storing full entry");
- RunLoop::main().dispatch([this, &write] {
- dispatchFullWriteOperation(write);
- });
- return;
- }
-
- auto channel = openFileForKey(write.record.key, IOChannel::Type::Write, cachePathCapture.string());
- channel->write(0, headerData, [this, &write](int error) {
- LOG(NetworkCacheStorage, "(NetworkProcess) update complete error=%d", error);
-
- if (error)
- remove(write.record.key);
-
- write.completionHandler(!error, { });
-
- ASSERT(m_activeWriteOperations.contains(&write));
- m_activeWriteOperations.remove(&write);
- dispatchPendingWriteOperations();
- });
- });
-}
-
</del><span class="cx"> void Storage::setCapacity(size_t capacity)
</span><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="lines">@@ -633,7 +597,7 @@
</span><span class="cx">
</span><span class="cx"> StringCapture directoryPathCapture(m_directoryPath);
</span><span class="cx">
</span><del>- ioQueue().dispatch([directoryPathCapture] {
</del><ins>+ ioQueue().dispatch([this, directoryPathCapture] {
</ins><span class="cx"> String directoryPath = directoryPathCapture.string();
</span><span class="cx"> traverseDirectory(directoryPath, DT_DIR, [&directoryPath](const String& subdirName) {
</span><span class="cx"> String subdirPath = WebCore::pathByAppendingComponent(directoryPath, subdirName);
</span><span class="lines">@@ -642,6 +606,9 @@
</span><span class="cx"> });
</span><span class="cx"> WebCore::deleteEmptyDirectory(subdirPath);
</span><span class="cx"> });
</span><ins>+
+ // This cleans unreferences blobs.
+ m_blobStorage.synchronize();
</ins><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -660,24 +627,30 @@
</span><span class="cx"> return duration<double>(accessAge) / age;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-
-static double deletionProbability(FileTimes times)
</del><ins>+static double deletionProbability(FileTimes times, unsigned bodyShareCount)
</ins><span class="cx"> {
</span><span class="cx"> static const double maximumProbability { 0.33 };
</span><ins>+ static const unsigned maximumEffectiveShareCount { 5 };
</ins><span class="cx">
</span><span class="cx"> auto worth = computeRecordWorth(times);
</span><span class="cx">
</span><span class="cx"> // Adjust a bit so the most valuable entries don't get deleted at all.
</span><span class="cx"> auto effectiveWorth = std::min(1.1 * worth, 1.);
</span><span class="cx">
</span><del>- return (1 - effectiveWorth) * maximumProbability;
</del><ins>+ auto probability = (1 - effectiveWorth) * maximumProbability;
+
+ // It is less useful to remove an entry that shares its body data.
+ if (bodyShareCount)
+ probability /= std::min(bodyShareCount, maximumEffectiveShareCount);
+
+ return probability;
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void Storage::shrinkIfNeeded()
</span><span class="cx"> {
</span><span class="cx"> ASSERT(RunLoop::isMain());
</span><span class="cx">
</span><del>- if (m_approximateSize > m_capacity)
</del><ins>+ if (approximateSize() > m_capacity)
</ins><span class="cx"> shrink();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -689,22 +662,27 @@
</span><span class="cx"> return;
</span><span class="cx"> m_shrinkInProgress = true;
</span><span class="cx">
</span><del>- LOG(NetworkCacheStorage, "(NetworkProcess) shrinking cache approximateSize=%zu capacity=%zu", static_cast<size_t>(m_approximateSize), m_capacity);
</del><ins>+ LOG(NetworkCacheStorage, "(NetworkProcess) shrinking cache approximateSize=%zu capacity=%zu", approximateSize(), m_capacity);
</ins><span class="cx">
</span><span class="cx"> StringCapture cachePathCapture(m_directoryPath);
</span><span class="cx"> backgroundIOQueue().dispatch([this, cachePathCapture] {
</span><span class="cx"> String cachePath = cachePathCapture.string();
</span><del>- traverseCacheFiles(cachePath, [](const String& fileName, const String& partitionPath) {
- auto filePath = WebCore::pathByAppendingComponent(partitionPath, fileName);
</del><ins>+ traverseCacheFiles(cachePath, [this](const String& fileName, const String& partitionPath) {
+ auto recordPath = WebCore::pathByAppendingComponent(partitionPath, fileName);
+ auto bodyPath = bodyPathForRecordPath(recordPath);
</ins><span class="cx">
</span><del>- auto times = fileTimes(filePath);
- auto probability = deletionProbability(times);
</del><ins>+ auto times = fileTimes(recordPath);
+ unsigned bodyShareCount = m_blobStorage.shareCount(bodyPath);
+ auto probability = deletionProbability(times, bodyShareCount);
+
</ins><span class="cx"> bool shouldDelete = randomNumber() < probability;
</span><span class="cx">
</span><del>- LOG(NetworkCacheStorage, "Deletion probability=%f shouldDelete=%d", probability, shouldDelete);
</del><ins>+ LOG(NetworkCacheStorage, "Deletion probability=%f bodyLinkCount=%d shouldDelete=%d", probability, bodyShareCount, shouldDelete);
</ins><span class="cx">
</span><del>- if (shouldDelete)
- WebCore::deleteFile(filePath);
</del><ins>+ if (shouldDelete) {
+ WebCore::deleteFile(recordPath);
+ m_blobStorage.remove(bodyPath);
+ }
</ins><span class="cx"> });
</span><span class="cx">
</span><span class="cx"> // Let system figure out if they are really empty.
</span><span class="lines">@@ -739,6 +717,7 @@
</span><span class="cx"> WebCore::deleteEmptyDirectory(partitionPath);
</span><span class="cx"> });
</span><span class="cx"> });
</span><ins>+ // FIXME: Delete V2 cache.
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceWebKit2NetworkProcesscacheNetworkCacheStorageh"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/NetworkProcess/cache/NetworkCacheStorage.h 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -28,6 +28,7 @@
</span><span class="cx">
</span><span class="cx"> #if ENABLE(NETWORK_CACHE)
</span><span class="cx">
</span><ins>+#include "NetworkCacheBlobStorage.h"
</ins><span class="cx"> #include "NetworkCacheData.h"
</span><span class="cx"> #include "NetworkCacheKey.h"
</span><span class="cx"> #include <wtf/BloomFilter.h>
</span><span class="lines">@@ -57,27 +58,30 @@
</span><span class="cx"> typedef std::function<bool (std::unique_ptr<Record>)> RetrieveCompletionHandler;
</span><span class="cx"> void retrieve(const Key&, unsigned priority, RetrieveCompletionHandler&&);
</span><span class="cx">
</span><del>- typedef std::function<void (bool success, const Data& mappedBody)> StoreCompletionHandler;
- void store(const Record&, StoreCompletionHandler&&);
- void update(const Record& updateRecord, const Record& existingRecord, StoreCompletionHandler&&);
</del><ins>+ typedef std::function<void (const Data& mappedBody)> MappedBodyHandler;
+ void store(const Record&, MappedBodyHandler&&);
</ins><span class="cx">
</span><span class="cx"> void remove(const Key&);
</span><span class="cx">
</span><span class="cx"> struct RecordInfo {
</span><span class="cx"> size_t bodySize { 0 };
</span><span class="cx"> double worth { -1 }; // 0-1 where 1 is the most valuable.
</span><ins>+ unsigned bodyShareCount { 0 };
+ String bodyHash;
</ins><span class="cx"> };
</span><span class="cx"> enum TraverseFlag {
</span><span class="cx"> ComputeWorth = 1 << 0,
</span><ins>+ ShareCount = 1 << 1,
</ins><span class="cx"> };
</span><span class="cx"> typedef unsigned TraverseFlags;
</span><span class="cx"> // Null record signals end.
</span><span class="cx"> void traverse(TraverseFlags, std::function<void (const Record*, const RecordInfo&)>&&);
</span><span class="cx">
</span><span class="cx"> void setCapacity(size_t);
</span><ins>+ size_t approximateSize() const;
</ins><span class="cx"> void clear();
</span><span class="cx">
</span><del>- static const unsigned version = 2;
</del><ins>+ static const unsigned version = 3;
</ins><span class="cx">
</span><span class="cx"> const String& baseDirectoryPath() const { return m_baseDirectoryPath; }
</span><span class="cx"> const String& directoryPath() const { return m_directoryPath; }
</span><span class="lines">@@ -96,17 +100,17 @@
</span><span class="cx"> };
</span><span class="cx"> void dispatchReadOperation(const ReadOperation&);
</span><span class="cx"> void dispatchPendingReadOperations();
</span><ins>+ void finishReadOperation(const ReadOperation&, std::unique_ptr<Record>);
</ins><span class="cx">
</span><span class="cx"> struct WriteOperation {
</span><span class="cx"> Record record;
</span><del>- Optional<Record> existingRecord;
- StoreCompletionHandler completionHandler;
</del><ins>+ MappedBodyHandler mappedBodyHandler;
</ins><span class="cx"> };
</span><del>- void dispatchFullWriteOperation(const WriteOperation&);
- void dispatchHeaderWriteOperation(const WriteOperation&);
</del><ins>+ void dispatchWriteOperation(const WriteOperation&);
</ins><span class="cx"> void dispatchPendingWriteOperations();
</span><ins>+ void finishWriteOperation(const WriteOperation&);
</ins><span class="cx">
</span><del>- void updateFileModificationTime(IOChannel&);
</del><ins>+ void updateFileModificationTime(const String& path);
</ins><span class="cx">
</span><span class="cx"> WorkQueue& ioQueue() { return m_ioQueue.get(); }
</span><span class="cx"> WorkQueue& backgroundIOQueue() { return m_backgroundIOQueue.get(); }
</span><span class="lines">@@ -141,6 +145,8 @@
</span><span class="cx"> Ref<WorkQueue> m_ioQueue;
</span><span class="cx"> Ref<WorkQueue> m_backgroundIOQueue;
</span><span class="cx"> Ref<WorkQueue> m_serialBackgroundIOQueue;
</span><ins>+
+ BlobStorage m_blobStorage;
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceWebKit2WebKit2xcodeprojprojectpbxproj"></a>
<div class="modfile"><h4>Modified: trunk/Source/WebKit2/WebKit2.xcodeproj/project.pbxproj (182802 => 182803)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/WebKit2/WebKit2.xcodeproj/project.pbxproj 2015-04-14 17:57:36 UTC (rev 182802)
+++ trunk/Source/WebKit2/WebKit2.xcodeproj/project.pbxproj 2015-04-14 18:40:27 UTC (rev 182803)
</span><span class="lines">@@ -1796,6 +1796,8 @@
</span><span class="cx"> E489D28E1A0A2DB80078C06A /* NetworkCacheDecoder.h in Headers */ = {isa = PBXBuildFile; fileRef = E489D2871A0A2DB80078C06A /* NetworkCacheDecoder.h */; };
</span><span class="cx"> E489D28F1A0A2DB80078C06A /* NetworkCacheEncoder.cpp in Sources */ = {isa = PBXBuildFile; fileRef = E489D2881A0A2DB80078C06A /* NetworkCacheEncoder.cpp */; };
</span><span class="cx"> E489D2901A0A2DB80078C06A /* NetworkCacheEncoder.h in Headers */ = {isa = PBXBuildFile; fileRef = E489D2891A0A2DB80078C06A /* NetworkCacheEncoder.h */; };
</span><ins>+ E49D40D71AD3FB170066B7B9 /* NetworkCacheBlobStorage.h in Headers */ = {isa = PBXBuildFile; fileRef = E49D40D61AD3FB170066B7B9 /* NetworkCacheBlobStorage.h */; };
+ E49D40D91AD3FB210066B7B9 /* NetworkCacheBlobStorage.cpp in Sources */ = {isa = PBXBuildFile; fileRef = E49D40D81AD3FB210066B7B9 /* NetworkCacheBlobStorage.cpp */; };
</ins><span class="cx"> ED82A7F2128C6FAF004477B3 /* WKBundlePageOverlay.h in Headers */ = {isa = PBXBuildFile; fileRef = 1A22F0FF1289FCD90085E74F /* WKBundlePageOverlay.h */; settings = {ATTRIBUTES = (Private, ); }; };
</span><span class="cx"> EDCA71B7128DDA8C00201B26 /* WKBundlePageOverlay.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1A22F1001289FCD90085E74F /* WKBundlePageOverlay.cpp */; };
</span><span class="cx"> F036978815F4BF0500C3A80E /* WebColorPicker.cpp in Sources */ = {isa = PBXBuildFile; fileRef = F036978715F4BF0500C3A80E /* WebColorPicker.cpp */; };
</span><span class="lines">@@ -4059,6 +4061,8 @@
</span><span class="cx"> E489D2871A0A2DB80078C06A /* NetworkCacheDecoder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkCacheDecoder.h; sourceTree = "<group>"; };
</span><span class="cx"> E489D2881A0A2DB80078C06A /* NetworkCacheEncoder.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NetworkCacheEncoder.cpp; sourceTree = "<group>"; };
</span><span class="cx"> E489D2891A0A2DB80078C06A /* NetworkCacheEncoder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkCacheEncoder.h; sourceTree = "<group>"; };
</span><ins>+ E49D40D61AD3FB170066B7B9 /* NetworkCacheBlobStorage.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = NetworkCacheBlobStorage.h; sourceTree = "<group>"; };
+ E49D40D81AD3FB210066B7B9 /* NetworkCacheBlobStorage.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = NetworkCacheBlobStorage.cpp; sourceTree = "<group>"; };
</ins><span class="cx"> F036978715F4BF0500C3A80E /* WebColorPicker.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebColorPicker.cpp; sourceTree = "<group>"; };
</span><span class="cx"> F6113E24126CE1820057D0A7 /* APIUserContentURLPattern.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APIUserContentURLPattern.h; sourceTree = "<group>"; };
</span><span class="cx"> F6113E26126CE19B0057D0A7 /* WKUserContentURLPattern.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WKUserContentURLPattern.cpp; sourceTree = "<group>"; };
</span><span class="lines">@@ -7498,6 +7502,8 @@
</span><span class="cx"> E489D2821A0A2BE80078C06A /* cache */ = {
</span><span class="cx"> isa = PBXGroup;
</span><span class="cx"> children = (
</span><ins>+ E49D40D81AD3FB210066B7B9 /* NetworkCacheBlobStorage.cpp */,
+ E49D40D61AD3FB170066B7B9 /* NetworkCacheBlobStorage.h */,
</ins><span class="cx"> E4436EBE1A0CFDB200EAD204 /* NetworkCache.cpp */,
</span><span class="cx"> E4436EBF1A0CFDB200EAD204 /* NetworkCache.h */,
</span><span class="cx"> E489D2831A0A2DB80078C06A /* NetworkCacheCoder.h */,
</span><span class="lines">@@ -8149,6 +8155,7 @@
</span><span class="cx"> 1AAF08A2192681D100B6390C /* WebUserContentControllerProxy.h in Headers */,
</span><span class="cx"> 7C361D79192803BD0036A59D /* WebUserContentControllerProxyMessages.h in Headers */,
</span><span class="cx"> 3F889D15188778C900FEADAF /* WebVideoFullscreenManagerProxy.h in Headers */,
</span><ins>+ E49D40D71AD3FB170066B7B9 /* NetworkCacheBlobStorage.h in Headers */,
</ins><span class="cx"> 29CD55AA128E294F00133C85 /* WKAccessibilityWebPageObjectBase.h in Headers */,
</span><span class="cx"> 29232DF418B29D6800D0596F /* WKAccessibilityWebPageObjectMac.h in Headers */,
</span><span class="cx"> 2D0730A319F9C7DA00E9D9C4 /* WKActionMenuController.h in Headers */,
</span><span class="lines">@@ -9512,6 +9519,7 @@
</span><span class="cx"> CD67D30E15C08F9A00843ADF /* InjectedBundlePageDiagnosticLoggingClient.cpp in Sources */,
</span><span class="cx"> E1EE53E711F8CFFB00CCBEE4 /* InjectedBundlePageEditorClient.cpp in Sources */,
</span><span class="cx"> BC14E109120B905E00826C0C /* InjectedBundlePageFormClient.cpp in Sources */,
</span><ins>+ E49D40D91AD3FB210066B7B9 /* NetworkCacheBlobStorage.cpp in Sources */,
</ins><span class="cx"> CD5C66A0134B9D38004FE2A8 /* InjectedBundlePageFullScreenClient.cpp in Sources */,
</span><span class="cx"> BCA8C6A811E3BA5F00812FB7 /* InjectedBundlePageLoaderClient.cpp in Sources */,
</span><span class="cx"> BC8147AA12F64CDA007B2C32 /* InjectedBundlePagePolicyClient.cpp in Sources */,
</span></span></pre>
</div>
</div>
</body>
</html>