<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[214065] trunk/Websites/perf.webkit.org</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/214065">214065</a></dd>
<dt>Author</dt> <dd>rniwa@webkit.org</dd>
<dt>Date</dt> <dd>2017-03-16 13:53:35 -0700 (Thu, 16 Mar 2017)</dd>
</dl>
<h3>Log Message</h3>
<pre>Add the file uploading capability to the perf dashboard.
https://bugs.webkit.org/show_bug.cgi?id=169737
Reviewed by Chris Dumez.
Added /privileged-api/upload-file to upload a file, and /api/uploaded-file/ to download the file
and retrieve its meta data based on its SHA256. We treat two files with the identical SHA256 as
identical since anyone who can upload a file using this mechanism can execute arbitrary code in
our bots anyway. This is important for avoiding uploading a large darwinup roots multiple times
to the server, saving both user's time/bandwidth and server's disk space.
* config.json: Added uploadDirectory, uploadFileLimitInMB, and uploadUserQuotaInMB as options.
* init-database.sql: Added uploaded_files table.
* public/api/uploaded-file.php: Added.
(main): /api/uploaded-file/N would download uploaded_file with id=N. /api/uploaded-file/?sha256=X
would return the meta data for uploaded_file with sha256=X.
(stream_file_content): Streams the file content in 64KB chunks. We support Range & If-Range HTTP
request headers so that browsers can pause and resume downloading of a large root file.
(parse_range_header): Parses Range HTTP request header.
* public/include/json-header.php:
(remote_user_name): Use the default argument of NULL.
* public/include/manifest-generator.php:
(ManifestGenerator::generate): Include the maximum upload size in the manifest file to let the
frontend code preemptively check the file size before attempting to submit a file.
* public/include/uploaded-file-helpers.php: Added.
(format_uploaded_file):
(uploaded_file_path_for_row):
* public/privileged-api/upload-file-form.html: Added. For debugging purposes.
(fetchCSRFfToken):
(upload):
* public/privileged-api/upload-file.php: Added.
(main):
(query_total_file_size):
(create_uploaded_file_from_form_data):
* public/shared/common-remote.js:
(CommonRemoteAPI.prototype.postFormData): Added.
(CommonRemoteAPI.prototype.postFormDataWithStatus): Added.
(CommonRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
(CommonRemoteAPI.prototype._asJSON): Throw an exception instead of calling a non-existent reject.
* public/v3/models/uploaded-file.js: Added.
(UploadedFile): Added.
(UploadedFile.uploadFile): Added.
(UploadedFile.fetchUnloadedFileWithIdenticalHash): Added. Finds the file with the same SHA256 in
the server to avoid uploading a large custom root multiple times.
(UploadedFile._computeSHA256Hash): Added.
* public/v3/privileged-api.js:
(PrivilegedAPI.prototype.sendRequest): Added the options dictionary as a third argument. For now,
only support useFormData boolean.
* public/v3/remote.js:
(BrowserRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
* server-tests/api-manifest.js: Updated per the inclusion of fileUploadSizeLimit in the manifest.
* server-tests/api-uploaded-file.js: Added.
* server-tests/privileged-api-upload-file-tests.js: Added.
* server-tests/resources/temporary-file.js: Added.
(TemporaryFile): Added. A helper class for creating a temporary file to upload.
(TemporaryFile.makeTemporaryFileOfSizeInMB):
(TemporaryFile.makeTemporaryFile):
(TemporaryFile.inject):
* server-tests/resources/test-server.conf: Set upload_max_filesize and post_max_size for testing.
* server-tests/resources/test-server.js:
(TestServer.prototype.testConfig): Use uploadFileLimitInMB and uploadUserQuotaInMB of 2MB and 5MB.
(TestServer.prototype._ensureDataDirectory): Create a directory to store uploaded files inside
the data directory. In a production server, we can place it outside ServerRoot / DocumentRoot.
(TestServer.prototype.cleanDataDirectory): Delete the aforementioned directory as needed.
* tools/js/database.js:
(tableToPrefixMap): Added uploaded_files.
* tools/js/remote.js:
(NodeRemoteAPI.prototype.sendHttpRequest): Added a dictionary to specify request headers and
a callback to process the response as arguments. Fixed the bug that any 2xx code other than 200
was resulting in a rejected promise. Also include the response headers in the result for tests.
Finally, when content is a function, call that instead of writing the content since FormData
requires a custom logic.
(NodeRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
* tools/js/v3-models.js: Include uploaded-file.js.
* tools/run-tests.py:
(main): Add form-data as a new dependency.</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgChangeLog">trunk/Websites/perf.webkit.org/ChangeLog</a></li>
<li><a href="#trunkWebsitesperfwebkitorgconfigjson">trunk/Websites/perf.webkit.org/config.json</a></li>
<li><a href="#trunkWebsitesperfwebkitorginitdatabasesql">trunk/Websites/perf.webkit.org/init-database.sql</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicincludejsonheaderphp">trunk/Websites/perf.webkit.org/public/include/json-header.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicincludemanifestgeneratorphp">trunk/Websites/perf.webkit.org/public/include/manifest-generator.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicsharedcommonremotejs">trunk/Websites/perf.webkit.org/public/shared/common-remote.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv3privilegedapijs">trunk/Websites/perf.webkit.org/public/v3/privileged-api.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv3remotejs">trunk/Websites/perf.webkit.org/public/v3/remote.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsapimanifestjs">trunk/Websites/perf.webkit.org/server-tests/api-manifest.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsresourcestestserverconf">trunk/Websites/perf.webkit.org/server-tests/resources/test-server.conf</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsresourcestestserverjs">trunk/Websites/perf.webkit.org/server-tests/resources/test-server.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsjsdatabasejs">trunk/Websites/perf.webkit.org/tools/js/database.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsjsremotejs">trunk/Websites/perf.webkit.org/tools/js/remote.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsjsv3modelsjs">trunk/Websites/perf.webkit.org/tools/js/v3-models.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgtoolsruntestspy">trunk/Websites/perf.webkit.org/tools/run-tests.py</a></li>
</ul>
<h3>Added Paths</h3>
<ul>
<li><a href="#trunkWebsitesperfwebkitorgpublicapiuploadedfilephp">trunk/Websites/perf.webkit.org/public/api/uploaded-file.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicincludeuploadedfilehelpersphp">trunk/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicprivilegedapiuploadfileformhtml">trunk/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicprivilegedapiuploadfilephp">trunk/Websites/perf.webkit.org/public/privileged-api/upload-file.php</a></li>
<li><a href="#trunkWebsitesperfwebkitorgpublicv3modelsuploadedfilejs">trunk/Websites/perf.webkit.org/public/v3/models/uploaded-file.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsapiuploadedfilejs">trunk/Websites/perf.webkit.org/server-tests/api-uploaded-file.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsprivilegedapiuploadfiletestsjs">trunk/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js</a></li>
<li><a href="#trunkWebsitesperfwebkitorgservertestsresourcestemporaryfilejs">trunk/Websites/perf.webkit.org/server-tests/resources/temporary-file.js</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkWebsitesperfwebkitorgChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/ChangeLog (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/ChangeLog 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -1,3 +1,99 @@
</span><ins>+2017-03-16 Ryosuke Niwa <rniwa@webkit.org>
+
+ Add the file uploading capability to the perf dashboard.
+ https://bugs.webkit.org/show_bug.cgi?id=169737
+
+ Reviewed by Chris Dumez.
+
+ Added /privileged-api/upload-file to upload a file, and /api/uploaded-file/ to download the file
+ and retrieve its meta data based on its SHA256. We treat two files with the identical SHA256 as
+ identical since anyone who can upload a file using this mechanism can execute arbitrary code in
+ our bots anyway. This is important for avoiding uploading a large darwinup roots multiple times
+ to the server, saving both user's time/bandwidth and server's disk space.
+
+ * config.json: Added uploadDirectory, uploadFileLimitInMB, and uploadUserQuotaInMB as options.
+ * init-database.sql: Added uploaded_files table.
+
+ * public/api/uploaded-file.php: Added.
+ (main): /api/uploaded-file/N would download uploaded_file with id=N. /api/uploaded-file/?sha256=X
+ would return the meta data for uploaded_file with sha256=X.
+ (stream_file_content): Streams the file content in 64KB chunks. We support Range & If-Range HTTP
+ request headers so that browsers can pause and resume downloading of a large root file.
+ (parse_range_header): Parses Range HTTP request header.
+
+ * public/include/json-header.php:
+ (remote_user_name): Use the default argument of NULL.
+
+ * public/include/manifest-generator.php:
+ (ManifestGenerator::generate): Include the maximum upload size in the manifest file to let the
+ frontend code preemptively check the file size before attempting to submit a file.
+
+ * public/include/uploaded-file-helpers.php: Added.
+ (format_uploaded_file):
+ (uploaded_file_path_for_row):
+
+ * public/privileged-api/upload-file-form.html: Added. For debugging purposes.
+ (fetchCSRFfToken):
+ (upload):
+
+ * public/privileged-api/upload-file.php: Added.
+ (main):
+ (query_total_file_size):
+ (create_uploaded_file_from_form_data):
+
+ * public/shared/common-remote.js:
+ (CommonRemoteAPI.prototype.postFormData): Added.
+ (CommonRemoteAPI.prototype.postFormDataWithStatus): Added.
+ (CommonRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+ (CommonRemoteAPI.prototype._asJSON): Throw an exception instead of calling a non-existent reject.
+
+ * public/v3/models/uploaded-file.js: Added.
+ (UploadedFile): Added.
+ (UploadedFile.uploadFile): Added.
+ (UploadedFile.fetchUnloadedFileWithIdenticalHash): Added. Finds the file with the same SHA256 in
+ the server to avoid uploading a large custom root multiple times.
+ (UploadedFile._computeSHA256Hash): Added.
+
+ * public/v3/privileged-api.js:
+ (PrivilegedAPI.prototype.sendRequest): Added the options dictionary as a third argument. For now,
+ only support useFormData boolean.
+
+ * public/v3/remote.js:
+ (BrowserRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+
+ * server-tests/api-manifest.js: Updated per the inclusion of fileUploadSizeLimit in the manifest.
+ * server-tests/api-uploaded-file.js: Added.
+ * server-tests/privileged-api-upload-file-tests.js: Added.
+
+ * server-tests/resources/temporary-file.js: Added.
+ (TemporaryFile): Added. A helper class for creating a temporary file to upload.
+ (TemporaryFile.makeTemporaryFileOfSizeInMB):
+ (TemporaryFile.makeTemporaryFile):
+ (TemporaryFile.inject):
+
+ * server-tests/resources/test-server.conf: Set upload_max_filesize and post_max_size for testing.
+ * server-tests/resources/test-server.js:
+ (TestServer.prototype.testConfig): Use uploadFileLimitInMB and uploadUserQuotaInMB of 2MB and 5MB.
+ (TestServer.prototype._ensureDataDirectory): Create a directory to store uploaded files inside
+ the data directory. In a production server, we can place it outside ServerRoot / DocumentRoot.
+ (TestServer.prototype.cleanDataDirectory): Delete the aforementioned directory as needed.
+
+ * tools/js/database.js:
+ (tableToPrefixMap): Added uploaded_files.
+
+ * tools/js/remote.js:
+ (NodeRemoteAPI.prototype.sendHttpRequest): Added a dictionary to specify request headers and
+ a callback to process the response as arguments. Fixed the bug that any 2xx code other than 200
+ was resulting in a rejected promise. Also include the response headers in the result for tests.
+ Finally, when content is a function, call that instead of writing the content since FormData
+ requires a custom logic.
+ (NodeRemoteAPI.prototype.sendHttpRequestWithFormData): Added.
+
+ * tools/js/v3-models.js: Include uploaded-file.js.
+
+ * tools/run-tests.py:
+ (main): Add form-data as a new dependency.
+
</ins><span class="cx"> 2017-03-15 Dewei Zhu <dewei_zhu@apple.com>
</span><span class="cx">
</span><span class="cx"> Fix unit test and bug fix for 'pull-os-versions.js' script.
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgconfigjson"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/config.json (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/config.json 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/config.json 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -3,6 +3,9 @@
</span><span class="cx"> "debug": true,
</span><span class="cx"> "jsonCacheMaxAge": 600,
</span><span class="cx"> "dataDirectory": "public/data/",
</span><ins>+ "uploadDirectory": "uploaded",
+ "uploadFileLimitInMB": 800,
+ "uploadUserQuotaInMB": 8192,
</ins><span class="cx"> "database": {
</span><span class="cx"> "host": "localhost",
</span><span class="cx"> "port": "5432",
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorginitdatabasesql"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/init-database.sql (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/init-database.sql 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/init-database.sql 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -24,6 +24,7 @@
</span><span class="cx"> DROP TABLE IF EXISTS build_triggerables CASCADE;
</span><span class="cx"> DROP TABLE IF EXISTS triggerable_configurations CASCADE;
</span><span class="cx"> DROP TABLE IF EXISTS triggerable_repositories CASCADE;
</span><ins>+DROP TABLE IF EXISTS uploaded_files CASCADE;
</ins><span class="cx"> DROP TABLE IF EXISTS bugs CASCADE;
</span><span class="cx"> DROP TABLE IF EXISTS analysis_test_groups CASCADE;
</span><span class="cx"> DROP TABLE IF EXISTS commit_sets CASCADE;
</span><span class="lines">@@ -241,6 +242,19 @@
</span><span class="cx"> trigconfig_triggerable integer REFERENCES build_triggerables NOT NULL,
</span><span class="cx"> CONSTRAINT triggerable_must_be_unique_for_test_and_platform UNIQUE(trigconfig_test, trigconfig_platform));
</span><span class="cx">
</span><ins>+CREATE TABLE uploaded_files (
+ file_id serial PRIMARY KEY,
+ file_created_at timestamp NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'UTC'),
+ file_deleted_at timestamp,
+ file_mime varchar(64),
+ file_filename varchar(1024) NOT NULL,
+ file_extension varchar(16),
+ file_author varchar(256),
+ file_size bigint NOT NULL,
+ file_sha256 char(64) NOT NULL);
+CREATE INDEX file_author_index ON uploaded_files(file_author);
+CREATE UNIQUE INDEX file_sha256_index ON uploaded_files(file_sha256) WHERE file_deleted_at is NULL;
+
</ins><span class="cx"> CREATE TABLE analysis_test_groups (
</span><span class="cx"> testgroup_id serial PRIMARY KEY,
</span><span class="cx"> testgroup_task integer REFERENCES analysis_tasks NOT NULL,
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicapiuploadedfilephp"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/public/api/uploaded-file.php (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/api/uploaded-file.php (rev 0)
+++ trunk/Websites/perf.webkit.org/public/api/uploaded-file.php 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,122 @@
</span><ins>+<?php
+
+require('../include/json-header.php');
+require('../include/uploaded-file-helpers.php');
+
+function main($path)
+{
+ if (count($path) > 1)
+ exit_with_error('InvalidRequest');
+
+ $db = connect();
+ if (count($path) && $path[0]) {
+ $file_id = intval($path[0]);
+ $file_row = $db->select_first_row('uploaded_files', 'file', array('id' => $file_id));
+ if (!$file_row)
+ exit_with_404();
+ $file_path = uploaded_file_path_for_row($file_row);
+ return stream_file_content($file_path, $file_row['file_sha256'], $file_row['file_filename']);
+ }
+
+ $sha256 = array_get($_GET, 'sha256');
+ if ($sha256) {
+ $file_row = $db->select_first_row('uploaded_files', 'file', array('sha256' => $sha256, 'deleted_at' => null));
+ if (!$file_row)
+ exit_with_error('NotFound');
+ exit_with_success(array('uploadedFile' => format_uploaded_file($file_row)));
+ }
+ exit_with_error('InvalidArguments');
+}
+
+define('STREAM_CHUNK_SIZE', 64 * 1024);
+
+function stream_file_content($uploaded_file, $etag, $disposition_name)
+{
+ if (!file_exists($uploaded_file))
+ exit_with_404();
+
+ $file_size = filesize($uploaded_file);
+ $last_modified = gmdate('D, d M Y H:i:s', filemtime($uploaded_file)) . ' GMT';
+ $file_handle = fopen($uploaded_file, "rb");
+ if (!$file_handle)
+ exit_with_404();
+
+ $headers = getallheaders();
+
+ // We don't support multi-part range request. e.g. bytes=1-3,4-5
+ $range = parse_range_header(array_get($headers, 'Range'), $file_size);
+ if ($range && (!array_key_exists('If-Range', $headers) || $headers['If-Range'] == $last_modified || $headers['If-Range'] == $etag)) {
+ assert($range['start'] >= 0);
+ if ($range['start'] > $range['end'] || $range['end'] >= $file_size) {
+ header('HTTP/1.1 416 Range Not Satisfiable');
+ header("Content-Range: bytes */$file_size");
+ exit(416);
+ }
+ $start = $range['start'];
+ $end = $range['end'];
+ $content_length = $end - $start + 1;
+ header('HTTP/1.1 206 Partial Content');
+ header("Content-Range: bytes $start-$end/$file_size");
+ fseek($file_handle, $start);
+ } else {
+ $content_length = $file_size;
+ header("Accept-Ranges: bytes");
+ header("ETag: $etag");
+ }
+
+ $output_buffer = fopen('php://output', 'wb');
+ $encoded_filename = urlencode($disposition_name);
+
+ header('Content-Type: application/octet-stream');
+ header('Content-Length: ' . $content_length);
+ header("Content-Disposition: attachment; filename*=utf-8''$encoded_filename");
+ header("Last-Modified: $last_modified");
+
+ set_time_limit(0);
+ while (!feof($file_handle) && $content_length) {
+ $is_end = $content_length < STREAM_CHUNK_SIZE;
+ $chunk_size = $is_end ? $content_length : STREAM_CHUNK_SIZE;
+ $chunk = fread($file_handle, $chunk_size);
+ $content_length -= $chunk_size;
+ fwrite($output_buffer, $chunk, $chunk_size);
+ flush();
+ }
+
+ exit(0);
+}
+
+function parse_range_header($range_header, $file_size)
+{
+ // We don't support multi-part range request. e.g. bytes=1-3,4-5
+ $matches = array();
+ $end_byte = $file_size;
+ if (!$range_header || !preg_match('/^\s*bytes\s*=\s*((\d+)-(\d*)|-(\d+))\s*$/', $range_header, $matches))
+ return NULL;
+
+ $end_byte = $file_size - 1;
+ if ($matches[2]) {
+ $start_byte = intval($matches[2]);
+ if ($matches[3])
+ $end_byte = intval($matches[3]);
+ else
+ $end_byte = $file_size - 1;
+ } else {
+ $suffix_length = intval($matches[4]);
+ if ($file_size < $suffix_length)
+ $start_byte = 0;
+ else
+ $start_byte = $file_size - $suffix_length;
+ }
+
+ return array('start' => $start_byte, 'end' => $end_byte);
+}
+
+function exit_with_404()
+{
+ header($_SERVER['SERVER_PROTOCOL'] . ' 404 Not Found');
+ exit_with_error('NotFound');
+}
+
+main(array_key_exists('PATH_INFO', $_SERVER) ? explode('/', trim($_SERVER['PATH_INFO'], '/')) : array());
+
+?>
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicincludejsonheaderphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/include/json-header.php (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/include/json-header.php 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/public/include/json-header.php 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -1,6 +1,7 @@
</span><span class="cx"> <?php
</span><span class="cx">
</span><span class="cx"> require_once('db.php');
</span><ins>+
</ins><span class="cx"> require_once('test-path-resolver.php');
</span><span class="cx">
</span><span class="cx"> header('Content-type: application/json');
</span><span class="lines">@@ -116,8 +117,8 @@
</span><span class="cx"> return $data;
</span><span class="cx"> }
</span><span class="cx">
</span><del>-function remote_user_name($data) {
- return should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
</del><ins>+function remote_user_name($data = NULL) {
+ return $data && should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> function should_authenticate_as_slave($data) {
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicincludemanifestgeneratorphp"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/include/manifest-generator.php (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/include/manifest-generator.php 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/public/include/manifest-generator.php 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -43,6 +43,7 @@
</span><span class="cx"> 'triggerables'=> (object)$this->triggerables(),
</span><span class="cx"> 'dashboards' => (object)config('dashboards'),
</span><span class="cx"> 'summaryPages' => config('summaryPages'),
</span><ins>+ 'fileUploadSizeLimit' => config('uploadFileLimitInMB', 0) * 1024 * 1024,
</ins><span class="cx"> );
</span><span class="cx">
</span><span class="cx"> $this->manifest['elapsedTime'] = (microtime(true) - $start_time) * 1000;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicincludeuploadedfilehelpersphp"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php (rev 0)
+++ trunk/Websites/perf.webkit.org/public/include/uploaded-file-helpers.php 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,20 @@
</span><ins>+<?php
+
+function format_uploaded_file($file_row)
+{
+ return array(
+ 'id' => $file_row['file_id'],
+ 'size' => $file_row['file_size'],
+ 'createdAt' => $file_row['file_created_at'],
+ 'mime' => $file_row['file_mime'],
+ 'filename' => $file_row['file_filename'],
+ 'author' => $file_row['file_author'],
+ 'sha256' => $file_row['file_sha256']);
+}
+
+function uploaded_file_path_for_row($file_row)
+{
+ return config_path('uploadDirectory', $file_row['file_id'] . $file_row['file_extension']);
+}
+
+?>
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicprivilegedapiuploadfileformhtml"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html (rev 0)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/upload-file-form.html 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,59 @@
</span><ins>+<!DOCTYPE html>
+<html>
+<body>
+<script>
+
+function fetchCSRFfToken(fileInput, progressElement) {
+ return new Promise((resolve, reject) => {
+ const xhr = new XMLHttpRequest();
+ xhr.open('POST', 'generate-csrf-token', true);
+ xhr.onload = () => {
+ let content;
+ try {
+ content = JSON.parse(xhr.responseText);
+ } catch (error) {
+ return reject(error + ':' + xhr.responseText);
+ }
+ if (content['status'] != 'OK')
+ reject(content['status']);
+ else
+ resolve(content['token']);
+ }
+ xhr.onerror = reject;
+ xhr.send('{}');
+ });
+}
+
+function upload(fileInput, progressElement) {
+ fetchCSRFfToken().then((token) => {
+ const xhr = new XMLHttpRequest();
+ const formData = new FormData();
+ formData.append('token', token);
+ formData.append('newFile', fileInput.files[0]);
+
+ xhr.open('POST', 'upload-file', true);
+ xhr.onload = function () {
+ alert(xhr.response);
+ }
+ xhr.onerror = function () {
+ alert('error: ' + xhr.response);
+ }
+ xhr.upload.onprogress = function (event) {
+ if (event.lengthComputable) {
+ progressElement.max = event.total;
+ progressElement.value = event.loaded;
+ }
+ }
+ xhr.send(formData);
+ }, (error) => {
+ alert(`Failed to fetch the CSRF token: ${error}`);
+ });
+}
+
+</script>
+<p>Upload a new custom root. <b>This is for debuging purpose only. The uploaded file will be deleted.</b></p>
+<input id=file type="file">
+<button onclick="upload(file, p)">Upload</button>
+<progress id=p></progress>
+</body>
+</html>
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicprivilegedapiuploadfilephp"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/public/privileged-api/upload-file.php (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/privileged-api/upload-file.php (rev 0)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/upload-file.php 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,99 @@
</span><ins>+<?php
+
+ini_set('upload_max_filesize', '1025M');
+ini_set('post_max_size', '1025M');
+require_once('../include/json-header.php');
+require_once('../include/uploaded-file-helpers.php');
+
+define('MEGABYTES', 1024 * 1024);
+
+function main()
+{
+ if (array_get($_SERVER, 'CONTENT_LENGTH') && empty($_POST) && empty($_FILES))
+ exit_with_error('FileSizeLimitExceeded2');
+
+ if (!verify_token(array_get($_POST, 'token')))
+ exit_with_error('InvalidToken');
+
+ if (!is_dir(config_path('uploadDirectory', '')))
+ exit_with_error('NotSupported');
+
+ $input_file = array_get($_FILES, 'newFile');
+ if (!$input_file)
+ exit_with_error('NoFileSpecified');
+
+ if ($input_file['error'] != UPLOAD_ERR_OK)
+ exit_with_error('FailedToUploadFile', array('name' => $input_file['name'], 'error' => $input_file['error']));
+
+ if (config('uploadFileLimitInMB') * MEGABYTES < $input_file['size'])
+ exit_with_error('FileSizeLimitExceeded');
+
+ $uploaded_file = create_uploaded_file_from_form_data($input_file);
+
+ $current_user = remote_user_name();
+ $db = connect();
+
+ // FIXME: Cleanup old files.
+
+ if (config('uploadUserQuotaInMB') * MEGABYTES - query_total_file_size($db, $current_user) < $input_file['size'])
+ exit_with_error('FileSizeQuotaExceeded');
+
+ $db->begin_transaction();
+ $file_row = $db->select_or_insert_row('uploaded_files', 'file',
+ array('sha256' => $uploaded_file['sha256'], 'deleted_at' => null), $uploaded_file, '*');
+ if (!$file_row)
+ exit_with_error('FailedToInsertFileData');
+
+ // A concurrent session may have inserted another file.
+ if (config('uploadUserQuotaInMB') * MEGABYTES < query_total_file_size($db, $current_user)) {
+ $db->rollback_transaction();
+ exit_with_error('FileSizeQuotaExceeded');
+ }
+
+ $new_path = uploaded_file_path_for_row($file_row);
+ if (!move_uploaded_file($input_file['tmp_name'], $new_path)) {
+ $db->rollback_transaction();
+ exit_with_error('FailedToMoveUploadedFile');
+ }
+ $db->commit_transaction();
+
+ exit_with_success(array('uploadedFile' => format_uploaded_file($file_row)));
+}
+
+function query_total_file_size($db, $user)
+{
+ if ($user)
+ $count_result = $db->query_and_fetch_all('SELECT sum(file_size) as "sum" FROM uploaded_files WHERE file_deleted_at IS NULL AND file_author = $1', array($user));
+ else
+ $count_result = $db->query_and_fetch_all('SELECT sum(file_size) as "sum" FROM uploaded_files WHERE file_deleted_at IS NULL AND file_author IS NULL');
+ if (!$count_result)
+ return FALSE;
+ return intval($count_result[0]["sum"]);
+}
+
+function create_uploaded_file_from_form_data($input_file)
+{
+ $file_sha256 = hash_file('sha256', $input_file['tmp_name']);
+ if (!$file_sha256)
+ exit_with_error('FailedToComputeSHA256');
+
+ $matches = array();
+ $file_extension = null;
+ if (preg_match('/(\.[a-zA-Z0-9]{1,5}){1,2}$/', $input_file['name'], $matches)) {
+ $file_extension = $matches[0];
+ assert(strlen($file_extension) <= 16);
+ }
+
+ return array(
+ 'author' => remote_user_name(),
+ 'filename' => $input_file['name'],
+ 'extension' => $file_extension,
+ 'mime' => $input_file['type'], // Sanitize MIME types.
+ 'size' => $input_file['size'],
+ 'sha256' => $file_sha256
+ );
+}
+
+main();
+
+?>
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicsharedcommonremotejs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/shared/common-remote.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/shared/common-remote.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/public/shared/common-remote.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -11,6 +11,19 @@
</span><span class="cx"> return this._checkStatus(this.postJSON(path, data));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ postFormData(path, data)
+ {
+ const formData = new FormData();
+ for (let key in data)
+ formData.append(key, data[key]);
+ return this._asJSON(this.sendHttpRequestWithFormData(path, formData));
+ }
+
+ postFormDataWithStatus(path, data)
+ {
+ return this._checkStatus(this.postFormData(path, data));
+ }
+
</ins><span class="cx"> getJSON(path)
</span><span class="cx"> {
</span><span class="cx"> return this._asJSON(this.sendHttpRequest(path, 'GET', null, null));
</span><span class="lines">@@ -26,6 +39,11 @@
</span><span class="cx"> throw 'NotImplemented';
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ sendHttpRequestWithFormData(path, formData)
+ {
+ throw 'NotImplemented';
+ }
+
</ins><span class="cx"> _asJSON(promise)
</span><span class="cx"> {
</span><span class="cx"> return promise.then((result) => {
</span><span class="lines">@@ -33,7 +51,7 @@
</span><span class="cx"> return JSON.parse(result.responseText);
</span><span class="cx"> } catch (error) {
</span><span class="cx"> console.error(result.responseText);
</span><del>- reject(result.statusCode + ', ' + error);
</del><ins>+ throw `{result.statusCode}: ${error}`;
</ins><span class="cx"> }
</span><span class="cx"> });
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv3modelsuploadedfilejs"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/public/v3/models/uploaded-file.js (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v3/models/uploaded-file.js (rev 0)
+++ trunk/Websites/perf.webkit.org/public/v3/models/uploaded-file.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,54 @@
</span><ins>+
+class UploadedFile extends DataModelObject {
+
+ constructor(id, object)
+ {
+ super(id, object);
+ this._createdAt = new Date(object.createdAt);
+ this._filename = object.filename;
+ this._author = object.author;
+ this._size = object.size;
+ this._sha256 = object.sha256;
+ this.ensureNamedStaticMap('sha256')[object.sha256] = this;
+ }
+
+ static uploadFile(file)
+ {
+ return PrivilegedAPI.sendRequest('upload-file', {'newFile': file}, {useFormData: true}).then((rawData) => {
+ return UploadedFile.ensureSingleton(rawData['uploadedFile'].id, rawData['uploadedFile']);
+ });
+ }
+
+ static fetchUnloadedFileWithIdenticalHash(file)
+ {
+ return new Promise((resolve, reject) => {
+ const reader = new FileReader();
+ reader.onload = () => resolve(reader.result);
+ reader.onerror = () => reject();
+ reader.readAsArrayBuffer(file);
+ }).then((content) => {
+ return this._computeSHA256Hash(content);
+ }).then((sha256) => {
+ const map = this.namedStaticMap('sha256');
+ if (map && sha256 in map)
+ return map[sha256];
+ return RemoteAPI.getJSONWithStatus(`../api/uploaded-file?sha256=${sha256}`).then((rawData) => {
+ if (!rawData['uploadedFile'])
+ return null;
+ return UploadedFile.ensureSingleton(rawData['uploadedFile'].id, rawData['uploadedFile']);
+ });
+ });
+ }
+
+ static _computeSHA256Hash(content)
+ {
+ return crypto.subtle.digest('SHA-256', content).then((digest) => {
+ return Array.from(new Uint8Array(digest)).map((byte) => {
+ if (byte < 0x10)
+ return '0' + byte.toString(16);
+ return byte.toString(16);
+ }).join('');
+ });
+ }
+
+}
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv3privilegedapijs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v3/privileged-api.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v3/privileged-api.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/public/v3/privileged-api.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -2,7 +2,7 @@
</span><span class="cx">
</span><span class="cx"> class PrivilegedAPI {
</span><span class="cx">
</span><del>- static sendRequest(path, data)
</del><ins>+ static sendRequest(path, data, options = {useFormData: false})
</ins><span class="cx"> {
</span><span class="cx"> const clonedData = {};
</span><span class="cx"> for (let key in data)
</span><span class="lines">@@ -9,7 +9,9 @@
</span><span class="cx"> clonedData[key] = data[key];
</span><span class="cx">
</span><span class="cx"> const fullPath = '/privileged-api/' + path;
</span><del>- const post = () => RemoteAPI.postJSONWithStatus(fullPath, clonedData);
</del><ins>+ const post = options.useFormData
+ ? () => RemoteAPI.postFormDataWithStatus(fullPath, clonedData)
+ : () => RemoteAPI.postJSONWithStatus(fullPath, clonedData);
</ins><span class="cx">
</span><span class="cx"> return this.requestCSRFToken().then((token) => {
</span><span class="cx"> clonedData['token'] = token;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgpublicv3remotejs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/public/v3/remote.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/public/v3/remote.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/public/v3/remote.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -36,6 +36,11 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ sendHttpRequestWithFormData(path, formData)
+ {
+ return this.sendHttpRequest(path, 'POST', null, formData); // Content-type is set by the browser.
+ }
+
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> const RemoteAPI = new BrowserRemoteAPI;
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsapimanifestjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/server-tests/api-manifest.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/api-manifest.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/server-tests/api-manifest.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -14,7 +14,7 @@
</span><span class="cx"> it("should generate an empty manifest when database is empty", () => {
</span><span class="cx"> return TestServer.remoteAPI().getJSON('/api/manifest').then((manifest) => {
</span><span class="cx"> assert.deepEqual(Object.keys(manifest).sort(), ['all', 'bugTrackers', 'builders', 'dashboard', 'dashboards',
</span><del>- 'elapsedTime', 'metrics', 'repositories', 'siteTitle', 'status', 'summaryPages', 'tests', 'triggerables']);
</del><ins>+ 'elapsedTime', 'fileUploadSizeLimit', 'metrics', 'repositories', 'siteTitle', 'status', 'summaryPages', 'tests', 'triggerables']);
</ins><span class="cx">
</span><span class="cx"> assert.equal(typeof(manifest.elapsedTime), 'number');
</span><span class="cx"> delete manifest.elapsedTime;
</span><span class="lines">@@ -26,6 +26,7 @@
</span><span class="cx"> builders: {},
</span><span class="cx"> dashboard: {},
</span><span class="cx"> dashboards: {},
</span><ins>+ fileUploadSizeLimit: 2097152, // 2MB during testing.
</ins><span class="cx"> metrics: {},
</span><span class="cx"> repositories: {},
</span><span class="cx"> tests: {},
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsapiuploadedfilejs"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/server-tests/api-uploaded-file.js (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/api-uploaded-file.js (rev 0)
+++ trunk/Websites/perf.webkit.org/server-tests/api-uploaded-file.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,309 @@
</span><ins>+'use strict';
+
+require('../tools/js/v3-models.js');
+
+const assert = require('assert');
+global.FormData = require('form-data');
+
+const TestServer = require('./resources/test-server.js');
+const TemporaryFile = require('./resources/temporary-file.js').TemporaryFile;
+
+describe('/api/uploaded-file', function () {
+ this.timeout(5000);
+ TestServer.inject();
+
+ TemporaryFile.inject();
+
+ it('should return "InvalidArguments" when neither path nor sha256 query is set', () => {
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file').then((content) => {
+ assert.equal(content['status'], 'InvalidArguments');
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file/');
+ }).then((content) => {
+ assert.equal(content['status'], 'InvalidArguments');
+ });
+ });
+
+ it('should return 404 when there is no file with the specified ID', () => {
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file/1').then((content) => {
+ assert(false, 'should never be reached');
+ }, (error) => {
+ assert.equal(error, 404);
+ });
+ });
+
+ it('should return 404 when the specified ID is not a valid integer', () => {
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file/foo').then((content) => {
+ assert(false, 'should never be reached');
+ }, (error) => {
+ assert.equal(error, 404);
+ });
+ });
+
+ it('should return the file content matching the specified file ID', () => {
+ let uploadedFile;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile = response['uploadedFile'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${uploadedFile['id']}`, 'GET', null, null);
+ }).then((response) => {
+ assert.equal(response.responseText, 'some content');
+ });
+ });
+
+ it('should return "NotFound" when the specified SHA256 is invalid', () => {
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file/?sha256=abc').then((content) => {
+ assert.equal(content['status'], 'NotFound');
+ });
+ });
+
+ it('should return "NotFound" when there is no file matching the specified SHA256 ', () => {
+ return TestServer.remoteAPI().getJSON('/api/uploaded-file/?sha256=5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5').then((content) => {
+ assert.equal(content['status'], 'NotFound');
+ });
+ });
+
+ it('should return the meta data of the file with the specified SHA256', () => {
+ let uploadedFile;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile = response['uploadedFile'];
+ return TestServer.remoteAPI().getJSON(`/api/uploaded-file/?sha256=${uploadedFile['sha256']}`);
+ }).then((response) => {
+ assert.deepEqual(uploadedFile, response['uploadedFile']);
+ });
+ });
+
+ it('should return "NotFound" when the file matching the specified SHA256 had already been deleted', () => {
+ let uploadedFile;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile = response['uploadedFile'];
+ const db = TestServer.database();
+ return db.connect().then(() => db.query(`UPDATE uploaded_files SET file_deleted_at = now() at time zone 'utc'`));
+ }).then(() => {
+ return TestServer.remoteAPI().getJSON(`/api/uploaded-file/?sha256=${uploadedFile['sha256']}`);
+ }).then((content) => {
+ assert.equal(content['status'], 'NotFound');
+ });
+ });
+
+
+ it('should respond with ETag, Acccept-Ranges, Content-Disposition, Content-Length, and Last-Modified headers', () => {
+ let uploadedFile;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile = response['uploadedFile'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${uploadedFile['id']}`, 'GET', null, null);
+ }).then((response) => {
+ const headers = response.headers;
+
+ assert(Object.keys(headers).includes('etag'));
+ assert.equal(headers['etag'], uploadedFile['sha256']);
+
+ assert(Object.keys(headers).includes('accept-ranges'));
+ assert.equal(headers['accept-ranges'], 'bytes');
+
+ assert(Object.keys(headers).includes('content-disposition'));
+ assert.equal(headers['content-disposition'], `attachment; filename*=utf-8''some.dat`);
+
+ assert(Object.keys(headers).includes('content-length'));
+ assert.equal(headers['content-length'], uploadedFile['size']);
+
+ assert(Object.keys(headers).includes('last-modified'));
+ });
+ });
+
+ it('should respond with the same Last-Modified each time', () => {
+ let id;
+ let lastModified;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+ }).then((response) => {
+ lastModified = response.headers['last-modified'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+ }).then((response) => {
+ assert.equal(response.headers['last-modified'], lastModified);
+ });
+ });
+
+ it('should respond with Content-Range when requested after X bytes', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=5-'});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 5-11/12');
+ assert.equal(response.responseText, 'content');
+ });
+ });
+
+ it('should respond with Content-Range when requested between X-Y bytes', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=4-9'});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 4-9/12');
+ assert.equal(response.responseText, ' conte');
+ });
+ });
+
+ it('should respond with Content-Range when requested for the last X bytes', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=-4'});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 8-11/12');
+ assert.equal(response.responseText, 'tent');
+ });
+ });
+
+ it('should respond with Content-Range for the whole content when the suffix length is larger than the content', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=-100'});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 0-11/12');
+ assert.equal(response.responseText, 'some content');
+ });
+ });
+
+ it('should return 416 when the starting byte is after the file size', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=12-'})
+ .then(() => assert(false, 'should never be reached'), (error) => assert.equal(error, 416));
+ });
+ });
+
+ it('should return 416 when the starting byte after the ending byte', () => {
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ const id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null, {'Range': 'bytes=2-1'})
+ .then(() => assert(false, 'should never be reached'), (error) => assert.equal(error, 416));
+ });
+ });
+
+ it('should respond with Content-Range when If-Range matches the last modified date', () => {
+ let id;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+ }).then((response) => {
+ assert.equal(response.statusCode, 200);
+ assert.equal(response.responseText, 'some content');
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+ {'Range': 'bytes = 9-10', 'If-Range': response.headers['last-modified']});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 9-10/12');
+ assert.equal(response.responseText, 'en');
+ });
+ });
+
+ it('should respond with Content-Range when If-Range matches ETag', () => {
+ let id;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+ }).then((response) => {
+ assert.equal(response.statusCode, 200);
+ assert.equal(response.responseText, 'some content');
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+ {'Range': 'bytes = 9-10', 'If-Range': response.headers['etag']});
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], 'bytes 9-10/12');
+ assert.equal(response.responseText, 'en');
+ });
+ });
+
+ it('should return the full content when If-Range does not match the last modified date or ETag', () => {
+ let id;
+ return TemporaryFile.makeTemporaryFile('some.dat', 'some content').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null);
+ }).then((response) => {
+ assert.equal(response.statusCode, 200);
+ assert.equal(response.responseText, 'some content');
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+ {'Range': 'bytes = 9-10', 'If-Range': 'foo'});
+ }).then((response) => {
+ assert.equal(response.statusCode, 200);
+ assert.equal(response.responseText, 'some content');
+ });
+ });
+
+ it('should respond with Content-Range across 64KB streaming chunks', () => {
+ let id;
+ const fileSize = 256 * 1024;
+ const tokens = "0123456789abcdefghijklmnopqrstuvwxyz";
+ let buffer = Buffer.allocUnsafe(fileSize);
+ for (let i = 0; i < fileSize; i++)
+ buffer[i] = Math.floor(Math.random() * 256);
+ let startByte = 63 * 1024;
+ let endByte = 128 * 1024 - 1;
+
+ let responseBufferList = [];
+ const responseHandler = (response) => {
+ response.on('data', (chunk) => responseBufferList.push(chunk));
+ };
+
+ function verifyBuffer()
+ {
+ const responseBuffer = Buffer.concat(responseBufferList);
+ for (let i = 0; i < endByte - startByte + 1; i++) {
+ const actual = responseBuffer[i];
+ const expected = buffer[startByte + i];
+ assert.equal(actual, expected, `The byte at index ${i} should be identical. Expected ${expected} but got ${actual}`);
+ }
+ }
+
+ return TemporaryFile.makeTemporaryFile('some.dat', buffer).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ id = response['uploadedFile']['id'];
+ return TestServer.remoteAPI().sendHttpRequest(`/api/uploaded-file/${id}`, 'GET', null, null,
+ {'Range': `bytes = ${startByte}-${endByte}`}, responseHandler);
+ }).then((response) => {
+ const headers = response.headers;
+ assert.equal(response.statusCode, 206);
+ assert.equal(headers['content-range'], `bytes ${startByte}-${endByte}/${fileSize}`);
+ verifyBuffer();
+ });
+ });
+
+});
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsprivilegedapiuploadfiletestsjs"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js (rev 0)
+++ trunk/Websites/perf.webkit.org/server-tests/privileged-api-upload-file-tests.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,192 @@
</span><ins>+'use strict';
+
+require('../tools/js/v3-models.js');
+
+const assert = require('assert');
+global.FormData = require('form-data');
+
+const TestServer = require('./resources/test-server.js');
+const TemporaryFile = require('./resources/temporary-file.js').TemporaryFile;
+
+describe('/privileged-api/upload-file', function () {
+ this.timeout(5000);
+ TestServer.inject();
+
+ TemporaryFile.inject();
+
+ it('should return "NotFileSpecified" when newFile not is specified', () => {
+ return PrivilegedAPI.sendRequest('upload-file', {}, {useFormData: true}).then(() => {
+ assert(false, 'should never be reached');
+ }, (error) => {
+ assert.equal(error, 'NoFileSpecified');
+ });
+ });
+
+ it('should return "FileSizeLimitExceeded" when the file is too big', () => {
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', TestServer.testConfig().uploadFileLimitInMB + 1).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true}).then(() => {
+ assert(false, 'should never be reached');
+ }, (error) => {
+ assert.equal(error, 'FileSizeLimitExceeded');
+ });
+ });
+ });
+
+ it('should upload a file when the filesize is smaller than the limit', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ let uploadedFile;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile = response['uploadedFile'];
+ return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+ }).then((rows) => {
+ assert.equal(rows.length, 1);
+ assert.equal(rows[0].id, uploadedFile.id);
+ assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+ assert.equal(rows[0].size, uploadedFile.size);
+ assert.equal(rows[0].filename, 'some.dat');
+ assert.equal(rows[0].filename, uploadedFile.filename);
+ assert.equal(rows[0].extension, '.dat');
+ assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+ assert.equal(rows[0].sha256, uploadedFile.sha256);
+ });
+ });
+
+ it('should not create a duplicate files when the identical files are uploaded', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ let uploadedFile1;
+ let uploadedFile2;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile1 = response['uploadedFile'];
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+ }).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile2 = response['uploadedFile'];
+ return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+ }).then((rows) => {
+ assert.deepEqual(uploadedFile1, uploadedFile2);
+ assert.equal(rows.length, 1);
+ assert.equal(rows[0].id, uploadedFile1.id);
+ assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+ assert.equal(rows[0].size, uploadedFile1.size);
+ assert.equal(rows[0].filename, 'some.dat');
+ assert.equal(rows[0].filename, uploadedFile1.filename);
+ assert.equal(rows[0].extension, '.dat');
+ assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+ assert.equal(rows[0].sha256, uploadedFile1.sha256);
+ });
+ });
+
+ it('should not create a duplicate files when the identical files are uploaded', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ let uploadedFile1;
+ let uploadedFile2;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile1 = response['uploadedFile'];
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+ }).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile2 = response['uploadedFile'];
+ return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+ }).then((rows) => {
+ assert.deepEqual(uploadedFile1, uploadedFile2);
+ assert.equal(rows.length, 1);
+ assert.equal(rows[0].id, uploadedFile1.id);
+ assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+ assert.equal(rows[0].size, uploadedFile1.size);
+ assert.equal(rows[0].filename, 'some.dat');
+ assert.equal(rows[0].filename, uploadedFile1.filename);
+ assert.equal(rows[0].extension, '.dat');
+ assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+ assert.equal(rows[0].sha256, uploadedFile1.sha256);
+ });
+ });
+
+ it('should re-upload the file when the previously uploaded file had been deleted', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ let uploadedFile1;
+ let uploadedFile2;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile1 = response['uploadedFile'];
+ return db.connect().then(() => db.query(`UPDATE uploaded_files SET file_deleted_at = now() at time zone 'utc'`));
+ }).then(() => {
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB);
+ }).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then((response) => {
+ uploadedFile2 = response['uploadedFile'];
+ return db.connect().then(() => db.selectAll('uploaded_files', 'id'));
+ }).then((rows) => {
+ assert.notEqual(uploadedFile1.id, uploadedFile2.id);
+ assert.equal(rows.length, 2);
+ assert.equal(rows[0].id, uploadedFile1.id);
+ assert.equal(rows[1].id, uploadedFile2.id);
+
+ assert.equal(rows[0].filename, 'some.dat');
+ assert.equal(rows[0].filename, uploadedFile1.filename);
+ assert.equal(rows[1].filename, 'other.dat');
+ assert.equal(rows[1].filename, uploadedFile2.filename);
+
+ assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+ assert.equal(rows[0].size, uploadedFile1.size);
+ assert.equal(rows[0].size, uploadedFile2.size);
+ assert.equal(rows[0].size, rows[1].size);
+ assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+ assert.equal(rows[0].sha256, uploadedFile1.sha256);
+ assert.equal(rows[0].sha256, uploadedFile2.sha256);
+ assert.equal(rows[0].sha256, rows[1].sha256);
+ assert.equal(rows[0].extension, '.dat');
+ assert.equal(rows[1].extension, '.dat');
+ });
+ });
+
+ it('should pick up at most two file extensions', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.other.tar.gz', limitInMB).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then(() => {
+ return db.connect().then(() => db.selectAll('uploaded_files', 'id'))
+ }).then((rows) => {
+ assert.equal(rows.length, 1);
+ assert.equal(rows[0].size, limitInMB * 1024 * 1024);
+ assert.equal(rows[0].mime, 'application/octet-stream');
+ assert.equal(rows[0].filename, 'some.other.tar.gz');
+ assert.equal(rows[0].extension, '.tar.gz');
+ assert.equal(rows[0].sha256, '5256ec18f11624025905d057d6befb03d77b243511ac5f77ed5e0221ce6d84b5');
+ });
+ });
+
+ it('should return "FileSizeQuotaExceeded" when the total file size exceeds the quota allowed per user', () => {
+ const db = TestServer.database();
+ const limitInMB = TestServer.testConfig().uploadFileLimitInMB;
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('some.dat', limitInMB, 'a').then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then(() => {
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB, 'b');
+ }).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true});
+ }).then(() => {
+ return TemporaryFile.makeTemporaryFileOfSizeInMB('other.dat', limitInMB, 'c');
+ }).then((stream) => {
+ return PrivilegedAPI.sendRequest('upload-file', {newFile: stream}, {useFormData: true}).then(() => {
+ assert(false, 'should never be reached');
+ }, (error) => {
+ assert.equal(error, 'FileSizeQuotaExceeded');
+ });
+ });
+ });
+});
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsresourcestemporaryfilejs"></a>
<div class="addfile"><h4>Added: trunk/Websites/perf.webkit.org/server-tests/resources/temporary-file.js (0 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/resources/temporary-file.js (rev 0)
+++ trunk/Websites/perf.webkit.org/server-tests/resources/temporary-file.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -0,0 +1,49 @@
</span><ins>+
+const assert = require('assert');
+const childProcess = require('child_process');
+const fs = require('fs');
+const path = require('path');
+
+const Config = require('../../tools/js/config.js');
+
+class TemporaryFile {
+ static makeTemporaryFileOfSizeInMB(name, sizeInMB, characterToFill = 'a')
+ {
+ let megabyteString = characterToFill;
+ for (let i = 0; i < 20; i++)
+ megabyteString = megabyteString + megabyteString;
+ assert.equal(megabyteString.length, 1024 * 1024);
+
+ let content = '';
+ for (let i = 0; i < sizeInMB; i++)
+ content += megabyteString;
+
+ return this.makeTemporaryFile(name, content);
+ }
+
+ static makeTemporaryFile(name, content)
+ {
+ const newPath = path.resolve(TemporaryFile._tempDir, name);
+ return new Promise((resolve) => {
+ return fs.writeFile(newPath, content, () => {
+ resolve(fs.createReadStream(newPath));
+ });
+ });
+ }
+
+ static inject()
+ {
+ beforeEach(() => {
+ this._tempDir = fs.mkdtempSync(path.resolve(Config.path('dataDirectory'), 'temp/'));
+ });
+
+ afterEach(() => {
+ childProcess.execFileSync('rm', ['-rf', this._tempDir]);
+ this._tempDir = null;
+ });
+ }
+}
+TemporaryFile._tempDir = null;
+
+if (typeof module != 'undefined')
+ module.exports.TemporaryFile = TemporaryFile;
</ins></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsresourcestestserverconf"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/server-tests/resources/test-server.conf (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/resources/test-server.conf 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/server-tests/resources/test-server.conf 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -52,6 +52,9 @@
</span><span class="cx"> <IfModule php5_module>
</span><span class="cx"> AddType application/x-httpd-php .php
</span><span class="cx"> AddType application/x-httpd-php-source .phps
</span><ins>+
+ php_value upload_max_filesize 5M
+ php_value post_max_size 5M
</ins><span class="cx"> </IfModule>
</span><span class="cx">
</span><span class="cx"> Include /private/etc/apache2/extra/httpd-mpm.conf
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgservertestsresourcestestserverjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/server-tests/resources/test-server.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/server-tests/resources/test-server.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/server-tests/resources/test-server.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -76,6 +76,9 @@
</span><span class="cx"> 'password': Config.value('database.password'),
</span><span class="cx"> 'name': Config.value('testDatabaseName'),
</span><span class="cx"> },
</span><ins>+ 'uploadFileLimitInMB': 2,
+ 'uploadUserQuotaInMB': 5,
+ 'uploadDirectory': Config.value('dataDirectory') + '/uploaded',
</ins><span class="cx"> 'universalSlavePassword': null,
</span><span class="cx"> 'maintenanceMode': false,
</span><span class="cx"> 'clusterStart': [2000, 1, 1, 0, 0],
</span><span class="lines">@@ -96,6 +99,7 @@
</span><span class="cx"> } else if (fs.existsSync(backupPath)) // Assume this is a backup from the last failed run
</span><span class="cx"> this._backupDataPath = backupPath;
</span><span class="cx"> fs.mkdirSync(this._dataDirectory, 0o755);
</span><ins>+ fs.mkdirSync(path.resolve(this._dataDirectory, 'uploaded'), 0o755);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> _restoreDataDirectory()
</span><span class="lines">@@ -108,8 +112,13 @@
</span><span class="cx"> cleanDataDirectory()
</span><span class="cx"> {
</span><span class="cx"> let fileList = fs.readdirSync(this._dataDirectory);
</span><ins>+ for (let filename of fileList) {
+ if (filename != 'uploaded')
+ fs.unlinkSync(path.resolve(this._dataDirectory, filename));
+ }
+ fileList = fs.readdirSync(path.resolve(this._dataDirectory, 'uploaded'));
</ins><span class="cx"> for (let filename of fileList)
</span><del>- fs.unlinkSync(path.resolve(this._dataDirectory, filename));
</del><ins>+ fs.unlinkSync(path.resolve(this._dataDirectory, 'uploaded', filename));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> _ensureTestDatabase()
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsjsdatabasejs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/js/database.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/js/database.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/tools/js/database.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -152,6 +152,7 @@
</span><span class="cx"> 'commit_sets': 'commitset',
</span><span class="cx"> 'commit_set_relationships': 'commitset',
</span><span class="cx"> 'run_iterations': 'iteration',
</span><ins>+ 'uploaded_files': 'file',
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> if (typeof module != 'undefined')
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsjsremotejs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/js/remote.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/js/remote.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/tools/js/remote.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -64,7 +64,7 @@
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="cx">
</span><del>- sendHttpRequest(path, method, contentType, content)
</del><ins>+ sendHttpRequest(path, method, contentType, content, headers = {}, responseHandler = null)
</ins><span class="cx"> {
</span><span class="cx"> let server = this._server;
</span><span class="cx"> return new Promise((resolve, reject) => {
</span><span class="lines">@@ -78,10 +78,14 @@
</span><span class="cx">
</span><span class="cx"> let request = (server.scheme == 'http' ? http : https).request(options, (response) => {
</span><span class="cx"> let responseText = '';
</span><del>- response.setEncoding('utf8');
- response.on('data', (chunk) => { responseText += chunk; });
</del><ins>+ if (responseHandler)
+ responseHandler(response);
+ else {
+ response.setEncoding('utf8');
+ response.on('data', (chunk) => { responseText += chunk; });
+ }
</ins><span class="cx"> response.on('end', () => {
</span><del>- if (response.statusCode != 200)
</del><ins>+ if (response.statusCode < 200 || response.statusCode >= 300)
</ins><span class="cx"> return reject(response.statusCode);
</span><span class="cx">
</span><span class="cx"> if ('set-cookie' in response.headers) {
</span><span class="lines">@@ -90,7 +94,7 @@
</span><span class="cx"> this._cookies.set(nameValue[0], nameValue[1]);
</span><span class="cx"> }
</span><span class="cx"> }
</span><del>- resolve({statusCode: response.statusCode, responseText: responseText});
</del><ins>+ resolve({statusCode: response.statusCode, responseText: responseText, headers: response.headers});
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx">
</span><span class="lines">@@ -102,12 +106,26 @@
</span><span class="cx"> if (this._cookies.size)
</span><span class="cx"> request.setHeader('Cookie', Array.from(this._cookies.keys()).map((key) => `${key}=${this._cookies.get(key)}`).join('; '));
</span><span class="cx">
</span><del>- if (content)
- request.write(content);
</del><ins>+ for (let headerName in headers)
+ request.setHeader(headerName, headers[headerName]);
</ins><span class="cx">
</span><del>- request.end();
</del><ins>+ if (content instanceof Function)
+ content(request);
+ else {
+ if (content)
+ request.write(content);
+ request.end();
+ }
</ins><span class="cx"> });
</span><span class="cx"> }
</span><ins>+
+ sendHttpRequestWithFormData(path, formData)
+ {
+ return this.sendHttpRequest(path, 'POST', `multipart/form-data; boundary=${formData.getBoundary()}`, (request) => {
+ formData.pipe(request);
+ });
+ }
+
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> if (typeof module != 'undefined')
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsjsv3modelsjs"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/js/v3-models.js (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/js/v3-models.js 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/tools/js/v3-models.js 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -28,6 +28,7 @@
</span><span class="cx"> importFromV3('models/test-group.js', 'TestGroup');
</span><span class="cx"> importFromV3('models/time-series.js', 'TimeSeries');
</span><span class="cx"> importFromV3('models/triggerable.js', 'Triggerable');
</span><ins>+importFromV3('models/uploaded-file.js', 'UploadedFile');
</ins><span class="cx">
</span><span class="cx"> importFromV3('privileged-api.js', 'PrivilegedAPI');
</span><span class="cx"> importFromV3('instrumentation.js', 'Instrumentation');
</span></span></pre></div>
<a id="trunkWebsitesperfwebkitorgtoolsruntestspy"></a>
<div class="modfile"><h4>Modified: trunk/Websites/perf.webkit.org/tools/run-tests.py (214064 => 214065)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Websites/perf.webkit.org/tools/run-tests.py 2017-03-16 20:50:36 UTC (rev 214064)
+++ trunk/Websites/perf.webkit.org/tools/run-tests.py 2017-03-16 20:53:35 UTC (rev 214065)
</span><span class="lines">@@ -11,7 +11,7 @@
</span><span class="cx"> node_modules_dir = os.path.join(root_dir, 'node_modules')
</span><span class="cx">
</span><span class="cx"> os.chdir(root_dir)
</span><del>- packages = ['mocha', 'pg']
</del><ins>+ packages = ['mocha', 'pg', 'form-data']
</ins><span class="cx"> for package_name in packages:
</span><span class="cx"> target_dir = os.path.join(node_modules_dir, package_name)
</span><span class="cx"> if not os.path.isdir(target_dir):
</span></span></pre>
</div>
</div>
</body>
</html>