<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[277680] trunk/Source/JavaScriptCore</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/277680">277680</a></dd>
<dt>Author</dt> <dd>sbarati@apple.com</dd>
<dt>Date</dt> <dd>2021-05-18 14:30:48 -0700 (Tue, 18 May 2021)</dd>
</dl>
<h3>Log Message</h3>
<pre>Add Data Call ICs that don't repatch and use them in the baseline JIT
https://bugs.webkit.org/show_bug.cgi?id=225321
<rdar://problem/77773796>
Reviewed by Michael Saboff.
This patch adds Data ICs for calls. Data ICs for calls work by loading a code
pointer from CallLinkInfo, and indirect calling that pointer. This means that
to repatch such an IC, all we need to do is replace a code pointer inside
CallLinkInfo. No need to repatch the JIT code.
The current implementation only does this for monomorphic calls. We still
repatch the JIT code for polymorphic calls. In a followup, we will also
opt polymorphic call ICs into data-based calls:
https://bugs.webkit.org/show_bug.cgi?id=225793
This patch only uses Data Call ICs for the Baseline JIT. Even with that, it
reduces the number of calls to cacheFlush by ~45% on JetStream2.
Performance is neutral on AS Macs, but it paves the way towards doing
unlinked JITting in JSC.
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::addLateLinkTask):
* assembler/LinkBuffer.cpp:
(JSC::LinkBuffer::linkCode):
(JSC::LinkBuffer::performFinalization):
* assembler/LinkBuffer.h:
* bytecode/AccessCase.cpp:
(JSC::AccessCase::generateImpl):
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::CallLinkInfo):
(JSC::CallLinkInfo::unlink):
(JSC::CallLinkInfo::fastPathStart):
(JSC::CallLinkInfo::slowPathStart):
(JSC::CallLinkInfo::doneLocation):
(JSC::CallLinkInfo::setMonomorphicCallee):
(JSC::CallLinkInfo::clearCallee):
(JSC::CallLinkInfo::emitFirstInstructionForDataIC):
(JSC::CallLinkInfo::emitFastPathImpl):
(JSC::CallLinkInfo::emitFastPath):
(JSC::CallLinkInfo::emitTailCallFastPath):
(JSC::CallLinkInfo::emitSlowPath):
(JSC::CallLinkInfo::emitDirectFastPath):
(JSC::CallLinkInfo::emitDirectTailCallFastPath):
(JSC::CallLinkInfo::initializeDirectCall):
(JSC::CallLinkInfo::setDirectCallTarget):
(JSC::CallLinkInfo::setSlowPathCallDestination):
(JSC::CallLinkInfo::revertCallToStub):
(JSC::CallLinkInfo::setStub):
(JSC::CallLinkInfo::callReturnLocation): Deleted.
(JSC::CallLinkInfo::patchableJump): Deleted.
(JSC::CallLinkInfo::hotPathBegin): Deleted.
(JSC::CallLinkInfo::setCallee): Deleted.
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::calleeGPR const):
(JSC::CallLinkInfo::isDataIC const):
(JSC::CallLinkInfo::setUsesDataICs):
(JSC::CallLinkInfo::setCodeLocations):
(JSC::CallLinkInfo::offsetOfCallee):
(JSC::CallLinkInfo::offsetOfMonomorphicCallDestination):
(JSC::CallLinkInfo::offsetOfSlowPathCallDestination):
(JSC::CallLinkInfo::setCallLocations): Deleted.
(JSC::CallLinkInfo::hotPathOther): Deleted.
(JSC::CallLinkInfo::setStub): Deleted.
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::link):
* dfg/DFGJITCompiler.h:
(JSC::DFG::JITCompiler::addJSCall):
(JSC::DFG::JITCompiler::addJSDirectCall):
(JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
(JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
(JSC::DFG::JITCompiler::addJSDirectTailCall): Deleted.
(JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord): Deleted.
* dfg/DFGOSRExitCompilerCommon.cpp:
(JSC::DFG::callerReturnPC):
* dfg/DFGOperations.cpp:
(JSC::DFG::JSC_DEFINE_JIT_OPERATION):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
* jit/CCallHelpers.cpp:
(JSC::CCallHelpers::emitJITCodeOver):
* jit/CCallHelpers.h:
* jit/JIT.cpp:
(JSC::JIT::link):
* jit/JIT.h:
* jit/JITCall.cpp:
(JSC::JIT::compileTailCall):
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):
* jit/JITCall32_64.cpp:
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):
* jit/JITOperations.cpp:
(JSC::JSC_DEFINE_JIT_OPERATION):
* jit/PolymorphicCallStubRoutine.cpp:
(JSC::PolymorphicCallNode::unlink):
* jit/Repatch.cpp:
(JSC::linkSlowPathTo):
(JSC::linkSlowFor):
(JSC::linkMonomorphicCall):
(JSC::linkDirectCall):
(JSC::revertCall):
(JSC::unlinkCall):
(JSC::linkPolymorphicCall):
(JSC::linkFor): Deleted.
(JSC::linkDirectFor): Deleted.
(JSC::unlinkFor): Deleted.
* jit/Repatch.h:
* wasm/js/WasmToJS.cpp:
(JSC::Wasm::wasmToJS):</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerAbstractMacroAssemblerh">trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerLinkBuffercpp">trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerLinkBufferh">trunk/Source/JavaScriptCore/assembler/LinkBuffer.h</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeAccessCasecpp">trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCallLinkInfocpp">trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCallLinkInfoh">trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGJITCompilercpp">trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGJITCompilerh">trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOSRExitCompilerCommoncpp">trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOperationscpp">trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJIT32_64cpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp">trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitCCallHelperscpp">trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitCCallHelpersh">trunk/Source/JavaScriptCore/jit/CCallHelpers.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITcpp">trunk/Source/JavaScriptCore/jit/JIT.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITh">trunk/Source/JavaScriptCore/jit/JIT.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITCallcpp">trunk/Source/JavaScriptCore/jit/JITCall.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITCall32_64cpp">trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITOperationscpp">trunk/Source/JavaScriptCore/jit/JITOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitPolymorphicCallStubRoutinecpp">trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitRepatchcpp">trunk/Source/JavaScriptCore/jit/Repatch.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitRepatchh">trunk/Source/JavaScriptCore/jit/Repatch.h</a></li>
<li><a href="#trunkSourceJavaScriptCorewasmjsWasmToJScpp">trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/ChangeLog 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1,3 +1,125 @@
</span><ins>+2021-05-18 Saam Barati <sbarati@apple.com>
+
+ Add Data Call ICs that don't repatch and use them in the baseline JIT
+ https://bugs.webkit.org/show_bug.cgi?id=225321
+ <rdar://problem/77773796>
+
+ Reviewed by Michael Saboff.
+
+ This patch adds Data ICs for calls. Data ICs for calls work by loading a code
+ pointer from CallLinkInfo, and indirect calling that pointer. This means that
+ to repatch such an IC, all we need to do is replace a code pointer inside
+ CallLinkInfo. No need to repatch the JIT code.
+
+ The current implementation only does this for monomorphic calls. We still
+ repatch the JIT code for polymorphic calls. In a followup, we will also
+ opt polymorphic call ICs into data-based calls:
+ https://bugs.webkit.org/show_bug.cgi?id=225793
+
+ This patch only uses Data Call ICs for the Baseline JIT. Even with that, it
+ reduces the number of calls to cacheFlush by ~45% on JetStream2.
+
+ Performance is neutral on AS Macs, but it paves the way towards doing
+ unlinked JITting in JSC.
+
+ * assembler/AbstractMacroAssembler.h:
+ (JSC::AbstractMacroAssembler::addLateLinkTask):
+ * assembler/LinkBuffer.cpp:
+ (JSC::LinkBuffer::linkCode):
+ (JSC::LinkBuffer::performFinalization):
+ * assembler/LinkBuffer.h:
+ * bytecode/AccessCase.cpp:
+ (JSC::AccessCase::generateImpl):
+ * bytecode/CallLinkInfo.cpp:
+ (JSC::CallLinkInfo::CallLinkInfo):
+ (JSC::CallLinkInfo::unlink):
+ (JSC::CallLinkInfo::fastPathStart):
+ (JSC::CallLinkInfo::slowPathStart):
+ (JSC::CallLinkInfo::doneLocation):
+ (JSC::CallLinkInfo::setMonomorphicCallee):
+ (JSC::CallLinkInfo::clearCallee):
+ (JSC::CallLinkInfo::emitFirstInstructionForDataIC):
+ (JSC::CallLinkInfo::emitFastPathImpl):
+ (JSC::CallLinkInfo::emitFastPath):
+ (JSC::CallLinkInfo::emitTailCallFastPath):
+ (JSC::CallLinkInfo::emitSlowPath):
+ (JSC::CallLinkInfo::emitDirectFastPath):
+ (JSC::CallLinkInfo::emitDirectTailCallFastPath):
+ (JSC::CallLinkInfo::initializeDirectCall):
+ (JSC::CallLinkInfo::setDirectCallTarget):
+ (JSC::CallLinkInfo::setSlowPathCallDestination):
+ (JSC::CallLinkInfo::revertCallToStub):
+ (JSC::CallLinkInfo::setStub):
+ (JSC::CallLinkInfo::callReturnLocation): Deleted.
+ (JSC::CallLinkInfo::patchableJump): Deleted.
+ (JSC::CallLinkInfo::hotPathBegin): Deleted.
+ (JSC::CallLinkInfo::setCallee): Deleted.
+ * bytecode/CallLinkInfo.h:
+ (JSC::CallLinkInfo::calleeGPR const):
+ (JSC::CallLinkInfo::isDataIC const):
+ (JSC::CallLinkInfo::setUsesDataICs):
+ (JSC::CallLinkInfo::setCodeLocations):
+ (JSC::CallLinkInfo::offsetOfCallee):
+ (JSC::CallLinkInfo::offsetOfMonomorphicCallDestination):
+ (JSC::CallLinkInfo::offsetOfSlowPathCallDestination):
+ (JSC::CallLinkInfo::setCallLocations): Deleted.
+ (JSC::CallLinkInfo::hotPathOther): Deleted.
+ (JSC::CallLinkInfo::setStub): Deleted.
+ * dfg/DFGJITCompiler.cpp:
+ (JSC::DFG::JITCompiler::link):
+ * dfg/DFGJITCompiler.h:
+ (JSC::DFG::JITCompiler::addJSCall):
+ (JSC::DFG::JITCompiler::addJSDirectCall):
+ (JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
+ (JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
+ (JSC::DFG::JITCompiler::addJSDirectTailCall): Deleted.
+ (JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord): Deleted.
+ * dfg/DFGOSRExitCompilerCommon.cpp:
+ (JSC::DFG::callerReturnPC):
+ * dfg/DFGOperations.cpp:
+ (JSC::DFG::JSC_DEFINE_JIT_OPERATION):
+ * dfg/DFGSpeculativeJIT32_64.cpp:
+ (JSC::DFG::SpeculativeJIT::emitCall):
+ * dfg/DFGSpeculativeJIT64.cpp:
+ (JSC::DFG::SpeculativeJIT::emitCall):
+ * ftl/FTLLowerDFGToB3.cpp:
+ (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
+ (JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
+ (JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
+ (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread):
+ (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
+ * jit/CCallHelpers.cpp:
+ (JSC::CCallHelpers::emitJITCodeOver):
+ * jit/CCallHelpers.h:
+ * jit/JIT.cpp:
+ (JSC::JIT::link):
+ * jit/JIT.h:
+ * jit/JITCall.cpp:
+ (JSC::JIT::compileTailCall):
+ (JSC::JIT::compileOpCall):
+ (JSC::JIT::compileOpCallSlowCase):
+ * jit/JITCall32_64.cpp:
+ (JSC::JIT::compileOpCall):
+ (JSC::JIT::compileOpCallSlowCase):
+ * jit/JITOperations.cpp:
+ (JSC::JSC_DEFINE_JIT_OPERATION):
+ * jit/PolymorphicCallStubRoutine.cpp:
+ (JSC::PolymorphicCallNode::unlink):
+ * jit/Repatch.cpp:
+ (JSC::linkSlowPathTo):
+ (JSC::linkSlowFor):
+ (JSC::linkMonomorphicCall):
+ (JSC::linkDirectCall):
+ (JSC::revertCall):
+ (JSC::unlinkCall):
+ (JSC::linkPolymorphicCall):
+ (JSC::linkFor): Deleted.
+ (JSC::linkDirectFor): Deleted.
+ (JSC::unlinkFor): Deleted.
+ * jit/Repatch.h:
+ * wasm/js/WasmToJS.cpp:
+ (JSC::Wasm::wasmToJS):
+
</ins><span class="cx"> 2021-05-18 Commit Queue <commit-queue@webkit.org>
</span><span class="cx">
</span><span class="cx"> Unreviewed, reverting r276655.
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerAbstractMacroAssemblerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -971,6 +971,12 @@
</span><span class="cx"> m_linkTasks.append(createSharedTask<void(LinkBuffer&)>(functor));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ template<typename Functor>
+ void addLateLinkTask(const Functor& functor) // Run after all link tasks
+ {
+ m_lateLinkTasks.append(createSharedTask<void(LinkBuffer&)>(functor));
+ }
+
</ins><span class="cx"> #if COMPILER(GCC)
</span><span class="cx"> // Workaround for GCC demanding that memcpy "must be the name of a function with external linkage".
</span><span class="cx"> static void* memcpy(void* dst, const void* src, size_t size)
</span><span class="lines">@@ -1114,6 +1120,7 @@
</span><span class="cx"> bool m_allowScratchRegister { true };
</span><span class="cx">
</span><span class="cx"> Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_linkTasks;
</span><ins>+ Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_lateLinkTasks;
</ins><span class="cx">
</span><span class="cx"> friend class LinkBuffer;
</span><span class="cx"> }; // class AbstractMacroAssembler
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerLinkBuffercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -439,6 +439,7 @@
</span><span class="cx"> #endif // !ENABLE(BRANCH_COMPACTION)
</span><span class="cx">
</span><span class="cx"> m_linkTasks = WTFMove(macroAssembler.m_linkTasks);
</span><ins>+ m_lateLinkTasks = WTFMove(macroAssembler.m_lateLinkTasks);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void LinkBuffer::allocate(MacroAssembler& macroAssembler, JITCompilationEffort effort)
</span><span class="lines">@@ -471,6 +472,8 @@
</span><span class="cx"> {
</span><span class="cx"> for (auto& task : m_linkTasks)
</span><span class="cx"> task->run(*this);
</span><ins>+ for (auto& task : m_lateLinkTasks)
+ task->run(*this);
</ins><span class="cx">
</span><span class="cx"> #ifndef NDEBUG
</span><span class="cx"> ASSERT(m_isJumpIsland || !isCompilationThread());
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerLinkBufferh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/LinkBuffer.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/LinkBuffer.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/LinkBuffer.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -403,6 +403,7 @@
</span><span class="cx"> Profile m_profile { Profile::Uncategorized };
</span><span class="cx"> MacroAssemblerCodePtr<LinkBufferPtrTag> m_code;
</span><span class="cx"> Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_linkTasks;
</span><ins>+ Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_lateLinkTasks;
</ins><span class="cx">
</span><span class="cx"> static size_t s_profileCummulativeLinkedSizes[numberOfProfiles];
</span><span class="cx"> };
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeAccessCasecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1580,11 +1580,6 @@
</span><span class="cx"> // Stuff for custom getters/setters.
</span><span class="cx"> CCallHelpers::Call operationCall;
</span><span class="cx">
</span><del>- // Stuff for JS getters/setters.
- CCallHelpers::DataLabelPtr addressOfLinkFunctionCheck;
- CCallHelpers::Call fastPathCall;
- CCallHelpers::Call slowPathCall;
-
</del><span class="cx"> // This also does the necessary calculations of whether or not we're an
</span><span class="cx"> // exception handling call site.
</span><span class="cx"> AccessGenerationState::SpillState spillState = state.preserveLiveRegistersToStackForCall();
</span><span class="lines">@@ -1701,24 +1696,22 @@
</span><span class="cx"> virtualRegisterForArgumentIncludingThis(1).offset() * sizeof(Register)));
</span><span class="cx"> }
</span><span class="cx">
</span><del>- CCallHelpers::Jump slowCase = jit.branchPtrWithPatch(
- CCallHelpers::NotEqual, loadedValueGPR, addressOfLinkFunctionCheck,
- CCallHelpers::TrustedImmPtr(nullptr));
</del><ins>+ auto slowCase = access.callLinkInfo()->emitFastPath(jit, loadedValueGPR, loadedValueGPR == GPRInfo::regT2 ? GPRInfo::regT0 : GPRInfo::regT2, JITCode::isOptimizingJIT(codeBlock->jitType()) ? CallLinkInfo::UseDataIC::No : CallLinkInfo::UseDataIC::Yes);
+ auto doneLocation = jit.label();
</ins><span class="cx">
</span><del>- fastPathCall = jit.nearCall();
</del><span class="cx"> if (m_type == Getter)
</span><span class="cx"> jit.setupResults(valueRegs);
</span><span class="cx"> done.append(jit.jump());
</span><span class="cx">
</span><span class="cx"> slowCase.link(&jit);
</span><ins>+ auto slowPathStart = jit.label();
</ins><span class="cx"> jit.move(loadedValueGPR, GPRInfo::regT0);
</span><span class="cx"> #if USE(JSVALUE32_64)
</span><span class="cx"> // We *always* know that the getter/setter, if non-null, is a cell.
</span><span class="cx"> jit.move(CCallHelpers::TrustedImm32(JSValue::CellTag), GPRInfo::regT1);
</span><span class="cx"> #endif
</span><del>- jit.move(CCallHelpers::TrustedImmPtr(access.callLinkInfo()), GPRInfo::regT2);
</del><span class="cx"> jit.move(CCallHelpers::TrustedImmPtr(globalObject), GPRInfo::regT3);
</span><del>- slowPathCall = jit.nearCall();
</del><ins>+ access.callLinkInfo()->emitSlowPath(vm, jit);
</ins><span class="cx"> if (m_type == Getter)
</span><span class="cx"> jit.setupResults(valueRegs);
</span><span class="cx"> done.append(jit.jump());
</span><span class="lines">@@ -1735,15 +1728,10 @@
</span><span class="cx"> bool callHasReturnValue = isGetter();
</span><span class="cx"> restoreLiveRegistersFromStackForCall(spillState, callHasReturnValue);
</span><span class="cx">
</span><del>- jit.addLinkTask([=, &vm] (LinkBuffer& linkBuffer) {
- this->as<GetterSetterAccessCase>().callLinkInfo()->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowPathCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(addressOfLinkFunctionCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastPathCall));
-
- linkBuffer.link(
- slowPathCall,
- CodeLocationLabel<JITThunkPtrTag>(vm.getCTIStub(linkCallThunkGenerator).code()));
</del><ins>+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ this->as<GetterSetterAccessCase>().callLinkInfo()->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> });
</span><span class="cx"> } else {
</span><span class="cx"> ASSERT(m_type == CustomValueGetter || m_type == CustomAccessorGetter || m_type == CustomValueSetter || m_type == CustomAccessorSetter);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCallLinkInfocpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -26,11 +26,15 @@
</span><span class="cx"> #include "config.h"
</span><span class="cx"> #include "CallLinkInfo.h"
</span><span class="cx">
</span><ins>+#include "CCallHelpers.h"
</ins><span class="cx"> #include "CallFrameShuffleData.h"
</span><ins>+#include "DisallowMacroScratchRegisterUsage.h"
</ins><span class="cx"> #include "FunctionCodeBlock.h"
</span><span class="cx"> #include "JSCellInlines.h"
</span><ins>+#include "LinkBuffer.h"
</ins><span class="cx"> #include "Opcode.h"
</span><span class="cx"> #include "Repatch.h"
</span><ins>+#include "ThunkGenerators.h"
</ins><span class="cx"> #include <wtf/ListDump.h>
</span><span class="cx">
</span><span class="cx"> #if ENABLE(JIT)
</span><span class="lines">@@ -77,6 +81,7 @@
</span><span class="cx"> , m_allowStubs(true)
</span><span class="cx"> , m_clearedByJettison(false)
</span><span class="cx"> , m_callType(None)
</span><ins>+ , m_useDataIC(static_cast<unsigned>(UseDataIC::Yes))
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -102,7 +107,7 @@
</span><span class="cx"> // We could be called even if we're not linked anymore because of how polymorphic calls
</span><span class="cx"> // work. Each callsite within the polymorphic call stub may separately ask us to unlink().
</span><span class="cx"> if (isLinked())
</span><del>- unlinkFor(vm, *this);
</del><ins>+ unlinkCall(vm, *this);
</ins><span class="cx">
</span><span class="cx"> // Either we were unlinked, in which case we should not have been on any list, or we unlinked
</span><span class="cx"> // ourselves so that we're not on any list anymore.
</span><span class="lines">@@ -109,34 +114,34 @@
</span><span class="cx"> RELEASE_ASSERT(!isOnList());
</span><span class="cx"> }
</span><span class="cx">
</span><del>-CodeLocationNearCall<JSInternalPtrTag> CallLinkInfo::callReturnLocation()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::fastPathStart()
</ins><span class="cx"> {
</span><del>- RELEASE_ASSERT(!isDirect());
- return CodeLocationNearCall<JSInternalPtrTag>(m_callReturnLocationOrPatchableJump, NearCallMode::Regular);
</del><ins>+ return CodeLocationDataLabelPtr<JSInternalPtrTag>(m_fastPathStart);
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-CodeLocationJump<JSInternalPtrTag> CallLinkInfo::patchableJump()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::slowPathStart()
</ins><span class="cx"> {
</span><del>- RELEASE_ASSERT(callType() == DirectTailCall);
- return CodeLocationJump<JSInternalPtrTag>(m_callReturnLocationOrPatchableJump);
</del><ins>+ RELEASE_ASSERT(!isDataIC());
+ return u.codeIC.m_slowPathStart;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-CodeLocationDataLabelPtr<JSInternalPtrTag> CallLinkInfo::hotPathBegin()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::doneLocation()
</ins><span class="cx"> {
</span><span class="cx"> RELEASE_ASSERT(!isDirect());
</span><del>- return CodeLocationDataLabelPtr<JSInternalPtrTag>(m_hotPathBeginOrSlowPathStart);
</del><ins>+ return m_doneLocation;
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::slowPathStart()
</del><ins>+void CallLinkInfo::setMonomorphicCallee(VM& vm, JSCell* owner, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</ins><span class="cx"> {
</span><del>- RELEASE_ASSERT(isDirect());
- return m_hotPathBeginOrSlowPathStart;
-}
-
-void CallLinkInfo::setCallee(VM& vm, JSCell* owner, JSObject* callee)
-{
</del><span class="cx"> RELEASE_ASSERT(!isDirect());
</span><span class="cx"> m_calleeOrCodeBlock.set(vm, owner, callee);
</span><ins>+
+ if (isDataIC())
+ u.dataIC.m_monomorphicCallDestination = codePtr;
+ else {
+ MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, CodeLocationLabel<JSEntryPtrTag>(codePtr));
+ MacroAssembler::repatchPointer(u.codeIC.m_calleeLocation, callee);
+ }
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> void CallLinkInfo::clearCallee()
</span><span class="lines">@@ -143,6 +148,10 @@
</span><span class="cx"> {
</span><span class="cx"> RELEASE_ASSERT(!isDirect());
</span><span class="cx"> m_calleeOrCodeBlock.clear();
</span><ins>+ if (isDataIC())
+ u.dataIC.m_monomorphicCallDestination = nullptr;
+ else if (!clearedByJettison())
+ MacroAssembler::repatchPointer(u.codeIC.m_calleeLocation, nullptr);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> JSObject* CallLinkInfo::callee()
</span><span class="lines">@@ -284,6 +293,206 @@
</span><span class="cx"> m_frameShuffleData->shrinkToFit();
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void CallLinkInfo::emitFirstInstructionForDataIC(CCallHelpers& jit, GPRReg callLinkInfoGPR)
+{
+ GPRReg scratchGPR = jit.scratchRegister();
+ DisallowMacroScratchRegisterUsage disallowScratch(jit);
+ // When repatching, we just overwrite the first instruction back, since it'll have been replaced with a jump to the polymorphic call stub.
+ size_t startSize = jit.m_assembler.buffer().codeSize();
+ jit.loadPtr(CCallHelpers::Address(callLinkInfoGPR, offsetOfCallee()), scratchGPR);
+ size_t loadSize = jit.m_assembler.buffer().codeSize() - startSize;
+ if (loadSize < static_cast<size_t>(CCallHelpers::patchableJumpSize()))
+ jit.emitNops(static_cast<size_t>(CCallHelpers::patchableJumpSize()) - loadSize);
+}
+
+MacroAssembler::JumpList CallLinkInfo::emitFastPathImpl(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC, WTF::Function<void()> prepareForTailCall)
+{
+ setUsesDataICs(useDataIC);
+ if (isDataIC()) {
+ RELEASE_ASSERT(callLinkInfoGPR != GPRReg::InvalidGPRReg);
+ jit.move(CCallHelpers::TrustedImmPtr(this), callLinkInfoGPR);
+ u.dataIC.m_callLinkInfoGPR = callLinkInfoGPR;
+ }
+
+ auto fastPathStart = jit.label();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+ });
+
+ CCallHelpers::JumpList slowPath;
+
+ if (isDataIC()) {
+ GPRReg scratchGPR = jit.scratchRegister();
+ emitFirstInstructionForDataIC(jit, callLinkInfoGPR);
+ {
+ DisallowMacroScratchRegisterUsage disallowScratch(jit);
+ slowPath.append(jit.branchPtr(CCallHelpers::NotEqual, scratchGPR, calleeGPR));
+ }
+ if (isTailCall()) {
+ prepareForTailCall();
+ jit.farJump(CCallHelpers::Address(callLinkInfoGPR, offsetOfMonomorphicCallDestination()), JSEntryPtrTag);
+ } else
+ jit.call(CCallHelpers::Address(callLinkInfoGPR, offsetOfMonomorphicCallDestination()), JSEntryPtrTag);
+ } else {
+ CCallHelpers::DataLabelPtr calleeCheck;
+ slowPath.append(jit.branchPtrWithPatch(CCallHelpers::NotEqual, calleeGPR, calleeCheck, CCallHelpers::TrustedImmPtr(nullptr)));
+
+ CCallHelpers::Call call;
+ if (isTailCall()) {
+ prepareForTailCall();
+ call = jit.nearTailCall();
+ } else
+ call = jit.nearCall();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+ u.codeIC.m_calleeLocation = linkBuffer.locationOf<JSInternalPtrTag>(calleeCheck);
+ });
+ }
+
+ return slowPath;
+}
+
+CCallHelpers::JumpList CallLinkInfo::emitFastPath(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC)
+{
+ RELEASE_ASSERT(!isTailCall());
+ return emitFastPathImpl(jit, calleeGPR, callLinkInfoGPR, useDataIC, nullptr);
+}
+
+MacroAssembler::JumpList CallLinkInfo::emitTailCallFastPath(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC, WTF::Function<void()> prepareForTailCall)
+{
+ RELEASE_ASSERT(isTailCall());
+ return emitFastPathImpl(jit, calleeGPR, callLinkInfoGPR, useDataIC, WTFMove(prepareForTailCall));
+}
+
+void CallLinkInfo::emitSlowPath(VM& vm, CCallHelpers& jit)
+{
+ setSlowPathCallDestination(vm.getCTIStub(linkCallThunkGenerator).template retaggedCode<JSEntryPtrTag>());
+ jit.move(CCallHelpers::TrustedImmPtr(this), GPRInfo::regT2);
+ jit.call(CCallHelpers::Address(GPRInfo::regT2, offsetOfSlowPathCallDestination()), JSEntryPtrTag);
+}
+
+void CallLinkInfo::emitDirectFastPath(CCallHelpers& jit)
+{
+ RELEASE_ASSERT(!isTailCall());
+
+ setUsesDataICs(UseDataIC::No);
+
+ auto fastPathStart = jit.label();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+ });
+
+ auto call = jit.nearCall();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+ });
+ jit.addLateLinkTask([this] (LinkBuffer&) {
+ initializeDirectCall();
+ });
+}
+
+void CallLinkInfo::emitDirectTailCallFastPath(CCallHelpers& jit, WTF::Function<void()> prepareForTailCall)
+{
+ RELEASE_ASSERT(isTailCall());
+
+ setUsesDataICs(UseDataIC::No);
+
+ auto fastPathStart = jit.label();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+ });
+
+ // - If we're not yet linked, this is a jump to the slow path.
+ // - Once we're linked to a fast path, this goes back to being nops so we fall through to the linked jump.
+ jit.emitNops(CCallHelpers::patchableJumpSize());
+
+ prepareForTailCall();
+ auto call = jit.nearTailCall();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+ });
+ jit.addLateLinkTask([this] (LinkBuffer&) {
+ initializeDirectCall();
+ });
+}
+
+void CallLinkInfo::initializeDirectCall()
+{
+ RELEASE_ASSERT(isDirect());
+ ASSERT(u.codeIC.m_callLocation);
+ if (isTailCall()) {
+ RELEASE_ASSERT(fastPathStart());
+ CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+ auto jump = jit.jump();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ linkBuffer.link(jump, slowPathStart());
+ });
+ }, "initialize direct call");
+ } else
+ MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, slowPathStart());
+}
+
+void CallLinkInfo::setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag> target)
+{
+ RELEASE_ASSERT(isDirect());
+
+ if (isTailCall()) {
+ RELEASE_ASSERT(fastPathStart());
+ CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+ // We reserved this many bytes for the jump at fastPathStart(). Make that
+ // code nops now so we fall through to the jump to the fast path.
+ jit.emitNops(CCallHelpers::patchableJumpSize());
+ }, "Setting direct call target");
+ }
+
+ MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, target);
+}
+
+void CallLinkInfo::setSlowPathCallDestination(MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
+{
+ m_slowPathCallDestination = codePtr;
+}
+
+void CallLinkInfo::revertCallToStub()
+{
+ RELEASE_ASSERT(stub());
+ // The start of our JIT code is now a jump to the polymorphic stub. Rewrite the first instruction
+ // to be what we need for non stub ICs.
+
+ // this runs into some branch compaction crap I'd like to avoid for now. Essentially, the branch
+ // doesn't know if it can be compacted or not. So we end up with 28 bytes of machine code, for
+ // what in all likelihood fits in 24. So we just splat out the first instruction. Long term, we
+ // need something cleaner. But this works on arm64 for now.
+
+ CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+ if (isDataIC())
+ emitFirstInstructionForDataIC(jit, u.dataIC.m_callLinkInfoGPR);
+ else {
+ CCallHelpers::revertJumpReplacementToBranchPtrWithPatch(
+ CCallHelpers::startOfBranchPtrWithPatchOnRegister(u.codeIC.m_calleeLocation), calleeGPR(), nullptr);
+ }
+ }, "Resetting PolymorphicCall stubbed CallLinkInfo");
+}
+
+void CallLinkInfo::setStub(Ref<PolymorphicCallStubRoutine>&& newStub)
+{
+ clearStub();
+ m_stub = WTFMove(newStub);
+
+ if (isDataIC()) {
+ CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+ auto jump = jit.jump();
+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ linkBuffer.link(jump, CodeLocationLabel<JITStubRoutinePtrTag>(m_stub->code().code()));
+ });
+ }, "Patching call to PolymorphicCallStubRoutine in CallLinkInfo");
+ } else {
+ MacroAssembler::replaceWithJump(
+ MacroAssembler::startOfBranchPtrWithPatchOnRegister(u.codeIC.m_calleeLocation),
+ CodeLocationLabel<JITStubRoutinePtrTag>(m_stub->code().code()));
+ }
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx"> #endif // ENABLE(JIT)
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCallLinkInfoh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx"> #include "CodeSpecializationKind.h"
</span><span class="cx"> #include "PolymorphicCallStubRoutine.h"
</span><span class="cx"> #include "WriteBarrier.h"
</span><ins>+#include <wtf/Function.h>
</ins><span class="cx"> #include <wtf/SentinelLinkedList.h>
</span><span class="cx">
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -36,6 +37,7 @@
</span><span class="cx">
</span><span class="cx"> #if ENABLE(JIT)
</span><span class="cx">
</span><ins>+class CCallHelpers;
</ins><span class="cx"> class FunctionCodeBlock;
</span><span class="cx"> class JSFunction;
</span><span class="cx"> enum OpcodeID : unsigned;
</span><span class="lines">@@ -163,16 +165,38 @@
</span><span class="cx"> m_calleeGPR = calleeGPR;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- void setCallLocations(
- CodeLocationLabel<JSInternalPtrTag> callReturnLocationOrPatchableJump,
- CodeLocationLabel<JSInternalPtrTag> hotPathBeginOrSlowPathStart,
- CodeLocationNearCall<JSInternalPtrTag> hotPathOther)
</del><ins>+ GPRReg calleeGPR() const { return m_calleeGPR; }
+
+ enum class UseDataIC : uint8_t {
+ Yes,
+ No
+ };
+
+private:
+ void emitFirstInstructionForDataIC(CCallHelpers&, GPRReg callLinkInfoGPR);
+ MacroAssembler::JumpList emitFastPathImpl(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC, WTF::Function<void()> prepareForTailCall) WARN_UNUSED_RETURN;
+public:
+ MacroAssembler::JumpList emitFastPath(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC) WARN_UNUSED_RETURN;
+ MacroAssembler::JumpList emitTailCallFastPath(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC, WTF::Function<void()> prepareForTailCall) WARN_UNUSED_RETURN;
+ void emitDirectFastPath(CCallHelpers&);
+ void emitDirectTailCallFastPath(CCallHelpers&, WTF::Function<void()> prepareForTailCall);
+ void emitSlowPath(VM&, CCallHelpers&);
+ void revertCallToStub();
+
+ bool isDataIC() const { return static_cast<UseDataIC>(m_useDataIC) == UseDataIC::Yes; }
+ void setUsesDataICs(UseDataIC useDataIC) { m_useDataIC = static_cast<unsigned>(useDataIC); }
+
+ void setCodeLocations(
+ CodeLocationLabel<JSInternalPtrTag> slowPathStart,
+ CodeLocationLabel<JSInternalPtrTag> doneLocation)
</ins><span class="cx"> {
</span><del>- m_callReturnLocationOrPatchableJump = callReturnLocationOrPatchableJump;
- m_hotPathBeginOrSlowPathStart = hotPathBeginOrSlowPathStart;
- m_hotPathOther = hotPathOther;
</del><ins>+ if (!isDataIC())
+ u.codeIC.m_slowPathStart = slowPathStart;
+ m_doneLocation = doneLocation;
</ins><span class="cx"> }
</span><span class="cx">
</span><ins>+ void initializeDirectCall();
+
</ins><span class="cx"> bool allowStubs() const { return m_allowStubs; }
</span><span class="cx">
</span><span class="cx"> void disallowStubs()
</span><span class="lines">@@ -180,17 +204,12 @@
</span><span class="cx"> m_allowStubs = false;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- CodeLocationNearCall<JSInternalPtrTag> callReturnLocation();
- CodeLocationJump<JSInternalPtrTag> patchableJump();
- CodeLocationDataLabelPtr<JSInternalPtrTag> hotPathBegin();
</del><ins>+ CodeLocationLabel<JSInternalPtrTag> fastPathStart();
</ins><span class="cx"> CodeLocationLabel<JSInternalPtrTag> slowPathStart();
</span><ins>+ CodeLocationLabel<JSInternalPtrTag> doneLocation();
</ins><span class="cx">
</span><del>- CodeLocationNearCall<JSInternalPtrTag> hotPathOther()
- {
- return m_hotPathOther;
- }
-
- void setCallee(VM&, JSCell*, JSObject* callee);
</del><ins>+ void setMonomorphicCallee(VM&, JSCell*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
+ void setSlowPathCallDestination(MacroAssemblerCodePtr<JSEntryPtrTag>);
</ins><span class="cx"> void clearCallee();
</span><span class="cx"> JSObject* callee();
</span><span class="cx">
</span><span class="lines">@@ -197,6 +216,7 @@
</span><span class="cx"> void setCodeBlock(VM&, JSCell*, FunctionCodeBlock*);
</span><span class="cx"> void clearCodeBlock();
</span><span class="cx"> FunctionCodeBlock* codeBlock();
</span><ins>+ void setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag>);
</ins><span class="cx">
</span><span class="cx"> void setLastSeenCallee(VM&, const JSCell* owner, JSObject* callee);
</span><span class="cx"> void clearLastSeenCallee();
</span><span class="lines">@@ -206,12 +226,7 @@
</span><span class="cx"> void setExecutableDuringCompilation(ExecutableBase*);
</span><span class="cx"> ExecutableBase* executable();
</span><span class="cx">
</span><del>- void setStub(Ref<PolymorphicCallStubRoutine>&& newStub)
- {
- clearStub();
- m_stub = WTFMove(newStub);
- }
-
</del><ins>+ void setStub(Ref<PolymorphicCallStubRoutine>&&);
</ins><span class="cx"> void clearStub();
</span><span class="cx">
</span><span class="cx"> PolymorphicCallStubRoutine* stub() const
</span><span class="lines">@@ -311,6 +326,21 @@
</span><span class="cx"> return OBJECT_OFFSETOF(CallLinkInfo, m_slowPathCount);
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+ static ptrdiff_t offsetOfCallee()
+ {
+ return OBJECT_OFFSETOF(CallLinkInfo, m_calleeOrCodeBlock);
+ }
+
+ static ptrdiff_t offsetOfMonomorphicCallDestination()
+ {
+ return OBJECT_OFFSETOF(CallLinkInfo, u) + OBJECT_OFFSETOF(UnionType, dataIC.m_monomorphicCallDestination);
+ }
+
+ static ptrdiff_t offsetOfSlowPathCallDestination()
+ {
+ return OBJECT_OFFSETOF(CallLinkInfo, m_slowPathCallDestination);
+ }
+
</ins><span class="cx"> GPRReg calleeGPR()
</span><span class="cx"> {
</span><span class="cx"> return m_calleeGPR;
</span><span class="lines">@@ -352,10 +382,25 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> private:
</span><del>- uint32_t m_maxArgumentCountIncludingThis { 0 }; // For varargs: the profiled maximum number of arguments. For direct: the number of stack slots allocated for arguments.
- CodeLocationLabel<JSInternalPtrTag> m_callReturnLocationOrPatchableJump;
- CodeLocationLabel<JSInternalPtrTag> m_hotPathBeginOrSlowPathStart;
- CodeLocationNearCall<JSInternalPtrTag> m_hotPathOther;
</del><ins>+ CodeLocationLabel<JSInternalPtrTag> m_fastPathStart;
+ CodeLocationLabel<JSInternalPtrTag> m_doneLocation;
+ MacroAssemblerCodePtr<JSEntryPtrTag> m_slowPathCallDestination;
+ union UnionType {
+ UnionType()
+ : dataIC { nullptr, InvalidGPRReg }
+ { }
+ struct DataIC {
+ MacroAssemblerCodePtr<JSEntryPtrTag> m_monomorphicCallDestination;
+ GPRReg m_callLinkInfoGPR;
+ } dataIC;
+
+ struct {
+ CodeLocationNearCall<JSInternalPtrTag> m_callLocation;
+ CodeLocationDataLabelPtr<JSInternalPtrTag> m_calleeLocation;
+ CodeLocationLabel<JSInternalPtrTag> m_slowPathStart;
+ } codeIC;
+ } u;
+
</ins><span class="cx"> WriteBarrier<JSCell> m_calleeOrCodeBlock;
</span><span class="cx"> WriteBarrier<JSCell> m_lastSeenCalleeOrExecutable;
</span><span class="cx"> RefPtr<PolymorphicCallStubRoutine> m_stub;
</span><span class="lines">@@ -369,8 +414,10 @@
</span><span class="cx"> bool m_allowStubs : 1;
</span><span class="cx"> bool m_clearedByJettison : 1;
</span><span class="cx"> unsigned m_callType : 4; // CallType
</span><ins>+ unsigned m_useDataIC : 1; // UseDataIC
</ins><span class="cx"> GPRReg m_calleeGPR { InvalidGPRReg };
</span><span class="cx"> uint32_t m_slowPathCount { 0 };
</span><ins>+ uint32_t m_maxArgumentCountIncludingThis { 0 }; // For varargs: the profiled maximum number of arguments. For direct: the number of stack slots allocated for arguments.
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> inline CodeOrigin getCallLinkInfoCodeOrigin(CallLinkInfo& callLinkInfo)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGJITCompilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -257,32 +257,19 @@
</span><span class="cx"> finalizeInlineCaches(m_instanceOfs, linkBuffer);
</span><span class="cx"> finalizeInlineCaches(m_privateBrandAccesses, linkBuffer);
</span><span class="cx">
</span><del>- auto linkCallThunk = FunctionPtr<NoPtrTag>(vm().getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
</del><span class="cx"> for (auto& record : m_jsCalls) {
</span><span class="cx"> CallLinkInfo& info = *record.info;
</span><del>- linkBuffer.link(record.slowCall, linkCallThunk);
- info.setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.slowCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(record.targetToCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.fastCall));
</del><ins>+ info.setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(record.slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(record.doneLocation));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- for (JSDirectCallRecord& record : m_jsDirectCalls) {
</del><ins>+ for (auto& record : m_jsDirectCalls) {
</ins><span class="cx"> CallLinkInfo& info = *record.info;
</span><del>- linkBuffer.link(record.call, linkBuffer.locationOf<NoPtrTag>(record.slowPath));
- info.setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(),
</del><ins>+ info.setCodeLocations(
</ins><span class="cx"> linkBuffer.locationOf<JSInternalPtrTag>(record.slowPath),
</span><del>- linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.call));
</del><ins>+ CodeLocationLabel<JSInternalPtrTag>());
</ins><span class="cx"> }
</span><del>-
- for (JSDirectTailCallRecord& record : m_jsDirectTailCalls) {
- CallLinkInfo& info = *record.info;
- info.setCallLocations(
- linkBuffer.locationOf<JSInternalPtrTag>(record.patchableJump),
- linkBuffer.locationOf<JSInternalPtrTag>(record.slowPath),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.call));
- }
</del><span class="cx">
</span><span class="cx"> #if ENABLE(EXTRA_CTI_THUNKS)
</span><span class="cx"> if (!m_exceptionChecks.empty())
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGJITCompilerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -219,21 +219,16 @@
</span><span class="cx"> m_privateBrandAccesses.append(InlineCacheWrapper<JITPrivateBrandAccessGenerator>(gen, slowPath));
</span><span class="cx"> }
</span><span class="cx">
</span><del>- void addJSCall(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
</del><ins>+ void addJSCall(Label slowPathStart, Label doneLocation, CallLinkInfo* info)
</ins><span class="cx"> {
</span><del>- m_jsCalls.append(JSCallRecord(fastCall, slowCall, targetToCheck, info));
</del><ins>+ m_jsCalls.append(JSCallRecord(slowPathStart, doneLocation, info));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- void addJSDirectCall(Call call, Label slowPath, CallLinkInfo* info)
</del><ins>+ void addJSDirectCall(Label slowPath, CallLinkInfo* info)
</ins><span class="cx"> {
</span><del>- m_jsDirectCalls.append(JSDirectCallRecord(call, slowPath, info));
</del><ins>+ m_jsDirectCalls.append(JSDirectCallRecord(slowPath, info));
</ins><span class="cx"> }
</span><span class="cx">
</span><del>- void addJSDirectTailCall(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
- {
- m_jsDirectTailCalls.append(JSDirectTailCallRecord(patchableJump, call, slowPath, info));
- }
-
</del><span class="cx"> void addWeakReference(JSCell* target)
</span><span class="cx"> {
</span><span class="cx"> m_graph.m_plan.weakReferences().addLazily(target);
</span><span class="lines">@@ -328,53 +323,29 @@
</span><span class="cx">
</span><span class="cx">
</span><span class="cx"> struct JSCallRecord {
</span><del>- JSCallRecord(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
- : fastCall(fastCall)
- , slowCall(slowCall)
- , targetToCheck(targetToCheck)
</del><ins>+ JSCallRecord(Label slowPathStart, Label doneLocation, CallLinkInfo* info)
+ : slowPathStart(slowPathStart)
+ , doneLocation(doneLocation)
</ins><span class="cx"> , info(info)
</span><span class="cx"> {
</span><del>- ASSERT(fastCall.isFlagSet(Call::Near));
- ASSERT(slowCall.isFlagSet(Call::Near));
</del><span class="cx"> }
</span><span class="cx">
</span><del>- Call fastCall;
- Call slowCall;
- DataLabelPtr targetToCheck;
</del><ins>+ Label slowPathStart;
+ Label doneLocation;
</ins><span class="cx"> CallLinkInfo* info;
</span><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> struct JSDirectCallRecord {
</span><del>- JSDirectCallRecord(Call call, Label slowPath, CallLinkInfo* info)
- : call(call)
- , slowPath(slowPath)
</del><ins>+ JSDirectCallRecord(Label slowPath, CallLinkInfo* info)
+ : slowPath(slowPath)
</ins><span class="cx"> , info(info)
</span><span class="cx"> {
</span><del>- ASSERT(call.isFlagSet(Call::Near));
</del><span class="cx"> }
</span><span class="cx">
</span><del>- Call call;
</del><span class="cx"> Label slowPath;
</span><span class="cx"> CallLinkInfo* info;
</span><span class="cx"> };
</span><span class="cx">
</span><del>- struct JSDirectTailCallRecord {
- JSDirectTailCallRecord(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
- : patchableJump(patchableJump)
- , call(call)
- , slowPath(slowPath)
- , info(info)
- {
- ASSERT(call.isFlagSet(Call::Near) && call.isFlagSet(Call::Tail));
- }
-
- PatchableJump patchableJump;
- Call call;
- Label slowPath;
- CallLinkInfo* info;
- };
-
-
</del><span class="cx"> Vector<InlineCacheWrapper<JITGetByIdGenerator>, 4> m_getByIds;
</span><span class="cx"> Vector<InlineCacheWrapper<JITGetByIdWithThisGenerator>, 4> m_getByIdsWithThis;
</span><span class="cx"> Vector<InlineCacheWrapper<JITGetByValGenerator>, 4> m_getByVals;
</span><span class="lines">@@ -386,7 +357,6 @@
</span><span class="cx"> Vector<InlineCacheWrapper<JITPrivateBrandAccessGenerator>, 4> m_privateBrandAccesses;
</span><span class="cx"> Vector<JSCallRecord, 4> m_jsCalls;
</span><span class="cx"> Vector<JSDirectCallRecord, 4> m_jsDirectCalls;
</span><del>- Vector<JSDirectTailCallRecord, 4> m_jsDirectTailCalls;
</del><span class="cx"> SegmentedVector<OSRExitCompilationInfo, 4> m_exitCompilationInfo;
</span><span class="cx"> Vector<Vector<Label>> m_exitSiteLabels;
</span><span class="cx"> Vector<DFG::OSREntryData> m_osrEntry;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOSRExitCompilerCommoncpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -209,7 +209,7 @@
</span><span class="cx"> baselineCodeBlockForCaller->getCallLinkInfoForBytecodeIndex(callBytecodeIndex);
</span><span class="cx"> RELEASE_ASSERT(callLinkInfo);
</span><span class="cx">
</span><del>- jumpTarget = callLinkInfo->callReturnLocation().retagged<JSEntryPtrTag>();
</del><ins>+ jumpTarget = callLinkInfo->doneLocation().retagged<JSEntryPtrTag>();
</ins><span class="cx"> break;
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -3616,7 +3616,7 @@
</span><span class="cx"> codePtr = functionExecutable->entrypointFor(kind, ArityCheckNotRequired);
</span><span class="cx"> }
</span><span class="cx">
</span><del>- linkDirectFor(callFrame, *callLinkInfo, codeBlock, codePtr);
</del><ins>+ linkDirectCall(callFrame, *callLinkInfo, codeBlock, codePtr);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> JSC_DEFINE_JIT_OPERATION(operationTriggerReoptimizationNow, void, (CodeBlock* codeBlock, CodeBlock* optimizedCodeBlock, OSRExitBase* exit))
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJIT32_64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -838,19 +838,16 @@
</span><span class="cx"> if (isTail) {
</span><span class="cx"> RELEASE_ASSERT(node->op() == DirectTailCall);
</span><span class="cx">
</span><del>- JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
</del><span class="cx"> JITCompiler::Label mainPath = m_jit.label();
</span><span class="cx">
</span><span class="cx"> m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">
</span><del>- info->setFrameShuffleData(shuffleData);
- CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-
- JITCompiler::Call call = m_jit.nearTailCall();
-
</del><ins>+ info->emitDirectTailCallFastPath(m_jit, [&] {
+ info->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+ });
+
</ins><span class="cx"> JITCompiler::Label slowPath = m_jit.label();
</span><del>- patchableJump.m_jump.linkTo(slowPath, &m_jit);
-
</del><span class="cx"> silentSpillAllRegisters(InvalidGPRReg);
</span><span class="cx"> callOperation(operationLinkDirectCall, info, calleePayloadGPR);
</span><span class="cx"> silentFillAllRegisters();
</span><span class="lines">@@ -859,7 +856,7 @@
</span><span class="cx">
</span><span class="cx"> useChildren(node);
</span><span class="cx">
</span><del>- m_jit.addJSDirectTailCall(patchableJump, call, slowPath, info);
</del><ins>+ m_jit.addJSDirectCall(slowPath, info);
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -867,7 +864,7 @@
</span><span class="cx">
</span><span class="cx"> m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">
</span><del>- JITCompiler::Call call = m_jit.nearCall();
</del><ins>+ info->emitDirectFastPath(m_jit);
</ins><span class="cx"> JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">
</span><span class="cx"> JITCompiler::Label slowPath = m_jit.label();
</span><span class="lines">@@ -879,7 +876,7 @@
</span><span class="cx">
</span><span class="cx"> setResultAndResetStack();
</span><span class="cx">
</span><del>- m_jit.addJSDirectCall(call, slowPath, info);
</del><ins>+ m_jit.addJSDirectCall(slowPath, info);
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -888,21 +885,25 @@
</span><span class="cx"> slowPath.append(m_jit.branchIfNotCell(JSValueRegs(calleeTagGPR, calleePayloadGPR)));
</span><span class="cx"> slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
</span><span class="cx">
</span><ins>+ CCallHelpers::JumpList slowCases;
</ins><span class="cx"> if (isTail) {
</span><del>- if (node->op() == TailCall) {
- info->setFrameShuffleData(shuffleData);
- CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
- } else {
- m_jit.emitRestoreCalleeSaves();
- m_jit.prepareForTailCallSlow();
- }
- }
</del><ins>+ slowCases = info->emitTailCallFastPath(m_jit, calleePayloadGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+ if (node->op() == TailCall) {
+ info->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+ } else {
+ m_jit.emitRestoreCalleeSaves();
+ m_jit.prepareForTailCallSlow();
+ }
+ });
+ } else
+ slowCases = info->emitFastPath(m_jit, calleePayloadGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx">
</span><del>- JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
-
</del><span class="cx"> JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">
</span><span class="cx"> slowPath.link(&m_jit);
</span><ins>+ slowCases.link(&m_jit);
+ auto slowPathStart = m_jit.label();
</ins><span class="cx">
</span><span class="cx"> if (node->op() == TailCall) {
</span><span class="cx"> CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
</span><span class="lines">@@ -927,11 +928,11 @@
</span><span class="cx"> m_jit.emitRestoreCalleeSaves();
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_jit.move(TrustedImmPtr(info), GPRInfo::regT2);
</del><span class="cx"> m_jit.move(TrustedImmPtr::weakPointer(m_graph, globalObject), GPRInfo::regT3);
</span><del>- JITCompiler::Call slowCall = m_jit.nearCall();
</del><ins>+ info->emitSlowPath(vm(), m_jit);
</ins><span class="cx">
</span><span class="cx"> done.link(&m_jit);
</span><ins>+ auto doneLocation = m_jit.label();
</ins><span class="cx">
</span><span class="cx"> if (isTail)
</span><span class="cx"> m_jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="lines">@@ -938,7 +939,7 @@
</span><span class="cx"> else
</span><span class="cx"> setResultAndResetStack();
</span><span class="cx">
</span><del>- m_jit.addJSCall(fastCall, slowCall, targetToCheck, info);
</del><ins>+ m_jit.addJSCall(slowPathStart, doneLocation, info);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> template<bool strict>
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -830,18 +830,14 @@
</span><span class="cx"> if (isTail) {
</span><span class="cx"> RELEASE_ASSERT(node->op() == DirectTailCall);
</span><span class="cx">
</span><del>- JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
</del><span class="cx"> JITCompiler::Label mainPath = m_jit.label();
</span><del>-
</del><span class="cx"> m_jit.emitStoreCallSiteIndex(callSite);
</span><del>-
- callLinkInfo->setFrameShuffleData(shuffleData);
- CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-
- JITCompiler::Call call = m_jit.nearTailCall();
-
</del><ins>+
+ callLinkInfo->emitDirectTailCallFastPath(m_jit, [&] {
+ callLinkInfo->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+ });
</ins><span class="cx"> JITCompiler::Label slowPath = m_jit.label();
</span><del>- patchableJump.m_jump.linkTo(slowPath, &m_jit);
</del><span class="cx">
</span><span class="cx"> silentSpillAllRegisters(InvalidGPRReg);
</span><span class="cx"> callOperation(operationLinkDirectCall, callLinkInfo, calleeGPR);
</span><span class="lines">@@ -851,15 +847,13 @@
</span><span class="cx">
</span><span class="cx"> useChildren(node);
</span><span class="cx">
</span><del>- m_jit.addJSDirectTailCall(patchableJump, call, slowPath, callLinkInfo);
</del><ins>+ m_jit.addJSDirectCall(slowPath, callLinkInfo);
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> JITCompiler::Label mainPath = m_jit.label();
</span><del>-
</del><span class="cx"> m_jit.emitStoreCallSiteIndex(callSite);
</span><del>-
- JITCompiler::Call call = m_jit.nearCall();
</del><ins>+ callLinkInfo->emitDirectFastPath(m_jit);
</ins><span class="cx"> JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">
</span><span class="cx"> JITCompiler::Label slowPath = m_jit.label();
</span><span class="lines">@@ -874,30 +868,29 @@
</span><span class="cx">
</span><span class="cx"> setResultAndResetStack();
</span><span class="cx">
</span><del>- m_jit.addJSDirectCall(call, slowPath, callLinkInfo);
</del><ins>+ m_jit.addJSDirectCall(slowPath, callLinkInfo);
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">
</span><del>- JITCompiler::DataLabelPtr targetToCheck;
- JITCompiler::Jump slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, TrustedImmPtr(nullptr));
-
</del><ins>+ CCallHelpers::JumpList slowCases;
</ins><span class="cx"> if (isTail) {
</span><del>- if (node->op() == TailCall) {
- callLinkInfo->setFrameShuffleData(shuffleData);
- CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
- } else {
- m_jit.emitRestoreCalleeSaves();
- m_jit.prepareForTailCallSlow();
- }
- }
-
- JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
-
</del><ins>+ slowCases = callLinkInfo->emitTailCallFastPath(m_jit, calleeGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+ if (node->op() == TailCall) {
+ callLinkInfo->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+ } else {
+ m_jit.emitRestoreCalleeSaves();
+ m_jit.prepareForTailCallSlow();
+ }
+ });
+ } else
+ slowCases = callLinkInfo->emitFastPath(m_jit, calleeGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx"> JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">
</span><del>- slowPath.link(&m_jit);
</del><ins>+ slowCases.link(&m_jit);
+ auto slowPathStart = m_jit.label();
</ins><span class="cx">
</span><span class="cx"> if (node->op() == TailCall) {
</span><span class="cx"> CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
</span><span class="lines">@@ -905,16 +898,15 @@
</span><span class="cx"> callFrameShuffler.prepareForSlowPath();
</span><span class="cx"> } else {
</span><span class="cx"> m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0
</span><del>-
</del><span class="cx"> if (isTail)
</span><span class="cx"> m_jit.emitRestoreCalleeSaves(); // This needs to happen after we moved calleeGPR to regT0
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_jit.move(TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2
</del><span class="cx"> m_jit.move(TrustedImmPtr::weakPointer(m_graph, globalObject), GPRInfo::regT3); // JSGlobalObject needs to be in regT3
</span><del>- JITCompiler::Call slowCall = m_jit.nearCall();
</del><ins>+ callLinkInfo->emitSlowPath(vm(), m_jit);
</ins><span class="cx">
</span><span class="cx"> done.link(&m_jit);
</span><ins>+ auto doneLocation = m_jit.label();
</ins><span class="cx">
</span><span class="cx"> if (isTail)
</span><span class="cx"> m_jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="lines">@@ -921,7 +913,7 @@
</span><span class="cx"> else
</span><span class="cx"> setResultAndResetStack();
</span><span class="cx">
</span><del>- m_jit.addJSCall(fastCall, slowCall, targetToCheck, callLinkInfo);
</del><ins>+ m_jit.addJSCall(slowPathStart, doneLocation, callLinkInfo);
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> // Clang should allow unreachable [[clang::fallthrough]] in template functions if any template expansion uses it
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -9717,24 +9717,20 @@
</span><span class="cx"> CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><span class="cx">
</span><span class="cx"> CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><ins>+ callLinkInfo->setUpCall(
+ node->op() == Construct ? CallLinkInfo::Construct : CallLinkInfo::Call, GPRInfo::regT0);
</ins><span class="cx">
</span><del>- CCallHelpers::DataLabelPtr targetToCheck;
- CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
- CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
- CCallHelpers::TrustedImmPtr(nullptr));
-
- CCallHelpers::Call fastCall = jit.nearCall();
</del><ins>+ auto slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx"> CCallHelpers::Jump done = jit.jump();
</span><span class="cx">
</span><span class="cx"> slowPath.link(&jit);
</span><ins>+ auto slowPathStart = jit.label();
+ jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
+ callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">
</span><del>- jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
- jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
- CCallHelpers::Call slowCall = jit.nearCall();
</del><span class="cx"> done.link(&jit);
</span><span class="cx">
</span><del>- callLinkInfo->setUpCall(
- node->op() == Construct ? CallLinkInfo::Construct : CallLinkInfo::Call, GPRInfo::regT0);
</del><ins>+ auto doneLocation = jit.label();
</ins><span class="cx">
</span><span class="cx"> jit.addPtr(
</span><span class="cx"> CCallHelpers::TrustedImm32(-params.proc().frameSize()),
</span><span class="lines">@@ -9742,13 +9738,9 @@
</span><span class="cx">
</span><span class="cx"> jit.addLinkTask(
</span><span class="cx"> [=] (LinkBuffer& linkBuffer) {
</span><del>- MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
- linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
- callLinkInfo->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx">
</span><span class="lines">@@ -9857,63 +9849,50 @@
</span><span class="cx"> shuffleData.setupCalleeSaveRegisters(jit.codeBlock());
</span><span class="cx">
</span><span class="cx"> CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><ins>+ callLinkInfo->setUpCall(CallLinkInfo::DirectTailCall, InvalidGPRReg);
</ins><span class="cx">
</span><del>- CCallHelpers::PatchableJump patchableJump = jit.patchableJump();
</del><span class="cx"> CCallHelpers::Label mainPath = jit.label();
</span><del>-
</del><span class="cx"> jit.store32(
</span><span class="cx"> CCallHelpers::TrustedImm32(callSiteIndex.bits()),
</span><span class="cx"> CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><del>-
- callLinkInfo->setFrameShuffleData(shuffleData);
- CallFrameShuffler(jit, shuffleData).prepareForTailCall();
</del><ins>+ callLinkInfo->emitDirectTailCallFastPath(jit, [&] {
+ callLinkInfo->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(jit, shuffleData).prepareForTailCall();
+ });
</ins><span class="cx">
</span><del>- CCallHelpers::Call call = jit.nearTailCall();
-
</del><span class="cx"> jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">
</span><span class="cx"> CCallHelpers::Label slowPath = jit.label();
</span><del>- patchableJump.m_jump.linkTo(slowPath, &jit);
</del><span class="cx"> callOperation(
</span><span class="cx"> *state, toSave, jit,
</span><span class="cx"> node->origin.semantic, exceptions.get(), operationLinkDirectCall,
</span><span class="cx"> InvalidGPRReg, CCallHelpers::TrustedImmPtr(callLinkInfo), calleeGPR).call();
</span><span class="cx"> jit.jump().linkTo(mainPath, &jit);
</span><del>-
- callLinkInfo->setUpCall(CallLinkInfo::DirectTailCall, InvalidGPRReg);
</del><span class="cx"> callLinkInfo->setExecutableDuringCompilation(executable);
</span><span class="cx"> if (numAllocatedArgs > numPassedArgs)
</span><span class="cx"> callLinkInfo->setMaxArgumentCountIncludingThis(numAllocatedArgs);
</span><span class="cx">
</span><del>- jit.addLinkTask(
- [=] (LinkBuffer& linkBuffer) {
- CodeLocationLabel<JSInternalPtrTag> patchableJumpLocation = linkBuffer.locationOf<JSInternalPtrTag>(patchableJump);
- CodeLocationNearCall<JSInternalPtrTag> callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
- CodeLocationLabel<JSInternalPtrTag> slowPathLocation = linkBuffer.locationOf<JSInternalPtrTag>(slowPath);
-
- callLinkInfo->setCallLocations(
- patchableJumpLocation,
- slowPathLocation,
- callLocation);
- });
</del><ins>+ jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPath),
+ CodeLocationLabel<JSInternalPtrTag>());
+ });
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><del>-
</del><ins>+ callLinkInfo->setUpCall(
+ isConstruct ? CallLinkInfo::DirectConstruct : CallLinkInfo::DirectCall, InvalidGPRReg);
+
</ins><span class="cx"> CCallHelpers::Label mainPath = jit.label();
</span><del>-
</del><span class="cx"> jit.store32(
</span><span class="cx"> CCallHelpers::TrustedImm32(callSiteIndex.bits()),
</span><span class="cx"> CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><del>-
- CCallHelpers::Call call = jit.nearCall();
</del><ins>+ callLinkInfo->emitDirectFastPath(jit);
</ins><span class="cx"> jit.addPtr(
</span><span class="cx"> CCallHelpers::TrustedImm32(-params.proc().frameSize()),
</span><span class="cx"> GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="cx">
</span><del>- callLinkInfo->setUpCall(
- isConstruct ? CallLinkInfo::DirectConstruct : CallLinkInfo::DirectCall, InvalidGPRReg);
</del><span class="cx"> callLinkInfo->setExecutableDuringCompilation(executable);
</span><span class="cx"> if (numAllocatedArgs > numPassedArgs)
</span><span class="cx"> callLinkInfo->setMaxArgumentCountIncludingThis(numAllocatedArgs);
</span><span class="lines">@@ -9935,15 +9914,9 @@
</span><span class="cx">
</span><span class="cx"> jit.addLinkTask(
</span><span class="cx"> [=] (LinkBuffer& linkBuffer) {
</span><del>- CodeLocationNearCall<JSInternalPtrTag> callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
- CodeLocationLabel<JSInternalPtrTag> slowPathLocation = linkBuffer.locationOf<JSInternalPtrTag>(slowPath);
-
- linkBuffer.link(call, slowPathLocation);
-
- callLinkInfo->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(),
- slowPathLocation,
- callLocation);
</del><ins>+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPath),
+ CodeLocationLabel<JSInternalPtrTag>());
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx"> });
</span><span class="lines">@@ -10034,40 +10007,30 @@
</span><span class="cx"> shuffleData.setupCalleeSaveRegisters(jit.codeBlock());
</span><span class="cx">
</span><span class="cx"> CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(codeOrigin);
</span><ins>+ callLinkInfo->setUpCall(CallLinkInfo::TailCall, GPRInfo::regT0);
</ins><span class="cx">
</span><del>- CCallHelpers::DataLabelPtr targetToCheck;
- CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
- CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
- CCallHelpers::TrustedImmPtr(nullptr));
</del><ins>+ auto slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+ callLinkInfo->setFrameShuffleData(shuffleData);
+ CallFrameShuffler(jit, shuffleData).prepareForTailCall();
+ });
</ins><span class="cx">
</span><del>- callLinkInfo->setFrameShuffleData(shuffleData);
- CallFrameShuffler(jit, shuffleData).prepareForTailCall();
-
- CCallHelpers::Call fastCall = jit.nearTailCall();
-
</del><span class="cx"> slowPath.link(&jit);
</span><del>-
</del><ins>+ auto slowPathStart = jit.label();
</ins><span class="cx"> CallFrameShuffler slowPathShuffler(jit, shuffleData);
</span><span class="cx"> slowPathShuffler.setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT0));
</span><span class="cx"> slowPathShuffler.prepareForSlowPath();
</span><span class="cx">
</span><del>- jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx"> jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>- CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+ callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">
</span><ins>+ auto doneLocation = jit.label();
</ins><span class="cx"> jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">
</span><del>- callLinkInfo->setUpCall(CallLinkInfo::TailCall, GPRInfo::regT0);
-
</del><span class="cx"> jit.addLinkTask(
</span><span class="cx"> [=] (LinkBuffer& linkBuffer) {
</span><del>- MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
- linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
- callLinkInfo->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx"> }
</span><span class="lines">@@ -10320,41 +10283,39 @@
</span><span class="cx"> else
</span><span class="cx"> callType = CallLinkInfo::CallVarargs;
</span><span class="cx">
</span><ins>+ callLinkInfo->setUpCall(callType, GPRInfo::regT0);
+
</ins><span class="cx"> bool isTailCall = CallLinkInfo::callModeFor(callType) == CallMode::Tail;
</span><del>-
- CCallHelpers::DataLabelPtr targetToCheck;
- CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
- CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
- CCallHelpers::TrustedImmPtr(nullptr));
-
- CCallHelpers::Call fastCall;
</del><ins>+
+ ASSERT(!usedRegisters.get(GPRInfo::regT2)); // Used on the slow path.
+
+ CCallHelpers::JumpList slowPath;
</ins><span class="cx"> CCallHelpers::Jump done;
</span><del>-
</del><span class="cx"> if (isTailCall) {
</span><del>- jit.emitRestoreCalleeSaves();
- jit.prepareForTailCallSlow();
- fastCall = jit.nearTailCall();
</del><ins>+ slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+ jit.emitRestoreCalleeSaves();
+ jit.prepareForTailCallSlow();
+ });
</ins><span class="cx"> } else {
</span><del>- fastCall = jit.nearCall();
</del><ins>+ slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx"> done = jit.jump();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> slowPath.link(&jit);
</span><ins>+ auto slowPathStart = jit.label();
</ins><span class="cx">
</span><span class="cx"> if (isTailCall)
</span><span class="cx"> jit.emitRestoreCalleeSaves();
</span><del>- ASSERT(!usedRegisters.get(GPRInfo::regT2));
- jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx"> jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>- CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+ callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">
</span><span class="cx"> if (isTailCall)
</span><span class="cx"> jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx"> else
</span><span class="cx"> done.link(&jit);
</span><del>-
- callLinkInfo->setUpCall(callType, GPRInfo::regT0);
</del><span class="cx">
</span><ins>+ auto doneLocation = jit.label();
+
</ins><span class="cx"> jit.addPtr(
</span><span class="cx"> CCallHelpers::TrustedImm32(-originalStackHeight),
</span><span class="cx"> GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="lines">@@ -10361,13 +10322,9 @@
</span><span class="cx">
</span><span class="cx"> jit.addLinkTask(
</span><span class="cx"> [=] (LinkBuffer& linkBuffer) {
</span><del>- MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
- linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
- callLinkInfo->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx">
</span><span class="lines">@@ -10607,40 +10564,37 @@
</span><span class="cx"> else
</span><span class="cx"> callType = CallLinkInfo::CallVarargs;
</span><span class="cx">
</span><ins>+ callLinkInfo->setUpCall(callType, GPRInfo::regT0);
+
</ins><span class="cx"> bool isTailCall = CallLinkInfo::callModeFor(callType) == CallMode::Tail;
</span><span class="cx">
</span><del>- CCallHelpers::DataLabelPtr targetToCheck;
- CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
- CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
- CCallHelpers::TrustedImmPtr(nullptr));
-
- CCallHelpers::Call fastCall;
</del><ins>+ CCallHelpers::JumpList slowPath;
</ins><span class="cx"> CCallHelpers::Jump done;
</span><del>-
</del><span class="cx"> if (isTailCall) {
</span><del>- jit.emitRestoreCalleeSaves();
- jit.prepareForTailCallSlow();
- fastCall = jit.nearTailCall();
</del><ins>+ slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+ jit.emitRestoreCalleeSaves();
+ jit.prepareForTailCallSlow();
+ });
</ins><span class="cx"> } else {
</span><del>- fastCall = jit.nearCall();
</del><ins>+ slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx"> done = jit.jump();
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> slowPath.link(&jit);
</span><ins>+ auto slowPathStart = jit.label();
</ins><span class="cx">
</span><span class="cx"> if (isTailCall)
</span><span class="cx"> jit.emitRestoreCalleeSaves();
</span><del>- jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx"> jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>- CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+ callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">
</span><span class="cx"> if (isTailCall)
</span><span class="cx"> jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx"> else
</span><span class="cx"> done.link(&jit);
</span><ins>+
+ auto doneLocation = jit.label();
</ins><span class="cx">
</span><del>- callLinkInfo->setUpCall(callType, GPRInfo::regT0);
-
</del><span class="cx"> jit.addPtr(
</span><span class="cx"> CCallHelpers::TrustedImm32(-originalStackHeight),
</span><span class="cx"> GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="lines">@@ -10647,13 +10601,9 @@
</span><span class="cx">
</span><span class="cx"> jit.addLinkTask(
</span><span class="cx"> [=] (LinkBuffer& linkBuffer) {
</span><del>- MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
- linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
- callLinkInfo->setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
- CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
- linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+ callLinkInfo->setCodeLocations(
+ linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> });
</span><span class="cx"> });
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitCCallHelperscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -71,6 +71,16 @@
</span><span class="cx"> storePtr(scratch2, Address(scratch1NonArgGPR));
</span><span class="cx"> }
</span><span class="cx">
</span><ins>+void CCallHelpers::emitJITCodeOver(MacroAssemblerCodePtr<JSInternalPtrTag> where, WTF::Function<void(CCallHelpers&)> emitCode, const char* description)
+{
+ CCallHelpers jit;
+ emitCode(jit);
+
+ constexpr bool needsBranchCompaction = false;
+ LinkBuffer linkBuffer(jit, where, jit.m_assembler.buffer().codeSize(), LinkBuffer::Profile::InlineCache, JITCompilationMustSucceed, needsBranchCompaction);
+ FINALIZE_CODE(linkBuffer, NoPtrTag, description);
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx">
</span><span class="cx"> #endif // ENABLE(JIT)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitCCallHelpersh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/CCallHelpers.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/CCallHelpers.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/CCallHelpers.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -818,6 +818,8 @@
</span><span class="cx"> void logShadowChickenTailPacket(GPRReg shadowPacket, JSValueRegs thisRegs, GPRReg scope, CodeBlock*, CallSiteIndex);
</span><span class="cx"> // Leaves behind a pointer to the Packet we should write to in shadowPacket.
</span><span class="cx"> void ensureShadowChickenPacket(VM&, GPRReg shadowPacket, GPRReg scratch1NonArgGPR, GPRReg scratch2);
</span><ins>+
+ static void emitJITCodeOver(MacroAssemblerCodePtr<JSInternalPtrTag> where, WTF::Function<void(CCallHelpers&)>, const char*);
</ins><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JIT.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JIT.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JIT.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -963,10 +963,9 @@
</span><span class="cx">
</span><span class="cx"> for (auto& compilationInfo : m_callCompilationInfo) {
</span><span class="cx"> CallLinkInfo& info = *compilationInfo.callLinkInfo;
</span><del>- info.setCallLocations(
- CodeLocationLabel<JSInternalPtrTag>(patchBuffer.locationOfNearCall<JSInternalPtrTag>(compilationInfo.callReturnLocation)),
- CodeLocationLabel<JSInternalPtrTag>(patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.hotPathBegin)),
- patchBuffer.locationOfNearCall<JSInternalPtrTag>(compilationInfo.hotPathOther));
</del><ins>+ info.setCodeLocations(
+ patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.slowPathStart),
+ patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.doneLocation));
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JIT.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JIT.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JIT.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -186,9 +186,8 @@
</span><span class="cx"> };
</span><span class="cx">
</span><span class="cx"> struct CallCompilationInfo {
</span><del>- MacroAssembler::DataLabelPtr hotPathBegin;
- MacroAssembler::Call hotPathOther;
- MacroAssembler::Call callReturnLocation;
</del><ins>+ MacroAssembler::Label slowPathStart;
+ MacroAssembler::Label doneLocation;
</ins><span class="cx"> CallLinkInfo* callLinkInfo;
</span><span class="cx"> };
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITCallcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITCall.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITCall.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITCall.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -188,8 +188,14 @@
</span><span class="cx"> ValueRecovery::inGPR(regT0, DataFormatJS);
</span><span class="cx"> shuffleData.setupCalleeSaveRegisters(m_codeBlock);
</span><span class="cx"> info->setFrameShuffleData(shuffleData);
</span><del>- CallFrameShuffler(*this, shuffleData).prepareForTailCall();
- m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+
+ JumpList slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+ CallFrameShuffler(*this, shuffleData).prepareForTailCall();
+ });
+ addSlowCase(slowPaths);
+ auto doneLocation = label();
+ m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
+
</ins><span class="cx"> return true;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -228,29 +234,31 @@
</span><span class="cx"> if (compileCallEval(bytecode))
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- DataLabelPtr addressOfLinkedFunctionCheck;
- Jump slowCase = branchPtrWithPatch(NotEqual, regT0, addressOfLinkedFunctionCheck, TrustedImmPtr(nullptr));
- addSlowCase(slowCase);
-
</del><span class="cx"> ASSERT(m_callCompilationInfo.size() == callLinkInfoIndex);
</span><span class="cx"> info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</span><span class="cx"> m_callCompilationInfo.append(CallCompilationInfo());
</span><del>- m_callCompilationInfo[callLinkInfoIndex].hotPathBegin = addressOfLinkedFunctionCheck;
</del><span class="cx"> m_callCompilationInfo[callLinkInfoIndex].callLinkInfo = info;
</span><span class="cx">
</span><del>- if (compileTailCall(bytecode, info, callLinkInfoIndex)) {
</del><ins>+ if (compileTailCall(bytecode, info, callLinkInfoIndex))
</ins><span class="cx"> return;
</span><del>- }
</del><span class="cx">
</span><span class="cx"> if (opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</span><del>- emitRestoreCalleeSaves();
- prepareForTailCallSlow();
- m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+ auto slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+ emitRestoreCalleeSaves();
+ prepareForTailCallSlow(regT2);
+ });
+ addSlowCase(slowPaths);
+ auto doneLocation = label();
+ m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearCall();
</del><ins>+ auto slowPaths = info->emitFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes);
+ auto doneLocation = label();
+ addSlowCase(slowPaths);
</ins><span class="cx">
</span><ins>+ m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
+
</ins><span class="cx"> addPtr(TrustedImm32(stackPointerOffsetFor(m_codeBlock) * sizeof(Register)), callFrameRegister, stackPointerRegister);
</span><span class="cx"> checkStackPointerAlignment();
</span><span class="cx">
</span><span class="lines">@@ -265,16 +273,15 @@
</span><span class="cx">
</span><span class="cx"> linkAllSlowCases(iter);
</span><span class="cx">
</span><ins>+ m_callCompilationInfo[callLinkInfoIndex].slowPathStart = label();
+
</ins><span class="cx"> if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments)
</span><span class="cx"> emitRestoreCalleeSaves();
</span><span class="cx">
</span><span class="cx"> move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
</span><del>- move(TrustedImmPtr(m_callCompilationInfo[callLinkInfoIndex].callLinkInfo), regT2);
</del><ins>+ m_callCompilationInfo[callLinkInfoIndex].callLinkInfo->emitSlowPath(*m_vm, *this);
</ins><span class="cx">
</span><del>- m_callCompilationInfo[callLinkInfoIndex].callReturnLocation =
- emitNakedNearCall(m_vm->getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
-
- if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) {
</del><ins>+ if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</ins><span class="cx"> abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx"> return;
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITCall32_64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -301,30 +301,28 @@
</span><span class="cx"> if (compileCallEval(bytecode))
</span><span class="cx"> return;
</span><span class="cx">
</span><del>- if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs)
- emitRestoreCalleeSaves();
-
- addSlowCase(branchIfNotCell(regT1));
-
- DataLabelPtr addressOfLinkedFunctionCheck;
- Jump slowCase = branchPtrWithPatch(NotEqual, regT0, addressOfLinkedFunctionCheck, TrustedImmPtr(nullptr));
-
- addSlowCase(slowCase);
-
</del><ins>+ info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</ins><span class="cx"> ASSERT(m_callCompilationInfo.size() == callLinkInfoIndex);
</span><del>- info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</del><span class="cx"> m_callCompilationInfo.append(CallCompilationInfo());
</span><del>- m_callCompilationInfo[callLinkInfoIndex].hotPathBegin = addressOfLinkedFunctionCheck;
</del><span class="cx"> m_callCompilationInfo[callLinkInfoIndex].callLinkInfo = info;
</span><span class="cx">
</span><ins>+ addSlowCase(branchIfNotCell(regT1));
+
</ins><span class="cx"> checkStackPointerAlignment();
</span><span class="cx"> if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</span><del>- prepareForTailCallSlow();
- m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+ auto slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+ emitRestoreCalleeSaves();
+ prepareForTailCallSlow(regT2);
+ });
+ addSlowCase(slowPaths);
+ auto doneLocation = label();
+ m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><del>- m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearCall();
</del><ins>+ auto slowPaths = info->emitFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes);
+ addSlowCase(slowPaths);
+ m_callCompilationInfo[callLinkInfoIndex].doneLocation = label();
</ins><span class="cx">
</span><span class="cx"> addPtr(TrustedImm32(stackPointerOffsetFor(m_codeBlock) * sizeof(Register)), callFrameRegister, stackPointerRegister);
</span><span class="cx"> checkStackPointerAlignment();
</span><span class="lines">@@ -343,16 +341,15 @@
</span><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> linkAllSlowCases(iter);
</span><ins>+ m_callCompilationInfo[callLinkInfoIndex].slowPathStart = label();
</ins><span class="cx">
</span><del>- move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
- move(TrustedImmPtr(m_callCompilationInfo[callLinkInfoIndex].callLinkInfo), regT2);
-
</del><span class="cx"> if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments)
</span><span class="cx"> emitRestoreCalleeSaves();
</span><span class="cx">
</span><del>- m_callCompilationInfo[callLinkInfoIndex].callReturnLocation = emitNakedNearCall(m_vm->getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
</del><ins>+ move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
+ m_callCompilationInfo[callLinkInfoIndex].callLinkInfo->emitSlowPath(*m_vm, *this);
</ins><span class="cx">
</span><del>- if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) {
</del><ins>+ if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</ins><span class="cx"> abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx"> return;
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITOperations.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITOperations.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITOperations.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1386,7 +1386,7 @@
</span><span class="cx"> if (!callLinkInfo->seenOnce())
</span><span class="cx"> callLinkInfo->setSeen();
</span><span class="cx"> else
</span><del>- linkFor(vm, calleeFrame, *callLinkInfo, nullptr, internalFunction, codePtr);
</del><ins>+ linkMonomorphicCall(vm, calleeFrame, *callLinkInfo, nullptr, internalFunction, codePtr);
</ins><span class="cx">
</span><span class="cx"> void* linkedTarget = codePtr.executableAddress();
</span><span class="cx"> return encodeResult(linkedTarget, reinterpret_cast<void*>(callLinkInfo->callMode() == CallMode::Tail ? ReuseTheFrame : KeepTheFrame));
</span><span class="lines">@@ -1434,7 +1434,7 @@
</span><span class="cx"> if (!callLinkInfo->seenOnce())
</span><span class="cx"> callLinkInfo->setSeen();
</span><span class="cx"> else
</span><del>- linkFor(vm, calleeFrame, *callLinkInfo, codeBlock, callee, codePtr);
</del><ins>+ linkMonomorphicCall(vm, calleeFrame, *callLinkInfo, codeBlock, callee, codePtr);
</ins><span class="cx">
</span><span class="cx"> return encodeResult(codePtr.executableAddress(), reinterpret_cast<void*>(callLinkInfo->callMode() == CallMode::Tail ? ReuseTheFrame : KeepTheFrame));
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitPolymorphicCallStubRoutinecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -45,7 +45,7 @@
</span><span class="cx"> void PolymorphicCallNode::unlink(VM& vm)
</span><span class="cx"> {
</span><span class="cx"> if (m_callLinkInfo) {
</span><del>- dataLogLnIf(Options::dumpDisassembly(), "Unlinking polymorphic call at ", m_callLinkInfo->callReturnLocation(), ", bc#", m_callLinkInfo->codeOrigin().bytecodeIndex());
</del><ins>+ dataLogLnIf(Options::dumpDisassembly(), "Unlinking polymorphic call at ", m_callLinkInfo->doneLocation(), ", bc#", m_callLinkInfo->codeOrigin().bytecodeIndex());
</ins><span class="cx"> m_callLinkInfo->unlink(vm);
</span><span class="cx"> }
</span><span class="cx">
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitRepatchcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/Repatch.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/Repatch.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/Repatch.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1188,20 +1188,20 @@
</span><span class="cx"> ftlThunkAwareRepatchCall(codeBlock, stubInfo.slowPathCallLocation, operationInstanceOfGeneric);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-static void linkSlowFor(VM&, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef<JITStubRoutinePtrTag> codeRef)
</del><ins>+static void linkSlowPathTo(VM&, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef<JITStubRoutinePtrTag> codeRef)
</ins><span class="cx"> {
</span><del>- MacroAssembler::repatchNearCall(callLinkInfo.callReturnLocation(), CodeLocationLabel<JITStubRoutinePtrTag>(codeRef.code()));
</del><ins>+ callLinkInfo.setSlowPathCallDestination(codeRef.code().template retagged<JSEntryPtrTag>());
</ins><span class="cx"> }
</span><span class="cx">
</span><del>-static void linkSlowFor(VM& vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
</del><ins>+static void linkSlowPathTo(VM& vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
</ins><span class="cx"> {
</span><del>- linkSlowFor(vm, callLinkInfo, vm.getCTIStub(generator).retagged<JITStubRoutinePtrTag>());
</del><ins>+ linkSlowPathTo(vm, callLinkInfo, vm.getCTIStub(generator).retagged<JITStubRoutinePtrTag>());
</ins><span class="cx"> }
</span><span class="cx">
</span><span class="cx"> static void linkSlowFor(VM& vm, CallLinkInfo& callLinkInfo)
</span><span class="cx"> {
</span><span class="cx"> MacroAssemblerCodeRef<JITStubRoutinePtrTag> virtualThunk = virtualThunkFor(vm, callLinkInfo);
</span><del>- linkSlowFor(vm, callLinkInfo, virtualThunk);
</del><ins>+ linkSlowPathTo(vm, callLinkInfo, virtualThunk);
</ins><span class="cx"> callLinkInfo.setSlowStub(GCAwareJITStubRoutine::create(virtualThunk, vm));
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -1217,7 +1217,7 @@
</span><span class="cx"> #endif // ENABLE(WEBASSEMBLY)
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void linkFor(
</del><ins>+void linkMonomorphicCall(
</ins><span class="cx"> VM& vm, CallFrame* callFrame, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock,
</span><span class="cx"> JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</span><span class="cx"> {
</span><span class="lines">@@ -1235,19 +1235,17 @@
</span><span class="cx"> ASSERT(owner);
</span><span class="cx">
</span><span class="cx"> ASSERT(!callLinkInfo.isLinked());
</span><del>- callLinkInfo.setCallee(vm, owner, callee);
- MacroAssembler::repatchPointer(callLinkInfo.hotPathBegin(), callee);
</del><ins>+ callLinkInfo.setMonomorphicCallee(vm, owner, callee, codePtr);
</ins><span class="cx"> callLinkInfo.setLastSeenCallee(vm, owner, callee);
</span><ins>+
</ins><span class="cx"> if (shouldDumpDisassemblyFor(callerCodeBlock))
</span><span class="cx"> dataLog("Linking call in ", FullCodeOrigin(callerCodeBlock, callLinkInfo.codeOrigin()), " to ", pointerDump(calleeCodeBlock), ", entrypoint at ", codePtr, "\n");
</span><span class="cx">
</span><del>- MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), CodeLocationLabel<JSEntryPtrTag>(codePtr));
-
</del><span class="cx"> if (calleeCodeBlock)
</span><span class="cx"> calleeCodeBlock->linkIncomingCall(callerFrame, &callLinkInfo);
</span><span class="cx">
</span><span class="cx"> if (callLinkInfo.specializationKind() == CodeForCall && callLinkInfo.allowStubs()) {
</span><del>- linkSlowFor(vm, callLinkInfo, linkPolymorphicCallThunkGenerator);
</del><ins>+ linkSlowPathTo(vm, callLinkInfo, linkPolymorphicCallThunkGenerator);
</ins><span class="cx"> return;
</span><span class="cx"> }
</span><span class="cx">
</span><span class="lines">@@ -1254,7 +1252,7 @@
</span><span class="cx"> linkSlowFor(vm, callLinkInfo);
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void linkDirectFor(
</del><ins>+void linkDirectCall(
</ins><span class="cx"> CallFrame* callFrame, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock,
</span><span class="cx"> MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</span><span class="cx"> {
</span><span class="lines">@@ -1269,9 +1267,7 @@
</span><span class="cx"> if (shouldDumpDisassemblyFor(callerCodeBlock))
</span><span class="cx"> dataLog("Linking call in ", FullCodeOrigin(callerCodeBlock, callLinkInfo.codeOrigin()), " to ", pointerDump(calleeCodeBlock), ", entrypoint at ", codePtr, "\n");
</span><span class="cx">
</span><del>- if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
- MacroAssembler::repatchJumpToNop(callLinkInfo.patchableJump());
- MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), CodeLocationLabel<JSEntryPtrTag>(codePtr));
</del><ins>+ callLinkInfo.setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag>(codePtr));
</ins><span class="cx">
</span><span class="cx"> if (calleeCodeBlock)
</span><span class="cx"> calleeCodeBlock->linkIncomingCall(callFrame, &callLinkInfo);
</span><span class="lines">@@ -1289,21 +1285,16 @@
</span><span class="cx"> {
</span><span class="cx"> if (callLinkInfo.isDirect()) {
</span><span class="cx"> callLinkInfo.clearCodeBlock();
</span><del>- if (!callLinkInfo.clearedByJettison()) {
- if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
- MacroAssembler::repatchJump(callLinkInfo.patchableJump(), callLinkInfo.slowPathStart());
- else
- MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), callLinkInfo.slowPathStart());
- }
</del><ins>+ if (!callLinkInfo.clearedByJettison())
+ callLinkInfo.initializeDirectCall();
</ins><span class="cx"> } else {
</span><span class="cx"> if (!callLinkInfo.clearedByJettison()) {
</span><del>- MacroAssembler::revertJumpReplacementToBranchPtrWithPatch(
- MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
- callLinkInfo.calleeGPR(), nullptr);
- linkSlowFor(vm, callLinkInfo, codeRef);
- MacroAssembler::repatchPointer(callLinkInfo.hotPathBegin(), nullptr);
</del><ins>+ linkSlowPathTo(vm, callLinkInfo, codeRef);
+
+ if (callLinkInfo.stub())
+ callLinkInfo.revertCallToStub();
</ins><span class="cx"> }
</span><del>- callLinkInfo.clearCallee();
</del><ins>+ callLinkInfo.clearCallee(); // This also clears the inline cache both for data and code-based caches.
</ins><span class="cx"> }
</span><span class="cx"> callLinkInfo.clearSeen();
</span><span class="cx"> callLinkInfo.clearStub();
</span><span class="lines">@@ -1312,9 +1303,9 @@
</span><span class="cx"> callLinkInfo.remove();
</span><span class="cx"> }
</span><span class="cx">
</span><del>-void unlinkFor(VM& vm, CallLinkInfo& callLinkInfo)
</del><ins>+void unlinkCall(VM& vm, CallLinkInfo& callLinkInfo)
</ins><span class="cx"> {
</span><del>- dataLogLnIf(Options::dumpDisassembly(), "Unlinking call at ", callLinkInfo.hotPathOther());
</del><ins>+ dataLogLnIf(Options::dumpDisassembly(), "Unlinking call at ", callLinkInfo.fastPathStart());
</ins><span class="cx">
</span><span class="cx"> revertCall(vm, callLinkInfo, vm.getCTIStub(linkCallThunkGenerator).retagged<JITStubRoutinePtrTag>());
</span><span class="cx"> }
</span><span class="lines">@@ -1580,7 +1571,7 @@
</span><span class="cx"> }
</span><span class="cx"> stubJit.move(CCallHelpers::TrustedImmPtr(globalObject), GPRInfo::regT3);
</span><span class="cx"> stubJit.move(CCallHelpers::TrustedImmPtr(&callLinkInfo), GPRInfo::regT2);
</span><del>- stubJit.move(CCallHelpers::TrustedImmPtr(callLinkInfo.callReturnLocation().untaggedExecutableAddress()), GPRInfo::regT4);
</del><ins>+ stubJit.move(CCallHelpers::TrustedImmPtr(callLinkInfo.doneLocation().untaggedExecutableAddress()), GPRInfo::regT4);
</ins><span class="cx">
</span><span class="cx"> stubJit.restoreReturnAddressBeforeReturn(GPRInfo::regT4);
</span><span class="cx"> AssemblyHelpers::Jump slow = stubJit.jump();
</span><span class="lines">@@ -1603,10 +1594,7 @@
</span><span class="cx"> patchBuffer.link(callToCodePtr.call, FunctionPtr<JSEntryPtrTag>(callToCodePtr.codePtr));
</span><span class="cx"> #endif
</span><span class="cx"> }
</span><del>- if (isWebAssembly || JITCode::isOptimizingJIT(callerCodeBlock->jitType()))
- patchBuffer.link(done, callLinkInfo.callReturnLocation().labelAtOffset(0));
- else
- patchBuffer.link(done, callLinkInfo.hotPathOther().labelAtOffset(0));
</del><ins>+ patchBuffer.link(done, callLinkInfo.doneLocation());
</ins><span class="cx"> patchBuffer.link(slow, CodeLocationLabel<JITThunkPtrTag>(vm.getCTIStub(linkPolymorphicCallThunkGenerator).code()));
</span><span class="cx">
</span><span class="cx"> auto stubRoutine = adoptRef(*new PolymorphicCallStubRoutine(
</span><span class="lines">@@ -1613,19 +1601,16 @@
</span><span class="cx"> FINALIZE_CODE_FOR(
</span><span class="cx"> callerCodeBlock, patchBuffer, JITStubRoutinePtrTag,
</span><span class="cx"> "Polymorphic call stub for %s, return point %p, targets %s",
</span><del>- isWebAssembly ? "WebAssembly" : toCString(*callerCodeBlock).data(), callLinkInfo.callReturnLocation().labelAtOffset(0).executableAddress(),
</del><ins>+ isWebAssembly ? "WebAssembly" : toCString(*callerCodeBlock).data(), callLinkInfo.doneLocation().executableAddress(),
</ins><span class="cx"> toCString(listDump(callCases)).data()),
</span><span class="cx"> vm, owner, callFrame->callerFrame(), callLinkInfo, callCases,
</span><span class="cx"> WTFMove(fastCounts)));
</span><del>-
- MacroAssembler::replaceWithJump(
- MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
- CodeLocationLabel<JITStubRoutinePtrTag>(stubRoutine->code().code()));
</del><ins>+
</ins><span class="cx"> // The original slow path is unreachable on 64-bits, but still
</span><span class="cx"> // reachable on 32-bits since a non-cell callee will always
</span><span class="cx"> // trigger the slow path
</span><span class="cx"> linkSlowFor(vm, callLinkInfo);
</span><del>-
</del><ins>+
</ins><span class="cx"> // If there had been a previous stub routine, that one will die as soon as the GC runs and sees
</span><span class="cx"> // that it's no longer on stack.
</span><span class="cx"> callLinkInfo.setStub(WTFMove(stubRoutine));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitRepatchh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/Repatch.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/Repatch.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/Repatch.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -56,10 +56,10 @@
</span><span class="cx"> void repatchCheckPrivateBrand(JSGlobalObject*, CodeBlock*, JSObject*, CacheableIdentifier, StructureStubInfo&);
</span><span class="cx"> void repatchSetPrivateBrand(JSGlobalObject*, CodeBlock*, JSObject*, Structure*, CacheableIdentifier, StructureStubInfo&);
</span><span class="cx"> void repatchInstanceOf(JSGlobalObject*, CodeBlock*, JSValue value, JSValue prototype, StructureStubInfo&, bool wasFound);
</span><del>-void linkFor(VM&, CallFrame*, CallLinkInfo&, CodeBlock*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
-void linkDirectFor(CallFrame*, CallLinkInfo&, CodeBlock*, MacroAssemblerCodePtr<JSEntryPtrTag>);
</del><ins>+void linkMonomorphicCall(VM&, CallFrame*, CallLinkInfo&, CodeBlock*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
+void linkDirectCall(CallFrame*, CallLinkInfo&, CodeBlock*, MacroAssemblerCodePtr<JSEntryPtrTag>);
</ins><span class="cx"> void linkSlowFor(CallFrame*, CallLinkInfo&);
</span><del>-void unlinkFor(VM&, CallLinkInfo&);
</del><ins>+void unlinkCall(VM&, CallLinkInfo&);
</ins><span class="cx"> void linkPolymorphicCall(JSGlobalObject*, CallFrame*, CallLinkInfo&, CallVariant);
</span><span class="cx"> void resetGetBy(CodeBlock*, StructureStubInfo&, GetByKind);
</span><span class="cx"> void resetPutByID(CodeBlock*, StructureStubInfo&);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorewasmjsWasmToJScpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -261,19 +261,18 @@
</span><span class="cx">
</span><span class="cx"> CallLinkInfo* callLinkInfo = callLinkInfos.add(CodeOrigin());
</span><span class="cx"> callLinkInfo->setUpCall(CallLinkInfo::Call, importJSCellGPRReg);
</span><del>- JIT::DataLabelPtr targetToCheck;
- JIT::TrustedImmPtr initialRightValue(nullptr);
- JIT::Jump slowPath = jit.branchPtrWithPatch(MacroAssembler::NotEqual, importJSCellGPRReg, targetToCheck, initialRightValue);
- JIT::Call fastCall = jit.nearCall();
</del><ins>+ auto slowPath = callLinkInfo->emitFastPath(jit, importJSCellGPRReg, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
+
</ins><span class="cx"> JIT::Jump done = jit.jump();
</span><span class="cx"> slowPath.link(&jit);
</span><ins>+ auto slowPathStart = jit.label();
</ins><span class="cx"> // Callee needs to be in regT0 here.
</span><del>- jit.move(MacroAssembler::TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2.
</del><span class="cx"> jit.loadWasmContextInstance(GPRInfo::regT3);
</span><span class="cx"> jit.loadPtr(CCallHelpers::Address(GPRInfo::regT3, Instance::offsetOfOwner()), GPRInfo::regT3);
</span><span class="cx"> jit.loadPtr(CCallHelpers::Address(GPRInfo::regT3, JSWebAssemblyInstance::offsetOfGlobalObject()), GPRInfo::regT3);
</span><del>- JIT::Call slowCall = jit.nearCall();
</del><ins>+ callLinkInfo->emitSlowPath(vm, jit);
</ins><span class="cx"> done.link(&jit);
</span><ins>+ auto doneLocation = jit.label();
</ins><span class="cx">
</span><span class="cx"> if (signature.returnCount() == 1) {
</span><span class="cx"> switch (signature.returnType(0).kind) {
</span><span class="lines">@@ -421,11 +420,9 @@
</span><span class="cx"> if (UNLIKELY(patchBuffer.didFailToAllocate()))
</span><span class="cx"> return makeUnexpected(BindingFailure::OutOfMemory);
</span><span class="cx">
</span><del>- patchBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(vm.getCTIStub(linkCallThunkGenerator).code()));
- CodeLocationLabel<JSInternalPtrTag> callReturnLocation(patchBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall));
- CodeLocationLabel<JSInternalPtrTag> hotPathBegin(patchBuffer.locationOf<JSInternalPtrTag>(targetToCheck));
- CodeLocationNearCall<JSInternalPtrTag> hotPathOther = patchBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall);
- callLinkInfo->setCallLocations(callReturnLocation, hotPathBegin, hotPathOther);
</del><ins>+ callLinkInfo->setCodeLocations(
+ patchBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+ patchBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">
</span><span class="cx"> return FINALIZE_WASM_CODE(patchBuffer, WasmEntryPtrTag, "WebAssembly->JavaScript import[%i] %s", importIndex, signature.toString().ascii().data());
</span><span class="cx"> }
</span></span></pre>
</div>
</div>
</body>
</html>