<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[277680] trunk/Source/JavaScriptCore</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/277680">277680</a></dd>
<dt>Author</dt> <dd>sbarati@apple.com</dd>
<dt>Date</dt> <dd>2021-05-18 14:30:48 -0700 (Tue, 18 May 2021)</dd>
</dl>

<h3>Log Message</h3>
<pre>Add Data Call ICs that don't repatch and use them in the baseline JIT
https://bugs.webkit.org/show_bug.cgi?id=225321
<rdar://problem/77773796>

Reviewed by Michael Saboff.

This patch adds Data ICs for calls. Data ICs for calls work by loading a code
pointer from CallLinkInfo, and indirect calling that pointer. This means that
to repatch such an IC, all we need to do is replace a code pointer inside
CallLinkInfo. No need to repatch the JIT code.

The current implementation only does this for monomorphic calls. We still
repatch the JIT code for polymorphic calls. In a followup, we will also
opt polymorphic call ICs into data-based calls:
https://bugs.webkit.org/show_bug.cgi?id=225793

This patch only uses Data Call ICs for the Baseline JIT. Even with that, it
reduces the number of calls to cacheFlush by ~45% on JetStream2.

Performance is neutral on AS Macs, but it paves the way towards doing
unlinked JITting in JSC.

* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::addLateLinkTask):
* assembler/LinkBuffer.cpp:
(JSC::LinkBuffer::linkCode):
(JSC::LinkBuffer::performFinalization):
* assembler/LinkBuffer.h:
* bytecode/AccessCase.cpp:
(JSC::AccessCase::generateImpl):
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::CallLinkInfo):
(JSC::CallLinkInfo::unlink):
(JSC::CallLinkInfo::fastPathStart):
(JSC::CallLinkInfo::slowPathStart):
(JSC::CallLinkInfo::doneLocation):
(JSC::CallLinkInfo::setMonomorphicCallee):
(JSC::CallLinkInfo::clearCallee):
(JSC::CallLinkInfo::emitFirstInstructionForDataIC):
(JSC::CallLinkInfo::emitFastPathImpl):
(JSC::CallLinkInfo::emitFastPath):
(JSC::CallLinkInfo::emitTailCallFastPath):
(JSC::CallLinkInfo::emitSlowPath):
(JSC::CallLinkInfo::emitDirectFastPath):
(JSC::CallLinkInfo::emitDirectTailCallFastPath):
(JSC::CallLinkInfo::initializeDirectCall):
(JSC::CallLinkInfo::setDirectCallTarget):
(JSC::CallLinkInfo::setSlowPathCallDestination):
(JSC::CallLinkInfo::revertCallToStub):
(JSC::CallLinkInfo::setStub):
(JSC::CallLinkInfo::callReturnLocation): Deleted.
(JSC::CallLinkInfo::patchableJump): Deleted.
(JSC::CallLinkInfo::hotPathBegin): Deleted.
(JSC::CallLinkInfo::setCallee): Deleted.
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::calleeGPR const):
(JSC::CallLinkInfo::isDataIC const):
(JSC::CallLinkInfo::setUsesDataICs):
(JSC::CallLinkInfo::setCodeLocations):
(JSC::CallLinkInfo::offsetOfCallee):
(JSC::CallLinkInfo::offsetOfMonomorphicCallDestination):
(JSC::CallLinkInfo::offsetOfSlowPathCallDestination):
(JSC::CallLinkInfo::setCallLocations): Deleted.
(JSC::CallLinkInfo::hotPathOther): Deleted.
(JSC::CallLinkInfo::setStub): Deleted.
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::link):
* dfg/DFGJITCompiler.h:
(JSC::DFG::JITCompiler::addJSCall):
(JSC::DFG::JITCompiler::addJSDirectCall):
(JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
(JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
(JSC::DFG::JITCompiler::addJSDirectTailCall): Deleted.
(JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord): Deleted.
* dfg/DFGOSRExitCompilerCommon.cpp:
(JSC::DFG::callerReturnPC):
* dfg/DFGOperations.cpp:
(JSC::DFG::JSC_DEFINE_JIT_OPERATION):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::emitCall):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
* jit/CCallHelpers.cpp:
(JSC::CCallHelpers::emitJITCodeOver):
* jit/CCallHelpers.h:
* jit/JIT.cpp:
(JSC::JIT::link):
* jit/JIT.h:
* jit/JITCall.cpp:
(JSC::JIT::compileTailCall):
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):
* jit/JITCall32_64.cpp:
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):
* jit/JITOperations.cpp:
(JSC::JSC_DEFINE_JIT_OPERATION):
* jit/PolymorphicCallStubRoutine.cpp:
(JSC::PolymorphicCallNode::unlink):
* jit/Repatch.cpp:
(JSC::linkSlowPathTo):
(JSC::linkSlowFor):
(JSC::linkMonomorphicCall):
(JSC::linkDirectCall):
(JSC::revertCall):
(JSC::unlinkCall):
(JSC::linkPolymorphicCall):
(JSC::linkFor): Deleted.
(JSC::linkDirectFor): Deleted.
(JSC::unlinkFor): Deleted.
* jit/Repatch.h:
* wasm/js/WasmToJS.cpp:
(JSC::Wasm::wasmToJS):</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerAbstractMacroAssemblerh">trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerLinkBuffercpp">trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreassemblerLinkBufferh">trunk/Source/JavaScriptCore/assembler/LinkBuffer.h</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeAccessCasecpp">trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCallLinkInfocpp">trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorebytecodeCallLinkInfoh">trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGJITCompilercpp">trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGJITCompilerh">trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOSRExitCompilerCommoncpp">trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGOperationscpp">trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJIT32_64cpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp">trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp">trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitCCallHelperscpp">trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitCCallHelpersh">trunk/Source/JavaScriptCore/jit/CCallHelpers.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITcpp">trunk/Source/JavaScriptCore/jit/JIT.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITh">trunk/Source/JavaScriptCore/jit/JIT.h</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITCallcpp">trunk/Source/JavaScriptCore/jit/JITCall.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITCall32_64cpp">trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitJITOperationscpp">trunk/Source/JavaScriptCore/jit/JITOperations.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitPolymorphicCallStubRoutinecpp">trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitRepatchcpp">trunk/Source/JavaScriptCore/jit/Repatch.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCorejitRepatchh">trunk/Source/JavaScriptCore/jit/Repatch.h</a></li>
<li><a href="#trunkSourceJavaScriptCorewasmjsWasmToJScpp">trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog    2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/ChangeLog       2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1,3 +1,125 @@
</span><ins>+2021-05-18  Saam Barati  <sbarati@apple.com>
+
+        Add Data Call ICs that don't repatch and use them in the baseline JIT
+        https://bugs.webkit.org/show_bug.cgi?id=225321
+        <rdar://problem/77773796>
+
+        Reviewed by Michael Saboff.
+
+        This patch adds Data ICs for calls. Data ICs for calls work by loading a code 
+        pointer from CallLinkInfo, and indirect calling that pointer. This means that
+        to repatch such an IC, all we need to do is replace a code pointer inside
+        CallLinkInfo. No need to repatch the JIT code.
+        
+        The current implementation only does this for monomorphic calls. We still
+        repatch the JIT code for polymorphic calls. In a followup, we will also
+        opt polymorphic call ICs into data-based calls:
+        https://bugs.webkit.org/show_bug.cgi?id=225793
+        
+        This patch only uses Data Call ICs for the Baseline JIT. Even with that, it
+        reduces the number of calls to cacheFlush by ~45% on JetStream2.
+        
+        Performance is neutral on AS Macs, but it paves the way towards doing
+        unlinked JITting in JSC.
+
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::addLateLinkTask):
+        * assembler/LinkBuffer.cpp:
+        (JSC::LinkBuffer::linkCode):
+        (JSC::LinkBuffer::performFinalization):
+        * assembler/LinkBuffer.h:
+        * bytecode/AccessCase.cpp:
+        (JSC::AccessCase::generateImpl):
+        * bytecode/CallLinkInfo.cpp:
+        (JSC::CallLinkInfo::CallLinkInfo):
+        (JSC::CallLinkInfo::unlink):
+        (JSC::CallLinkInfo::fastPathStart):
+        (JSC::CallLinkInfo::slowPathStart):
+        (JSC::CallLinkInfo::doneLocation):
+        (JSC::CallLinkInfo::setMonomorphicCallee):
+        (JSC::CallLinkInfo::clearCallee):
+        (JSC::CallLinkInfo::emitFirstInstructionForDataIC):
+        (JSC::CallLinkInfo::emitFastPathImpl):
+        (JSC::CallLinkInfo::emitFastPath):
+        (JSC::CallLinkInfo::emitTailCallFastPath):
+        (JSC::CallLinkInfo::emitSlowPath):
+        (JSC::CallLinkInfo::emitDirectFastPath):
+        (JSC::CallLinkInfo::emitDirectTailCallFastPath):
+        (JSC::CallLinkInfo::initializeDirectCall):
+        (JSC::CallLinkInfo::setDirectCallTarget):
+        (JSC::CallLinkInfo::setSlowPathCallDestination):
+        (JSC::CallLinkInfo::revertCallToStub):
+        (JSC::CallLinkInfo::setStub):
+        (JSC::CallLinkInfo::callReturnLocation): Deleted.
+        (JSC::CallLinkInfo::patchableJump): Deleted.
+        (JSC::CallLinkInfo::hotPathBegin): Deleted.
+        (JSC::CallLinkInfo::setCallee): Deleted.
+        * bytecode/CallLinkInfo.h:
+        (JSC::CallLinkInfo::calleeGPR const):
+        (JSC::CallLinkInfo::isDataIC const):
+        (JSC::CallLinkInfo::setUsesDataICs):
+        (JSC::CallLinkInfo::setCodeLocations):
+        (JSC::CallLinkInfo::offsetOfCallee):
+        (JSC::CallLinkInfo::offsetOfMonomorphicCallDestination):
+        (JSC::CallLinkInfo::offsetOfSlowPathCallDestination):
+        (JSC::CallLinkInfo::setCallLocations): Deleted.
+        (JSC::CallLinkInfo::hotPathOther): Deleted.
+        (JSC::CallLinkInfo::setStub): Deleted.
+        * dfg/DFGJITCompiler.cpp:
+        (JSC::DFG::JITCompiler::link):
+        * dfg/DFGJITCompiler.h:
+        (JSC::DFG::JITCompiler::addJSCall):
+        (JSC::DFG::JITCompiler::addJSDirectCall):
+        (JSC::DFG::JITCompiler::JSCallRecord::JSCallRecord):
+        (JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
+        (JSC::DFG::JITCompiler::addJSDirectTailCall): Deleted.
+        (JSC::DFG::JITCompiler::JSDirectTailCallRecord::JSDirectTailCallRecord): Deleted.
+        * dfg/DFGOSRExitCompilerCommon.cpp:
+        (JSC::DFG::callerReturnPC):
+        * dfg/DFGOperations.cpp:
+        (JSC::DFG::JSC_DEFINE_JIT_OPERATION):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
+        (JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
+        (JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread):
+        (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
+        * jit/CCallHelpers.cpp:
+        (JSC::CCallHelpers::emitJITCodeOver):
+        * jit/CCallHelpers.h:
+        * jit/JIT.cpp:
+        (JSC::JIT::link):
+        * jit/JIT.h:
+        * jit/JITCall.cpp:
+        (JSC::JIT::compileTailCall):
+        (JSC::JIT::compileOpCall):
+        (JSC::JIT::compileOpCallSlowCase):
+        * jit/JITCall32_64.cpp:
+        (JSC::JIT::compileOpCall):
+        (JSC::JIT::compileOpCallSlowCase):
+        * jit/JITOperations.cpp:
+        (JSC::JSC_DEFINE_JIT_OPERATION):
+        * jit/PolymorphicCallStubRoutine.cpp:
+        (JSC::PolymorphicCallNode::unlink):
+        * jit/Repatch.cpp:
+        (JSC::linkSlowPathTo):
+        (JSC::linkSlowFor):
+        (JSC::linkMonomorphicCall):
+        (JSC::linkDirectCall):
+        (JSC::revertCall):
+        (JSC::unlinkCall):
+        (JSC::linkPolymorphicCall):
+        (JSC::linkFor): Deleted.
+        (JSC::linkDirectFor): Deleted.
+        (JSC::unlinkFor): Deleted.
+        * jit/Repatch.h:
+        * wasm/js/WasmToJS.cpp:
+        (JSC::Wasm::wasmToJS):
+
</ins><span class="cx"> 2021-05-18  Commit Queue  <commit-queue@webkit.org>
</span><span class="cx"> 
</span><span class="cx">         Unreviewed, reverting r276655.
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerAbstractMacroAssemblerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h   2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h      2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -971,6 +971,12 @@
</span><span class="cx">         m_linkTasks.append(createSharedTask<void(LinkBuffer&)>(functor));
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    template<typename Functor>
+    void addLateLinkTask(const Functor& functor) // Run after all link tasks
+    {
+        m_lateLinkTasks.append(createSharedTask<void(LinkBuffer&)>(functor));
+    }
+
</ins><span class="cx"> #if COMPILER(GCC)
</span><span class="cx">     // Workaround for GCC demanding that memcpy "must be the name of a function with external linkage".
</span><span class="cx">     static void* memcpy(void* dst, const void* src, size_t size)
</span><span class="lines">@@ -1114,6 +1120,7 @@
</span><span class="cx">     bool m_allowScratchRegister { true };
</span><span class="cx"> 
</span><span class="cx">     Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_linkTasks;
</span><ins>+    Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_lateLinkTasks;
</ins><span class="cx"> 
</span><span class="cx">     friend class LinkBuffer;
</span><span class="cx"> }; // class AbstractMacroAssembler
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerLinkBuffercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp     2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/LinkBuffer.cpp        2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -439,6 +439,7 @@
</span><span class="cx"> #endif // !ENABLE(BRANCH_COMPACTION)
</span><span class="cx"> 
</span><span class="cx">     m_linkTasks = WTFMove(macroAssembler.m_linkTasks);
</span><ins>+    m_lateLinkTasks = WTFMove(macroAssembler.m_lateLinkTasks);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void LinkBuffer::allocate(MacroAssembler& macroAssembler, JITCompilationEffort effort)
</span><span class="lines">@@ -471,6 +472,8 @@
</span><span class="cx"> {
</span><span class="cx">     for (auto& task : m_linkTasks)
</span><span class="cx">         task->run(*this);
</span><ins>+    for (auto& task : m_lateLinkTasks)
+        task->run(*this);
</ins><span class="cx"> 
</span><span class="cx"> #ifndef NDEBUG
</span><span class="cx">     ASSERT(m_isJumpIsland || !isCompilationThread());
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreassemblerLinkBufferh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/assembler/LinkBuffer.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/assembler/LinkBuffer.h       2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/assembler/LinkBuffer.h  2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -403,6 +403,7 @@
</span><span class="cx">     Profile m_profile { Profile::Uncategorized };
</span><span class="cx">     MacroAssemblerCodePtr<LinkBufferPtrTag> m_code;
</span><span class="cx">     Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_linkTasks;
</span><ins>+    Vector<RefPtr<SharedTask<void(LinkBuffer&)>>> m_lateLinkTasks;
</ins><span class="cx"> 
</span><span class="cx">     static size_t s_profileCummulativeLinkedSizes[numberOfProfiles];
</span><span class="cx"> };
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeAccessCasecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp      2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/AccessCase.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1580,11 +1580,6 @@
</span><span class="cx">         // Stuff for custom getters/setters.
</span><span class="cx">         CCallHelpers::Call operationCall;
</span><span class="cx"> 
</span><del>-        // Stuff for JS getters/setters.
-        CCallHelpers::DataLabelPtr addressOfLinkFunctionCheck;
-        CCallHelpers::Call fastPathCall;
-        CCallHelpers::Call slowPathCall;
-
</del><span class="cx">         // This also does the necessary calculations of whether or not we're an
</span><span class="cx">         // exception handling call site.
</span><span class="cx">         AccessGenerationState::SpillState spillState = state.preserveLiveRegistersToStackForCall();
</span><span class="lines">@@ -1701,24 +1696,22 @@
</span><span class="cx">                         virtualRegisterForArgumentIncludingThis(1).offset() * sizeof(Register)));
</span><span class="cx">             }
</span><span class="cx"> 
</span><del>-            CCallHelpers::Jump slowCase = jit.branchPtrWithPatch(
-                CCallHelpers::NotEqual, loadedValueGPR, addressOfLinkFunctionCheck,
-                CCallHelpers::TrustedImmPtr(nullptr));
</del><ins>+            auto slowCase = access.callLinkInfo()->emitFastPath(jit, loadedValueGPR, loadedValueGPR == GPRInfo::regT2 ? GPRInfo::regT0 : GPRInfo::regT2, JITCode::isOptimizingJIT(codeBlock->jitType()) ? CallLinkInfo::UseDataIC::No : CallLinkInfo::UseDataIC::Yes);
+            auto doneLocation = jit.label();
</ins><span class="cx"> 
</span><del>-            fastPathCall = jit.nearCall();
</del><span class="cx">             if (m_type == Getter)
</span><span class="cx">                 jit.setupResults(valueRegs);
</span><span class="cx">             done.append(jit.jump());
</span><span class="cx"> 
</span><span class="cx">             slowCase.link(&jit);
</span><ins>+            auto slowPathStart = jit.label();
</ins><span class="cx">             jit.move(loadedValueGPR, GPRInfo::regT0);
</span><span class="cx"> #if USE(JSVALUE32_64)
</span><span class="cx">             // We *always* know that the getter/setter, if non-null, is a cell.
</span><span class="cx">             jit.move(CCallHelpers::TrustedImm32(JSValue::CellTag), GPRInfo::regT1);
</span><span class="cx"> #endif
</span><del>-            jit.move(CCallHelpers::TrustedImmPtr(access.callLinkInfo()), GPRInfo::regT2);
</del><span class="cx">             jit.move(CCallHelpers::TrustedImmPtr(globalObject), GPRInfo::regT3);
</span><del>-            slowPathCall = jit.nearCall();
</del><ins>+            access.callLinkInfo()->emitSlowPath(vm, jit);
</ins><span class="cx">             if (m_type == Getter)
</span><span class="cx">                 jit.setupResults(valueRegs);
</span><span class="cx">             done.append(jit.jump());
</span><span class="lines">@@ -1735,15 +1728,10 @@
</span><span class="cx">             bool callHasReturnValue = isGetter();
</span><span class="cx">             restoreLiveRegistersFromStackForCall(spillState, callHasReturnValue);
</span><span class="cx"> 
</span><del>-            jit.addLinkTask([=, &vm] (LinkBuffer& linkBuffer) {
-                this->as<GetterSetterAccessCase>().callLinkInfo()->setCallLocations(
-                    CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowPathCall)),
-                    CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(addressOfLinkFunctionCheck)),
-                    linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastPathCall));
-
-                linkBuffer.link(
-                    slowPathCall,
-                    CodeLocationLabel<JITThunkPtrTag>(vm.getCTIStub(linkCallThunkGenerator).code()));
</del><ins>+            jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+                this->as<GetterSetterAccessCase>().callLinkInfo()->setCodeLocations(
+                    linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+                    linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">             });
</span><span class="cx">         } else {
</span><span class="cx">             ASSERT(m_type == CustomValueGetter || m_type == CustomAccessorGetter || m_type == CustomValueSetter || m_type == CustomAccessorSetter);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCallLinkInfocpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp    2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp       2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -26,11 +26,15 @@
</span><span class="cx"> #include "config.h"
</span><span class="cx"> #include "CallLinkInfo.h"
</span><span class="cx"> 
</span><ins>+#include "CCallHelpers.h"
</ins><span class="cx"> #include "CallFrameShuffleData.h"
</span><ins>+#include "DisallowMacroScratchRegisterUsage.h"
</ins><span class="cx"> #include "FunctionCodeBlock.h"
</span><span class="cx"> #include "JSCellInlines.h"
</span><ins>+#include "LinkBuffer.h"
</ins><span class="cx"> #include "Opcode.h"
</span><span class="cx"> #include "Repatch.h"
</span><ins>+#include "ThunkGenerators.h"
</ins><span class="cx"> #include <wtf/ListDump.h>
</span><span class="cx"> 
</span><span class="cx"> #if ENABLE(JIT)
</span><span class="lines">@@ -77,6 +81,7 @@
</span><span class="cx">     , m_allowStubs(true)
</span><span class="cx">     , m_clearedByJettison(false)
</span><span class="cx">     , m_callType(None)
</span><ins>+    , m_useDataIC(static_cast<unsigned>(UseDataIC::Yes))
</ins><span class="cx"> {
</span><span class="cx"> }
</span><span class="cx"> 
</span><span class="lines">@@ -102,7 +107,7 @@
</span><span class="cx">     // We could be called even if we're not linked anymore because of how polymorphic calls
</span><span class="cx">     // work. Each callsite within the polymorphic call stub may separately ask us to unlink().
</span><span class="cx">     if (isLinked())
</span><del>-        unlinkFor(vm, *this);
</del><ins>+        unlinkCall(vm, *this);
</ins><span class="cx"> 
</span><span class="cx">     // Either we were unlinked, in which case we should not have been on any list, or we unlinked
</span><span class="cx">     // ourselves so that we're not on any list anymore.
</span><span class="lines">@@ -109,34 +114,34 @@
</span><span class="cx">     RELEASE_ASSERT(!isOnList());
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-CodeLocationNearCall<JSInternalPtrTag> CallLinkInfo::callReturnLocation()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::fastPathStart()
</ins><span class="cx"> {
</span><del>-    RELEASE_ASSERT(!isDirect());
-    return CodeLocationNearCall<JSInternalPtrTag>(m_callReturnLocationOrPatchableJump, NearCallMode::Regular);
</del><ins>+    return CodeLocationDataLabelPtr<JSInternalPtrTag>(m_fastPathStart);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-CodeLocationJump<JSInternalPtrTag> CallLinkInfo::patchableJump()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::slowPathStart()
</ins><span class="cx"> {
</span><del>-    RELEASE_ASSERT(callType() == DirectTailCall);
-    return CodeLocationJump<JSInternalPtrTag>(m_callReturnLocationOrPatchableJump);
</del><ins>+    RELEASE_ASSERT(!isDataIC());
+    return u.codeIC.m_slowPathStart;
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-CodeLocationDataLabelPtr<JSInternalPtrTag> CallLinkInfo::hotPathBegin()
</del><ins>+CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::doneLocation()
</ins><span class="cx"> {
</span><span class="cx">     RELEASE_ASSERT(!isDirect());
</span><del>-    return CodeLocationDataLabelPtr<JSInternalPtrTag>(m_hotPathBeginOrSlowPathStart);
</del><ins>+    return m_doneLocation;
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-CodeLocationLabel<JSInternalPtrTag> CallLinkInfo::slowPathStart()
</del><ins>+void CallLinkInfo::setMonomorphicCallee(VM& vm, JSCell* owner, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</ins><span class="cx"> {
</span><del>-    RELEASE_ASSERT(isDirect());
-    return m_hotPathBeginOrSlowPathStart;
-}
-
-void CallLinkInfo::setCallee(VM& vm, JSCell* owner, JSObject* callee)
-{
</del><span class="cx">     RELEASE_ASSERT(!isDirect());
</span><span class="cx">     m_calleeOrCodeBlock.set(vm, owner, callee);
</span><ins>+
+    if (isDataIC()) 
+        u.dataIC.m_monomorphicCallDestination = codePtr;
+    else {
+        MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, CodeLocationLabel<JSEntryPtrTag>(codePtr));
+        MacroAssembler::repatchPointer(u.codeIC.m_calleeLocation, callee);
+    }
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> void CallLinkInfo::clearCallee()
</span><span class="lines">@@ -143,6 +148,10 @@
</span><span class="cx"> {
</span><span class="cx">     RELEASE_ASSERT(!isDirect());
</span><span class="cx">     m_calleeOrCodeBlock.clear();
</span><ins>+    if (isDataIC())
+        u.dataIC.m_monomorphicCallDestination = nullptr;
+    else if (!clearedByJettison())
+        MacroAssembler::repatchPointer(u.codeIC.m_calleeLocation, nullptr);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> JSObject* CallLinkInfo::callee()
</span><span class="lines">@@ -284,6 +293,206 @@
</span><span class="cx">     m_frameShuffleData->shrinkToFit();
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void CallLinkInfo::emitFirstInstructionForDataIC(CCallHelpers& jit, GPRReg callLinkInfoGPR)
+{
+    GPRReg scratchGPR = jit.scratchRegister();
+    DisallowMacroScratchRegisterUsage disallowScratch(jit);
+    // When repatching, we just overwrite the first instruction back, since it'll have been replaced with a jump to the polymorphic call stub.
+    size_t startSize = jit.m_assembler.buffer().codeSize();
+    jit.loadPtr(CCallHelpers::Address(callLinkInfoGPR, offsetOfCallee()), scratchGPR); 
+    size_t loadSize = jit.m_assembler.buffer().codeSize() - startSize;
+    if (loadSize < static_cast<size_t>(CCallHelpers::patchableJumpSize()))
+        jit.emitNops(static_cast<size_t>(CCallHelpers::patchableJumpSize()) - loadSize);
+}
+
+MacroAssembler::JumpList CallLinkInfo::emitFastPathImpl(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC, WTF::Function<void()> prepareForTailCall)
+{
+    setUsesDataICs(useDataIC);
+    if (isDataIC()) {
+        RELEASE_ASSERT(callLinkInfoGPR != GPRReg::InvalidGPRReg);
+        jit.move(CCallHelpers::TrustedImmPtr(this), callLinkInfoGPR);
+        u.dataIC.m_callLinkInfoGPR = callLinkInfoGPR;
+    }
+
+    auto fastPathStart = jit.label();
+    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+        m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+    });
+
+    CCallHelpers::JumpList slowPath;
+
+    if (isDataIC()) {
+        GPRReg scratchGPR = jit.scratchRegister();
+        emitFirstInstructionForDataIC(jit, callLinkInfoGPR);
+        {
+            DisallowMacroScratchRegisterUsage disallowScratch(jit);
+            slowPath.append(jit.branchPtr(CCallHelpers::NotEqual, scratchGPR, calleeGPR));
+        }
+        if (isTailCall()) {
+            prepareForTailCall();
+            jit.farJump(CCallHelpers::Address(callLinkInfoGPR, offsetOfMonomorphicCallDestination()), JSEntryPtrTag);
+        } else
+            jit.call(CCallHelpers::Address(callLinkInfoGPR, offsetOfMonomorphicCallDestination()), JSEntryPtrTag);
+    } else {
+        CCallHelpers::DataLabelPtr calleeCheck;
+        slowPath.append(jit.branchPtrWithPatch(CCallHelpers::NotEqual, calleeGPR, calleeCheck, CCallHelpers::TrustedImmPtr(nullptr)));
+
+        CCallHelpers::Call call;
+        if (isTailCall()) {
+            prepareForTailCall();
+            call = jit.nearTailCall();
+        } else
+            call = jit.nearCall();
+        jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+            u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+            u.codeIC.m_calleeLocation = linkBuffer.locationOf<JSInternalPtrTag>(calleeCheck);
+        });
+    }
+
+    return slowPath;
+}
+
+CCallHelpers::JumpList CallLinkInfo::emitFastPath(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC)
+{
+    RELEASE_ASSERT(!isTailCall());
+    return emitFastPathImpl(jit, calleeGPR, callLinkInfoGPR, useDataIC, nullptr);
+}
+
+MacroAssembler::JumpList CallLinkInfo::emitTailCallFastPath(CCallHelpers& jit, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC useDataIC, WTF::Function<void()> prepareForTailCall)
+{
+    RELEASE_ASSERT(isTailCall());
+    return emitFastPathImpl(jit, calleeGPR, callLinkInfoGPR, useDataIC, WTFMove(prepareForTailCall));
+}
+
+void CallLinkInfo::emitSlowPath(VM& vm, CCallHelpers& jit)
+{
+    setSlowPathCallDestination(vm.getCTIStub(linkCallThunkGenerator).template retaggedCode<JSEntryPtrTag>());
+    jit.move(CCallHelpers::TrustedImmPtr(this), GPRInfo::regT2);
+    jit.call(CCallHelpers::Address(GPRInfo::regT2, offsetOfSlowPathCallDestination()), JSEntryPtrTag);
+}
+
+void CallLinkInfo::emitDirectFastPath(CCallHelpers& jit)
+{
+    RELEASE_ASSERT(!isTailCall());
+
+    setUsesDataICs(UseDataIC::No);
+
+    auto fastPathStart = jit.label();
+    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+        m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+    });
+
+    auto call = jit.nearCall();
+    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+        u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+    });
+    jit.addLateLinkTask([this] (LinkBuffer&) {
+        initializeDirectCall();
+    });
+}
+
+void CallLinkInfo::emitDirectTailCallFastPath(CCallHelpers& jit, WTF::Function<void()> prepareForTailCall)
+{
+    RELEASE_ASSERT(isTailCall());
+
+    setUsesDataICs(UseDataIC::No);
+
+    auto fastPathStart = jit.label();
+    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+        m_fastPathStart = linkBuffer.locationOf<JSInternalPtrTag>(fastPathStart);
+    });
+
+    // - If we're not yet linked, this is a jump to the slow path.
+    // - Once we're linked to a fast path, this goes back to being nops so we fall through to the linked jump.
+    jit.emitNops(CCallHelpers::patchableJumpSize());
+
+    prepareForTailCall();
+    auto call = jit.nearTailCall();
+    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+        u.codeIC.m_callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
+    });
+    jit.addLateLinkTask([this] (LinkBuffer&) {
+        initializeDirectCall();
+    });
+}
+
+void CallLinkInfo::initializeDirectCall()
+{
+    RELEASE_ASSERT(isDirect());
+    ASSERT(u.codeIC.m_callLocation);
+    if (isTailCall()) {
+        RELEASE_ASSERT(fastPathStart());
+        CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+            auto jump = jit.jump();
+            jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+                linkBuffer.link(jump, slowPathStart());
+            });
+        }, "initialize direct call");
+    } else
+        MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, slowPathStart());
+}
+
+void CallLinkInfo::setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag> target)
+{
+    RELEASE_ASSERT(isDirect());
+
+    if (isTailCall()) {
+        RELEASE_ASSERT(fastPathStart());
+        CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+            // We reserved this many bytes for the jump at fastPathStart(). Make that
+            // code nops now so we fall through to the jump to the fast path.
+            jit.emitNops(CCallHelpers::patchableJumpSize()); 
+        }, "Setting direct call target");
+    }
+
+    MacroAssembler::repatchNearCall(u.codeIC.m_callLocation, target);
+}
+
+void CallLinkInfo::setSlowPathCallDestination(MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
+{
+    m_slowPathCallDestination = codePtr;
+}
+
+void CallLinkInfo::revertCallToStub()
+{
+    RELEASE_ASSERT(stub());
+    // The start of our JIT code is now a jump to the polymorphic stub. Rewrite the first instruction
+    // to be what we need for non stub ICs.
+
+    // this runs into some branch compaction crap I'd like to avoid for now. Essentially, the branch
+    // doesn't know if it can be compacted or not. So we end up with 28 bytes of machine code, for
+    // what in all likelihood fits in 24. So we just splat out the first instruction. Long term, we
+    // need something cleaner. But this works on arm64 for now.
+
+    CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+        if (isDataIC())
+            emitFirstInstructionForDataIC(jit, u.dataIC.m_callLinkInfoGPR);
+        else {
+            CCallHelpers::revertJumpReplacementToBranchPtrWithPatch(
+                CCallHelpers::startOfBranchPtrWithPatchOnRegister(u.codeIC.m_calleeLocation), calleeGPR(), nullptr);
+        }
+    },  "Resetting PolymorphicCall stubbed CallLinkInfo");
+}
+
+void CallLinkInfo::setStub(Ref<PolymorphicCallStubRoutine>&& newStub)
+{
+    clearStub();
+    m_stub = WTFMove(newStub);
+
+    if (isDataIC()) {
+        CCallHelpers::emitJITCodeOver(fastPathStart(), [&] (CCallHelpers& jit) {
+            auto jump = jit.jump();
+            jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+                linkBuffer.link(jump, CodeLocationLabel<JITStubRoutinePtrTag>(m_stub->code().code()));
+            });
+        }, "Patching call to PolymorphicCallStubRoutine in CallLinkInfo");
+    } else {
+        MacroAssembler::replaceWithJump(
+            MacroAssembler::startOfBranchPtrWithPatchOnRegister(u.codeIC.m_calleeLocation),
+            CodeLocationLabel<JITStubRoutinePtrTag>(m_stub->code().code()));
+    }
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx"> #endif // ENABLE(JIT)
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorebytecodeCallLinkInfoh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h      2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -30,6 +30,7 @@
</span><span class="cx"> #include "CodeSpecializationKind.h"
</span><span class="cx"> #include "PolymorphicCallStubRoutine.h"
</span><span class="cx"> #include "WriteBarrier.h"
</span><ins>+#include <wtf/Function.h>
</ins><span class="cx"> #include <wtf/SentinelLinkedList.h>
</span><span class="cx"> 
</span><span class="cx"> namespace JSC {
</span><span class="lines">@@ -36,6 +37,7 @@
</span><span class="cx"> 
</span><span class="cx"> #if ENABLE(JIT)
</span><span class="cx"> 
</span><ins>+class CCallHelpers;
</ins><span class="cx"> class FunctionCodeBlock;
</span><span class="cx"> class JSFunction;
</span><span class="cx"> enum OpcodeID : unsigned;
</span><span class="lines">@@ -163,16 +165,38 @@
</span><span class="cx">         m_calleeGPR = calleeGPR;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    void setCallLocations(
-        CodeLocationLabel<JSInternalPtrTag> callReturnLocationOrPatchableJump,
-        CodeLocationLabel<JSInternalPtrTag> hotPathBeginOrSlowPathStart,
-        CodeLocationNearCall<JSInternalPtrTag> hotPathOther)
</del><ins>+    GPRReg calleeGPR() const { return m_calleeGPR; }
+    
+    enum class UseDataIC : uint8_t {
+        Yes,
+        No
+    };
+
+private:
+    void emitFirstInstructionForDataIC(CCallHelpers&, GPRReg callLinkInfoGPR);
+    MacroAssembler::JumpList emitFastPathImpl(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC, WTF::Function<void()> prepareForTailCall) WARN_UNUSED_RETURN;
+public:
+    MacroAssembler::JumpList emitFastPath(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC) WARN_UNUSED_RETURN;
+    MacroAssembler::JumpList emitTailCallFastPath(CCallHelpers&, GPRReg calleeGPR, GPRReg callLinkInfoGPR, UseDataIC, WTF::Function<void()> prepareForTailCall) WARN_UNUSED_RETURN;
+    void emitDirectFastPath(CCallHelpers&);
+    void emitDirectTailCallFastPath(CCallHelpers&, WTF::Function<void()> prepareForTailCall);
+    void emitSlowPath(VM&, CCallHelpers&);
+    void revertCallToStub();
+
+    bool isDataIC() const { return static_cast<UseDataIC>(m_useDataIC) == UseDataIC::Yes; }
+    void setUsesDataICs(UseDataIC useDataIC) { m_useDataIC = static_cast<unsigned>(useDataIC); }
+
+    void setCodeLocations(
+        CodeLocationLabel<JSInternalPtrTag> slowPathStart,
+        CodeLocationLabel<JSInternalPtrTag> doneLocation)
</ins><span class="cx">     {
</span><del>-        m_callReturnLocationOrPatchableJump = callReturnLocationOrPatchableJump;
-        m_hotPathBeginOrSlowPathStart = hotPathBeginOrSlowPathStart;
-        m_hotPathOther = hotPathOther;
</del><ins>+        if (!isDataIC())
+            u.codeIC.m_slowPathStart = slowPathStart;
+        m_doneLocation = doneLocation;
</ins><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    void initializeDirectCall();
+
</ins><span class="cx">     bool allowStubs() const { return m_allowStubs; }
</span><span class="cx"> 
</span><span class="cx">     void disallowStubs()
</span><span class="lines">@@ -180,17 +204,12 @@
</span><span class="cx">         m_allowStubs = false;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    CodeLocationNearCall<JSInternalPtrTag> callReturnLocation();
-    CodeLocationJump<JSInternalPtrTag> patchableJump();
-    CodeLocationDataLabelPtr<JSInternalPtrTag> hotPathBegin();
</del><ins>+    CodeLocationLabel<JSInternalPtrTag> fastPathStart();
</ins><span class="cx">     CodeLocationLabel<JSInternalPtrTag> slowPathStart();
</span><ins>+    CodeLocationLabel<JSInternalPtrTag> doneLocation();
</ins><span class="cx"> 
</span><del>-    CodeLocationNearCall<JSInternalPtrTag> hotPathOther()
-    {
-        return m_hotPathOther;
-    }
-
-    void setCallee(VM&, JSCell*, JSObject* callee);
</del><ins>+    void setMonomorphicCallee(VM&, JSCell*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
+    void setSlowPathCallDestination(MacroAssemblerCodePtr<JSEntryPtrTag>);
</ins><span class="cx">     void clearCallee();
</span><span class="cx">     JSObject* callee();
</span><span class="cx"> 
</span><span class="lines">@@ -197,6 +216,7 @@
</span><span class="cx">     void setCodeBlock(VM&, JSCell*, FunctionCodeBlock*);
</span><span class="cx">     void clearCodeBlock();
</span><span class="cx">     FunctionCodeBlock* codeBlock();
</span><ins>+    void setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag>);
</ins><span class="cx"> 
</span><span class="cx">     void setLastSeenCallee(VM&, const JSCell* owner, JSObject* callee);
</span><span class="cx">     void clearLastSeenCallee();
</span><span class="lines">@@ -206,12 +226,7 @@
</span><span class="cx">     void setExecutableDuringCompilation(ExecutableBase*);
</span><span class="cx">     ExecutableBase* executable();
</span><span class="cx">     
</span><del>-    void setStub(Ref<PolymorphicCallStubRoutine>&& newStub)
-    {
-        clearStub();
-        m_stub = WTFMove(newStub);
-    }
-
</del><ins>+    void setStub(Ref<PolymorphicCallStubRoutine>&&);
</ins><span class="cx">     void clearStub();
</span><span class="cx"> 
</span><span class="cx">     PolymorphicCallStubRoutine* stub() const
</span><span class="lines">@@ -311,6 +326,21 @@
</span><span class="cx">         return OBJECT_OFFSETOF(CallLinkInfo, m_slowPathCount);
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    static ptrdiff_t offsetOfCallee()
+    {
+        return OBJECT_OFFSETOF(CallLinkInfo, m_calleeOrCodeBlock);
+    }
+
+    static ptrdiff_t offsetOfMonomorphicCallDestination()
+    {
+        return OBJECT_OFFSETOF(CallLinkInfo, u) + OBJECT_OFFSETOF(UnionType, dataIC.m_monomorphicCallDestination);
+    }
+
+    static ptrdiff_t offsetOfSlowPathCallDestination()
+    {
+        return OBJECT_OFFSETOF(CallLinkInfo, m_slowPathCallDestination);
+    }
+
</ins><span class="cx">     GPRReg calleeGPR()
</span><span class="cx">     {
</span><span class="cx">         return m_calleeGPR;
</span><span class="lines">@@ -352,10 +382,25 @@
</span><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx"> private:
</span><del>-    uint32_t m_maxArgumentCountIncludingThis { 0 }; // For varargs: the profiled maximum number of arguments. For direct: the number of stack slots allocated for arguments.
-    CodeLocationLabel<JSInternalPtrTag> m_callReturnLocationOrPatchableJump;
-    CodeLocationLabel<JSInternalPtrTag> m_hotPathBeginOrSlowPathStart;
-    CodeLocationNearCall<JSInternalPtrTag> m_hotPathOther;
</del><ins>+    CodeLocationLabel<JSInternalPtrTag> m_fastPathStart;
+    CodeLocationLabel<JSInternalPtrTag> m_doneLocation;
+    MacroAssemblerCodePtr<JSEntryPtrTag> m_slowPathCallDestination;
+    union UnionType {
+        UnionType() 
+            : dataIC { nullptr, InvalidGPRReg }
+        { }
+        struct DataIC {
+            MacroAssemblerCodePtr<JSEntryPtrTag> m_monomorphicCallDestination;
+            GPRReg m_callLinkInfoGPR;
+        } dataIC;
+
+        struct {
+            CodeLocationNearCall<JSInternalPtrTag> m_callLocation;
+            CodeLocationDataLabelPtr<JSInternalPtrTag> m_calleeLocation;
+            CodeLocationLabel<JSInternalPtrTag> m_slowPathStart;
+        } codeIC;
+    } u;
+
</ins><span class="cx">     WriteBarrier<JSCell> m_calleeOrCodeBlock;
</span><span class="cx">     WriteBarrier<JSCell> m_lastSeenCalleeOrExecutable;
</span><span class="cx">     RefPtr<PolymorphicCallStubRoutine> m_stub;
</span><span class="lines">@@ -369,8 +414,10 @@
</span><span class="cx">     bool m_allowStubs : 1;
</span><span class="cx">     bool m_clearedByJettison : 1;
</span><span class="cx">     unsigned m_callType : 4; // CallType
</span><ins>+    unsigned m_useDataIC : 1; // UseDataIC
</ins><span class="cx">     GPRReg m_calleeGPR { InvalidGPRReg };
</span><span class="cx">     uint32_t m_slowPathCount { 0 };
</span><ins>+    uint32_t m_maxArgumentCountIncludingThis { 0 }; // For varargs: the profiled maximum number of arguments. For direct: the number of stack slots allocated for arguments.
</ins><span class="cx"> };
</span><span class="cx"> 
</span><span class="cx"> inline CodeOrigin getCallLinkInfoCodeOrigin(CallLinkInfo& callLinkInfo)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGJITCompilercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp       2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp  2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -257,32 +257,19 @@
</span><span class="cx">     finalizeInlineCaches(m_instanceOfs, linkBuffer);
</span><span class="cx">     finalizeInlineCaches(m_privateBrandAccesses, linkBuffer);
</span><span class="cx"> 
</span><del>-    auto linkCallThunk = FunctionPtr<NoPtrTag>(vm().getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
</del><span class="cx">     for (auto& record : m_jsCalls) {
</span><span class="cx">         CallLinkInfo& info = *record.info;
</span><del>-        linkBuffer.link(record.slowCall, linkCallThunk);
-        info.setCallLocations(
-            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.slowCall)),
-            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(record.targetToCheck)),
-            linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.fastCall));
</del><ins>+        info.setCodeLocations(
+            linkBuffer.locationOf<JSInternalPtrTag>(record.slowPathStart),
+            linkBuffer.locationOf<JSInternalPtrTag>(record.doneLocation));
</ins><span class="cx">     }
</span><span class="cx">     
</span><del>-    for (JSDirectCallRecord& record : m_jsDirectCalls) {
</del><ins>+    for (auto& record : m_jsDirectCalls) {
</ins><span class="cx">         CallLinkInfo& info = *record.info;
</span><del>-        linkBuffer.link(record.call, linkBuffer.locationOf<NoPtrTag>(record.slowPath));
-        info.setCallLocations(
-            CodeLocationLabel<JSInternalPtrTag>(),
</del><ins>+        info.setCodeLocations(
</ins><span class="cx">             linkBuffer.locationOf<JSInternalPtrTag>(record.slowPath),
</span><del>-            linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.call));
</del><ins>+            CodeLocationLabel<JSInternalPtrTag>());
</ins><span class="cx">     }
</span><del>-    
-    for (JSDirectTailCallRecord& record : m_jsDirectTailCalls) {
-        CallLinkInfo& info = *record.info;
-        info.setCallLocations(
-            linkBuffer.locationOf<JSInternalPtrTag>(record.patchableJump),
-            linkBuffer.locationOf<JSInternalPtrTag>(record.slowPath),
-            linkBuffer.locationOfNearCall<JSInternalPtrTag>(record.call));
-    }
</del><span class="cx"> 
</span><span class="cx"> #if ENABLE(EXTRA_CTI_THUNKS)
</span><span class="cx">     if (!m_exceptionChecks.empty())
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGJITCompilerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.h    2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -219,21 +219,16 @@
</span><span class="cx">         m_privateBrandAccesses.append(InlineCacheWrapper<JITPrivateBrandAccessGenerator>(gen, slowPath));
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    void addJSCall(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
</del><ins>+    void addJSCall(Label slowPathStart, Label doneLocation, CallLinkInfo* info)
</ins><span class="cx">     {
</span><del>-        m_jsCalls.append(JSCallRecord(fastCall, slowCall, targetToCheck, info));
</del><ins>+        m_jsCalls.append(JSCallRecord(slowPathStart, doneLocation, info));
</ins><span class="cx">     }
</span><span class="cx">     
</span><del>-    void addJSDirectCall(Call call, Label slowPath, CallLinkInfo* info)
</del><ins>+    void addJSDirectCall(Label slowPath, CallLinkInfo* info)
</ins><span class="cx">     {
</span><del>-        m_jsDirectCalls.append(JSDirectCallRecord(call, slowPath, info));
</del><ins>+        m_jsDirectCalls.append(JSDirectCallRecord(slowPath, info));
</ins><span class="cx">     }
</span><span class="cx">     
</span><del>-    void addJSDirectTailCall(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
-    {
-        m_jsDirectTailCalls.append(JSDirectTailCallRecord(patchableJump, call, slowPath, info));
-    }
-    
</del><span class="cx">     void addWeakReference(JSCell* target)
</span><span class="cx">     {
</span><span class="cx">         m_graph.m_plan.weakReferences().addLazily(target);
</span><span class="lines">@@ -328,53 +323,29 @@
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx">     struct JSCallRecord {
</span><del>-        JSCallRecord(Call fastCall, Call slowCall, DataLabelPtr targetToCheck, CallLinkInfo* info)
-            : fastCall(fastCall)
-            , slowCall(slowCall)
-            , targetToCheck(targetToCheck)
</del><ins>+        JSCallRecord(Label slowPathStart, Label doneLocation, CallLinkInfo* info)
+            : slowPathStart(slowPathStart)
+            , doneLocation(doneLocation)
</ins><span class="cx">             , info(info)
</span><span class="cx">         {
</span><del>-            ASSERT(fastCall.isFlagSet(Call::Near));
-            ASSERT(slowCall.isFlagSet(Call::Near));
</del><span class="cx">         }
</span><span class="cx">         
</span><del>-        Call fastCall;
-        Call slowCall;
-        DataLabelPtr targetToCheck;
</del><ins>+        Label slowPathStart;
+        Label doneLocation;
</ins><span class="cx">         CallLinkInfo* info;
</span><span class="cx">     };
</span><span class="cx">     
</span><span class="cx">     struct JSDirectCallRecord {
</span><del>-        JSDirectCallRecord(Call call, Label slowPath, CallLinkInfo* info)
-            : call(call)
-            , slowPath(slowPath)
</del><ins>+        JSDirectCallRecord(Label slowPath, CallLinkInfo* info)
+            : slowPath(slowPath)
</ins><span class="cx">             , info(info)
</span><span class="cx">         {
</span><del>-            ASSERT(call.isFlagSet(Call::Near));
</del><span class="cx">         }
</span><span class="cx">         
</span><del>-        Call call;
</del><span class="cx">         Label slowPath;
</span><span class="cx">         CallLinkInfo* info;
</span><span class="cx">     };
</span><span class="cx">     
</span><del>-    struct JSDirectTailCallRecord {
-        JSDirectTailCallRecord(PatchableJump patchableJump, Call call, Label slowPath, CallLinkInfo* info)
-            : patchableJump(patchableJump)
-            , call(call)
-            , slowPath(slowPath)
-            , info(info)
-        {
-            ASSERT(call.isFlagSet(Call::Near) && call.isFlagSet(Call::Tail));
-        }
-        
-        PatchableJump patchableJump;
-        Call call;
-        Label slowPath;
-        CallLinkInfo* info;
-    };
-
-    
</del><span class="cx">     Vector<InlineCacheWrapper<JITGetByIdGenerator>, 4> m_getByIds;
</span><span class="cx">     Vector<InlineCacheWrapper<JITGetByIdWithThisGenerator>, 4> m_getByIdsWithThis;
</span><span class="cx">     Vector<InlineCacheWrapper<JITGetByValGenerator>, 4> m_getByVals;
</span><span class="lines">@@ -386,7 +357,6 @@
</span><span class="cx">     Vector<InlineCacheWrapper<JITPrivateBrandAccessGenerator>, 4> m_privateBrandAccesses;
</span><span class="cx">     Vector<JSCallRecord, 4> m_jsCalls;
</span><span class="cx">     Vector<JSDirectCallRecord, 4> m_jsDirectCalls;
</span><del>-    Vector<JSDirectTailCallRecord, 4> m_jsDirectTailCalls;
</del><span class="cx">     SegmentedVector<OSRExitCompilationInfo, 4> m_exitCompilationInfo;
</span><span class="cx">     Vector<Vector<Label>> m_exitSiteLabels;
</span><span class="cx">     Vector<DFG::OSREntryData> m_osrEntry;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOSRExitCompilerCommoncpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp     2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGOSRExitCompilerCommon.cpp        2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -209,7 +209,7 @@
</span><span class="cx">                 baselineCodeBlockForCaller->getCallLinkInfoForBytecodeIndex(callBytecodeIndex);
</span><span class="cx">             RELEASE_ASSERT(callLinkInfo);
</span><span class="cx"> 
</span><del>-            jumpTarget = callLinkInfo->callReturnLocation().retagged<JSEntryPtrTag>();
</del><ins>+            jumpTarget = callLinkInfo->doneLocation().retagged<JSEntryPtrTag>();
</ins><span class="cx">             break;
</span><span class="cx">         }
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp        2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp   2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -3616,7 +3616,7 @@
</span><span class="cx">             codePtr = functionExecutable->entrypointFor(kind, ArityCheckNotRequired);
</span><span class="cx">     }
</span><span class="cx">     
</span><del>-    linkDirectFor(callFrame, *callLinkInfo, codeBlock, codePtr);
</del><ins>+    linkDirectCall(callFrame, *callLinkInfo, codeBlock, codePtr);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> JSC_DEFINE_JIT_OPERATION(operationTriggerReoptimizationNow, void, (CodeBlock* codeBlock, CodeBlock* optimizedCodeBlock, OSRExitBase* exit))
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJIT32_64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp       2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp  2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -838,19 +838,16 @@
</span><span class="cx">         if (isTail) {
</span><span class="cx">             RELEASE_ASSERT(node->op() == DirectTailCall);
</span><span class="cx">             
</span><del>-            JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
</del><span class="cx">             JITCompiler::Label mainPath = m_jit.label();
</span><span class="cx">             
</span><span class="cx">             m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">             
</span><del>-            info->setFrameShuffleData(shuffleData);
-            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-            
-            JITCompiler::Call call = m_jit.nearTailCall();
-            
</del><ins>+            info->emitDirectTailCallFastPath(m_jit, [&] {
+                info->setFrameShuffleData(shuffleData);
+                CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            });
+
</ins><span class="cx">             JITCompiler::Label slowPath = m_jit.label();
</span><del>-            patchableJump.m_jump.linkTo(slowPath, &m_jit);
-            
</del><span class="cx">             silentSpillAllRegisters(InvalidGPRReg);
</span><span class="cx">             callOperation(operationLinkDirectCall, info, calleePayloadGPR);
</span><span class="cx">             silentFillAllRegisters();
</span><span class="lines">@@ -859,7 +856,7 @@
</span><span class="cx">             
</span><span class="cx">             useChildren(node);
</span><span class="cx">             
</span><del>-            m_jit.addJSDirectTailCall(patchableJump, call, slowPath, info);
</del><ins>+            m_jit.addJSDirectCall(slowPath, info);
</ins><span class="cx">             return;
</span><span class="cx">         }
</span><span class="cx">         
</span><span class="lines">@@ -867,7 +864,7 @@
</span><span class="cx">         
</span><span class="cx">         m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">         
</span><del>-        JITCompiler::Call call = m_jit.nearCall();
</del><ins>+        info->emitDirectFastPath(m_jit);
</ins><span class="cx">         JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">         
</span><span class="cx">         JITCompiler::Label slowPath = m_jit.label();
</span><span class="lines">@@ -879,7 +876,7 @@
</span><span class="cx">         
</span><span class="cx">         setResultAndResetStack();
</span><span class="cx">         
</span><del>-        m_jit.addJSDirectCall(call, slowPath, info);
</del><ins>+        m_jit.addJSDirectCall(slowPath, info);
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx">     
</span><span class="lines">@@ -888,21 +885,25 @@
</span><span class="cx">     slowPath.append(m_jit.branchIfNotCell(JSValueRegs(calleeTagGPR, calleePayloadGPR)));
</span><span class="cx">     slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
</span><span class="cx"> 
</span><ins>+    CCallHelpers::JumpList slowCases;
</ins><span class="cx">     if (isTail) {
</span><del>-        if (node->op() == TailCall) {
-            info->setFrameShuffleData(shuffleData);
-            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-        } else {
-            m_jit.emitRestoreCalleeSaves();
-            m_jit.prepareForTailCallSlow();
-        }
-    }
</del><ins>+        slowCases = info->emitTailCallFastPath(m_jit, calleePayloadGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+            if (node->op() == TailCall) {
+                info->setFrameShuffleData(shuffleData);
+                CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            } else {
+                m_jit.emitRestoreCalleeSaves();
+                m_jit.prepareForTailCallSlow();
+            }
+        });
+    } else
+        slowCases = info->emitFastPath(m_jit, calleePayloadGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx"> 
</span><del>-    JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
-
</del><span class="cx">     JITCompiler::Jump done = m_jit.jump();
</span><span class="cx"> 
</span><span class="cx">     slowPath.link(&m_jit);
</span><ins>+    slowCases.link(&m_jit);
+    auto slowPathStart = m_jit.label();
</ins><span class="cx"> 
</span><span class="cx">     if (node->op() == TailCall) {
</span><span class="cx">         CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
</span><span class="lines">@@ -927,11 +928,11 @@
</span><span class="cx">             m_jit.emitRestoreCalleeSaves();
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    m_jit.move(TrustedImmPtr(info), GPRInfo::regT2);
</del><span class="cx">     m_jit.move(TrustedImmPtr::weakPointer(m_graph, globalObject), GPRInfo::regT3);
</span><del>-    JITCompiler::Call slowCall = m_jit.nearCall();
</del><ins>+    info->emitSlowPath(vm(), m_jit);
</ins><span class="cx"> 
</span><span class="cx">     done.link(&m_jit);
</span><ins>+    auto doneLocation = m_jit.label();
</ins><span class="cx"> 
</span><span class="cx">     if (isTail)
</span><span class="cx">         m_jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="lines">@@ -938,7 +939,7 @@
</span><span class="cx">     else
</span><span class="cx">         setResultAndResetStack();
</span><span class="cx"> 
</span><del>-    m_jit.addJSCall(fastCall, slowCall, targetToCheck, info);
</del><ins>+    m_jit.addJSCall(slowPathStart, doneLocation, info);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> template<bool strict>
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoredfgDFGSpeculativeJIT64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp  2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp     2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -830,18 +830,14 @@
</span><span class="cx">         if (isTail) {
</span><span class="cx">             RELEASE_ASSERT(node->op() == DirectTailCall);
</span><span class="cx">             
</span><del>-            JITCompiler::PatchableJump patchableJump = m_jit.patchableJump();
</del><span class="cx">             JITCompiler::Label mainPath = m_jit.label();
</span><del>-            
</del><span class="cx">             m_jit.emitStoreCallSiteIndex(callSite);
</span><del>-            
-            callLinkInfo->setFrameShuffleData(shuffleData);
-            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-            
-            JITCompiler::Call call = m_jit.nearTailCall();
-            
</del><ins>+
+            callLinkInfo->emitDirectTailCallFastPath(m_jit, [&] {
+                callLinkInfo->setFrameShuffleData(shuffleData);
+                CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            });
</ins><span class="cx">             JITCompiler::Label slowPath = m_jit.label();
</span><del>-            patchableJump.m_jump.linkTo(slowPath, &m_jit);
</del><span class="cx">             
</span><span class="cx">             silentSpillAllRegisters(InvalidGPRReg);
</span><span class="cx">             callOperation(operationLinkDirectCall, callLinkInfo, calleeGPR);
</span><span class="lines">@@ -851,15 +847,13 @@
</span><span class="cx">             
</span><span class="cx">             useChildren(node);
</span><span class="cx">             
</span><del>-            m_jit.addJSDirectTailCall(patchableJump, call, slowPath, callLinkInfo);
</del><ins>+            m_jit.addJSDirectCall(slowPath, callLinkInfo);
</ins><span class="cx">             return;
</span><span class="cx">         }
</span><span class="cx">         
</span><span class="cx">         JITCompiler::Label mainPath = m_jit.label();
</span><del>-        
</del><span class="cx">         m_jit.emitStoreCallSiteIndex(callSite);
</span><del>-        
-        JITCompiler::Call call = m_jit.nearCall();
</del><ins>+        callLinkInfo->emitDirectFastPath(m_jit);
</ins><span class="cx">         JITCompiler::Jump done = m_jit.jump();
</span><span class="cx">         
</span><span class="cx">         JITCompiler::Label slowPath = m_jit.label();
</span><span class="lines">@@ -874,30 +868,29 @@
</span><span class="cx">         
</span><span class="cx">         setResultAndResetStack();
</span><span class="cx">         
</span><del>-        m_jit.addJSDirectCall(call, slowPath, callLinkInfo);
</del><ins>+        m_jit.addJSDirectCall(slowPath, callLinkInfo);
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx">     
</span><span class="cx">     m_jit.emitStoreCallSiteIndex(callSite);
</span><span class="cx">     
</span><del>-    JITCompiler::DataLabelPtr targetToCheck;
-    JITCompiler::Jump slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, TrustedImmPtr(nullptr));
-
</del><ins>+    CCallHelpers::JumpList slowCases;
</ins><span class="cx">     if (isTail) {
</span><del>-        if (node->op() == TailCall) {
-            callLinkInfo->setFrameShuffleData(shuffleData);
-            CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
-        } else {
-            m_jit.emitRestoreCalleeSaves();
-            m_jit.prepareForTailCallSlow();
-        }
-    }
-
-    JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall();
-
</del><ins>+        slowCases = callLinkInfo->emitTailCallFastPath(m_jit, calleeGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+            if (node->op() == TailCall) {
+                callLinkInfo->setFrameShuffleData(shuffleData);
+                CallFrameShuffler(m_jit, shuffleData).prepareForTailCall();
+            } else {
+                m_jit.emitRestoreCalleeSaves();
+                m_jit.prepareForTailCallSlow();
+            }
+        });
+    } else
+        slowCases = callLinkInfo->emitFastPath(m_jit, calleeGPR, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx">     JITCompiler::Jump done = m_jit.jump();
</span><span class="cx"> 
</span><del>-    slowPath.link(&m_jit);
</del><ins>+    slowCases.link(&m_jit);
+    auto slowPathStart = m_jit.label();
</ins><span class="cx"> 
</span><span class="cx">     if (node->op() == TailCall) {
</span><span class="cx">         CallFrameShuffler callFrameShuffler(m_jit, shuffleData);
</span><span class="lines">@@ -905,16 +898,15 @@
</span><span class="cx">         callFrameShuffler.prepareForSlowPath();
</span><span class="cx">     } else {
</span><span class="cx">         m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0
</span><del>-
</del><span class="cx">         if (isTail)
</span><span class="cx">             m_jit.emitRestoreCalleeSaves(); // This needs to happen after we moved calleeGPR to regT0
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    m_jit.move(TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2
</del><span class="cx">     m_jit.move(TrustedImmPtr::weakPointer(m_graph, globalObject), GPRInfo::regT3); // JSGlobalObject needs to be in regT3
</span><del>-    JITCompiler::Call slowCall = m_jit.nearCall();
</del><ins>+    callLinkInfo->emitSlowPath(vm(), m_jit);
</ins><span class="cx"> 
</span><span class="cx">     done.link(&m_jit);
</span><ins>+    auto doneLocation = m_jit.label();
</ins><span class="cx"> 
</span><span class="cx">     if (isTail)
</span><span class="cx">         m_jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="lines">@@ -921,7 +913,7 @@
</span><span class="cx">     else
</span><span class="cx">         setResultAndResetStack();
</span><span class="cx"> 
</span><del>-    m_jit.addJSCall(fastCall, slowCall, targetToCheck, callLinkInfo);
</del><ins>+    m_jit.addJSCall(slowPathStart, doneLocation, callLinkInfo);
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> // Clang should allow unreachable [[clang::fallthrough]] in template functions if any template expansion uses it
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreftlFTLLowerDFGToB3cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp      2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -9717,24 +9717,20 @@
</span><span class="cx">                     CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><span class="cx"> 
</span><span class="cx">                 CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><ins>+                callLinkInfo->setUpCall(
+                    node->op() == Construct ? CallLinkInfo::Construct : CallLinkInfo::Call, GPRInfo::regT0);
</ins><span class="cx"> 
</span><del>-                CCallHelpers::DataLabelPtr targetToCheck;
-                CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
-                    CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
-                    CCallHelpers::TrustedImmPtr(nullptr));
-
-                CCallHelpers::Call fastCall = jit.nearCall();
</del><ins>+                auto slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx">                 CCallHelpers::Jump done = jit.jump();
</span><span class="cx"> 
</span><span class="cx">                 slowPath.link(&jit);
</span><ins>+                auto slowPathStart = jit.label();
+                jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
+                callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx"> 
</span><del>-                jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
-                jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
-                CCallHelpers::Call slowCall = jit.nearCall();
</del><span class="cx">                 done.link(&jit);
</span><span class="cx"> 
</span><del>-                callLinkInfo->setUpCall(
-                    node->op() == Construct ? CallLinkInfo::Construct : CallLinkInfo::Call, GPRInfo::regT0);
</del><ins>+                auto doneLocation = jit.label();
</ins><span class="cx"> 
</span><span class="cx">                 jit.addPtr(
</span><span class="cx">                     CCallHelpers::TrustedImm32(-params.proc().frameSize()),
</span><span class="lines">@@ -9742,13 +9738,9 @@
</span><span class="cx"> 
</span><span class="cx">                 jit.addLinkTask(
</span><span class="cx">                     [=] (LinkBuffer& linkBuffer) {
</span><del>-                        MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
-                        linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
-                        callLinkInfo->setCallLocations(
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
-                            linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+                        callLinkInfo->setCodeLocations(
+                            linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+                            linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">                     });
</span><span class="cx">             });
</span><span class="cx"> 
</span><span class="lines">@@ -9857,63 +9849,50 @@
</span><span class="cx">                     shuffleData.setupCalleeSaveRegisters(jit.codeBlock());
</span><span class="cx">                     
</span><span class="cx">                     CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><ins>+                    callLinkInfo->setUpCall(CallLinkInfo::DirectTailCall, InvalidGPRReg);
</ins><span class="cx">                     
</span><del>-                    CCallHelpers::PatchableJump patchableJump = jit.patchableJump();
</del><span class="cx">                     CCallHelpers::Label mainPath = jit.label();
</span><del>-                    
</del><span class="cx">                     jit.store32(
</span><span class="cx">                         CCallHelpers::TrustedImm32(callSiteIndex.bits()),
</span><span class="cx">                         CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><del>-                
-                    callLinkInfo->setFrameShuffleData(shuffleData);
-                    CallFrameShuffler(jit, shuffleData).prepareForTailCall();
</del><ins>+                    callLinkInfo->emitDirectTailCallFastPath(jit, [&] {
+                        callLinkInfo->setFrameShuffleData(shuffleData);
+                        CallFrameShuffler(jit, shuffleData).prepareForTailCall();
+                    });
</ins><span class="cx">                     
</span><del>-                    CCallHelpers::Call call = jit.nearTailCall();
-                    
</del><span class="cx">                     jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">                     
</span><span class="cx">                     CCallHelpers::Label slowPath = jit.label();
</span><del>-                    patchableJump.m_jump.linkTo(slowPath, &jit);
</del><span class="cx">                     callOperation(
</span><span class="cx">                         *state, toSave, jit,
</span><span class="cx">                         node->origin.semantic, exceptions.get(), operationLinkDirectCall,
</span><span class="cx">                         InvalidGPRReg, CCallHelpers::TrustedImmPtr(callLinkInfo), calleeGPR).call();
</span><span class="cx">                     jit.jump().linkTo(mainPath, &jit);
</span><del>-                    
-                    callLinkInfo->setUpCall(CallLinkInfo::DirectTailCall, InvalidGPRReg);
</del><span class="cx">                     callLinkInfo->setExecutableDuringCompilation(executable);
</span><span class="cx">                     if (numAllocatedArgs > numPassedArgs)
</span><span class="cx">                         callLinkInfo->setMaxArgumentCountIncludingThis(numAllocatedArgs);
</span><span class="cx">                     
</span><del>-                    jit.addLinkTask(
-                        [=] (LinkBuffer& linkBuffer) {
-                            CodeLocationLabel<JSInternalPtrTag> patchableJumpLocation = linkBuffer.locationOf<JSInternalPtrTag>(patchableJump);
-                            CodeLocationNearCall<JSInternalPtrTag> callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
-                            CodeLocationLabel<JSInternalPtrTag> slowPathLocation = linkBuffer.locationOf<JSInternalPtrTag>(slowPath);
-
-                            callLinkInfo->setCallLocations(
-                                patchableJumpLocation,
-                                slowPathLocation,
-                                callLocation);
-                        });
</del><ins>+                    jit.addLinkTask([=] (LinkBuffer& linkBuffer) {
+                        callLinkInfo->setCodeLocations(
+                            linkBuffer.locationOf<JSInternalPtrTag>(slowPath),
+                            CodeLocationLabel<JSInternalPtrTag>());
+                    });
</ins><span class="cx">                     return;
</span><span class="cx">                 }
</span><span class="cx">                 
</span><span class="cx">                 CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(node->origin.semantic);
</span><del>-                
</del><ins>+                callLinkInfo->setUpCall(
+                    isConstruct ? CallLinkInfo::DirectConstruct : CallLinkInfo::DirectCall, InvalidGPRReg);
+
</ins><span class="cx">                 CCallHelpers::Label mainPath = jit.label();
</span><del>-
</del><span class="cx">                 jit.store32(
</span><span class="cx">                     CCallHelpers::TrustedImm32(callSiteIndex.bits()),
</span><span class="cx">                     CCallHelpers::tagFor(VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
</span><del>-                
-                CCallHelpers::Call call = jit.nearCall();
</del><ins>+                callLinkInfo->emitDirectFastPath(jit);
</ins><span class="cx">                 jit.addPtr(
</span><span class="cx">                     CCallHelpers::TrustedImm32(-params.proc().frameSize()),
</span><span class="cx">                     GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="cx">                 
</span><del>-                callLinkInfo->setUpCall(
-                    isConstruct ? CallLinkInfo::DirectConstruct : CallLinkInfo::DirectCall, InvalidGPRReg);
</del><span class="cx">                 callLinkInfo->setExecutableDuringCompilation(executable);
</span><span class="cx">                 if (numAllocatedArgs > numPassedArgs)
</span><span class="cx">                     callLinkInfo->setMaxArgumentCountIncludingThis(numAllocatedArgs);
</span><span class="lines">@@ -9935,15 +9914,9 @@
</span><span class="cx">                         
</span><span class="cx">                         jit.addLinkTask(
</span><span class="cx">                             [=] (LinkBuffer& linkBuffer) {
</span><del>-                                CodeLocationNearCall<JSInternalPtrTag> callLocation = linkBuffer.locationOfNearCall<JSInternalPtrTag>(call);
-                                CodeLocationLabel<JSInternalPtrTag> slowPathLocation = linkBuffer.locationOf<JSInternalPtrTag>(slowPath);
-
-                                linkBuffer.link(call, slowPathLocation);
-
-                                callLinkInfo->setCallLocations(
-                                    CodeLocationLabel<JSInternalPtrTag>(),
-                                    slowPathLocation,
-                                    callLocation);
</del><ins>+                                callLinkInfo->setCodeLocations(
+                                    linkBuffer.locationOf<JSInternalPtrTag>(slowPath), 
+                                    CodeLocationLabel<JSInternalPtrTag>());
</ins><span class="cx">                             });
</span><span class="cx">                     });
</span><span class="cx">             });
</span><span class="lines">@@ -10034,40 +10007,30 @@
</span><span class="cx">                 shuffleData.setupCalleeSaveRegisters(jit.codeBlock());
</span><span class="cx"> 
</span><span class="cx">                 CallLinkInfo* callLinkInfo = jit.codeBlock()->addCallLinkInfo(codeOrigin);
</span><ins>+                callLinkInfo->setUpCall(CallLinkInfo::TailCall, GPRInfo::regT0);
</ins><span class="cx"> 
</span><del>-                CCallHelpers::DataLabelPtr targetToCheck;
-                CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
-                    CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
-                    CCallHelpers::TrustedImmPtr(nullptr));
</del><ins>+                auto slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+                    callLinkInfo->setFrameShuffleData(shuffleData);
+                    CallFrameShuffler(jit, shuffleData).prepareForTailCall();
+                });
</ins><span class="cx"> 
</span><del>-                callLinkInfo->setFrameShuffleData(shuffleData);
-                CallFrameShuffler(jit, shuffleData).prepareForTailCall();
-
-                CCallHelpers::Call fastCall = jit.nearTailCall();
-
</del><span class="cx">                 slowPath.link(&jit);
</span><del>-
</del><ins>+                auto slowPathStart = jit.label();
</ins><span class="cx">                 CallFrameShuffler slowPathShuffler(jit, shuffleData);
</span><span class="cx">                 slowPathShuffler.setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT0));
</span><span class="cx">                 slowPathShuffler.prepareForSlowPath();
</span><span class="cx"> 
</span><del>-                jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx">                 jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>-                CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+                callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx"> 
</span><ins>+                auto doneLocation = jit.label();
</ins><span class="cx">                 jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx"> 
</span><del>-                callLinkInfo->setUpCall(CallLinkInfo::TailCall, GPRInfo::regT0);
-
</del><span class="cx">                 jit.addLinkTask(
</span><span class="cx">                     [=] (LinkBuffer& linkBuffer) {
</span><del>-                        MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
-                        linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-
-                        callLinkInfo->setCallLocations(
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
-                            linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+                        callLinkInfo->setCodeLocations(
+                            linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+                            linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">                     });
</span><span class="cx">             });
</span><span class="cx">     }
</span><span class="lines">@@ -10320,41 +10283,39 @@
</span><span class="cx">                 else
</span><span class="cx">                     callType = CallLinkInfo::CallVarargs;
</span><span class="cx">                 
</span><ins>+                callLinkInfo->setUpCall(callType, GPRInfo::regT0);
+
</ins><span class="cx">                 bool isTailCall = CallLinkInfo::callModeFor(callType) == CallMode::Tail;
</span><del>-                
-                CCallHelpers::DataLabelPtr targetToCheck;
-                CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
-                    CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
-                    CCallHelpers::TrustedImmPtr(nullptr));
-                
-                CCallHelpers::Call fastCall;
</del><ins>+
+                ASSERT(!usedRegisters.get(GPRInfo::regT2)); // Used on the slow path.
+
+                CCallHelpers::JumpList slowPath;
</ins><span class="cx">                 CCallHelpers::Jump done;
</span><del>-                
</del><span class="cx">                 if (isTailCall) {
</span><del>-                    jit.emitRestoreCalleeSaves();
-                    jit.prepareForTailCallSlow();
-                    fastCall = jit.nearTailCall();
</del><ins>+                    slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+                        jit.emitRestoreCalleeSaves();
+                        jit.prepareForTailCallSlow();
+                    });
</ins><span class="cx">                 } else {
</span><del>-                    fastCall = jit.nearCall();
</del><ins>+                    slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx">                     done = jit.jump();
</span><span class="cx">                 }
</span><span class="cx">                 
</span><span class="cx">                 slowPath.link(&jit);
</span><ins>+                auto slowPathStart = jit.label();
</ins><span class="cx"> 
</span><span class="cx">                 if (isTailCall)
</span><span class="cx">                     jit.emitRestoreCalleeSaves();
</span><del>-                ASSERT(!usedRegisters.get(GPRInfo::regT2));
-                jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx">                 jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>-                CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+                callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">                 
</span><span class="cx">                 if (isTailCall)
</span><span class="cx">                     jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">                 else
</span><span class="cx">                     done.link(&jit);
</span><del>-                
-                callLinkInfo->setUpCall(callType, GPRInfo::regT0);
</del><span class="cx"> 
</span><ins>+                auto doneLocation = jit.label();
+
</ins><span class="cx">                 jit.addPtr(
</span><span class="cx">                     CCallHelpers::TrustedImm32(-originalStackHeight),
</span><span class="cx">                     GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="lines">@@ -10361,13 +10322,9 @@
</span><span class="cx">                 
</span><span class="cx">                 jit.addLinkTask(
</span><span class="cx">                     [=] (LinkBuffer& linkBuffer) {
</span><del>-                        MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
-                        linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-                        
-                        callLinkInfo->setCallLocations(
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
-                            linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+                        callLinkInfo->setCodeLocations(
+                            linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+                            linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">                     });
</span><span class="cx">             });
</span><span class="cx"> 
</span><span class="lines">@@ -10607,40 +10564,37 @@
</span><span class="cx">                 else
</span><span class="cx">                     callType = CallLinkInfo::CallVarargs;
</span><span class="cx">                 
</span><ins>+                callLinkInfo->setUpCall(callType, GPRInfo::regT0);
+                
</ins><span class="cx">                 bool isTailCall = CallLinkInfo::callModeFor(callType) == CallMode::Tail;
</span><span class="cx">                 
</span><del>-                CCallHelpers::DataLabelPtr targetToCheck;
-                CCallHelpers::Jump slowPath = jit.branchPtrWithPatch(
-                    CCallHelpers::NotEqual, GPRInfo::regT0, targetToCheck,
-                    CCallHelpers::TrustedImmPtr(nullptr));
-                
-                CCallHelpers::Call fastCall;
</del><ins>+                CCallHelpers::JumpList slowPath;
</ins><span class="cx">                 CCallHelpers::Jump done;
</span><del>-                
</del><span class="cx">                 if (isTailCall) {
</span><del>-                    jit.emitRestoreCalleeSaves();
-                    jit.prepareForTailCallSlow();
-                    fastCall = jit.nearTailCall();
</del><ins>+                    slowPath = callLinkInfo->emitTailCallFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No, [&] {
+                        jit.emitRestoreCalleeSaves();
+                        jit.prepareForTailCallSlow();
+                    });
</ins><span class="cx">                 } else {
</span><del>-                    fastCall = jit.nearCall();
</del><ins>+                    slowPath = callLinkInfo->emitFastPath(jit, GPRInfo::regT0, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
</ins><span class="cx">                     done = jit.jump();
</span><span class="cx">                 }
</span><span class="cx">                 
</span><span class="cx">                 slowPath.link(&jit);
</span><ins>+                auto slowPathStart = jit.label();
</ins><span class="cx"> 
</span><span class="cx">                 if (isTailCall)
</span><span class="cx">                     jit.emitRestoreCalleeSaves();
</span><del>-                jit.move(CCallHelpers::TrustedImmPtr(callLinkInfo), GPRInfo::regT2);
</del><span class="cx">                 jit.move(CCallHelpers::TrustedImmPtr(jit.codeBlock()->globalObjectFor(node->origin.semantic)), GPRInfo::regT3);
</span><del>-                CCallHelpers::Call slowCall = jit.nearCall();
</del><ins>+                callLinkInfo->emitSlowPath(*vm, jit);
</ins><span class="cx">                 
</span><span class="cx">                 if (isTailCall)
</span><span class="cx">                     jit.abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">                 else
</span><span class="cx">                     done.link(&jit);
</span><ins>+
+                auto doneLocation = jit.label();
</ins><span class="cx">                 
</span><del>-                callLinkInfo->setUpCall(callType, GPRInfo::regT0);
-                
</del><span class="cx">                 jit.addPtr(
</span><span class="cx">                     CCallHelpers::TrustedImm32(-originalStackHeight),
</span><span class="cx">                     GPRInfo::callFrameRegister, CCallHelpers::stackPointerRegister);
</span><span class="lines">@@ -10647,13 +10601,9 @@
</span><span class="cx">                 
</span><span class="cx">                 jit.addLinkTask(
</span><span class="cx">                     [=] (LinkBuffer& linkBuffer) {
</span><del>-                        MacroAssemblerCodePtr<JITThunkPtrTag> linkCall = vm->getCTIStub(linkCallThunkGenerator).code();
-                        linkBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(linkCall));
-                        
-                        callLinkInfo->setCallLocations(
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall)),
-                            CodeLocationLabel<JSInternalPtrTag>(linkBuffer.locationOf<JSInternalPtrTag>(targetToCheck)),
-                            linkBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall));
</del><ins>+                        callLinkInfo->setCodeLocations(
+                            linkBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+                            linkBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx">                     });
</span><span class="cx">             });
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitCCallHelperscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/CCallHelpers.cpp    2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -71,6 +71,16 @@
</span><span class="cx">     storePtr(scratch2, Address(scratch1NonArgGPR));
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+void CCallHelpers::emitJITCodeOver(MacroAssemblerCodePtr<JSInternalPtrTag> where, WTF::Function<void(CCallHelpers&)> emitCode, const char* description)
+{
+    CCallHelpers jit;
+    emitCode(jit);
+
+    constexpr bool needsBranchCompaction = false;
+    LinkBuffer linkBuffer(jit, where, jit.m_assembler.buffer().codeSize(), LinkBuffer::Profile::InlineCache, JITCompilationMustSucceed, needsBranchCompaction);
+    FINALIZE_CODE(linkBuffer, NoPtrTag, description);
+}
+
</ins><span class="cx"> } // namespace JSC
</span><span class="cx"> 
</span><span class="cx"> #endif // ENABLE(JIT)
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitCCallHelpersh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/CCallHelpers.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/CCallHelpers.h   2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/CCallHelpers.h      2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -818,6 +818,8 @@
</span><span class="cx">     void logShadowChickenTailPacket(GPRReg shadowPacket, JSValueRegs thisRegs, GPRReg scope, CodeBlock*, CallSiteIndex);
</span><span class="cx">     // Leaves behind a pointer to the Packet we should write to in shadowPacket.
</span><span class="cx">     void ensureShadowChickenPacket(VM&, GPRReg shadowPacket, GPRReg scratch1NonArgGPR, GPRReg scratch2);
</span><ins>+
+    static void emitJITCodeOver(MacroAssemblerCodePtr<JSInternalPtrTag> where, WTF::Function<void(CCallHelpers&)>, const char*);
</ins><span class="cx"> };
</span><span class="cx"> 
</span><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JIT.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JIT.cpp  2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JIT.cpp     2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -963,10 +963,9 @@
</span><span class="cx"> 
</span><span class="cx">     for (auto& compilationInfo : m_callCompilationInfo) {
</span><span class="cx">         CallLinkInfo& info = *compilationInfo.callLinkInfo;
</span><del>-        info.setCallLocations(
-            CodeLocationLabel<JSInternalPtrTag>(patchBuffer.locationOfNearCall<JSInternalPtrTag>(compilationInfo.callReturnLocation)),
-            CodeLocationLabel<JSInternalPtrTag>(patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.hotPathBegin)),
-            patchBuffer.locationOfNearCall<JSInternalPtrTag>(compilationInfo.hotPathOther));
</del><ins>+        info.setCodeLocations(
+            patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.slowPathStart),
+            patchBuffer.locationOf<JSInternalPtrTag>(compilationInfo.doneLocation));
</ins><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     {
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JIT.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JIT.h    2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JIT.h       2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -186,9 +186,8 @@
</span><span class="cx">     };
</span><span class="cx"> 
</span><span class="cx">     struct CallCompilationInfo {
</span><del>-        MacroAssembler::DataLabelPtr hotPathBegin;
-        MacroAssembler::Call hotPathOther;
-        MacroAssembler::Call callReturnLocation;
</del><ins>+        MacroAssembler::Label slowPathStart;
+        MacroAssembler::Label doneLocation;
</ins><span class="cx">         CallLinkInfo* callLinkInfo;
</span><span class="cx">     };
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITCallcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITCall.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITCall.cpp      2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITCall.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -188,8 +188,14 @@
</span><span class="cx">         ValueRecovery::inGPR(regT0, DataFormatJS);
</span><span class="cx">     shuffleData.setupCalleeSaveRegisters(m_codeBlock);
</span><span class="cx">     info->setFrameShuffleData(shuffleData);
</span><del>-    CallFrameShuffler(*this, shuffleData).prepareForTailCall();
-    m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+
+    JumpList slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+        CallFrameShuffler(*this, shuffleData).prepareForTailCall();
+    });
+    addSlowCase(slowPaths);
+    auto doneLocation = label();
+    m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
+    
</ins><span class="cx">     return true;
</span><span class="cx"> }
</span><span class="cx"> 
</span><span class="lines">@@ -228,29 +234,31 @@
</span><span class="cx">     if (compileCallEval(bytecode))
</span><span class="cx">         return;
</span><span class="cx"> 
</span><del>-    DataLabelPtr addressOfLinkedFunctionCheck;
-    Jump slowCase = branchPtrWithPatch(NotEqual, regT0, addressOfLinkedFunctionCheck, TrustedImmPtr(nullptr));
-    addSlowCase(slowCase);
-
</del><span class="cx">     ASSERT(m_callCompilationInfo.size() == callLinkInfoIndex);
</span><span class="cx">     info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</span><span class="cx">     m_callCompilationInfo.append(CallCompilationInfo());
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].hotPathBegin = addressOfLinkedFunctionCheck;
</del><span class="cx">     m_callCompilationInfo[callLinkInfoIndex].callLinkInfo = info;
</span><span class="cx"> 
</span><del>-    if (compileTailCall(bytecode, info, callLinkInfoIndex)) {
</del><ins>+    if (compileTailCall(bytecode, info, callLinkInfoIndex))
</ins><span class="cx">         return;
</span><del>-    }
</del><span class="cx"> 
</span><span class="cx">     if (opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</span><del>-        emitRestoreCalleeSaves();
-        prepareForTailCallSlow();
-        m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+        auto slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+            emitRestoreCalleeSaves();
+            prepareForTailCallSlow(regT2);
+        });
+        addSlowCase(slowPaths);
+        auto doneLocation = label();
+        m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearCall();
</del><ins>+    auto slowPaths = info->emitFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes);
+    auto doneLocation = label();
+    addSlowCase(slowPaths);
</ins><span class="cx"> 
</span><ins>+    m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
+
</ins><span class="cx">     addPtr(TrustedImm32(stackPointerOffsetFor(m_codeBlock) * sizeof(Register)), callFrameRegister, stackPointerRegister);
</span><span class="cx">     checkStackPointerAlignment();
</span><span class="cx"> 
</span><span class="lines">@@ -265,16 +273,15 @@
</span><span class="cx"> 
</span><span class="cx">     linkAllSlowCases(iter);
</span><span class="cx"> 
</span><ins>+    m_callCompilationInfo[callLinkInfoIndex].slowPathStart = label();
+
</ins><span class="cx">     if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments)
</span><span class="cx">         emitRestoreCalleeSaves();
</span><span class="cx"> 
</span><span class="cx">     move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
</span><del>-    move(TrustedImmPtr(m_callCompilationInfo[callLinkInfoIndex].callLinkInfo), regT2);
</del><ins>+    m_callCompilationInfo[callLinkInfoIndex].callLinkInfo->emitSlowPath(*m_vm, *this);
</ins><span class="cx"> 
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].callReturnLocation =
-        emitNakedNearCall(m_vm->getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
-
-    if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) {
</del><ins>+    if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</ins><span class="cx">         abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">         return;
</span><span class="cx">     }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITCall32_64cpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp    2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -301,30 +301,28 @@
</span><span class="cx">     if (compileCallEval(bytecode))
</span><span class="cx">         return;
</span><span class="cx"> 
</span><del>-    if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs)
-        emitRestoreCalleeSaves();
-
-    addSlowCase(branchIfNotCell(regT1));
-
-    DataLabelPtr addressOfLinkedFunctionCheck;
-    Jump slowCase = branchPtrWithPatch(NotEqual, regT0, addressOfLinkedFunctionCheck, TrustedImmPtr(nullptr));
-
-    addSlowCase(slowCase);
-
</del><ins>+    info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</ins><span class="cx">     ASSERT(m_callCompilationInfo.size() == callLinkInfoIndex);
</span><del>-    info->setUpCall(CallLinkInfo::callTypeFor(opcodeID), regT0);
</del><span class="cx">     m_callCompilationInfo.append(CallCompilationInfo());
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].hotPathBegin = addressOfLinkedFunctionCheck;
</del><span class="cx">     m_callCompilationInfo[callLinkInfoIndex].callLinkInfo = info;
</span><span class="cx"> 
</span><ins>+    addSlowCase(branchIfNotCell(regT1));
+
</ins><span class="cx">     checkStackPointerAlignment();
</span><span class="cx">     if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</span><del>-        prepareForTailCallSlow();
-        m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearTailCall();
</del><ins>+        auto slowPaths = info->emitTailCallFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes, [&] {
+            emitRestoreCalleeSaves();
+            prepareForTailCallSlow(regT2);
+        });
+        addSlowCase(slowPaths);
+        auto doneLocation = label();
+        m_callCompilationInfo[callLinkInfoIndex].doneLocation = doneLocation;
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx"> 
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedNearCall();
</del><ins>+    auto slowPaths = info->emitFastPath(*this, regT0, regT2, CallLinkInfo::UseDataIC::Yes);
+    addSlowCase(slowPaths);
+    m_callCompilationInfo[callLinkInfoIndex].doneLocation = label();
</ins><span class="cx"> 
</span><span class="cx">     addPtr(TrustedImm32(stackPointerOffsetFor(m_codeBlock) * sizeof(Register)), callFrameRegister, stackPointerRegister);
</span><span class="cx">     checkStackPointerAlignment();
</span><span class="lines">@@ -343,16 +341,15 @@
</span><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     linkAllSlowCases(iter);
</span><ins>+    m_callCompilationInfo[callLinkInfoIndex].slowPathStart = label();
</ins><span class="cx"> 
</span><del>-    move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
-    move(TrustedImmPtr(m_callCompilationInfo[callLinkInfoIndex].callLinkInfo), regT2);
-
</del><span class="cx">     if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments)
</span><span class="cx">         emitRestoreCalleeSaves();
</span><span class="cx"> 
</span><del>-    m_callCompilationInfo[callLinkInfoIndex].callReturnLocation = emitNakedNearCall(m_vm->getCTIStub(linkCallThunkGenerator).retaggedCode<NoPtrTag>());
</del><ins>+    move(TrustedImmPtr(m_codeBlock->globalObject()), regT3);
+    m_callCompilationInfo[callLinkInfoIndex].callLinkInfo->emitSlowPath(*m_vm, *this);
</ins><span class="cx"> 
</span><del>-    if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) {
</del><ins>+    if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs || opcodeID == op_tail_call_forward_arguments) {
</ins><span class="cx">         abortWithReason(JITDidReturnFromTailCall);
</span><span class="cx">         return;
</span><span class="cx">     }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitJITOperationscpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/JITOperations.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/JITOperations.cpp        2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/JITOperations.cpp   2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1386,7 +1386,7 @@
</span><span class="cx">             if (!callLinkInfo->seenOnce())
</span><span class="cx">                 callLinkInfo->setSeen();
</span><span class="cx">             else
</span><del>-                linkFor(vm, calleeFrame, *callLinkInfo, nullptr, internalFunction, codePtr);
</del><ins>+                linkMonomorphicCall(vm, calleeFrame, *callLinkInfo, nullptr, internalFunction, codePtr);
</ins><span class="cx"> 
</span><span class="cx">             void* linkedTarget = codePtr.executableAddress();
</span><span class="cx">             return encodeResult(linkedTarget, reinterpret_cast<void*>(callLinkInfo->callMode() == CallMode::Tail ? ReuseTheFrame : KeepTheFrame));
</span><span class="lines">@@ -1434,7 +1434,7 @@
</span><span class="cx">     if (!callLinkInfo->seenOnce())
</span><span class="cx">         callLinkInfo->setSeen();
</span><span class="cx">     else
</span><del>-        linkFor(vm, calleeFrame, *callLinkInfo, codeBlock, callee, codePtr);
</del><ins>+        linkMonomorphicCall(vm, calleeFrame, *callLinkInfo, codeBlock, callee, codePtr);
</ins><span class="cx"> 
</span><span class="cx">     return encodeResult(codePtr.executableAddress(), reinterpret_cast<void*>(callLinkInfo->callMode() == CallMode::Tail ? ReuseTheFrame : KeepTheFrame));
</span><span class="cx"> }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitPolymorphicCallStubRoutinecpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp   2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/PolymorphicCallStubRoutine.cpp      2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -45,7 +45,7 @@
</span><span class="cx"> void PolymorphicCallNode::unlink(VM& vm)
</span><span class="cx"> {
</span><span class="cx">     if (m_callLinkInfo) {
</span><del>-        dataLogLnIf(Options::dumpDisassembly(), "Unlinking polymorphic call at ", m_callLinkInfo->callReturnLocation(), ", bc#", m_callLinkInfo->codeOrigin().bytecodeIndex());
</del><ins>+        dataLogLnIf(Options::dumpDisassembly(), "Unlinking polymorphic call at ", m_callLinkInfo->doneLocation(), ", bc#", m_callLinkInfo->codeOrigin().bytecodeIndex());
</ins><span class="cx">         m_callLinkInfo->unlink(vm);
</span><span class="cx">     }
</span><span class="cx"> 
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitRepatchcpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/Repatch.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/Repatch.cpp      2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/Repatch.cpp 2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -1188,20 +1188,20 @@
</span><span class="cx">         ftlThunkAwareRepatchCall(codeBlock, stubInfo.slowPathCallLocation, operationInstanceOfGeneric);
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-static void linkSlowFor(VM&, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef<JITStubRoutinePtrTag> codeRef)
</del><ins>+static void linkSlowPathTo(VM&, CallLinkInfo& callLinkInfo, MacroAssemblerCodeRef<JITStubRoutinePtrTag> codeRef)
</ins><span class="cx"> {
</span><del>-    MacroAssembler::repatchNearCall(callLinkInfo.callReturnLocation(), CodeLocationLabel<JITStubRoutinePtrTag>(codeRef.code()));
</del><ins>+    callLinkInfo.setSlowPathCallDestination(codeRef.code().template retagged<JSEntryPtrTag>());
</ins><span class="cx"> }
</span><span class="cx"> 
</span><del>-static void linkSlowFor(VM& vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
</del><ins>+static void linkSlowPathTo(VM& vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
</ins><span class="cx"> {
</span><del>-    linkSlowFor(vm, callLinkInfo, vm.getCTIStub(generator).retagged<JITStubRoutinePtrTag>());
</del><ins>+    linkSlowPathTo(vm, callLinkInfo, vm.getCTIStub(generator).retagged<JITStubRoutinePtrTag>());
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> static void linkSlowFor(VM& vm, CallLinkInfo& callLinkInfo)
</span><span class="cx"> {
</span><span class="cx">     MacroAssemblerCodeRef<JITStubRoutinePtrTag> virtualThunk = virtualThunkFor(vm, callLinkInfo);
</span><del>-    linkSlowFor(vm, callLinkInfo, virtualThunk);
</del><ins>+    linkSlowPathTo(vm, callLinkInfo, virtualThunk);
</ins><span class="cx">     callLinkInfo.setSlowStub(GCAwareJITStubRoutine::create(virtualThunk, vm));
</span><span class="cx"> }
</span><span class="cx"> 
</span><span class="lines">@@ -1217,7 +1217,7 @@
</span><span class="cx"> #endif // ENABLE(WEBASSEMBLY)
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void linkFor(
</del><ins>+void linkMonomorphicCall(
</ins><span class="cx">     VM& vm, CallFrame* callFrame, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock,
</span><span class="cx">     JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</span><span class="cx"> {
</span><span class="lines">@@ -1235,19 +1235,17 @@
</span><span class="cx">     ASSERT(owner);
</span><span class="cx"> 
</span><span class="cx">     ASSERT(!callLinkInfo.isLinked());
</span><del>-    callLinkInfo.setCallee(vm, owner, callee);
-    MacroAssembler::repatchPointer(callLinkInfo.hotPathBegin(), callee);
</del><ins>+    callLinkInfo.setMonomorphicCallee(vm, owner, callee, codePtr);
</ins><span class="cx">     callLinkInfo.setLastSeenCallee(vm, owner, callee);
</span><ins>+
</ins><span class="cx">     if (shouldDumpDisassemblyFor(callerCodeBlock))
</span><span class="cx">         dataLog("Linking call in ", FullCodeOrigin(callerCodeBlock, callLinkInfo.codeOrigin()), " to ", pointerDump(calleeCodeBlock), ", entrypoint at ", codePtr, "\n");
</span><span class="cx"> 
</span><del>-    MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), CodeLocationLabel<JSEntryPtrTag>(codePtr));
-
</del><span class="cx">     if (calleeCodeBlock)
</span><span class="cx">         calleeCodeBlock->linkIncomingCall(callerFrame, &callLinkInfo);
</span><span class="cx"> 
</span><span class="cx">     if (callLinkInfo.specializationKind() == CodeForCall && callLinkInfo.allowStubs()) {
</span><del>-        linkSlowFor(vm, callLinkInfo, linkPolymorphicCallThunkGenerator);
</del><ins>+        linkSlowPathTo(vm, callLinkInfo, linkPolymorphicCallThunkGenerator);
</ins><span class="cx">         return;
</span><span class="cx">     }
</span><span class="cx">     
</span><span class="lines">@@ -1254,7 +1252,7 @@
</span><span class="cx">     linkSlowFor(vm, callLinkInfo);
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void linkDirectFor(
</del><ins>+void linkDirectCall(
</ins><span class="cx">     CallFrame* callFrame, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock,
</span><span class="cx">     MacroAssemblerCodePtr<JSEntryPtrTag> codePtr)
</span><span class="cx"> {
</span><span class="lines">@@ -1269,9 +1267,7 @@
</span><span class="cx">     if (shouldDumpDisassemblyFor(callerCodeBlock))
</span><span class="cx">         dataLog("Linking call in ", FullCodeOrigin(callerCodeBlock, callLinkInfo.codeOrigin()), " to ", pointerDump(calleeCodeBlock), ", entrypoint at ", codePtr, "\n");
</span><span class="cx"> 
</span><del>-    if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
-        MacroAssembler::repatchJumpToNop(callLinkInfo.patchableJump());
-    MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), CodeLocationLabel<JSEntryPtrTag>(codePtr));
</del><ins>+    callLinkInfo.setDirectCallTarget(CodeLocationLabel<JSEntryPtrTag>(codePtr));
</ins><span class="cx"> 
</span><span class="cx">     if (calleeCodeBlock)
</span><span class="cx">         calleeCodeBlock->linkIncomingCall(callFrame, &callLinkInfo);
</span><span class="lines">@@ -1289,21 +1285,16 @@
</span><span class="cx"> {
</span><span class="cx">     if (callLinkInfo.isDirect()) {
</span><span class="cx">         callLinkInfo.clearCodeBlock();
</span><del>-        if (!callLinkInfo.clearedByJettison()) {
-            if (callLinkInfo.callType() == CallLinkInfo::DirectTailCall)
-                MacroAssembler::repatchJump(callLinkInfo.patchableJump(), callLinkInfo.slowPathStart());
-            else
-                MacroAssembler::repatchNearCall(callLinkInfo.hotPathOther(), callLinkInfo.slowPathStart());
-        }
</del><ins>+        if (!callLinkInfo.clearedByJettison())
+            callLinkInfo.initializeDirectCall();
</ins><span class="cx">     } else {
</span><span class="cx">         if (!callLinkInfo.clearedByJettison()) {
</span><del>-            MacroAssembler::revertJumpReplacementToBranchPtrWithPatch(
-                MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
-                callLinkInfo.calleeGPR(), nullptr);
-            linkSlowFor(vm, callLinkInfo, codeRef);
-            MacroAssembler::repatchPointer(callLinkInfo.hotPathBegin(), nullptr);
</del><ins>+            linkSlowPathTo(vm, callLinkInfo, codeRef);
+
+            if (callLinkInfo.stub())
+                callLinkInfo.revertCallToStub();
</ins><span class="cx">         }
</span><del>-        callLinkInfo.clearCallee();
</del><ins>+        callLinkInfo.clearCallee(); // This also clears the inline cache both for data and code-based caches.
</ins><span class="cx">     }
</span><span class="cx">     callLinkInfo.clearSeen();
</span><span class="cx">     callLinkInfo.clearStub();
</span><span class="lines">@@ -1312,9 +1303,9 @@
</span><span class="cx">         callLinkInfo.remove();
</span><span class="cx"> }
</span><span class="cx"> 
</span><del>-void unlinkFor(VM& vm, CallLinkInfo& callLinkInfo)
</del><ins>+void unlinkCall(VM& vm, CallLinkInfo& callLinkInfo)
</ins><span class="cx"> {
</span><del>-    dataLogLnIf(Options::dumpDisassembly(), "Unlinking call at ", callLinkInfo.hotPathOther());
</del><ins>+    dataLogLnIf(Options::dumpDisassembly(), "Unlinking call at ", callLinkInfo.fastPathStart());
</ins><span class="cx">     
</span><span class="cx">     revertCall(vm, callLinkInfo, vm.getCTIStub(linkCallThunkGenerator).retagged<JITStubRoutinePtrTag>());
</span><span class="cx"> }
</span><span class="lines">@@ -1580,7 +1571,7 @@
</span><span class="cx">     }
</span><span class="cx">     stubJit.move(CCallHelpers::TrustedImmPtr(globalObject), GPRInfo::regT3);
</span><span class="cx">     stubJit.move(CCallHelpers::TrustedImmPtr(&callLinkInfo), GPRInfo::regT2);
</span><del>-    stubJit.move(CCallHelpers::TrustedImmPtr(callLinkInfo.callReturnLocation().untaggedExecutableAddress()), GPRInfo::regT4);
</del><ins>+    stubJit.move(CCallHelpers::TrustedImmPtr(callLinkInfo.doneLocation().untaggedExecutableAddress()), GPRInfo::regT4);
</ins><span class="cx">     
</span><span class="cx">     stubJit.restoreReturnAddressBeforeReturn(GPRInfo::regT4);
</span><span class="cx">     AssemblyHelpers::Jump slow = stubJit.jump();
</span><span class="lines">@@ -1603,10 +1594,7 @@
</span><span class="cx">         patchBuffer.link(callToCodePtr.call, FunctionPtr<JSEntryPtrTag>(callToCodePtr.codePtr));
</span><span class="cx"> #endif
</span><span class="cx">     }
</span><del>-    if (isWebAssembly || JITCode::isOptimizingJIT(callerCodeBlock->jitType()))
-        patchBuffer.link(done, callLinkInfo.callReturnLocation().labelAtOffset(0));
-    else
-        patchBuffer.link(done, callLinkInfo.hotPathOther().labelAtOffset(0));
</del><ins>+    patchBuffer.link(done, callLinkInfo.doneLocation());
</ins><span class="cx">     patchBuffer.link(slow, CodeLocationLabel<JITThunkPtrTag>(vm.getCTIStub(linkPolymorphicCallThunkGenerator).code()));
</span><span class="cx">     
</span><span class="cx">     auto stubRoutine = adoptRef(*new PolymorphicCallStubRoutine(
</span><span class="lines">@@ -1613,19 +1601,16 @@
</span><span class="cx">         FINALIZE_CODE_FOR(
</span><span class="cx">             callerCodeBlock, patchBuffer, JITStubRoutinePtrTag,
</span><span class="cx">             "Polymorphic call stub for %s, return point %p, targets %s",
</span><del>-                isWebAssembly ? "WebAssembly" : toCString(*callerCodeBlock).data(), callLinkInfo.callReturnLocation().labelAtOffset(0).executableAddress(),
</del><ins>+                isWebAssembly ? "WebAssembly" : toCString(*callerCodeBlock).data(), callLinkInfo.doneLocation().executableAddress(),
</ins><span class="cx">                 toCString(listDump(callCases)).data()),
</span><span class="cx">         vm, owner, callFrame->callerFrame(), callLinkInfo, callCases,
</span><span class="cx">         WTFMove(fastCounts)));
</span><del>-    
-    MacroAssembler::replaceWithJump(
-        MacroAssembler::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin()),
-        CodeLocationLabel<JITStubRoutinePtrTag>(stubRoutine->code().code()));
</del><ins>+
</ins><span class="cx">     // The original slow path is unreachable on 64-bits, but still
</span><span class="cx">     // reachable on 32-bits since a non-cell callee will always
</span><span class="cx">     // trigger the slow path
</span><span class="cx">     linkSlowFor(vm, callLinkInfo);
</span><del>-    
</del><ins>+
</ins><span class="cx">     // If there had been a previous stub routine, that one will die as soon as the GC runs and sees
</span><span class="cx">     // that it's no longer on stack.
</span><span class="cx">     callLinkInfo.setStub(WTFMove(stubRoutine));
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorejitRepatchh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/jit/Repatch.h (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/jit/Repatch.h        2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/jit/Repatch.h   2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -56,10 +56,10 @@
</span><span class="cx"> void repatchCheckPrivateBrand(JSGlobalObject*, CodeBlock*, JSObject*, CacheableIdentifier, StructureStubInfo&);
</span><span class="cx"> void repatchSetPrivateBrand(JSGlobalObject*, CodeBlock*, JSObject*, Structure*, CacheableIdentifier, StructureStubInfo&);
</span><span class="cx"> void repatchInstanceOf(JSGlobalObject*, CodeBlock*, JSValue value, JSValue prototype, StructureStubInfo&, bool wasFound);
</span><del>-void linkFor(VM&, CallFrame*, CallLinkInfo&, CodeBlock*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
-void linkDirectFor(CallFrame*, CallLinkInfo&, CodeBlock*, MacroAssemblerCodePtr<JSEntryPtrTag>);
</del><ins>+void linkMonomorphicCall(VM&, CallFrame*, CallLinkInfo&, CodeBlock*, JSObject* callee, MacroAssemblerCodePtr<JSEntryPtrTag>);
+void linkDirectCall(CallFrame*, CallLinkInfo&, CodeBlock*, MacroAssemblerCodePtr<JSEntryPtrTag>);
</ins><span class="cx"> void linkSlowFor(CallFrame*, CallLinkInfo&);
</span><del>-void unlinkFor(VM&, CallLinkInfo&);
</del><ins>+void unlinkCall(VM&, CallLinkInfo&);
</ins><span class="cx"> void linkPolymorphicCall(JSGlobalObject*, CallFrame*, CallLinkInfo&, CallVariant);
</span><span class="cx"> void resetGetBy(CodeBlock*, StructureStubInfo&, GetByKind);
</span><span class="cx"> void resetPutByID(CodeBlock*, StructureStubInfo&);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCorewasmjsWasmToJScpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp (277679 => 277680)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp 2021-05-18 21:25:44 UTC (rev 277679)
+++ trunk/Source/JavaScriptCore/wasm/js/WasmToJS.cpp    2021-05-18 21:30:48 UTC (rev 277680)
</span><span class="lines">@@ -261,19 +261,18 @@
</span><span class="cx"> 
</span><span class="cx">     CallLinkInfo* callLinkInfo = callLinkInfos.add(CodeOrigin());
</span><span class="cx">     callLinkInfo->setUpCall(CallLinkInfo::Call, importJSCellGPRReg);
</span><del>-    JIT::DataLabelPtr targetToCheck;
-    JIT::TrustedImmPtr initialRightValue(nullptr);
-    JIT::Jump slowPath = jit.branchPtrWithPatch(MacroAssembler::NotEqual, importJSCellGPRReg, targetToCheck, initialRightValue);
-    JIT::Call fastCall = jit.nearCall();
</del><ins>+    auto slowPath = callLinkInfo->emitFastPath(jit, importJSCellGPRReg, InvalidGPRReg, CallLinkInfo::UseDataIC::No);
+
</ins><span class="cx">     JIT::Jump done = jit.jump();
</span><span class="cx">     slowPath.link(&jit);
</span><ins>+    auto slowPathStart = jit.label();
</ins><span class="cx">     // Callee needs to be in regT0 here.
</span><del>-    jit.move(MacroAssembler::TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2.
</del><span class="cx">     jit.loadWasmContextInstance(GPRInfo::regT3);
</span><span class="cx">     jit.loadPtr(CCallHelpers::Address(GPRInfo::regT3, Instance::offsetOfOwner()), GPRInfo::regT3);
</span><span class="cx">     jit.loadPtr(CCallHelpers::Address(GPRInfo::regT3, JSWebAssemblyInstance::offsetOfGlobalObject()), GPRInfo::regT3);
</span><del>-    JIT::Call slowCall = jit.nearCall();
</del><ins>+    callLinkInfo->emitSlowPath(vm, jit);
</ins><span class="cx">     done.link(&jit);
</span><ins>+    auto doneLocation = jit.label();
</ins><span class="cx"> 
</span><span class="cx">     if (signature.returnCount() == 1) {
</span><span class="cx">         switch (signature.returnType(0).kind) {
</span><span class="lines">@@ -421,11 +420,9 @@
</span><span class="cx">     if (UNLIKELY(patchBuffer.didFailToAllocate()))
</span><span class="cx">         return makeUnexpected(BindingFailure::OutOfMemory);
</span><span class="cx"> 
</span><del>-    patchBuffer.link(slowCall, FunctionPtr<JITThunkPtrTag>(vm.getCTIStub(linkCallThunkGenerator).code()));
-    CodeLocationLabel<JSInternalPtrTag> callReturnLocation(patchBuffer.locationOfNearCall<JSInternalPtrTag>(slowCall));
-    CodeLocationLabel<JSInternalPtrTag> hotPathBegin(patchBuffer.locationOf<JSInternalPtrTag>(targetToCheck));
-    CodeLocationNearCall<JSInternalPtrTag> hotPathOther = patchBuffer.locationOfNearCall<JSInternalPtrTag>(fastCall);
-    callLinkInfo->setCallLocations(callReturnLocation, hotPathBegin, hotPathOther);
</del><ins>+    callLinkInfo->setCodeLocations(
+        patchBuffer.locationOf<JSInternalPtrTag>(slowPathStart),
+        patchBuffer.locationOf<JSInternalPtrTag>(doneLocation));
</ins><span class="cx"> 
</span><span class="cx">     return FINALIZE_WASM_CODE(patchBuffer, WasmEntryPtrTag, "WebAssembly->JavaScript import[%i] %s", importIndex, signature.toString().ascii().data());
</span><span class="cx"> }
</span></span></pre>
</div>
</div>

</body>
</html>