<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[243948] trunk</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/243948">243948</a></dd>
<dt>Author</dt> <dd>ysuzuki@apple.com</dd>
<dt>Date</dt> <dd>2019-04-05 14:58:32 -0700 (Fri, 05 Apr 2019)</dd>
</dl>

<h3>Log Message</h3>
<pre>SIGSEGV in JSC::BytecodeGenerator::addStringConstant
https://bugs.webkit.org/show_bug.cgi?id=196486

Reviewed by Saam Barati.

JSTests:

* stress/arrow-function-and-use-strict-directive.js: Added.
* stress/arrow-function-syntax.js: Added. Checking EOF token handling.
(checkSyntax):
(checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.

Source/JavaScriptCore:

When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.

But now, we have ArrowFunctionExpression without braces `arrow => expr`. Let's consider the following code.

        arrow => expr
        "string!"

We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.

This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.

And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.

* parser/ASTBuilder.h:
(JSC::ASTBuilder::createString):
* parser/Lexer.cpp:
(JSC::Lexer<T>::parseMultilineComment):
(JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too.
(JSC::Lexer<T>::lex): Deleted.
* parser/Lexer.h:
(JSC::Lexer::hasLineTerminatorBeforeToken const):
(JSC::Lexer::setHasLineTerminatorBeforeToken):
(JSC::Lexer<T>::lex):
(JSC::Lexer::prevTerminator const): Deleted.
(JSC::Lexer::setTerminator): Deleted.
* parser/Parser.cpp:
(JSC::Parser<LexerType>::allowAutomaticSemicolon):
(JSC::Parser<LexerType>::parseSingleFunction):
(JSC::Parser<LexerType>::parseStatementListItem):
(JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement):
(JSC::Parser<LexerType>::parseFunctionInfo):
(JSC::Parser<LexerType>::parseClass):
(JSC::Parser<LexerType>::parseExportDeclaration):
(JSC::Parser<LexerType>::parseAssignmentExpression):
(JSC::Parser<LexerType>::parseYieldExpression):
(JSC::Parser<LexerType>::parseProperty):
(JSC::Parser<LexerType>::parsePrimaryExpression):
(JSC::Parser<LexerType>::parseMemberExpression):
* parser/Parser.h:
(JSC::Parser::nextWithoutClearingLineTerminator):
(JSC::Parser::lexCurrentTokenAgainUnderCurrentContext):
(JSC::Parser::internalSaveLexerState):
(JSC::Parser::restoreLexerState):</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkJSTestsChangeLog">trunk/JSTests/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreChangeLog">trunk/Source/JavaScriptCore/ChangeLog</a></li>
<li><a href="#trunkSourceJavaScriptCoreparserASTBuilderh">trunk/Source/JavaScriptCore/parser/ASTBuilder.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreparserLexercpp">trunk/Source/JavaScriptCore/parser/Lexer.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreparserLexerh">trunk/Source/JavaScriptCore/parser/Lexer.h</a></li>
<li><a href="#trunkSourceJavaScriptCoreparserParsercpp">trunk/Source/JavaScriptCore/parser/Parser.cpp</a></li>
<li><a href="#trunkSourceJavaScriptCoreparserParserh">trunk/Source/JavaScriptCore/parser/Parser.h</a></li>
</ul>

<h3>Added Paths</h3>
<ul>
<li><a href="#trunkJSTestsstressarrowfunctionandusestrictdirectivejs">trunk/JSTests/stress/arrow-function-and-use-strict-directive.js</a></li>
<li><a href="#trunkJSTestsstressarrowfunctionsyntaxjs">trunk/JSTests/stress/arrow-function-syntax.js</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkJSTestsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/JSTests/ChangeLog (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/JSTests/ChangeLog  2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/JSTests/ChangeLog     2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -1,3 +1,15 @@
</span><ins>+2019-04-05  Yusuke Suzuki  <ysuzuki@apple.com>
+
+        SIGSEGV in JSC::BytecodeGenerator::addStringConstant
+        https://bugs.webkit.org/show_bug.cgi?id=196486
+
+        Reviewed by Saam Barati.
+
+        * stress/arrow-function-and-use-strict-directive.js: Added.
+        * stress/arrow-function-syntax.js: Added. Checking EOF token handling.
+        (checkSyntax):
+        (checkSyntaxError): Currently not using it. But it is useful for testing more things related to arrow function syntax.
+
</ins><span class="cx"> 2019-04-05  Caitlin Potter  <caitp@igalia.com>
</span><span class="cx"> 
</span><span class="cx">         [JSC] Filter DontEnum properties in ProxyObject::getOwnPropertyNames()
</span></span></pre></div>
<a id="trunkJSTestsstressarrowfunctionandusestrictdirectivejs"></a>
<div class="addfile"><h4>Added: trunk/JSTests/stress/arrow-function-and-use-strict-directive.js (0 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/JSTests/stress/arrow-function-and-use-strict-directive.js                          (rev 0)
+++ trunk/JSTests/stress/arrow-function-and-use-strict-directive.js     2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -0,0 +1,18 @@
</span><ins>+// This test should not crash.
+
+dispatch => accessible.children()
+"use strict";
+
+dispatch2 => accessible.children()
+"use strict"
+
+var protected = 42;
+
+dispatch3 => "use strict"
+protected;
+
+async dispatch4 => hey
+"use strict";
+
+async dispatch4 => "use strict"
+protected;
</ins></span></pre></div>
<a id="trunkJSTestsstressarrowfunctionsyntaxjs"></a>
<div class="addfile"><h4>Added: trunk/JSTests/stress/arrow-function-syntax.js (0 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/JSTests/stress/arrow-function-syntax.js                            (rev 0)
+++ trunk/JSTests/stress/arrow-function-syntax.js       2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -0,0 +1,27 @@
</span><ins>+function checkSyntax(src) {
+    try {
+        eval(src);
+    } catch (error) {
+        if (error instanceof SyntaxError)
+            throw new Error("Syntax Error: " + String(error) + "\n script: `" + src + "`");
+    }
+}
+
+function checkSyntaxError(src, message) {
+    var bError = false;
+    try {
+        eval(src);
+    } catch (error) {
+        bError = error instanceof SyntaxError && (String(error) === message || typeof message === 'undefined');
+    }
+    if (!bError) {
+        throw new Error("Expected syntax Error: " + message + "\n in script: `" + src + "`");
+    }
+}
+
+checkSyntax(`()=>42`);
+checkSyntax(`()=>42
+`);
+checkSyntax(`()=>42//Hello`);
+checkSyntax(`()=>42//Hello
+`);
</ins></span></pre></div>
<a id="trunkSourceJavaScriptCoreChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/ChangeLog (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/ChangeLog    2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/ChangeLog       2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -1,3 +1,61 @@
</span><ins>+2019-04-05  Yusuke Suzuki  <ysuzuki@apple.com>
+
+        SIGSEGV in JSC::BytecodeGenerator::addStringConstant
+        https://bugs.webkit.org/show_bug.cgi?id=196486
+
+        Reviewed by Saam Barati.
+
+        When parsing a FunctionExpression / FunctionDeclaration etc., we use SyntaxChecker for the body of the function because we do not have any interest on the nodes of the body at that time.
+        The nodes will be parsed with the ASTBuilder when the function itself is parsed for code generation. This works well previously because all the function ends with "}" previously.
+        SyntaxChecker lexes this "}" token, and parser restores the context back to ASTBuilder and continues parsing.
+
+        But now, we have ArrowFunctionExpression without braces `arrow => expr`. Let's consider the following code.
+
+                arrow => expr
+                "string!"
+
+        We parse arrow function's body with SyntaxChecker. At that time, we lex "string!" token under the SyntaxChecker context. But this means that we may not build string content for this token
+        since SyntaxChecker may not have interest on string content itself in certain case. After the parser is back to ASTBuilder, we parse "string!" as ExpressionStatement with string constant,
+        generate StringNode with non-built identifier (nullptr), and we accidentally create StringNode with nullptr.
+
+        This patch fixes this problem. The root cause of this problem is that the last token lexed in the previous context is used. We add lexCurrentTokenAgainUnderCurrentContext which will re-lex
+        the current token under the current context (may be ASTBuilder). This should be done only when the caller's context is different from SyntaxChecker, which avoids unnecessary lexing.
+        We leverage existing SavePoint mechanism to implement lexCurrentTokenAgainUnderCurrentContext cleanly.
+
+        And we also fix the bug in the existing SavePoint mechanism, which is shown in the attached test script. When we save LexerState, we do not save line terminator status. This patch also introduces
+        lexWithoutClearingLineTerminator, which lex the token without clearing line terminator status.
+
+        * parser/ASTBuilder.h:
+        (JSC::ASTBuilder::createString):
+        * parser/Lexer.cpp:
+        (JSC::Lexer<T>::parseMultilineComment):
+        (JSC::Lexer<T>::lexWithoutClearingLineTerminator): EOF token also should record offset information. This offset information is correctly handled in Lexer::setOffset too.
+        (JSC::Lexer<T>::lex): Deleted.
+        * parser/Lexer.h:
+        (JSC::Lexer::hasLineTerminatorBeforeToken const):
+        (JSC::Lexer::setHasLineTerminatorBeforeToken):
+        (JSC::Lexer<T>::lex):
+        (JSC::Lexer::prevTerminator const): Deleted.
+        (JSC::Lexer::setTerminator): Deleted.
+        * parser/Parser.cpp:
+        (JSC::Parser<LexerType>::allowAutomaticSemicolon):
+        (JSC::Parser<LexerType>::parseSingleFunction):
+        (JSC::Parser<LexerType>::parseStatementListItem):
+        (JSC::Parser<LexerType>::maybeParseAsyncFunctionDeclarationStatement):
+        (JSC::Parser<LexerType>::parseFunctionInfo):
+        (JSC::Parser<LexerType>::parseClass):
+        (JSC::Parser<LexerType>::parseExportDeclaration):
+        (JSC::Parser<LexerType>::parseAssignmentExpression):
+        (JSC::Parser<LexerType>::parseYieldExpression):
+        (JSC::Parser<LexerType>::parseProperty):
+        (JSC::Parser<LexerType>::parsePrimaryExpression):
+        (JSC::Parser<LexerType>::parseMemberExpression):
+        * parser/Parser.h:
+        (JSC::Parser::nextWithoutClearingLineTerminator):
+        (JSC::Parser::lexCurrentTokenAgainUnderCurrentContext):
+        (JSC::Parser::internalSaveLexerState):
+        (JSC::Parser::restoreLexerState):
+
</ins><span class="cx"> 2019-04-05  Caitlin Potter  <caitp@igalia.com>
</span><span class="cx"> 
</span><span class="cx">         [JSC] Filter DontEnum properties in ProxyObject::getOwnPropertyNames()
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreparserASTBuilderh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/parser/ASTBuilder.h (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/parser/ASTBuilder.h  2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/parser/ASTBuilder.h     2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -241,6 +241,7 @@
</span><span class="cx"> 
</span><span class="cx">     ExpressionNode* createString(const JSTokenLocation& location, const Identifier* string)
</span><span class="cx">     {
</span><ins>+        ASSERT(string);
</ins><span class="cx">         incConstants();
</span><span class="cx">         return new (m_parserArena) StringNode(location, *string);
</span><span class="cx">     }
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreparserLexercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/parser/Lexer.cpp (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/parser/Lexer.cpp     2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/parser/Lexer.cpp        2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -1691,7 +1691,7 @@
</span><span class="cx"> 
</span><span class="cx">         if (isLineTerminator(m_current)) {
</span><span class="cx">             shiftLineTerminator();
</span><del>-            m_terminator = true;
</del><ins>+            m_hasLineTerminatorBeforeToken = true;
</ins><span class="cx">         } else
</span><span class="cx">             shift();
</span><span class="cx">     }
</span><span class="lines">@@ -1770,7 +1770,7 @@
</span><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> template <typename T>
</span><del>-JSTokenType Lexer<T>::lex(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
</del><ins>+JSTokenType Lexer<T>::lexWithoutClearingLineTerminator(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
</ins><span class="cx"> {
</span><span class="cx">     JSTokenData* tokenData = &tokenRecord->m_data;
</span><span class="cx">     JSTokenLocation* tokenLocation = &tokenRecord->m_location;
</span><span class="lines">@@ -1781,18 +1781,19 @@
</span><span class="cx">     ASSERT(m_buffer16.isEmpty());
</span><span class="cx"> 
</span><span class="cx">     JSTokenType token = ERRORTOK;
</span><del>-    m_terminator = false;
</del><span class="cx"> 
</span><span class="cx"> start:
</span><span class="cx">     skipWhitespace();
</span><span class="cx"> 
</span><del>-    if (atEnd())
-        return EOFTOK;
-    
</del><span class="cx">     tokenLocation->startOffset = currentOffset();
</span><span class="cx">     ASSERT(currentOffset() >= currentLineStartOffset());
</span><span class="cx">     tokenRecord->m_startPosition = currentPosition();
</span><span class="cx"> 
</span><ins>+    if (atEnd()) {
+        token = EOFTOK;
+        goto returnToken;
+    }
+
</ins><span class="cx">     CharacterType type;
</span><span class="cx">     if (LIKELY(isLatin1(m_current)))
</span><span class="cx">         type = static_cast<CharacterType>(typesOfLatin1Characters[m_current]);
</span><span class="lines">@@ -1902,7 +1903,7 @@
</span><span class="cx">         shift();
</span><span class="cx">         if (m_current == '+') {
</span><span class="cx">             shift();
</span><del>-            token = (!m_terminator) ? PLUSPLUS : AUTOPLUSPLUS;
</del><ins>+            token = (!m_hasLineTerminatorBeforeToken) ? PLUSPLUS : AUTOPLUSPLUS;
</ins><span class="cx">             break;
</span><span class="cx">         }
</span><span class="cx">         if (m_current == '=') {
</span><span class="lines">@@ -1916,13 +1917,13 @@
</span><span class="cx">         shift();
</span><span class="cx">         if (m_current == '-') {
</span><span class="cx">             shift();
</span><del>-            if ((m_atLineStart || m_terminator) && m_current == '>') {
</del><ins>+            if ((m_atLineStart || m_hasLineTerminatorBeforeToken) && m_current == '>') {
</ins><span class="cx">                 if (m_scriptMode == JSParserScriptMode::Classic) {
</span><span class="cx">                     shift();
</span><span class="cx">                     goto inSingleLineComment;
</span><span class="cx">                 }
</span><span class="cx">             }
</span><del>-            token = (!m_terminator) ? MINUSMINUS : AUTOMINUSMINUS;
</del><ins>+            token = (!m_hasLineTerminatorBeforeToken) ? MINUSMINUS : AUTOMINUSMINUS;
</ins><span class="cx">             break;
</span><span class="cx">         }
</span><span class="cx">         if (m_current == '=') {
</span><span class="lines">@@ -2293,7 +2294,7 @@
</span><span class="cx">         ASSERT(isLineTerminator(m_current));
</span><span class="cx">         shiftLineTerminator();
</span><span class="cx">         m_atLineStart = true;
</span><del>-        m_terminator = true;
</del><ins>+        m_hasLineTerminatorBeforeToken = true;
</ins><span class="cx">         m_lineStart = m_code;
</span><span class="cx">         goto start;
</span><span class="cx">     case CharacterPrivateIdentifierStart:
</span><span class="lines">@@ -2333,13 +2334,16 @@
</span><span class="cx">         auto endPosition = currentPosition();
</span><span class="cx"> 
</span><span class="cx">         while (!isLineTerminator(m_current)) {
</span><del>-            if (atEnd())
-                return EOFTOK;
</del><ins>+            if (atEnd()) {
+                token = EOFTOK;
+                fillTokenInfo(tokenRecord, token, lineNumber, endOffset, lineStartOffset, endPosition);
+                return token;
+            }
</ins><span class="cx">             shift();
</span><span class="cx">         }
</span><span class="cx">         shiftLineTerminator();
</span><span class="cx">         m_atLineStart = true;
</span><del>-        m_terminator = true;
</del><ins>+        m_hasLineTerminatorBeforeToken = true;
</ins><span class="cx">         m_lineStart = m_code;
</span><span class="cx">         if (!lastTokenWasRestrKeyword())
</span><span class="cx">             goto start;
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreparserLexerh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/parser/Lexer.h (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/parser/Lexer.h       2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/parser/Lexer.h  2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -65,6 +65,7 @@
</span><span class="cx">     bool isReparsingFunction() const { return m_isReparsingFunction; }
</span><span class="cx"> 
</span><span class="cx">     JSTokenType lex(JSToken*, unsigned, bool strictMode);
</span><ins>+    JSTokenType lexWithoutClearingLineTerminator(JSToken*, unsigned, bool strictMode);
</ins><span class="cx">     bool nextTokenIsColon();
</span><span class="cx">     int lineNumber() const { return m_lineNumber; }
</span><span class="cx">     ALWAYS_INLINE int currentOffset() const { return offsetFromSourcePtr(m_code); }
</span><span class="lines">@@ -77,7 +78,7 @@
</span><span class="cx">     JSTokenLocation lastTokenLocation() const { return m_lastTokenLocation; }
</span><span class="cx">     void setLastLineNumber(int lastLineNumber) { m_lastLineNumber = lastLineNumber; }
</span><span class="cx">     int lastLineNumber() const { return m_lastLineNumber; }
</span><del>-    bool prevTerminator() const { return m_terminator; }
</del><ins>+    bool hasLineTerminatorBeforeToken() const { return m_hasLineTerminatorBeforeToken; }
</ins><span class="cx">     JSTokenType scanRegExp(JSToken*, UChar patternPrefix = 0);
</span><span class="cx">     enum class RawStringsBuildMode { BuildRawStrings, DontBuildRawStrings };
</span><span class="cx">     JSTokenType scanTemplateString(JSToken*, RawStringsBuildMode);
</span><span class="lines">@@ -110,9 +111,9 @@
</span><span class="cx">     {
</span><span class="cx">         m_lineNumber = line;
</span><span class="cx">     }
</span><del>-    void setTerminator(bool terminator)
</del><ins>+    void setHasLineTerminatorBeforeToken(bool terminator)
</ins><span class="cx">     {
</span><del>-        m_terminator = terminator;
</del><ins>+        m_hasLineTerminatorBeforeToken = terminator;
</ins><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     JSTokenType lexExpectIdentifier(JSToken*, unsigned, bool strictMode);
</span><span class="lines">@@ -202,7 +203,7 @@
</span><span class="cx">     Vector<LChar> m_buffer8;
</span><span class="cx">     Vector<UChar> m_buffer16;
</span><span class="cx">     Vector<UChar> m_bufferForRawTemplateString16;
</span><del>-    bool m_terminator;
</del><ins>+    bool m_hasLineTerminatorBeforeToken;
</ins><span class="cx">     int m_lastToken;
</span><span class="cx"> 
</span><span class="cx">     const SourceCode* m_source;
</span><span class="lines">@@ -403,4 +404,11 @@
</span><span class="cx">     return lex(tokenRecord, lexerFlags, strictMode);
</span><span class="cx"> }
</span><span class="cx"> 
</span><ins>+template <typename T>
+ALWAYS_INLINE JSTokenType Lexer<T>::lex(JSToken* tokenRecord, unsigned lexerFlags, bool strictMode)
+{
+    m_hasLineTerminatorBeforeToken = false;
+    return lexWithoutClearingLineTerminator(tokenRecord, lexerFlags, strictMode);
+}
+
</ins><span class="cx"> } // namespace JSC
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreparserParsercpp"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/parser/Parser.cpp (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/parser/Parser.cpp    2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/parser/Parser.cpp       2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -345,7 +345,7 @@
</span><span class="cx"> template <typename LexerType>
</span><span class="cx"> bool Parser<LexerType>::allowAutomaticSemicolon()
</span><span class="cx"> {
</span><del>-    return match(CLOSEBRACE) || match(EOFTOK) || m_lexer->prevTerminator();
</del><ins>+    return match(CLOSEBRACE) || match(EOFTOK) || m_lexer->hasLineTerminatorBeforeToken();
</ins><span class="cx"> }
</span><span class="cx"> 
</span><span class="cx"> template <typename LexerType>
</span><span class="lines">@@ -625,7 +625,7 @@
</span><span class="cx">     case IDENT:
</span><span class="cx">         if (*m_token.m_data.ident == m_vm->propertyNames->async && !m_token.m_data.escaped) {
</span><span class="cx">             next();
</span><del>-            failIfFalse(match(FUNCTION) && !m_lexer->prevTerminator(), "Cannot parse the async function");
</del><ins>+            failIfFalse(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken(), "Cannot parse the async function");
</ins><span class="cx">             statement = parseAsyncFunctionDeclaration(context, ExportType::NotExported, DeclarationDefaultContext::Standard, functionConstructorParametersEndPosition);
</span><span class="cx">             break;
</span><span class="cx">         }
</span><span class="lines">@@ -696,7 +696,7 @@
</span><span class="cx">             // but could be mistakenly parsed as an AsyncFunctionExpression.
</span><span class="cx">             SavePoint savePoint = createSavePoint();
</span><span class="cx">             next();
</span><del>-            if (UNLIKELY(match(FUNCTION) && !m_lexer->prevTerminator())) {
</del><ins>+            if (UNLIKELY(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken())) {
</ins><span class="cx">                 result = parseAsyncFunctionDeclaration(context);
</span><span class="cx">                 break;
</span><span class="cx">             }
</span><span class="lines">@@ -2026,7 +2026,7 @@
</span><span class="cx">     ASSERT(matchContextualKeyword(m_vm->propertyNames->async));
</span><span class="cx">     SavePoint savePoint = createSavePoint();
</span><span class="cx">     next();
</span><del>-    if (match(FUNCTION) && !m_lexer->prevTerminator()) {
</del><ins>+    if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken()) {
</ins><span class="cx">         const bool isAsync = true;
</span><span class="cx">         result = parseFunctionDeclarationStatement(context, isAsync, parentAllowsFunctionDeclarationAsStatement);
</span><span class="cx">         return true;
</span><span class="lines">@@ -2421,7 +2421,7 @@
</span><span class="cx"> 
</span><span class="cx">         matchOrFail(ARROWFUNCTION, "Expected a '=>' after arrow function parameter declaration");
</span><span class="cx"> 
</span><del>-        if (m_lexer->prevTerminator())
</del><ins>+        if (m_lexer->hasLineTerminatorBeforeToken())
</ins><span class="cx">             failDueToUnexpectedToken();
</span><span class="cx"> 
</span><span class="cx">         ASSERT(constructorKind == ConstructorKind::None);
</span><span class="lines">@@ -2612,6 +2612,8 @@
</span><span class="cx">         functionScope->fillParametersForSourceProviderCache(parameters, nonLocalCapturesFromParameterExpressions);
</span><span class="cx">         newInfo = SourceProviderCacheItem::create(parameters);
</span><span class="cx">     }
</span><ins>+
+    bool functionScopeWasStrictMode = functionScope->strictMode();
</ins><span class="cx">     
</span><span class="cx">     popScope(functionScope, TreeBuilder::NeedsFreeVariableInfo);
</span><span class="cx">     
</span><span class="lines">@@ -2618,6 +2620,15 @@
</span><span class="cx">     if (functionBodyType != ArrowFunctionBodyExpression) {
</span><span class="cx">         matchOrFail(CLOSEBRACE, "Expected a closing '}' after a ", stringForFunctionMode(mode), " body");
</span><span class="cx">         next();
</span><ins>+    } else {
+        // We need to lex the last token again because the last token is lexed under the different context because of the following possibilities.
+        // 1. which may have different strict mode.
+        // 2. which may not build strings for tokens.
+        // But (1) is not possible because we do not recognize the string literal in ArrowFunctionBodyExpression as directive and this is correct in terms of the spec (`value => "use strict"`).
+        // So we only check TreeBuilder's type here.
+        ASSERT_UNUSED(functionScopeWasStrictMode, functionScopeWasStrictMode == currentScope()->strictMode());
+        if (!std::is_same<TreeBuilder, SyntaxChecker>::value)
+            lexCurrentTokenAgainUnderCurrentContext();
</ins><span class="cx">     }
</span><span class="cx"> 
</span><span class="cx">     if (newInfo)
</span><span class="lines">@@ -2876,7 +2887,7 @@
</span><span class="cx">                 if (!isGeneratorMethodParseMode(parseMode) && !isAsyncMethodParseMode(parseMode)) {
</span><span class="cx">                     ident = m_token.m_data.ident;
</span><span class="cx">                     next();
</span><del>-                    if (match(OPENPAREN) || match(COLON) || match(EQUAL) || m_lexer->prevTerminator())
</del><ins>+                    if (match(OPENPAREN) || match(COLON) || match(EQUAL) || m_lexer->hasLineTerminatorBeforeToken())
</ins><span class="cx">                         break;
</span><span class="cx">                     if (UNLIKELY(consume(TIMES)))
</span><span class="cx">                         parseMode = SourceParseMode::AsyncGeneratorWrapperMethodMode;
</span><span class="lines">@@ -3396,7 +3407,7 @@
</span><span class="cx">         } else if (matchContextualKeyword(m_vm->propertyNames->async)) {
</span><span class="cx">             SavePoint savePoint = createSavePoint();
</span><span class="cx">             next();
</span><del>-            if (match(FUNCTION) && !m_lexer->prevTerminator()) {
</del><ins>+            if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken()) {
</ins><span class="cx">                 next();
</span><span class="cx">                 if (match(IDENT))
</span><span class="cx">                     localName = m_token.m_data.ident;
</span><span class="lines">@@ -3541,7 +3552,7 @@
</span><span class="cx">         case IDENT:
</span><span class="cx">             if (*m_token.m_data.ident == m_vm->propertyNames->async && !m_token.m_data.escaped) {
</span><span class="cx">                 next();
</span><del>-                semanticFailIfFalse(match(FUNCTION) && !m_lexer->prevTerminator(), "Expected 'function' keyword following 'async' keyword with no preceding line terminator");
</del><ins>+                semanticFailIfFalse(match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken(), "Expected 'function' keyword following 'async' keyword with no preceding line terminator");
</ins><span class="cx">                 DepthManager statementDepth(&m_statementDepth);
</span><span class="cx">                 m_statementDepth = 1;
</span><span class="cx">                 result = parseAsyncFunctionDeclaration(context, ExportType::Exported);
</span><span class="lines">@@ -3659,7 +3670,7 @@
</span><span class="cx">             if (UNLIKELY(classifier.indicatesPossibleAsyncArrowFunction())) {
</span><span class="cx">                 if (matchContextualKeyword(m_vm->propertyNames->async)) {
</span><span class="cx">                     next();
</span><del>-                    isAsyncArrow = !m_lexer->prevTerminator();
</del><ins>+                    isAsyncArrow = !m_lexer->hasLineTerminatorBeforeToken();
</ins><span class="cx">                 }
</span><span class="cx">             }
</span><span class="cx">             if (isArrowFunctionParameters()) {
</span><span class="lines">@@ -3776,7 +3787,7 @@
</span><span class="cx">     ASSERT(match(YIELD));
</span><span class="cx">     SavePoint savePoint = createSavePoint();
</span><span class="cx">     next();
</span><del>-    if (m_lexer->prevTerminator())
</del><ins>+    if (m_lexer->hasLineTerminatorBeforeToken())
</ins><span class="cx">         return context.createYield(location);
</span><span class="cx"> 
</span><span class="cx">     bool delegate = consume(TIMES);
</span><span class="lines">@@ -3936,7 +3947,7 @@
</span><span class="cx">                     goto namedProperty;
</span><span class="cx">                 }
</span><span class="cx"> 
</span><del>-                failIfTrue(m_lexer->prevTerminator(), "Expected a property name following keyword 'async'");
</del><ins>+                failIfTrue(m_lexer->hasLineTerminatorBeforeToken(), "Expected a property name following keyword 'async'");
</ins><span class="cx">                 if (UNLIKELY(consume(TIMES)))
</span><span class="cx">                     parseMode = SourceParseMode::AsyncGeneratorWrapperMethodMode;
</span><span class="cx">                 else
</span><span class="lines">@@ -4485,7 +4496,7 @@
</span><span class="cx">             const Identifier* ident = m_token.m_data.ident;
</span><span class="cx">             JSTokenLocation location(tokenLocation());
</span><span class="cx">             next();
</span><del>-            if (match(FUNCTION) && !m_lexer->prevTerminator())
</del><ins>+            if (match(FUNCTION) && !m_lexer->hasLineTerminatorBeforeToken())
</ins><span class="cx">                 return parseAsyncFunctionExpression(context);
</span><span class="cx"> 
</span><span class="cx">             // Avoid using variable if it is an arrow function parameter
</span><span class="lines">@@ -4751,7 +4762,7 @@
</span><span class="cx"> 
</span><span class="cx">         base = parsePrimaryExpression(context);
</span><span class="cx">         failIfFalse(base, "Cannot parse base expression");
</span><del>-        if (UNLIKELY(isAsync && context.isResolve(base) && !m_lexer->prevTerminator())) {
</del><ins>+        if (UNLIKELY(isAsync && context.isResolve(base) && !m_lexer->hasLineTerminatorBeforeToken())) {
</ins><span class="cx">             if (matchSpecIdentifier()) {
</span><span class="cx">                 // AsyncArrowFunction
</span><span class="cx">                 forceClassifyExpressionError(ErrorIndicatesAsyncArrowFunction);
</span></span></pre></div>
<a id="trunkSourceJavaScriptCoreparserParserh"></a>
<div class="modfile"><h4>Modified: trunk/Source/JavaScriptCore/parser/Parser.h (243947 => 243948)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Source/JavaScriptCore/parser/Parser.h      2019-04-05 21:52:51 UTC (rev 243947)
+++ trunk/Source/JavaScriptCore/parser/Parser.h 2019-04-05 21:58:32 UTC (rev 243948)
</span><span class="lines">@@ -1363,6 +1363,16 @@
</span><span class="cx">         m_token.m_type = m_lexer->lex(&m_token, lexerFlags, strictMode());
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    ALWAYS_INLINE void nextWithoutClearingLineTerminator(unsigned lexerFlags = 0)
+    {
+        int lastLine = m_token.m_location.line;
+        int lastTokenEnd = m_token.m_location.endOffset;
+        int lastTokenLineStart = m_token.m_location.lineStartOffset;
+        m_lastTokenEndPosition = JSTextPosition(lastLine, lastTokenEnd, lastTokenLineStart);
+        m_lexer->setLastLineNumber(lastLine);
+        m_token.m_type = m_lexer->lexWithoutClearingLineTerminator(&m_token, lexerFlags, strictMode());
+    }
+
</ins><span class="cx">     ALWAYS_INLINE void nextExpectIdentifier(unsigned lexerFlags = 0)
</span><span class="cx">     {
</span><span class="cx">         int lastLine = m_token.m_location.line;
</span><span class="lines">@@ -1373,6 +1383,12 @@
</span><span class="cx">         m_token.m_type = m_lexer->lexExpectIdentifier(&m_token, lexerFlags, strictMode());
</span><span class="cx">     }
</span><span class="cx"> 
</span><ins>+    ALWAYS_INLINE void lexCurrentTokenAgainUnderCurrentContext()
+    {
+        auto savePoint = createSavePoint();
+        restoreSavePoint(savePoint);
+    }
+
</ins><span class="cx">     ALWAYS_INLINE bool nextTokenIsColon()
</span><span class="cx">     {
</span><span class="cx">         return m_lexer->nextTokenIsColon();
</span><span class="lines">@@ -1762,6 +1778,7 @@
</span><span class="cx">         unsigned oldLineStartOffset;
</span><span class="cx">         unsigned oldLastLineNumber;
</span><span class="cx">         unsigned oldLineNumber;
</span><ins>+        bool hasLineTerminatorBeforeToken;
</ins><span class="cx">     };
</span><span class="cx"> 
</span><span class="cx">     // If you're using this directly, you probably should be using
</span><span class="lines">@@ -1775,6 +1792,7 @@
</span><span class="cx">         result.oldLineStartOffset = m_token.m_location.lineStartOffset;
</span><span class="cx">         result.oldLastLineNumber = m_lexer->lastLineNumber();
</span><span class="cx">         result.oldLineNumber = m_lexer->lineNumber();
</span><ins>+        result.hasLineTerminatorBeforeToken = m_lexer->hasLineTerminatorBeforeToken();
</ins><span class="cx">         ASSERT(static_cast<unsigned>(result.startOffset) >= result.oldLineStartOffset);
</span><span class="cx">         return result;
</span><span class="cx">     }
</span><span class="lines">@@ -1784,7 +1802,8 @@
</span><span class="cx">         // setOffset clears lexer errors.
</span><span class="cx">         m_lexer->setOffset(lexerState.startOffset, lexerState.oldLineStartOffset);
</span><span class="cx">         m_lexer->setLineNumber(lexerState.oldLineNumber);
</span><del>-        next();
</del><ins>+        m_lexer->setHasLineTerminatorBeforeToken(lexerState.hasLineTerminatorBeforeToken);
+        nextWithoutClearingLineTerminator();
</ins><span class="cx">         m_lexer->setLastLineNumber(lexerState.oldLastLineNumber);
</span><span class="cx">     }
</span><span class="cx"> 
</span></span></pre>
</div>
</div>

</body>
</html>