<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[286963] trunk/Tools</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.webkit.org/projects/webkit/changeset/286963">286963</a></dd>
<dt>Author</dt> <dd>angelos@igalia.com</dd>
<dt>Date</dt> <dd>2021-12-13 11:56:23 -0800 (Mon, 13 Dec 2021)</dd>
</dl>

<h3>Log Message</h3>
<pre>[JSC] Unify default/noisy tests definition
https://bugs.webkit.org/show_bug.cgi?id=233926

Reviewed by Yusuke Suzuki.

Currently, changing a test to be noisy (i.e. using
defaultRunNoisyTest) results in only a subset of the tests being
run.

Instead of manually changing the noisy test test handler, this
patch makes the mode definitions table-driven. However, for
compatibility with the existing code, it uses the table to define
the ruby methods that the testcase definitions (//@ comments,
yaml) and other callers in run-jsc-stress-tests (e.g. all
default*Run methods) expect.

Note that this slightly increases the number of tests executed (as
more test modes are enabled for the noisy tests).

This is an exploratory first step; other callers in
run-jsc-stress-tests could be simplified in the same way.

* Scripts/run-jsc-stress-tests:</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#trunkToolsChangeLog">trunk/Tools/ChangeLog</a></li>
<li><a href="#trunkToolsScriptsrunjscstresstests">trunk/Tools/Scripts/run-jsc-stress-tests</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="trunkToolsChangeLog"></a>
<div class="modfile"><h4>Modified: trunk/Tools/ChangeLog (286962 => 286963)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/ChangeLog    2021-12-13 19:55:06 UTC (rev 286962)
+++ trunk/Tools/ChangeLog       2021-12-13 19:56:23 UTC (rev 286963)
</span><span class="lines">@@ -1,3 +1,29 @@
</span><ins>+2021-12-13  Angelos Oikonomopoulos  <angelos@igalia.com>
+
+        [JSC] Unify default/noisy tests definition
+        https://bugs.webkit.org/show_bug.cgi?id=233926
+
+        Reviewed by Yusuke Suzuki.
+
+        Currently, changing a test to be noisy (i.e. using
+        defaultRunNoisyTest) results in only a subset of the tests being
+        run.
+
+        Instead of manually changing the noisy test test handler, this
+        patch makes the mode definitions table-driven. However, for
+        compatibility with the existing code, it uses the table to define
+        the ruby methods that the testcase definitions (//@ comments,
+        yaml) and other callers in run-jsc-stress-tests (e.g. all
+        default*Run methods) expect.
+
+        Note that this slightly increases the number of tests executed (as
+        more test modes are enabled for the noisy tests).
+
+        This is an exploratory first step; other callers in
+        run-jsc-stress-tests could be simplified in the same way.
+
+        * Scripts/run-jsc-stress-tests:
+
</ins><span class="cx"> 2021-12-13  Youenn Fablet  <youenn@apple.com>
</span><span class="cx"> 
</span><span class="cx">         Start websocket server for tests in http/tests that have websocket in their name
</span></span></pre></div>
<a id="trunkToolsScriptsrunjscstresstests"></a>
<div class="modfile"><h4>Modified: trunk/Tools/Scripts/run-jsc-stress-tests (286962 => 286963)</h4>
<pre class="diff"><span>
<span class="info">--- trunk/Tools/Scripts/run-jsc-stress-tests 2021-12-13 19:55:06 UTC (rev 286962)
+++ trunk/Tools/Scripts/run-jsc-stress-tests    2021-12-13 19:56:23 UTC (rev 286963)
</span><span class="lines">@@ -778,16 +778,23 @@
</span><span class="cx">     "#{$collectionName}/#{$benchmark}.#{kind}"
</span><span class="cx"> end
</span><span class="cx"> 
</span><del>-def addRunCommand(kind, command, outputHandler, errorHandler, *additionalEnv)
</del><ins>+def addRunCommandCfg(cfg, *additionalEnv)
+    [:kind, :command, :outputHandler, :errorHandler].each { |key|
+        if not cfg.has_key?(key)
+            raise "Missing #{key} in #{cfg}"
+        end
+    }
</ins><span class="cx">     $didAddRunCommand = true
</span><del>-    name = baseOutputName(kind)
</del><ins>+    name = baseOutputName(cfg[:kind])
</ins><span class="cx">     if $filter and name !~ $filter
</span><span class="cx">         return
</span><span class="cx">     end
</span><span class="cx">     plan = Plan.create(
</span><del>-        $benchmarkDirectory, command, "#{$collectionName}/#{$benchmark}", name, outputHandler,
-        errorHandler)
-    plan.additionalEnv.push(*additionalEnv)
</del><ins>+        $benchmarkDirectory, cfg[:command], "#{$collectionName}/#{$benchmark}", name, cfg[:outputHandler],
+        cfg[:errorHandler])
+    if cfg.has_key?(:additionalEnv)
+        plan.additionalEnv.push(*(cfg[:additionalEnv]))
+    end
</ins><span class="cx">     if $runCommandOptions[:serial]
</span><span class="cx">         # Add this to the list of tests to be run on their own, so
</span><span class="cx">         # that we can treat them specially when scheduling, but keep
</span><span class="lines">@@ -803,6 +810,17 @@
</span><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><ins>+def addRunCommand(kind, command, outputHandler, errorHandler, *additionalEnv)
+    cfg = {
+        :kind => kind,
+        :command => command,
+        :outputHandler => outputHandler,
+        :errorHandler => errorHandler,
+        :additionalEnv => additionalEnv,
+    }
+    addRunCommandCfg(cfg)
+end
+
</ins><span class="cx"> # Returns true if there were run commands found in the file ($benchmarkDirectory +
</span><span class="cx"> # $benchmark), in which case those run commands have already been executed. Otherwise
</span><span class="cx"> # returns false, in which case you're supposed to add your own run commands.
</span><span class="lines">@@ -900,12 +918,37 @@
</span><span class="cx">     $testSpecificRequiredOptions += options
</span><span class="cx"> end
</span><span class="cx"> 
</span><ins>+def runWithOptions(cfg, *options)
+    baseOptions = BASE_OPTIONS
+    if cfg.has_key?(:no_base_options)
+        baseOptions = []
+    end
+    commandPrefix = cfg.fetch(:command_prefix, [])
+    if cfg.has_key?(:place_benchmark_early)
+        cfg[:command] = commandPrefix + vmCommand + [$benchmark.to_s] + baseOptions + options + $testSpecificRequiredOptions
+    else
+        cfg[:command] = commandPrefix + vmCommand + baseOptions + options + $testSpecificRequiredOptions + [$benchmark.to_s]
+    end
+    addRunCommandCfg(cfg)
+end
+
</ins><span class="cx"> def runWithOutputHandler(kind, outputHandler, *options)
</span><del>-    addRunCommand(kind, vmCommand + BASE_OPTIONS + options + $testSpecificRequiredOptions + [$benchmark.to_s], outputHandler, simpleErrorHandler)
</del><ins>+    cfg = {
+        :kind => kind,
+        :outputHandler => outputHandler,
+        :errorHandler => simpleErrorHandler,
+    }
+    runWithOptions(cfg, *options)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><span class="cx"> def runWithOutputHandlerWithoutBaseOption(kind, outputHandler, *options)
</span><del>-    addRunCommand(kind, vmCommand + options + $testSpecificRequiredOptions + [$benchmark.to_s], outputHandler, simpleErrorHandler)
</del><ins>+    cfg = {
+        :kind => kind,
+        :outputHandler => outputHandler,
+        :errorHandler => simpleErrorHandler,
+        :no_base_options => true,
+    }
+    runWithOptions(cfg, *options)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><span class="cx"> def run(kind, *options)
</span><span class="lines">@@ -912,16 +955,25 @@
</span><span class="cx">     runWithOutputHandler(kind, silentOutputHandler, *options)
</span><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runWithoutBaseOption(kind, *options)
-    runWithOutputHandlerWithoutBaseOption(kind, silentOutputHandler, *options)
</del><ins>+def runInner(cfg, *options)
+    cfg = cfg.dup
+    if not cfg.has_key?(:outputHandler)
+        cfg[:outputHandler] = silentOutputHandler
+    end
+    if not cfg.has_key?(:errorHandler)
+        cfg[:errorHandler] = simpleErrorHandler
+    end
+    runWithOptions(cfg, *options)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runNoFTL(*optionalTestSpecificOptions)
-    run("no-ftl", *optionalTestSpecificOptions)
</del><ins>+def runWithoutBaseOptionCfg(cfg, *options)
+    cfg = cfg.dup
+    cfg[:no_base_options] = true
+    runWithOptions(cfg, *options)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runWithRAMSize(size, *optionalTestSpecificOptions)
-    run("ram-size-#{size}", "--forceRAMSize=#{size}", *optionalTestSpecificOptions)
</del><ins>+def runWithoutBaseOption(kind, *options)
+    runWithOutputHandlerWithoutBaseOption(kind, silentOutputHandler, *options)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><span class="cx"> def runOneLargeHeap(*optionalTestSpecificOptions)
</span><span class="lines">@@ -933,180 +985,517 @@
</span><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runNoJIT(*optionalTestSpecificOptions)
-    run("no-jit", "--useJIT=false", *optionalTestSpecificOptions)
-end
-
-def runNoLLInt(*optionalTestSpecificOptions)
-    if $jitTests
-        run("no-llint", "--useLLInt=false", *optionalTestSpecificOptions)
</del><ins>+def bytecodeCacheTemplate
+    if ($hostOS == "darwin")
+        return "bytecode-cache"
+    elsif ($hostOS == "linux" && $architecture != "mips")
+        # FIXME: need to fix https://bugs.webkit.org/show_bug.cgi?id=218703 to enable this on Linux/MIPS.
+        return "bytecode-cacheXXXXXX"
</ins><span class="cx">     end
</span><ins>+    nil
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-# NOTE: Tests rely on this using scribbleFreeCells.
-def runNoCJITValidate(*optionalTestSpecificOptions)
-    run("no-cjit", "--validateBytecode=true", "--validateGraph=true", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runNoCJITValidatePhases(*optionalTestSpecificOptions)
-    run("no-cjit-validate-phases", "--validateBytecode=true", "--validateGraphAtEachPhase=true", "--useSourceProviderCache=false", "--useRandomizingExecutableIslandAllocation=true", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runNoCJITCollectContinuously(*optionalTestSpecificOptions)
-    run("no-cjit-collect-continuously", *(NO_CJIT_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runDefault(*optionalTestSpecificOptions)
-    run("default", *(FTL_OPTIONS + optionalTestSpecificOptions))
-end
-
-# FIXME: need to fix https://bugs.webkit.org/show_bug.cgi?id=218703 to enable this on Linux/MIPS.
</del><span class="cx"> def runBytecodeCacheImpl(optionalTestSpecificOptions, *additionalEnv)
</span><del>-    if ($hostOS == "darwin")
-        fileTemplate = "bytecode-cache"
-    elsif ($hostOS == "linux" && $architecture != "mips")
-        fileTemplate = "bytecode-cacheXXXXXX"
-    else
-        skip
-        return
</del><ins>+    fileTemplate = bytecodeCacheTemplate
+    if fileTemplate.nil?
+        return nil
</ins><span class="cx">     end
</span><del>-
-    options = BASE_OPTIONS + FTL_OPTIONS + optionalTestSpecificOptions + $testSpecificRequiredOptions
-    addRunCommand("bytecode-cache", ["sh", (pathToHelpers + "bytecode-cache-test-helper.sh").to_s, fileTemplate.to_s, *vmCommand, $benchmark.to_s] + options, silentOutputHandler, simpleErrorHandler, *additionalEnv)
</del><ins>+    {
+        :cfg => {
+            :command_prefix => [
+                "sh",
+                (pathToHelpers + "bytecode-cache-test-helper.sh").to_s,
+                fileTemplate.to_s,
+            ],
+            :place_benchmark_early => true,
+            :additionalEnv => additionalEnv,
+        },
+        :testSpecificOptions => FTL_OPTIONS + optionalTestSpecificOptions,
+    }
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runBytecodeCache(*optionalTestSpecificOptions)
-    runBytecodeCacheImpl(optionalTestSpecificOptions)
-end
</del><span class="cx"> 
</span><del>-def runBytecodeCacheNoAssertion(*optionalTestSpecificOptions)
-    runBytecodeCacheImpl(optionalTestSpecificOptions, "JSC_forceDiskCache=false")
</del><ins>+def cfgInitializerPlain
+    Proc.new { |cfg, kind|
+        { :kind => kind}
+    }
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runFTLNoCJIT(*optionalTestSpecificOptions)
-    run("misc-ftl-no-cjit", "--useDataIC=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
</del><ins>+def cfgInitializerCfg
+    Proc.new { |cfg, kind|
+        cfg = cfg.dup
+        cfg[:kind] = kind
+        cfg
+    }
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runFTLNoCJITB3O0(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-b3o0", "--useArrayAllocationProfiling=false", "--forcePolyProto=true", "--useRandomizingExecutableIslandAllocation=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + B3O0_OPTIONS + FORCE_LLINT_EXIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+# For each base mode (defined below) we generate two kinds of functions:
+#
+# - a version which takes a cfg argument and passes it along, only
+#   setting the kind field
+# - a "plain" version which starts out with an empty cfg
+#
+# The plain  version is intended  for use in the  testcase definitions
+# (in `//@` comments and the like).
+#
+# The former version is used for plumbing. The caller may set various
+# fields in the cfg which will be respected.
+#
+# This way, we can
+# - define a set of test modes in defaultRunCfg
+# - have defaultRunCfg propagate the cfg argument to the run*Cfg
+#   functions it calls
+# - call defaultRunCfg from e.g. defaultRunNoisyTest with the output
+#   handlers appropriately set, in order to make sure we're running
+#   the exact same of tests.
+CfgKind = Struct.new(:extension, :expectCfg, :initializer)
+cfgKinds = [
+    CfgKind.new("", false, cfgInitializerPlain),
+    CfgKind.new("Cfg", true, cfgInitializerCfg),
+]
</ins><span class="cx"> 
</span><del>-def runFTLNoCJITValidate(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-validate-sampling-profiler", "--validateGraph=true", "--validateBCE=true", "--useSamplingProfiler=true", "--airForceIRCAllocator=true", "--useDataIC=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+# Define base test modes. Each mode is an array of [name, kind,
+# options]. The name is used to derive the ruby method names, the kind
+# is used for reporting (i.e. what you'd see in this script's
+# output). In the common case, options is a static array; if not, it's
+# a Proc that returns a dict that needs to be unpacked (see its use
+# site for a more detailed description).
+BASE_MODES = [
+    [
+        "NoCJIT",
+        "ftl-no-cjit",
+        [
+            "--validateBytecode=true", "--validateGraphAtEachPhase=true"
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS
+    ],
+    [
+        "FTLNoCJIT",
+        "misc-ftl-no-cjit",
+        [
+            "--useDataIC=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITB3O0",
+        "ftl-no-cjit-b3o0",
+        [
+            "--useArrayAllocationProfiling=false",
+            "--forcePolyProto=true",
+            "--useRandomizingExecutableIslandAllocation=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS +
+        B3O0_OPTIONS +
+        FORCE_LLINT_EXIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITValidate",
+        "ftl-no-cjit-validate-sampling-profiler",
+        [
+            "--validateGraph=true",
+            "--validateBCE=true",
+            "--useSamplingProfiler=true",
+            "--airForceIRCAllocator=true",
+            "--useDataIC=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITNoPutStackValidate",
+        "ftl-no-cjit-no-put-stack-validate",
+        [
+            "--validateGraph=true",
+            "--usePutStackSinking=false",
+            "--airForceIRCAllocator=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITNoInlineValidate",
+        "ftl-no-cjit-no-inline-validate",
+        [
+            "--validateGraph=true",
+            "--maximumInliningDepth=1",
+            "--airForceBriggsAllocator=true",
+            "--useB3HoistLoopInvariantValues=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITOSRValidation",
+        "ftl-no-cjit-osr-validation",
+        [
+            "--validateFTLOSRExitLiveness=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "DFGEager",
+        "dfg-eager",
+        EAGER_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS +
+        FORCE_LLINT_EXIT_OPTIONS
+    ],
+    [
+        "DFGEagerNoCJITValidate",
+        "dfg-eager-no-cjit-validate",
+        [
+            "--validateGraph=true",
+        ] +
+        NO_CJIT_OPTIONS +
+        EAGER_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS
+    ],
+    [
+        "FTLEager",
+        "ftl-eager",
+        [
+            "--airForceBriggsAllocator=true",
+            "--useRandomizingExecutableIslandAllocation=true",
+            "--forcePolyProto=true",
+            "--useDataIC=true",
+        ] +
+        FTL_OPTIONS +
+        EAGER_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS
+    ],
+    [
+        "FTLEagerNoCJITValidate",
+        "ftl-eager-no-cjit",
+        [
+            "--validateGraph=true",
+            "--validateBCE=true",
+            "--airForceIRCAllocator=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS +
+        EAGER_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS +
+        FORCE_LLINT_EXIT_OPTIONS +
+        EXECUTABLE_FUZZER_OPTIONS
+    ],
+    [
+        "FTLEagerNoCJITB3O1",
+        "ftl-eager-no-cjit-b3o1",
+        [
+            "--validateGraph=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS +
+        EAGER_OPTIONS +
+        B3O1_OPTIONS
+    ],
+    [
+        "FTLEagerNoCJITOSRValidation",
+        "ftl-eager-no-cjit-osr-validation",
+        [
+            "--validateFTLOSRExitLiveness=true",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS +
+        EAGER_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS
+    ],
+    [
+        "NoCJITNoASO",
+        "no-cjit-no-aso",
+        [
+            "--useArchitectureSpecificOptimizations=false",
+        ] +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "NoCJITNoAccessInlining",
+        "no-cjit-no-access-inlining",
+        [
+            "--useAccessInlining=false",
+        ] +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITNoAccessInlining",
+        "ftl-no-cjit-no-access-inlining",
+        [
+            "--useAccessInlining=false",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "FTLNoCJITSmallPool",
+        "ftl-no-cjit-small-pool",
+        [
+            "--jitMemoryReservationSize=202400",
+        ] +
+        FTL_OPTIONS +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "NoCJIT",
+        "no-cjit",
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "EagerJettisonNoCJIT",
+        "eager-jettison-no-cjit",
+        [
+            "--useRandomizingExecutableIslandAllocation=true",
+            "--forceCodeBlockToJettisonDueToOldAge=true",
+            "--verifyGC=true",
+        ] +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "ShadowChicken",
+        "shadow-chicken",
+        [
+            "--useDFGJIT=false",
+            "--alwaysUseShadowChicken=true",
+        ]
+    ],
+    [
+        "MiniMode",
+        "mini-mode",
+        [
+            "--forceMiniVMMode=true",
+        ]
+    ],
+    [
+        "LogicalAssignmentOperatorsEnabled",
+        "logical-assignment-operators-enabled",
+        [
+            "--useLogicalAssignmentOperators=true",
+        ] +
+        FTL_OPTIONS
+    ],
+    [
+        "NoJIT",
+        "no-jit",
+        [
+            "--useJIT=false",
+        ]
+    ],
+    [
+        # NOTE: Tests rely on this using scribbleFreeCells.
+        "NoCJITValidate",
+        "no-cjit",
+        [
+            "--validateBytecode=true",
+            "--validateGraph=true",
+        ] +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "NoCJITValidatePhases",
+        "no-cjit-validate-phases",
+        [
+            "--validateBytecode=true",
+            "--validateGraphAtEachPhase=true",
+            "--useSourceProviderCache=false",
+            "--useRandomizingExecutableIslandAllocation=true",
+        ] +
+        NO_CJIT_OPTIONS
+    ],
+    [
+        "NoCJITCollectContinuously",
+        "no-cjit-collect-continuously",
+        NO_CJIT_OPTIONS +
+        COLLECT_CONTINUOUSLY_OPTIONS
+    ],
+    [
+        "Default",
+        "default",
+        FTL_OPTIONS
+    ],
+    [
+        "NoFTL",
+        "no-ftl",
+        []
+    ],
+    [
+        "WithRAMSize",
+        nil, # Not used
+        Proc.new { |size, *optionalTestSpecificOptions|
+            {
+                :cfg => {
+                    :kind => "ram-size-#{size}",
+                },
+                :testSpecificOptions => [
+                    "--forceRAMSize=#{size}",
+                ] + optionalTestSpecificOptions
+            }
+        }
+    ],
+    [
+        "BytecodeCache",
+        "bytecode-cache",
+        Proc.new { |*optionalTestSpecificOptions|
+            runBytecodeCacheImpl(optionalTestSpecificOptions)
+        }
+    ],
+    [
+        "BytecodeCacheNoAssertion",
+        "bytecode-cache",
+        Proc.new { |*optionalTestSpecificOptions|
+            runBytecodeCacheImpl(optionalTestSpecificOptions, "JSC_forceDiskCache=false")
+        }
+    ],
+    [
+        "FTLEagerWatchdog",
+        nil,
+        Proc.new { |*optionalTestSpecificOptions|
+            timeout = rand(100)
+            {
+                :cfg => {
+                    :kind => "ftl-eager-watchdog-#{timeout}",
+                },
+                :testSpecificOptions => [
+                    "--watchdog=#{timeout}",
+                    "--watchdog-exception-ok",
+                ] +
+                FTL_OPTIONS +
+                EAGER_OPTIONS +
+                COLLECT_CONTINUOUSLY_OPTIONS +
+                optionalTestSpecificOptions
+            }
+        }
+    ],
+    [
+        "NoLLInt",
+        "no-llint",
+        Proc.new { |*optionalTestSpecificOptions|
+            return nil unless $jitTests
+            {
+                :cfg => {
+                },
+                :testSpecificOptions => [
+                    "--useLLInt=false",
+                ] + optionalTestSpecificOptions
+            }
+        }
+    ],
+    [
+        "OneLangeHeap",
+        "default",
+        Proc.new { |*optionalTestSpecificOptions|
+            if $memoryLimited
+                return nil
+            end
+            {
+                :testSpecificOptions => optionalTestSpecificOptions
+            }
+        }
+    ]
+]
</ins><span class="cx"> 
</span><del>-def runFTLNoCJITNoPutStackValidate(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-no-put-stack-validate", "--validateGraph=true", "--usePutStackSinking=false", "--airForceIRCAllocator=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+BASE_MODES.each { |mode|
+    name = "run#{mode[0]}"
+    kind = mode[1]
+    options = mode[2]
</ins><span class="cx"> 
</span><del>-def runFTLNoCJITNoInlineValidate(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-no-inline-validate", "--validateGraph=true", "--maximumInliningDepth=1", "--airForceBriggsAllocator=true", "--useB3HoistLoopInvariantValues=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+    # We need to define two variants, one expecting a cfg as the first
+    # argument, one not.
+    cfgKinds.each { |cfgKind|
+        methodName = "#{name}#{cfgKind.extension}".to_sym
+        define_method(methodName) { |*args|
+            cfg = nil
+            if cfgKind.expectCfg
+                # If we're defining a method that expects a cfg
+                # argument, pick it out of the args to pass to the
+                # initializer.
+                cfg = args.shift
+            end
+            # The cfg is initialized differently depending on whether
+            # we're in a run*Cfg method or not.
+            cfg = cfgKind.initializer.call(cfg, kind)
+            finalOptions = nil
+            if options.respond_to?(:call)
+                dynamicOptions = options.call(*args)
+                if dynamicOptions.nil?
+                    skip
+                    return
+                end
+                # The Proc object may override any cfg option passed
+                # in. This is used e.g. for dynamic test names as used
+                # by WithRAMSize and FTLEagerWatchdog.
+                cfg.merge!(dynamicOptions[:cfg])
+                # As the Proc may consume arguments, it's responsible
+                # for returning the final option list. Needed e.g. by
+                # WithRAMSize.
+                finalOptions = dynamicOptions[:testSpecificOptions]
+            else
+                finalOptions = options + args
+            end
+            runInner(cfg, *finalOptions)
+        }
+    }
+}
</ins><span class="cx"> 
</span><del>-def runFTLNoCJITOSRValidation(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-osr-validation", "--validateFTLOSRExitLiveness=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+CFG_NOISY = {
+            :outputHandler => noisyOutputHandler,
+            :errorHandler => noisyErrorHandler,
+}.freeze
</ins><span class="cx"> 
</span><del>-def runDFGEager(*optionalTestSpecificOptions)
-    run("dfg-eager", *(EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + FORCE_LLINT_EXIT_OPTIONS + optionalTestSpecificOptions))
-end
</del><ins>+BASE_MODES.each { |mode|
+    name = "runNoisyTest#{mode[0]}".to_sym
+    define_method(name) { |*args|
+        # For each base mode, define the "noisy" variant which simply
+        # calls the respective run#{name}Cfg, passing in the "noisy"
+        # cfg.
+        send("run#{mode[0]}Cfg", CFG_NOISY, "--validateBytecode=true", "--validateGraphAtEachPhase=true", *args)
+    }
+}
</ins><span class="cx"> 
</span><del>-def runDFGEagerNoCJITValidate(*optionalTestSpecificOptions)
-    run("dfg-eager-no-cjit-validate", "--validateGraph=true", *(NO_CJIT_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLEager(*optionalTestSpecificOptions)
-    run("ftl-eager", "--airForceBriggsAllocator=true", "--useRandomizingExecutableIslandAllocation=true", "--forcePolyProto=true", "--useDataIC=true", *(FTL_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLEagerWatchdog(*optionalTestSpecificOptions)
-    timeout = rand(100)
-    run("ftl-eager-watchdog-#{timeout}", "--watchdog=#{timeout}", "--watchdog-exception-ok", *(FTL_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLEagerNoCJITValidate(*optionalTestSpecificOptions)
-    run("ftl-eager-no-cjit", "--validateGraph=true", "--validateBCE=true", "--airForceIRCAllocator=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + FORCE_LLINT_EXIT_OPTIONS + EXECUTABLE_FUZZER_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLEagerNoCJITB3O1(*optionalTestSpecificOptions)
-    run("ftl-eager-no-cjit-b3o1", "--validateGraph=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + EAGER_OPTIONS + B3O1_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLEagerNoCJITOSRValidation(*optionalTestSpecificOptions)
-    run("ftl-eager-no-cjit-osr-validation", "--validateFTLOSRExitLiveness=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runNoCJITNoASO(*optionalTestSpecificOptions)
-    run("no-cjit-no-aso", "--useArchitectureSpecificOptimizations=false", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runNoCJITNoAccessInlining(*optionalTestSpecificOptions)
-    run("no-cjit-no-access-inlining", "--useAccessInlining=false", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLNoCJITNoAccessInlining(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-no-access-inlining", "--useAccessInlining=false", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runFTLNoCJITSmallPool(*optionalTestSpecificOptions)
-    run("ftl-no-cjit-small-pool", "--jitMemoryReservationSize=202400", *(FTL_OPTIONS + NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runNoCJIT(*optionalTestSpecificOptions)
-    run("no-cjit", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runEagerJettisonNoCJIT(*optionalTestSpecificOptions)
-    run("eager-jettison-no-cjit", "--useRandomizingExecutableIslandAllocation=true", "--forceCodeBlockToJettisonDueToOldAge=true", "--verifyGC=true", *(NO_CJIT_OPTIONS + optionalTestSpecificOptions))
-end
-
-def runShadowChicken(*optionalTestSpecificOptions)
-    run("shadow-chicken", "--useDFGJIT=false", "--alwaysUseShadowChicken=true", *optionalTestSpecificOptions)
-end
-
-def runMiniMode(*optionalTestSpecificOptions)
-    run("mini-mode", "--forceMiniVMMode=true", *optionalTestSpecificOptions)
-end
-
-def runLogicalAssignmentOperatorsEnabled(*optionalTestSpecificOptions)
-    run("logical-assignment-operators-enabled", "--useLogicalAssignmentOperators=true" , *(FTL_OPTIONS + optionalTestSpecificOptions))
-end
-
-def defaultRun
</del><ins>+# Default set of tests to run; propagates the cfg to every callee.
+def defaultRunCfg(cfg)
+    cfg.freeze
</ins><span class="cx">     if $mode == "quick"
</span><del>-        defaultQuickRun
</del><ins>+        defaultQuickRunCfg(cfg)
</ins><span class="cx">     else
</span><del>-        runDefault
-        runBytecodeCache
-        runMiniMode
</del><ins>+        runDefaultCfg(cfg)
+        runBytecodeCacheCfg(cfg)
+        runMiniModeCfg(cfg)
</ins><span class="cx">         if $jitTests
</span><del>-            runNoLLInt
-            runNoCJITValidatePhases
-            runNoCJITCollectContinuously if shouldCollectContinuously?
-            runDFGEager
</del><ins>+            runNoLLIntCfg(cfg)
+            runNoCJITValidatePhasesCfg(cfg)
+            runNoCJITCollectContinuouslyCfg(cfg) if shouldCollectContinuously?
+            runDFGEagerCfg(cfg)
</ins><span class="cx">             if $mode != "basic"
</span><del>-                runDFGEagerNoCJITValidate
-                runEagerJettisonNoCJIT
</del><ins>+                runDFGEagerNoCJITValidateCfg(cfg)
+                runEagerJettisonNoCJITCfg(cfg)
</ins><span class="cx">             end
</span><span class="cx"> 
</span><span class="cx">             return if !$isFTLPlatform
</span><span class="cx"> 
</span><del>-            runNoFTL
-            runFTLEager
-            runFTLEagerNoCJITValidate if $buildType == "release"
-            runFTLNoCJITSmallPool
</del><ins>+            runNoFTLCfg(cfg)
+            runFTLEagerCfg(cfg)
+            runFTLEagerNoCJITValidateCfg(cfg) if $buildType == "release"
+            runFTLNoCJITSmallPoolCfg(cfg)
</ins><span class="cx"> 
</span><span class="cx">             return if $mode == "basic"
</span><span class="cx"> 
</span><del>-            runFTLNoCJITValidate
-            runFTLNoCJITB3O0
-            runFTLNoCJITNoPutStackValidate
-            runFTLNoCJITNoInlineValidate
-            runFTLEagerNoCJITB3O1
</del><ins>+            runFTLNoCJITValidateCfg(cfg)
+            runFTLNoCJITB3O0Cfg(cfg)
+            runFTLNoCJITNoPutStackValidateCfg(cfg)
+            runFTLNoCJITNoInlineValidateCfg(cfg)
+            runFTLEagerNoCJITB3O1Cfg(cfg)
</ins><span class="cx">         end
</span><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><ins>+def defaultRun
+    defaultRunCfg({})
+end
+
</ins><span class="cx"> def defaultNoNoLLIntRun
</span><span class="cx">     if $mode == "quick"
</span><span class="cx">         defaultQuickRun
</span><span class="lines">@@ -1138,18 +1527,22 @@
</span><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><del>-def defaultQuickRun
-    runDefault
</del><ins>+def defaultQuickRunCfg(cfg)
+    runDefaultCfg(cfg)
</ins><span class="cx">     if $jitTests
</span><del>-        runNoCJITValidate
</del><ins>+        runNoCJITValidateCfg(cfg)
</ins><span class="cx"> 
</span><span class="cx">         return if !$isFTLPlatform
</span><span class="cx"> 
</span><del>-        runNoFTL
-        runFTLNoCJITValidate
</del><ins>+        runNoFTLCfg(cfg)
+        runFTLNoCJITValidateCfg(cfg)
</ins><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><ins>+def defaultQuickRun
+    defaultQuickRunCfg({})
+end
+
</ins><span class="cx"> def defaultSpotCheckNoMaximalFlush
</span><span class="cx">     defaultQuickRun
</span><span class="cx">     runNoCJITNoAccessInlining
</span><span class="lines">@@ -1800,46 +2193,19 @@
</span><span class="cx">     end
</span><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runNoisyTestImpl(kind, options, additionalEnv)
-    addRunCommand(kind, vmCommand + BASE_OPTIONS + options + $testSpecificRequiredOptions + [$benchmark.to_s], noisyOutputHandler, noisyErrorHandler, *additionalEnv)
-end
-
-def runNoisyTest(kind, *options)
-    runNoisyTestImpl(kind, options, [])
-end
-
</del><span class="cx"> def runNoisyTestWithEnv(kind, *additionalEnv)
</span><del>-    runNoisyTestImpl(kind, [], additionalEnv)
</del><ins>+    cfg = CFG_NOISY.dup
+    cfg[:kind] = kind
+    cfg[:additionalEnv] = additionalEnv
+    runDefaultCfg(cfg)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><del>-def runNoisyTestDefault
-    runNoisyTest("default", *FTL_OPTIONS)
-end
-
-def runNoisyTestNoFTL
-    runNoisyTest("no-ftl")
-end
-
-def runNoisyTestNoCJIT
-    runNoisyTest("ftl-no-cjit", "--validateBytecode=true", "--validateGraphAtEachPhase=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS))
-end
-
-def runNoisyTestNoCJITB3O1
-    runNoisyTest("ftl-no-cjit-b3o1", "--validateBytecode=true", "--validateGraphAtEachPhase=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + B3O1_OPTIONS))
-end
-
-def runNoisyTestEagerNoCJIT
-    runNoisyTest("ftl-eager-no-cjit", "--validateBytecode=true", "--validateGraphAtEachPhase=true", *(FTL_OPTIONS + NO_CJIT_OPTIONS + EAGER_OPTIONS + COLLECT_CONTINUOUSLY_OPTIONS))
-end
-
</del><span class="cx"> def defaultRunNoisyTest
</span><del>-    runNoisyTestDefault
-    if $jitTests and $isFTLPlatform
-        runNoisyTestNoFTL
-        runNoisyTestNoCJIT
-        runNoisyTestNoCJITB3O1
-        runNoisyTestEagerNoCJIT
-    end
</del><ins>+    cfg = {
+        :outputHandler => noisyOutputHandler,
+        :errorHandler => noisyErrorHandler,
+    }
+    defaultRunCfg(cfg)
</ins><span class="cx"> end
</span><span class="cx"> 
</span><span class="cx"> def skip
</span></span></pre>
</div>
</div>

</body>
</html>