[Webkit-unassigned] [Bug 95144] Skipping tests on platform mac due to WebCore::JSEventListener::jsFunction(WebCore::ScriptExecutionContext*) crashes

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=95144





--- Comment #2 from Mark Lam <mark.lam at apple.com>  2012-08-27 16:58:17 PST ---
How did I choose which tests to skip?
=======================
I manually sampled hundreds of the bot test failure results including going back to very early tests whose results are still on the server but are not conveniently presented in the waterfall results.  I did this by going to the following URL for Apple Lion WK1 debug:

http://build.webkit.org/builders/Apple%20Lion%20Debug%20WK1%20%28Tests%29/builds/<build number>

This is how I found build 1713 which was the earliest build I saw this crash manifest on.

For each occurrence of the crash stack trace I encountered, I recorded the build number and the failing test that the crash stack trace was observed on.  Based on these records, I was able to observe that there is a consistency in which tests fail due to this crash.

Subsequently, I run each of those tests individually and repeatedly on my local machine like so:

oss/webkit4$ perl ./Tools/Scripts/run-webkit-tests --no-launch-safari --no-new-test-results --no-sample-on-timeout --results-directory ./layout-test-results --use-remote-links-to-tests --debug --no-build --exit-after-n-crashes-or-timeouts 20 --exit-after-n-failures 500 -v --repeat-each 1000 <test name>

If the test crashes as expected with the same stack trace, then I consider it a worthy candidate to be skipped.  The fact that it crashes suggests that this test has the trigger / originating condition for manifesting the crash.

There is no guarantee that once we exclude these tests that the crashes will go away altogether.  Since the crashes are intermittent, there may be a few more straggles who just haven't manifested with this symptom yet.  We would expect the set of the stragglers to be less in count than the set I am now skipping.

Alternatively, there's a chance that the symptom will just start manifesting on other tests.  But if that case happens, then we will see a whole new set of failures with this crash of about the same size as the original set.  If that occurs, we will now have evidence that the crash can occur just anywhere, and we can and should revert this current change set for skipping these tests.

Note: there were some tests that were seen to manifest the crash failure, but when run alone, do not manifest the crash.  This suggests that there are tests that do not have the trigger / originating condition of the crash.  As a result, if we successfully skip all the tests that do trigger the crash, there is a possibility that the crash will cease to manifest.

Also, the test that do not manifest the crash when run alone, though previously seen in reported bot failures, are excluded from the skip list.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.