[Webkit-unassigned] [Bug 210976] New: [GStreamer] media/track/track-load-error-readyState.html passes only when accompanied by some other tests

Fri Apr 24 09:57:51 PDT 2020

https://bugs.webkit.org/show_bug.cgi?id=210976

            Bug ID: 210976
           Summary: [GStreamer]
                    media/track/track-load-error-readyState.html passes
                    only when accompanied by some other tests
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: WebKitGTK
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: aboya at igalia.com
                CC: bugs-noreply at webkitgtk.org

Ok, this is a weird one. This is media/track/track-load-error-readyState.html, the whole thing:

<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

        <script src=../media-file.js></script>
        <script src=../video-test.js></script>
        <script>

            function trackError()
            {
                consoleWrite("EVENT(error)");
                track = document.getElementById('testTrackError'); 
                testExpected("track.readyState", HTMLTrackElement.ERROR);
                endTest();
            }

        </script>
    </head>
    <body>
        <p>Tests the error event on HTMLTrackElement and ERROR readyState on TextTrack.</p>
        <video id="videotests">
            <track id="testTrackError" src="junk" onerror="trackError()" default>
        </video>
    </body>
</html>

Run it in a minibrowser and it will timeout every time. The onerror() handler will not be called and no text will be printed. 100% reproducible.

Run it in singly run-webkit-tests, and indeed, you will get a timeout, 100% of the time:

    $ run-webkit-tests --gtk --debug media/track/track-load-error-readyState.html
    Using port 'gtk-wk2'
    Test configuration: <, x86, debug>
    Placing test results in /webkit/WebKitBuild/Debug/layout-test-results
    Using Debug build
    Pixel tests disabled
    Regular timeout: 30000, slow test timeout: 150000
    Command line: /webkit/Tools/jhbuild/jhbuild-wrapper --gtk run /webkit/WebKitBuild/Debug/bin/WebKitTestRunner -

    Found 1 test; running 1, skipping 0.

    Verbose baseline search path: platform/gtk -> platform/wk2 -> generic

    Baseline search path: platform/gtk -> platform/wk2 -> generic

    Running 1 test

    Running 1 WebKitTestRunner.     

    [1/1] media/track/track-load-error-readyState.html failed unexpectedly (test timed out, text diff)

    Retrying 1 unexpected failure ...

    Running 1 WebKitTestRunner.

    [1/1] media/track/track-load-error-readyState.html failed unexpectedly (test timed out, text diff)

    0 tests ran as expected, 1 didn't:

    Regressions: Unexpected timeouts (1)
      media/track/track-load-error-readyState.html [ Timeout ]

But here comes the surprise. Run it within certain sets of tests, and it will pass, 100% of the time. For instance:

    $ run-webkit-tests --gtk --debug media/track/track-in-band-style.html media/track/track-in-band-subtitles-too-large.html media/track/track-kind.html media/track/track-legacyapi-with-automatic-mode.html media/track/track-load-error-readyState.html
    Using port 'gtk-wk2'
    Test configuration: <, x86, debug>
    Placing test results in /webkit/WebKitBuild/Debug/layout-test-results
    Using Debug build
    Pixel tests disabled
    Regular timeout: 30000, slow test timeout: 150000
    Command line: /webkit/Tools/jhbuild/jhbuild-wrapper --gtk run /webkit/WebKitBuild/Debug/bin/WebKitTestRunner -

    Found 5 tests; running 5, skipping 0.

    Verbose baseline search path: platform/gtk -> platform/wk2 -> generic

    Baseline search path: platform/gtk -> platform/wk2 -> generic

    Running 5 tests

    Running 1 WebKitTestRunner.     

    All 5 tests ran as expected.   

This is also the case when running all the tests. Therefore, according to the bots, this test passes.

I made changes to the test file to confirm that it is in fact being run. The onerror callback is called, and modifying the code there modifies the output which makes the test fail as expected.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20200424/0e54658b/attachment-0001.htm>