[Webkit-unassigned] [Bug 37396] new-run-webkit-tests should log the order tests are run in

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=37396





--- Comment #5 from Dirk Pranke <dpranke at chromium.org>  2012-12-01 17:45:16 PST ---
Note that we currently do do this in the tests_run*.txt files written into layout-test-results (one file per worker). 

However, the file does not contain DRT/WTR pids so you can't tell when the workers crash or are otherwise restarted, and even the --debug-rwt-logging doesn't give you enough information to fix that. It also doesn't contain any timestamp information to help you determine which tests were running concurrently.

More importantly, it's hard (though not impossible) to do something useful with the data in the tests_run*.txt files, since you can't easily feed it back in to NRWT or control how things are sharded ( you can use --test-list to feed in a single list of tests, and at least now with --order=none it'll honor that, but that won't help across multiple workers).

I think ideally we'd merge all of the tests_run* files into a single file and add a --replay <path to file> option or something that would make this easier. I think there used to be a flag that would do a simpler version of this (--retry-last-failures or something?) but I'm not seeing it there now.

I'm closing this bug for now (since we do at leaset record the order) and going to file a new one for the --replay enhancement.

Regarding comment #4, I'm not sure that a random number seed would be needed or useful here. The nondeterminism comes from test timing and contention, not from using a random order. (You could of course specify the seed used when intentionally randomizing the tests, but that's a whole different thing).

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.