[Webkit-unassigned] [Bug 32954] have run-webkit-tests output the JSON files for tracking layout test flakiness

Wed Mar 30 21:55:41 PDT 2011

https://bugs.webkit.org/show_bug.cgi?id=32954

--- Comment #9 from Adam Roben (:aroben) <aroben at apple.com>  2011-03-30 21:55:41 PST ---
Some info gleaned via IRC:

<aroben> ojan: I think my main problem is not understanding the format of incremental_results.json or how it's generated
<ojan> aroben: yeah...it's confusing...
<ojan> aroben: you could get a small version of incremental_results.json...
<ojan> aroben: with a local test run...
<ojan> aroben: let me get you the right commandline to run
<aroben> ojan: I guess that's true!
<ojan> aroben: new-run-webkit-tests editing/selection --build-number 5 --builder-name MyBuilderName
<ojan> aroben: puts the files in WebKitBuild/Debug/layout-test-results
<ojan> aroben: there's 4 json files it spits out
<ojan> aroben: you can ignore unexpected_results.json for now
<ojan> aroben: it has the same format as full_results.json fwiw
<ojan> aroben: one catch, it will only include individual tests there if the test actually fails
<ojan> aroben: so you'll want to force a few tests to fail/timeout/crash to get a better sense of what the format is
<aroben> ojan: which file are you referring to?
<ojan> aroben: incremental_results.json
<aroben> ojan: how accurate is https://sites.google.com/a/chromium.org/dev/developers/design-documents/layout-tests-results-dashboard ?
<ojan> aroben: it was accurate when i wrote it almost a year ago, but some things have changed
<ojan> aroben: it should be 100% accurate for expectations.json
<ojan> aroben: but for incremental_results.json it's a bit outdated...
<ojan> aroben: incremtnal_results.json didn't exist when i wrote that
<ojan> aroben: it's the same format, but it only contains the data for one run
<aroben> ojan: same as results.json, you mean?
<ojan> then, ont he server-side it gets merged into results.json
<ojan> aroben: and we'll need to figure out what the right values are for the *Count values, but that can be done later.
<aroben> ojan: so incremental_results.json contains information about tests that didn't match expectations for the current test run?
<aroben> ojan: is that a fair description?
<ojan> aroben: exactly
<ojan> aroben: + a little extra metadata not specific to individual tests
<ojan> aroben: e.g. the webkitRevision, buildNumber, time of the test run, etc

<ojan> aroben: in an ideal world, we would synthesize all the arguments needed by http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/layout_package/json_layout_results_generator.py from webkit-patch
<ojan> aroben: so we could just use that code
<aroben> ojan: seems like that should be possible
<ojan> aroben: yeah...i tthink that'd be easiest
<ojan> aroben: most of those arguments are straightforward
<aroben> ojan: (though I might make it a separate command rather than reusing webkit-patch; this isn't very "patch"-related)
<ojan> aroben: yeah, makes sense
<ojan> aroben: some of the arguments will need to be passed on the commandline obviously since they depend on the bot (e.g. builder_name, build_number, etc)
<aroben> ojan: build_number is just the buildbot build number?
<ojan> yeah
<ojan> the confusing one is builder_name vs build_name
<ojan> :(
<aroben> ojan: does NRWT always spit out these JSON files, even when an engineer runs the tests?
<ojan> aroben: yes
<ojan> aroben: but it doesn't do anything with them unless you pass a test-results-server argument
<ojan> aroben: except "webkit-patch rebaseline-server" uses unexpected_results.json
<aroben> ojan: so what's build_name?
<ojan> aroben: lemme figure that out...i always need to look at the code to remember
<ojan> aroben: for the build.webkit.org bots it might be the same a builder_name
<ojan> aroben: for reasons i don't udnerstand, the chromium bots store the layout test results at a path with a different name than the builder_name
<ojan> aroben: e.g. http://build.chromium.org/f/chromium/layout_test_results/Webkit_Win__deps__dbg__1_/79958/
<ojan> is for Webkit Win (deps)(dbg)(1)
<aroben> ojan: I see
<ojan> aroben: but it looks like the build.webkit.org bots don't have that distinction
<aroben> ojan: ok, we'll leave it the same for now and see what happens
<ojan> aroben: yup :)
<aroben> ojan: there are other differences between build.webkit.org and build.chromium.org's URLs
<aroben> ojan: I wonder if they will matter
<ojan> aroben: they won't matter for the JSON data
<ojan> aroben: we'll need to fix a bunch of assumptions in the dashboard itself
<aroben> ojan: where does that code live?
<ojan> aroben: the dashboard code?
<aroben> ojan: yeah
<ojan> aroben: it's in the chromium repo
<ojan> aroben: http://src.chromium.org/viewvc/chrome/trunk/tools/dashboards/
<ojan> aroben: the dashboard is just a handful of static html/js files that pull in the JSON files, so there's no server really
<ojan> aroben: test-results.appspot.com is just a dumb file server except for the fact that it knows how to merge incremental_results.json files
<ojan> aroben: fwiw, i don't think anyone would be opposed to moving the dashboard code into the webkit tree as long as it's still OK for it to have some support for the chromium tests
<ojan> aroben: the test-results.appspot.com code is in the webkit tree already
<aroben> ojan: I don't really care where it lives, as long as we can get this all working
<aroben> ojan: huh?
<ojan> aroben: http://trac.webkit.org/browser/trunk/Tools/TestResultServer
<aroben> ojan: oh, but the dashboard code is separate
<ojan> yeah
<ojan> aroben: the dashboard code could be served up from anywhere
<aroben> ojan: seems like it would make sense to have it all in one place
<ojan> makes it easy to "run a local server" since you just load the html file from your local checkout in a browser :)
<ojan> aroben: yeah...there just wasn't much motivation since only the chromium bots were supported anyways
<ojan> aroben: we should prob move the code once we start hooking bits together
<ojan> aroben: but i don't really care either way :)
<aroben> ojan: I only care if I personally need to make changes to the dashboard
<aroben> ojan: in which case svn.webkit.org is much easier for me

<aroben> ojan: what's "fixable" in full_results.json?
<aroben> ojan: oh, your docs page explains it
<aroben> ojan: for us it will just be the number of failures I think
<ojan> aroben: yes
<ojan> aroben: and most of the *Count values will always be 0 for you
<ojan> aroben: yeah, i think fixableCount is the only Count value that makes any sense for non-Chromium ports

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.