[Webkit-unassigned] [Bug 50098] New webkit-patch rebaseline2 command.

Thu Dec 2 18:24:23 PST 2010

https://bugs.webkit.org/show_bug.cgi?id=50098

--- Comment #19 from Dirk Pranke <dpranke at chromium.org>  2010-12-02 18:24:23 PST ---
(In reply to comment #16)
> I see, so the mac fallback order [mac-tiger, mac-leopard, mac-snowleopard, mac] is meant to be searching from the oldest revisions to the newest revisions of mac. Thanks for the explanation.
> 
> So this would imply that nothing in the mac directory will ever get overwritten by this rebaseline script because we don't have mac-lion builders? IE: "SnowLeopard Intel Release (Tests)" would have its results sent to mac-snowleopard?
> 

Well, it's conceivable that someone might have access to a Lion builder and want to use this script. More generally, I was trying to describe the overall method we use for handling operating system versions.

Now, SL/Lion is an interesting case since we know it will arrive "soonish". I just asked on #webkit to attempt to clarify this. The answer I got back was that platform/mac should be used to refer to "the newest version of the mac operating system". In particular, those of us who don't have access to Lion have no way of knowing whether or not our tests (or results) are SL-specific, or if they will be the same for both Lion and SL (in which case, they should be in platform/mac).

Basically, only people w/ access to Lion who know what they are doing should ever put something into mac-snowleopard. The rest of us should assume mac-snowleopard is read/delete-only (not write).

As to how you address this in the code, I have no brilliant idea at the moment. The best idea I have is to distinguish the "read" path from the "write" path, maybe through a flag passed to baseline_search_path. I suggest that for now, for this patch, assume either mac == lion and we have access to it, or mac == snow-leopard, and pretend that mac-snowleopard doesn't exist.

However, I just clarified the i
> > 
> > > > > > WebKitTools/Scripts/webkitpy/layout_tests/rebaseline/result.py:35
> > > > > > +class Result(object):
> > > > > > +    """Represents the result of a single test on a single platform"""
> > > > > 
> > > > > We already have an abstraction for Result.
> > > > 
> > > > Are you referring to the webkitpy/layout_tests/test_types module? Those classes seem like they have a different usage to the ones I'm introducing here - they have no state and perform functions on filenames and file contents passed in to them. There is no point in having more than one instance of those classes, whereas the Result classes defined here are stateful and are meant to represent Results (local or remote) for the purposes of comparing them.
> > > 
> > 
> > Ouch. My head hurts here at introducing this concept. I think what you actually want is something closer to the layout_tests.layout_package.test_output concept, possibly combined with the expected_* methods on the Port class.
> > 
> > Also, please don't call this "Result" - that is too generic and overloaded a term. Even if you were to call it a "TestResult", that's confusing, because is the object referring to the actual expected output, or the results of running the test and comparing against the expectations (e.g., "PASS"/"FAIL", etc.)? 
> 
> Maybe we can use the term 'baseline' to mean the thing that this class represents, ie: something that the output of a test can be compared against to determine PASS/FAIL?
> 
> That seems to fit with the general phrase 'rebaselining' - it's taking baselines from builders and using them to overwrite our existing baselines. What do you think?
> 
> > 
> > I've been trying to morph the layout tests code into using something a bit more consistent, so we should line things up here if we can. I would prefer something like "TestOutput" here.
> 
> Could you be a bit more specific? How could I change Results to be more suitable?
> 
> > 
> > As to the overlap between test_output, test_types, and the port/* routines, those ned to be cleaned up as well. My current thinking is that the test_types classes are legacy objects from much earlier versions of new-run-webkit-tests (predating the port/* abstraction completely), and just need to go away. I.e., it's an abstraction that is confusing things.
> > 
> > In the very near future we'll need to add support for audio files and tests, and this aspect of the design should really be cleaned up (I'm expecting by me) before then.
> > 
> > Also, the work we're doing with reftests will impact this as well, so it would be good to get up to speed on that if you aren't already (and I've cc'ed Ito-san on this to make sure he's aware of it as well).
> 
> I'm not familiar with reftests. Could you point me to somewhere I could learn more about them?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.