[Webkit-unassigned] [Bug 64142] New: test-webkitpy needs to support integration tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=64142

           Summary: test-webkitpy needs to support integration tests
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: dpranke at chromium.org
                CC: eric at webkit.org, aroben at apple.com, tony at chromium.org,
                    abarth at webkit.org, ojan at chromium.org


There is a class of tests in webkitpy that are "integration tests": they talk to real filesystems, require code to have been built, and start and stop servers. We have no way to automatically run and test them all.

I could have sworn I had either already implemented support for this in test-wekbitpy, or at least filed a bug, but I'm not seeing either now.

At any rate, there are several issues we should probably agree on:

1) What exactly is the difference between a unit test and an integration test? Is it that a unit test is very fast, or that the unit test does not require any system-dependent real resources, or that the test is tightly scoped to as few methods/modules as necessary?

For example, most of the tests in run_webkit_tests_integrationtest.py do not require real resources, and while they individually run relatively fast, there are a bunch of them. Perhaps more importantly, I'll call these "functional" tests rather than unit or integration tests, because they are testing what happens if you run NRWT with a particular set of command line options (but don't use real resources).

2) Can a single python file contain both integration and unit tests? I think the answer might want to be yes, because it's easy to imagine wanting to write tests that can run in both modes to get better coverage.

3) What should the test files be called? Currently we have _unittest.py for files that are (mostly?) unit tests, and _integrationtest.py for files that are (mostly?) integration tests.

4) Where do we want to run these tests? On the bots, and optionally interactively but not by default interactively?

I propose the following:

unit tests are platform-independent. integration tests are platform-dependent and may not function correctly if all of the dependencies aren't available.

Separately, we should separate our tests into "fast" and "slow", where we can run a "fast" set of tests that gets some basic, shallow coverage but may not be comprehensive. I would argue that any test that takes, say, more than 10ms to complete is "slow", and we can annotate those with a descriptor like @slow or something.

There should be a way to run either only unit tests, or unit + integration tests.

There should be an orthogonal way to run only the fast subset vs. the slow, comprehensive subset. The bots should run the slow set of tests, and all integration tests (although this may preclude running multiple test-webkitpys concurrently, I think it's important that at least some bots should cover this)

It is easy for me to write patches for whatever the desired combination of answers to the above are, but I haven't done so yet.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.