[wpe-webkit] Webkit tests in WPE

[Michael Catanzaro]

On Wed, Jul 17, 2019 at 9:55 AM, Joshua Watt <jpewhacker at gmail.com> 
wrote:
> Hello,
> 
> I've been attempting to make some changes to the WPE port of WebKit. 
> As part of my changes, I've tried running the test suite with 
> |run-webkit-tests, but I get a lot off diff errors and timeouts. Is 
> this expected for the WPE port, or am I doing something wrong?

Yeah it's expected. You might be able to run a subset of the tests 
locally, but the entire testsuite is only expected to work properly on 
the bots.

Well, of course it's *supposed* to work properly everywhere, but it 
never has. Differences in system libraries will cause differences in 
test output, and that means failing tests. To mitigate this, we try to 
put every library that can affect test output into the jhbuild 
environment to ensure we are running the same software. 
(run-webkit-tests automatically uses the jhbuild environment that you 
previously built with update-webkitwpe-libs.) Probably broken tests 
correspond to something missing from the jhbuild environment that we 
don't know about.

Another problem I've seen before is Apache modules missing from the 
install-dependencies script leading to test failures. (Dependencies 
like Apache that aren't supposed to affect test results don't need to 
be in the jhbuild, but... sometimes theory doesn't work so well in 
practice.)

Finally, some tests are just broken and missing test expectations to 
mark them as such. You can see these here: 
https://build.webkit.org/dashboard/. (Tip: disable the Mac and Windows 
tests using the cog in the upper left to make the page load faster.) If 
you're getting similar test output to what our bots are seeing then 
your local environment is probably good.

Besides the layout tests ('run-webkit-tests --wpe') the other very 
important testsuite is the API tests ('run-wpe-tests'). It looks like 
we just pushed a commit that broke the API tests a couple hours ago... 
I'll look at that now.

Michael