[webkit-dev] Deployment of new EWS Non-Unified builder

Wed Jun 1 16:08:31 PDT 2022

I feel like this has been discussed adequately in the past, but one more time for good measure:

Any two platforms which don't build the exact same set of files will undergo unification differently. That means that unification shifts are an inherent part of working on WebKit, embedder or otherwise.

The only way to be certain that includes are done correctly in a given patch is to perform a non-unified build. This would be an unreasonable burden for local development, but that's exactly why an EWS builder is desirable.

To have this appear like a non-issue is to ignore the work that Sony and Igalia have continually performed through the 5(?) years since unified builds were introduced. From experience, I know that it can take a person about a day per month to clean up includes on behalf of the entire WebKit community.

Each patch should be responsible for getting its own includes correct. This has not been possible in the past, but Igalia's generosity in providing an EWS builder means that it now could be. They deserve our gratitude.

Ross
________________________________
From: Alexey Proskuryakov <ap at webkit.org>
Sent: Wednesday, June 1, 2022 3:26:50 PM
To: Kirsling, Ross <Ross.Kirsling at sony.com>
Cc: dpino <dpino at igalia.com>; webkit-dev at lists.webkit.org <webkit-dev at lists.webkit.org>
Subject: Re: [webkit-dev] Deployment of new EWS Non-Unified builder

Hi,

I'm not sure if we have a consensus on whether it is a project goal to keep non-unified build working at all times. As discussed last year, setting up post-commit bots is a pre-requisite for having EWS, so this part is resolved. But proactively maintaining the non-unified build is strictly more work than fixing it on demand. So the immediate response continues to be that we shouldn't be doing more work when we can do less.

You mention that embedders who build with non-default flags are more likely to hit issues. Building with non-default flags would be resulting in missing includes for non-unified builds too, do you have an estimate of how much this problem grows due to unified builds? How do we decide if everyone is responsible for the convenience of downstream embedders?

It sounds like none of actively supported ports builds non-unified by default, which I understand from the fact that a special post-commit queue had to be set up for that.

Perhaps part of the argument is that even though proactively maintaining the non-unified build is more work, this work is somehow easier than fixing builds on demand. If so, can you elaborate on why this is the case?

- Alexey

21 мая 2022 г., в 12:10 AM, Kirsling, Ross via webkit-dev <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>> написал(а):

This is wonderful news―thanks Diego!

Ross
________________________________
From: dpino via webkit-dev <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Sent: Friday, May 20, 2022 9:17:56 PM
To: webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org> <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: [webkit-dev] Deployment of new EWS Non-Unified builder

Hi,

Last year we started a thread to discuss the possibility of deploying a
new EWS bot that builds WebKit with Non-Unified sources [1]. This thread
explains the technical reasons why a non-unified build bot is desirable.
If you're aware of the problems introduced by unified sources
compilation, skip the next paragraph.

Unified sources compilation consists of merging several source files
together into one big file to improve building times. Usually the same
sources files are grouped together, but compilation flags or downstream
changes can create different source file groups. When that happens, you
may hit a build error. Since unified sources group files together,
missing headers in one file can be satisfied by another file in the same
group. What's happening in WebKit development currently is that source
code that breaks non-unified compilation frequently lands without even
being noticed. The breaks are usually related to missing headers.
Embedders that need to support WebKit builds with different flags or
maintain downstream changes are more likely to hit these compilation
errors.

Last year's attempt to deploy an EWS Non-Unified builder ended up in a
deployment of a post-commit bot plus two workers running in the UAT. It
was actually hard to get the Non-Unified builder working [2], something
that we achieved at the beginning of this year (kudos to Adrián and
Lauro). Since then we have been maintaining the post-commit bot very
closely. There are periods of time where the bot has remained green for
a long period of time.

But lately maintaining the bot green has become harder and harder,
specially due to the big refactorizations that have happened in WebKit
source code lately. The bot very rarely stays green longer than 2 or 3
days without close maintenance. We believe that we all should share the
responsibility of keeping the non-unified build working, and therefore
we want to proceed with the deployment of a EWS Non-Unified builder.

Initially, we'll provide two workers on a modern host with 8 cores
assigned each. We have found this setup faster for most patches than
many of the existing EWS queues, as it only runs a build in non-unified
mode, without test or upload steps. If this were not fast enough, we
would consider increasing the number of workers in the queue. We set up
a page[3] with a rough comparison to the regular (unified) bot and the
build times are pretty similar.

Diego

[1]
https://urldefense.com/v3/__https://www.mail-archive.com/webkit-dev@lists.webkit.org/msg30077.html__;!!JmoZiZGBv3RvKRSx!6YGOGRuK21OCDj8_MKm9gtNOYvBg9Ry10BMna8HXDHWUcUpVeDPBeA4QBT-nC3xj3j8wgt_mjpqbrrOsbTOSWjtVsXQ$
[2] https://urldefense.com/v3/__https://bugs.webkit.org/show_bug.cgi?id=226088__;!!JmoZiZGBv3RvKRSx!6YGOGRuK21OCDj8_MKm9gtNOYvBg9Ry10BMna8HXDHWUcUpVeDPBeA4QBT-nC3xj3j8wgt_mjpqbrrOsbTOSqxngQkY$
[3] https://urldefense.com/v3/__https://people.igalia.com/lmoura/webkit-bots-dashboard/unified.html__;!!JmoZiZGBv3RvKRSx!6YGOGRuK21OCDj8_MKm9gtNOYvBg9Ry10BMna8HXDHWUcUpVeDPBeA4QBT-nC3xj3j8wgt_mjpqbrrOsbTOSPFziSrI$
_______________________________________________
webkit-dev mailing list
webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>
https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!6YGOGRuK21OCDj8_MKm9gtNOYvBg9Ry10BMna8HXDHWUcUpVeDPBeA4QBT-nC3xj3j8wgt_mjpqbrrOsbTOS8gKEMAU$
_______________________________________________
webkit-dev mailing list
webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>
https://lists.webkit.org/mailman/listinfo/webkit-dev<https://urldefense.com/v3/__https://lists.webkit.org/mailman/listinfo/webkit-dev__;!!JmoZiZGBv3RvKRSx!4WbNlVZgYm9gwcwv8Co61EO1GFMnB8BZLzjvT_llnekVSvVVoSwG1ExtXWdwmZQ1MNNd35P2daPP$>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20220601/a481a705/attachment.htm>