Reducing / removing use of Makefile based DerivedSources.make
Hi webkit-dev, I’d like to propose, and gauge feedback on, reducing (with the goal of ultimately removing) the use of Makefile based DerivedSources.make. My understanding is that currently only the Xcode based ports still use DerivedSources.make, as all the CMake based ones have moved derived source generation to within CMake, so that should limit the scope of who this might affect. Why do we use Makefiles today? While I can’t recall the initial reasons for using Makefiles for derived sources, over the years there have been a number of advantages to it that I do know of. One clear advantage, that is no longer applicable, was code sharing, as earlier in the project, at least the Apple Windows port did utilize these Makefiles. Another was to work around some limitations in what dependencies Xcode was able to track with build rules. It seems at least some of the problems with build rules are no longer an issue, as we can now specify inputs to build rules, but I don’t if other problems will still be there, but for some prototyping I did, nothing yet has come up. What would we move to? As this only affects the Xcode based ports, we would move to distinct script phases and build rules in the Xcode project. Why make this change? What’s the benefit? There are few reasons to consider this. One advantage is simplifying the build system. Rather than two dependency systems (one for Xcode, one for the Makefile) we reduce it down to one. And with additional knowledge of the stages and dependencies, Xcode could potentially parallelize more phases. We would would also save some time by avoiding invoking make in the first place. We also have a bit of an issue with make itself, as due to system requirements, we are forever stuck with Make 3.81, which is coming up on being 15 years old. More than once in the last year I have tried to troubleshoot makefile issues, looking for resources on the web, only to be stymied because the solutions I found required newer make. What are the downsides? One potential downside will be that it will be a bit harder for those without Xcode to add new types of derived sources. I am not sure how much a real problem in practice this will be, as editing project.pbxproj files is already required for just adding new files, but I want to call it out anyway. What are your thoughts on this? Are there additional reasons we might want to stick with or move away from Makefile based derived sources? Thanks, -Sam
On Oct 17, 2020, at 12:34 PM, Fujii Hironori <fujii.hironori@gmail.com <mailto:fujii.hironori@gmail.com>> wrote:
On Sun, Oct 18, 2020 at 2:00 AM Sam Weinig <weinig@webkit.org <mailto:weinig@webkit.org>> wrote:
What are your thoughts on this?
I'm disappointed. I want to use CMake for all ports.
I hear you, it would indeed be nice to have a single build system, but I’d really like to keep this thread specifically about the explicit issue of DerivedSources.make use for the current Xcode based builds. I encourage you to start a separate thread about moving to a CMake only world. There are plenty of seperate challenges there (e.g. generating identical output, maintaining Xcode based work flows, etc.), and I am not sure how practical it is, but I think enumerating the challenging and discussing them is important. Thanks, - Sam
On Sat, Oct 17, 2020 at 10:00 AM Sam Weinig <weinig@webkit.org> wrote:
Hi webkit-dev,
I’d like to propose, and gauge feedback on, reducing (with the goal of ultimately removing) the use of Makefile based DerivedSources.make.
My understanding is that currently only the Xcode based ports still use DerivedSources.make, as all the CMake based ones have moved derived source generation to within CMake, so that should limit the scope of who this might affect.
Why do we use Makefiles today?
While I can’t recall the initial reasons for using Makefiles for derived sources, over the years there have been a number of advantages to it that I do know of. One clear advantage, that is no longer applicable, was code sharing, as earlier in the project, at least the Apple Windows port did utilize these Makefiles. Another was to work around some limitations in what dependencies Xcode was able to track with build rules. It seems at least some of the problems with build rules are no longer an issue, as we can now specify inputs to build rules, but I don’t if other problems will still be there, but for some prototyping I did, nothing yet has come up.
What would we move to?
As this only affects the Xcode based ports, we would move to distinct script phases and build rules in the Xcode project.
Why make this change? What’s the benefit?
There are few reasons to consider this. One advantage is simplifying the build system. Rather than two dependency systems (one for Xcode, one for the Makefile) we reduce it down to one. And with additional knowledge of the stages and dependencies, Xcode could potentially parallelize more phases. We would would also save some time by avoiding invoking make in the first place.
We also have a bit of an issue with make itself, as due to system requirements, we are forever stuck with Make 3.81, which is coming up on being 15 years old. More than once in the last year I have tried to troubleshoot makefile issues, looking for resources on the web, only to be stymied because the solutions I found required newer make.
What are the downsides?
One potential downside will be that it will be a bit harder for those without Xcode to add new types of derived sources. I am not sure how much a real problem in practice this will be, as editing project.pbxproj files is already required for just adding new files, but I want to call it out anyway.
Do we need to dig up some kind of bespoke Xcode UI to add a new IDL file after this? I always find all those build phase things in Xcode to be impossible to edit.
What are your thoughts on this? Are there additional reasons we might want to stick with or move away from Makefile based derived sources?
Is there some way we can use something like Sources.txt to list the files from which derived sources are created? DerivedSources.make is annoying to edit but the equivalent CMake file is equally annoying to edit. If we had a simple list of files like we do for unified sources, it would make an average WebKit contributor's life way easier. - R. Niwa
Please note that this is a Cocoa-only, Xcode-only conversation. As Sam pointed out, there’s a whole different “will you please switch to CMake” conversation. That’s not an easy switch to make for programmers working on the Apple platforms, but we might do it at some point if we can work out the issues Sam mentioned.
On Oct 17, 2020, at 10:00 AM, Sam Weinig <weinig@webkit.org> wrote:
While I can’t recall the initial reasons for using Makefiles for derived sources, over the years there have been a number of advantages to it that I do know of. One clear advantage, that is no longer applicable, was code sharing, as earlier in the project, at least the Apple Windows port did utilize these Makefiles.
Pretty sure I was the one who did this. And I do recall the initial reasons. There were only two. 1) Code sharing I believed at the time that the rules for building derived sources were tricky and hard to get right, and proposed that we would use the lowest-common-denominator build system, make, across all platforms. This strategy didn’t work because right from the beginning people working with various build systems used copies of the derived sources rules that I wrote rather than invoking make, and never tried sharing code. I think this was a missed opportunity to share complicated code, but it happened a very long time ago, and I made my peace with it back then. 2) Xcode build system deficiencies At that time, Xcode’s build system had too many problems for these kinds of complex rules. Many years later, that may no longer be true. Since we’re not even trying to do the code sharing, I think it’s fine to stop using makefiles now. If we’re deeply committed to switching to CMake at some point in future, one way to take a step in that direction would be to return to the original code sharing goal, and use CMake for derived sources, so we have code sharing. I don’t know how modular CMake is and how practical it is to use it for part, but not all, of the build process with Xcode. People using CMake for everything might be frustrated with the constraints this could place on how CMake is used. I think an important third goal that I did not have at the start, but have now, is the one Ryosuke mentioned: 3) Minimize the number of places files are listed, so it’s easy to add, remove, and rename files of various types. It would be wonderful to drastically cut down the repeated mentions of the same filenames. People not on Xcode platforms probably focus primarily on Xcode projects for this frustration. But I find it ridiculous how many places I have to add things. For example, to add AbstractRange I had to add filenames to: Shared: Sources.txt CMake: CMakeLists.txt Headers.cmake Xcode: WebCore.xcodeproj/project.pbxproj And also, the checked-in generated files (that we’d need regardless of build systems, due to Apple build requirements): DerivedSources-input.xcfilelist DerivedSources-output.xcfilelist — Darin
It would be wonderful to drastically cut down the repeated mentions of the same filenames. People not on Xcode platforms probably focus primarily on Xcode projects for this frustration. But I find it ridiculous how many places I have to add things. For example, to add AbstractRange I had to add filenames to: Shared - Sources.txt: Two source files and one IDL-generated file. CMake: - CMakeLists.txt: IDL file. - Headers.cmake: One source file and one IDL-generated file. Xcode: - DerivedSources.make: IDL file - WebCore.xcodeproj/project.pbxproj: IDL file, two source files, and Two IDL-generated files. And also, updating these checked-in generated files (that we’d need regardless of build system, due to Apple build requirements): - DerivedSources-input.xcfilelist - DerivedSources-output.xcfilelist It seems like we should have a goal that you should not have to list IDL-generated file names. That will be hard to accomplish in Xcode, though. Maybe the reason we haven’t been so ambitious about getting this right for CMake is that it’s so unreachable a goal for Xcode, given that we want the files to show up in a clean way in the IDE? And it would be good to find a way to share the lists of IDL files like we do the list of source files. That seems relevant to this discussion of changing DerivedSources.make. — Darin
One direct benefit of moving away from DerivedSources.make would likely be (I say likely, as the details of how it works out are far from certain in my mind) removing at least one place that a IDL file needs to be listed as we would not need to explicitly the list the IDL file in DerivedSources.make, and would only need to ensure it was in some *input.xcfilelist. But the idea of sharing “input” lists is very compelling. Just focusing on the IDL files themselves, I wonder if we could construct an input file that could serve both CMake and Xcode. It seems like CMake has the ability to read files into variables (https://cmake.org/cmake/help/latest/command/file.html#reading <https://cmake.org/cmake/help/latest/command/file.html#reading>) so perhaps we could construct IDLs.txt that worked both as an input CMake and Xcode. One issue that I don’t know how to solve in a good way is how to make adding files to the Xcode project easier. Perhaps this is somewhere that a script could help? Rather than adding the files to Sources.txt and then manually adding them to the Xcode project, perhaps we should just have a script that does this for us (either via manually invoking it or with the post processing that generates the xcfilelists?) - Sam
On Oct 18, 2020, at 10:38 AM, Darin Adler <darin@apple.com> wrote:
It would be wonderful to drastically cut down the repeated mentions of the same filenames. People not on Xcode platforms probably focus primarily on Xcode projects for this frustration. But I find it ridiculous how many places I have to add things. For example, to add AbstractRange I had to add filenames to:
Shared
- Sources.txt: Two source files and one IDL-generated file.
CMake:
- CMakeLists.txt: IDL file. - Headers.cmake: One source file and one IDL-generated file.
Xcode:
- DerivedSources.make: IDL file - WebCore.xcodeproj/project.pbxproj: IDL file, two source files, and Two IDL-generated files.
And also, updating these checked-in generated files (that we’d need regardless of build system, due to Apple build requirements):
- DerivedSources-input.xcfilelist - DerivedSources-output.xcfilelist
It seems like we should have a goal that you should not have to list IDL-generated file names. That will be hard to accomplish in Xcode, though. Maybe the reason we haven’t been so ambitious about getting this right for CMake is that it’s so unreachable a goal for Xcode, given that we want the files to show up in a clean way in the IDE?
And it would be good to find a way to share the lists of IDL files like we do the list of source files. That seems relevant to this discussion of changing DerivedSources.make.
— Darin _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Hi Sam,
On Oct 18, 2020, at 11:01, Sam Weinig <weinig@apple.com> wrote:
One direct benefit of moving away from DerivedSources.make would likely be (I say likely, as the details of how it works out are far from certain in my mind) removing at least one place that a IDL file needs to be listed as we would not need to explicitly the list the IDL file in DerivedSources.make, and would only need to ensure it was in some *input.xcfilelist.
The way I imagine this working is that the existing .idl file references in the (e.g., WebCore) project would become members in a target with a custom build rule for “*.idl”. Target membership is itself another list inside the .xcodeproj file, just one you manage through checkboxes in Xcode (and can be done reasonably quickly as part of adding a new file reference). Whether or not this actually creates one fewer “place a IDL file needs to be listed” (which I think is a worthwhile goal), I think you should still remove the Makefile build phases from Xcode for the reasons you originally cited. Andy
On Oct 20, 2020, at 10:51, Andy Estes <aestes@apple.com> wrote:
On Oct 18, 2020, at 11:01, Sam Weinig <weinig@apple.com> wrote:
One direct benefit of moving away from DerivedSources.make would likely be (I say likely, as the details of how it works out are far from certain in my mind) removing at least one place that a IDL file needs to be listed as we would not need to explicitly the list the IDL file in DerivedSources.make, and would only need to ensure it was in some *input.xcfilelist.
The way I imagine this working is that the existing .idl file references in the (e.g., WebCore) project would become members in a target with a custom build rule for “*.idl”. Target membership is itself another list inside the .xcodeproj file, just one you manage through checkboxes in Xcode (and can be done reasonably quickly as part of adding a new file reference).
Whether or not this actually creates one fewer “place a IDL file needs to be listed” (which I think is a worthwhile goal), I think you should still remove the Makefile build phases from Xcode for the reasons you originally cited.
Andy
Overall, this sounds like an awesome endeavor. The initial reasons for using Makefiles were sound ones (as Darin described), but there are also the incumbent issues, such as maintainability and efficiency. I wasn’t clear on what the alternative to Makefiles would look like, though. The options that occur to me are: * Custom build rules, as Andy describes. This seems to make sense, but I wonder if it is flexible enough for all the cases we might want to put it through. Our DerivedSources.make files process many types of files. I haven’t checked, but I could imagine that not all of these files could be uniquely distinguished by their suffixes. That is, there might be one set of *.foo files that should be processed one way, and another set that should processed another way. In those cases, I guess some suffixes could be changed to accommodate the different processing. * Using a set of Run Script build phases. That is, have a Run Script phase for processing a specific set of *.idl files, another for processing one set of *.foo files, a third for processing a different specific set of *.foo files, and so on. I’m against this approach, though. While it offers lots of flexibility, it’s not very maintainable. It would mean having to keep the set of script inputs and outputs up-to-date, something that is fragile, even with the help of the generate-xcfilelists utility script. Of these two, the one based on custom build rules seems to make the most sense. One need only add a file to an Xcode project and possibly a CMake file. Of note, the dreaded .xcfilelists are not affected (and, if this project can be taken to its extreme, we can get rid of the DerivedSources*.xcfilelist files). I do worry about some efficiency issues, though. For instance, one part of DerivedSources.make processes Platform.h (and all its children) in order to extract some configuration information. That processing is done once per DerivedSources.make invocation. If we move to a model where a script is invoked once per *.idl file, then the overhead is now multiplied by the number of *.idl files. We should watch for that. (Perhaps your recent configuration/preferences/settings changes obviate the need to pre-process Platform.h all the time?) Still, I’m open to a solution based on a single IDL.txt file that can be shared by Xcode and CMake. It all depends on the details, the resulting solution, and the trade-offs it makes. Regarding Xcode vs. CMake, I certainly understand the reasons people would like to see us move solely to CMake. But, for now, the plan is to see if we can get Apple-OS ports to build with CMake at all. Replacing Xcode and xcodebuild is a follow-on project with an interesting set of issues. I set up #cmake on Slack to provide a place where we can talk about this for anyone who’s interested. — Keith
On Oct 20, 2020, at 2:25 PM, Keith Rollin <krollin@apple.com> wrote:
On Oct 20, 2020, at 10:51, Andy Estes <aestes@apple.com> wrote:
On Oct 18, 2020, at 11:01, Sam Weinig <weinig@apple.com> wrote:
One direct benefit of moving away from DerivedSources.make would likely be (I say likely, as the details of how it works out are far from certain in my mind) removing at least one place that a IDL file needs to be listed as we would not need to explicitly the list the IDL file in DerivedSources.make, and would only need to ensure it was in some *input.xcfilelist.
The way I imagine this working is that the existing .idl file references in the (e.g., WebCore) project would become members in a target with a custom build rule for “*.idl”. Target membership is itself another list inside the .xcodeproj file, just one you manage through checkboxes in Xcode (and can be done reasonably quickly as part of adding a new file reference).
Whether or not this actually creates one fewer “place a IDL file needs to be listed” (which I think is a worthwhile goal), I think you should still remove the Makefile build phases from Xcode for the reasons you originally cited.
Andy
Overall, this sounds like an awesome endeavor. The initial reasons for using Makefiles were sound ones (as Darin described), but there are also the incumbent issues, such as maintainability and efficiency.
I wasn’t clear on what the alternative to Makefiles would look like, though. The options that occur to me are:
* Custom build rules, as Andy describes. This seems to make sense, but I wonder if it is flexible enough for all the cases we might want to put it through. Our DerivedSources.make files process many types of files. I haven’t checked, but I could imagine that not all of these files could be uniquely distinguished by their suffixes. That is, there might be one set of *.foo files that should be processed one way, and another set that should processed another way. In those cases, I guess some suffixes could be changed to accommodate the different processing.
* Using a set of Run Script build phases. That is, have a Run Script phase for processing a specific set of *.idl files, another for processing one set of *.foo files, a third for processing a different specific set of *.foo files, and so on. I’m against this approach, though. While it offers lots of flexibility, it’s not very maintainable. It would mean having to keep the set of script inputs and outputs up-to-date, something that is fragile, even with the help of the generate-xcfilelists utility script.
Of these two, the one based on custom build rules seems to make the most sense. One need only add a file to an Xcode project and possibly a CMake file. Of note, the dreaded .xcfilelists are not affected (and, if this project can be taken to its extreme, we can get rid of the DerivedSources*.xcfilelist files). I do worry about some efficiency issues, though. For instance, one part of DerivedSources.make processes Platform.h (and all its children) in order to extract some configuration information. That processing is done once per DerivedSources.make invocation. If we move to a model where a script is invoked once per *.idl file, then the overhead is now multiplied by the number of *.idl files. We should watch for that. (Perhaps your recent configuration/preferences/settings changes obviate the need to pre-process Platform.h all the time?)
My thought is that we would be using a combination of Run Script phases and Build Rules, depending on the thing being generated. There are quite a few one-off scripts like makeprop.pl, makevalues.pl, makeSelectorPseudoClassAndCompatibilityElementMap.py, etc, that don’t have their input or output change very often at all, so are really much better suited for a Run Script phase. Whereas things like the IDLs are better suited for build rules. For the Platform.h issue, I think we could probably engineer things in the short term to have a script phase that produces a Defines.txt that the other script phases and build rules depend on. Longer term, I would like to remove the need for it altogether, but that’s going to take additional work in the various scripts.
Still, I’m open to a solution based on a single IDL.txt file that can be shared by Xcode and CMake. It all depends on the details, the resulting solution, and the trade-offs it makes.
Regarding Xcode vs. CMake, I certainly understand the reasons people would like to see us move solely to CMake. But, for now, the plan is to see if we can get Apple-OS ports to build with CMake at all. Replacing Xcode and xcodebuild is a follow-on project with an interesting set of issues. I set up #cmake on Slack to provide a place where we can talk about this for anyone who’s interested.
Thanks for the feedback, - Sam
— Keith
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
On Oct 20, 2020, at 3:09 PM, Sam Weinig <weinig@apple.com> wrote:
For the Platform.h issue, I think we could probably engineer things in the short term to have a script phase that produces a Defines.txt that the other script phases and build rules depend on.
We definitely can, and it’s simpler than how we do it now. I have a patch that does this for the makefile partly done and set aside. My approach was to revise each script, one at a time, to take a file with the defines rather than requiring they be passed on the command line. It doesn’t work for the parts of the makefile itself that depend directly on checking the defines (grep for findstring.*FEATURE_AND_PLATFORM_DEFINES to see those). — Darin
On Oct 20, 2020, at 3:36 PM, Darin Adler <darin@apple.com> wrote:
On Oct 20, 2020, at 3:09 PM, Sam Weinig <weinig@apple.com> wrote:
For the Platform.h issue, I think we could probably engineer things in the short term to have a script phase that produces a Defines.txt that the other script phases and build rules depend on.
We definitely can, and it’s simpler than how we do it now. I have a patch that does this for the makefile partly done and set aside. My approach was to revise each script, one at a time, to take a file with the defines rather than requiring they be passed on the command line. It doesn’t work for the parts of the makefile itself that depend directly on checking the defines (grep for findstring.*FEATURE_AND_PLATFORM_DEFINES to see those).
Great. I think we can probably easily replace the makefile checking FEATURE_AND_PLATFORM_DEFINES itself ones. For the ones conditionalizing adding to ADDITIONAL_BINDING_IDLS, those IDLs should just have the appropriate Conditional=* extended attribute added, then they can be included unconditionally in the makefile. For the ones conditionalizing adding go USER_AGENT_STYLE_SHEETS, again, there doesn’t seem a real reason to keep that. The sheets are all preprocessed anyway, so we can just move those #ifdefs into the files. I filed https://bugs.webkit.org/show_bug.cgi?id=218001 <https://bugs.webkit.org/show_bug.cgi?id=218001> for this and will get that done regardless. -Sam
On Oct 20, 2020, at 4:08 PM, Sam Weinig <weinig@apple.com> wrote:
For the ones conditionalizing adding to ADDITIONAL_BINDING_IDLS, those IDLs should just have the appropriate Conditional=* extended attribute added, then they can be included unconditionally in the makefile.
For the ones conditionalizing adding go USER_AGENT_STYLE_SHEETS, again, there doesn’t seem a real reason to keep that. The sheets are all preprocessed anyway, so we can just move those #ifdefs into the files.
I filed https://bugs.webkit.org/show_bug.cgi?id=218001 <https://bugs.webkit.org/show_bug.cgi?id=218001> for this and will get that done regardless.
Some of these also make it possible to build without WebKitAdditions. We may need some different approach to that. — Darin
On Oct 17, 2020, at 10:00 AM, Sam Weinig <weinig@webkit.org> wrote:
Hi webkit-dev,
I’d like to propose, and gauge feedback on, reducing (with the goal of ultimately removing) the use of Makefile based DerivedSources.make.
My understanding is that currently only the Xcode based ports still use DerivedSources.make, as all the CMake based ones have moved derived source generation to within CMake, so that should limit the scope of who this might affect.
Why do we use Makefiles today?
While I can’t recall the initial reasons for using Makefiles for derived sources, over the years there have been a number of advantages to it that I do know of. One clear advantage, that is no longer applicable, was code sharing, as earlier in the project, at least the Apple Windows port did utilize these Makefiles. Another was to work around some limitations in what dependencies Xcode was able to track with build rules. It seems at least some of the problems with build rules are no longer an issue, as we can now specify inputs to build rules, but I don’t if other problems will still be there, but for some prototyping I did, nothing yet has come up.
I think I might be responsible for this, and I think the reason was that at the time, Xcode could not properly handle derived source dependencies, and ended up either gratuitously rebuilding, or miscomputing by failing to regenerate a source, or failing to rebuild it when needed. It’s possible, maybe even likely Xcode handling of derived sources has improved since then. But the last attempt to do this via Xcode caused subtle broken build issues and/ir gratuitous rebuilding, so we’d have to validate that it doesn’t have these problems any more. DerivedSources.make is also more human-editable without having to do so via the Xcode GUI.
What would we move to?
As this only affects the Xcode based ports, we would move to distinct script phases and build rules in the Xcode project.
Why make this change? What’s the benefit?
There are few reasons to consider this. One advantage is simplifying the build system. Rather than two dependency systems (one for Xcode, one for the Makefile) we reduce it down to one. And with additional knowledge of the stages and dependencies, Xcode could potentially parallelize more phases. We would would also save some time by avoiding invoking make in the first place.
We also have a bit of an issue with make itself, as due to system requirements, we are forever stuck with Make 3.81, which is coming up on being 15 years old. More than once in the last year I have tried to troubleshoot makefile issues, looking for resources on the web, only to be stymied because the solutions I found required newer make.
One question in my mind is whether this would potentially lead to faster builds. If so, and the reliability problems from older Xcode’s are gone, then this would probably be a worthwhile change. But see below...
What are the downsides?
One potential downside will be that it will be a bit harder for those without Xcode to add new types of derived sources. I am not sure how much a real problem in practice this will be, as editing project.pbxproj files is already required for just adding new files, but I want to call it out anyway.
What are your thoughts on this? Are there additional reasons we might want to stick with or move away from Makefile based derived sources?
One additional though, I think we hope to move the Apple-maintained ports to CMake, so the value of changes to the Xcode-based build system may be limited. We think this could speed up both incremental and full builds significantly. - Maciej
Thanks, -Sam
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
participants (9)
-
Andy Estes
-
Darin Adler
-
Fujii Hironori
-
Keith Rollin
-
Maciej Stachowiak
-
Ryosuke Niwa
-
Sam Weinig
-
Sam Weinig
-
Sam Weinig