From ticaiolima at gmail.com Mon Jul 1 09:19:58 2019 From: ticaiolima at gmail.com (Caio Lima) Date: Mon, 1 Jul 2019 13:19:58 -0300 Subject: [webkit-dev] Work on new JS language features Message-ID: Hi WebKittens, As some of you may know, my colleagues from Igalia have been working to implement new {Java|ECMA}Script language features in JavaScriptCore, including BigInt and Class Fields. We have a number of Class Fields patches awaiting review: - Instance Class Fields (https://bugs.webkit.org/show_bug.cgi?id=174212) - Private Methods (https://bugs.webkit.org/show_bug.cgi?id=194434) - Private Accessors (https://bugs.webkit.org/show_bug.cgi?id=194435) - Static Class Fields (https://bugs.webkit.org/show_bug.cgi?id=194095) And one BigInt patch: - BigInt operator "<<" in DFG (https://bugs.webkit.org/show_bug.cgi?id=192664) BR, Caio. From ticaiolima at gmail.com Tue Jul 9 09:42:18 2019 From: ticaiolima at gmail.com (Caio Lima) Date: Tue, 9 Jul 2019 13:42:18 -0300 Subject: [webkit-dev] Work on new JS language features In-Reply-To: References: Message-ID: Up on this thread. Em seg, 1 de jul de 2019 às 13:19, Caio Lima escreveu: > > Hi WebKittens, > > As some of you may know, my colleagues from Igalia have been working > to implement new {Java|ECMA}Script language features in > JavaScriptCore, including BigInt and Class Fields. > > We have a number of Class Fields patches awaiting review: > > - Instance Class Fields (https://bugs.webkit.org/show_bug.cgi?id=174212) > - Private Methods (https://bugs.webkit.org/show_bug.cgi?id=194434) > - Private Accessors (https://bugs.webkit.org/show_bug.cgi?id=194435) > - Static Class Fields (https://bugs.webkit.org/show_bug.cgi?id=194095) > > And one BigInt patch: > - BigInt operator "<<" in DFG > (https://bugs.webkit.org/show_bug.cgi?id=192664) > > BR, > Caio. From aakash_jain at apple.com Thu Jul 11 07:18:59 2019 From: aakash_jain at apple.com (Aakash Jain) Date: Thu, 11 Jul 2019 10:18:59 -0400 Subject: [webkit-dev] Recent EWS improvements In-Reply-To: References: <77F145E2-6ED8-4EDA-B76E-38FB74B1F225@apple.com>

Message-ID: Hi Everyone, I want to update everyone with the further improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly). New Features: - Launched 'Security EWS' - Added iOS-12 Builder queue on new EWS (moved from old to new EWS) - Added WPE and GTK queues on new EWS (moved from old to new EWS) - New EWS can now process large patches (larger than 640kb) https://webkit.org/b/198851 Infrastructure Improvements: - EWS now automatically retries builds in case of certain infrastructure issues and ToT compile failures https://bugs.webkit.org/show_bug.cgi?id=199025 - Improved stability by adding KillOldProcesses step before API and Layout tests https://webkit.org/b/199592 - Shared bots across queues to improve efficiency https://webkit.org/b/198370 - Added check for duplicate workers in config.json https://webkit.org/b/199240 - Patch link now opens the pretty-patch url https://webkit.org/b/199031 - Triggered builds should use same revision as parent build (fixes a corner case) https://webkit.org/b/198289 - Display pre-existing API test failures in build summary to help bot-watchers https://webkit.org/b/199525 - Send email notifications for failures (for maintenance/bot-watching) https://webkit.org/b/198919 - Remove unused buildbot tabs for better readability https://webkit.org/b/198108 - Allow skipping uploading built product for few builders https://webkit.org/b/199422 - EWS should provide option to download layout test results zip file https://webkit.org/b/199121 - Retry Layout test in case of failures https://webkit.org/b/199194 - Upload test results after running layout-tests https://webkit.org/b/199120 - Improved error message on performing certain actions (like Rebuild) which requires authentication - Added more unit-tests Bug fixes: - Results are clobbered in UploadTestResults and ExtractTestResults steps in case of multiple layout test runs https://webkit.org/b/199178 - New EWS: api-ios does not differentiate between patch compile failure and ToT compile failure https://webkit.org/b/197850 - Buildbot worker not pinged https://webkit.org/b/199438 - Make PrintConfiguration platform aware https://webkit.org/b/196657 - Status bubble should not turn orange when any build step is skipped https://webkit.org/b/199079 - Do not print worker environment variables in each build step https://webkit.org/b/197319 - Do not run unix commands for windows in PrintConfiguration https://webkit.org/b/199605 Thanks Aakash > On May 22, 2019, at 7:36 PM, Aakash Jain wrote: > > Hi Everyone, > > I just wanted to update everyone with the recent improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly). > > New Features: > EWS status-bubble now display position in queue while patch is waiting to be processed > Added webkitpy and bindings-tests EWS (moved from old to new EWS) > Status bubbles for webkitpy and bindings-tests EWS now display the exact test failures in hover-over message (https://webkit.org/b/197395 , https://webkit.org/b/197423 ) > Added support for 'new EWS' in webkit-patch tool > Added 'EWS Build Archives' (similar to 'WebKit Build Archives' https://webkit.org/blog/7978/introducing-webkit-build-archives/ ). For every patch uploaded to Bugzilla, EWS builders build the patch for various platforms (currently macOS and iOS) and upload the archives to S3. These archives are available to download by anyone (for 14 days). The S3 URL is in corresponding build (e.g.: notice 'uploaded archive' link in https://ews-build.webkit.org/#/builders/7/builds/2477 ). So, if for any reason, you want to get a built archive for your patch, you can simply upload the patch to Bugzilla. (Note that if there is interest in this, we can enhance it further) > > Infrastructure Improvements: > Flakiness in API tests has been reduced (thanks to many WebKit developers) > Infrastructure improvements to prevent build failure due to "worker not pinged" (e.g.: https://ews-build.webkit.org/#/builders/9/builds/332 ) > New EWS polls bugzilla more frequently https://webkit.org/b/197138 > Configured DEBUG mode appropriately for Production and Development env https://webkit.org/b/197700 > Ensured that Buildbot worker logs are not lost on restarting worker > Do not run clean build by default on EWS builders (to improve efficiency) https://webkit.org/b/196897 > build.webkit.org and ews-build.webkit.org starting sharing code (although very little as of now, however the plan is to share more code) > Added migrations file to repository https://webkit.org/b/197729 > Added EWS bots information to Internal scripts to easily monitor bots > Added more unit-tests > > Bug fixes: > Clicking 'submit to new ews' doesn't reload status-bubble https://webkit.org/b/196675 > Clicking on white bubble navigates to page with only bubbles https://webkit.org/b/197520 > Submit to EWS buttons are not aligned properly with status-bubbles https://webkit.org/b/197139 > Status bubble should turn orange when any build step fails https://webkit.org/b/197812 > Handle bug titles with unicode characters https://webkit.org/b/196802 > Scripts using Buildbot API have CORS error https://webkit.org/b/196709 > PrintConfiguration should display Xcode version instead of SDKVersion https://webkit.org/b/196780 > Trigger queues only after uploading the archive https://webkit.org/b/197180 > Do not upload archive when Compile Fails https://webkit.org/b/196674 > Exception while loading status-bubble when no build step has started https://webkit.org/b/196676 > Use singular verb in failure description in case of single api test failure https://webkit.org/b/197013 > EWS should clearly indicate flaky test failures https://webkit.org/b/196947 > Use explicit imports instead of wildcard imports https://webkit.org/b/197194 > New EWS: patches on recently added queues listed as #1 for older bugs https://webkit.org/b/197496 > Improved summary text for various build steps > > Interesting info: Since last month, 'EWS for API tests' prevented API test breakage on 50+ patches (https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains=new%20API%20Test&property=bug_id&order=-number ). > > Thanks > Aakash > >> On Apr 4, 2019, at 10:00 PM, Aakash Jain > wrote: >> >> Introducing brand new EWS >> >> with EWS for API Tests (for macOS and iOS) >> >> and EWS for WebKitPerl Tests! >> >> >> Starting today, when you upload a patch to bugs.webkit.org , you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag). >> >> The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS. >> >> >> Why new EWS? >> The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware. >> >> The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware. >> >> The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org ). >> >> >> How can you contribute: >> If you are interested in contributing, the source code is located at: >> ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build >> ews-app (web-app): Tools/BuildSlaveSupport/ews-app >> >> Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSystem >> >> >> Upcoming features: >> - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 >> - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 >> - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 >> - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605 >> >> >> If you notice any issue, please feel free to file bugs (and assign to me). >> >> Thanks & Regards >> Aakash Jain > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcatanzaro at igalia.com Fri Jul 12 07:37:58 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Fri, 12 Jul 2019 09:37:58 -0500 Subject: [webkit-dev] What's the current Safari UA? Message-ID: <1562942278.127379.0@igalia.com> Hi, Relevant to [1], could someone please check what Safari's current UA string is, please? This is important to help me make sure WebKitGTK fakes the Safari user agent as well as possible. In particular, I'd like to know the version numbers used in the UA. I had thought we were frozen on: Version/11.0 Safari/605.1.15 forever, but perhaps the Version/ component is not frozen and we're up to Version/12.0 or Version/12.1 now? I hope the 605.1.15, at least, is still frozen. Also, for those naughty websites that require we use some serious fakery and pretend to be macOS, we currently use the string "Macintosh; Intel Mac OS X 10_13_4". I understand that Apple's previous attempt to freeze that value was not successful, so we still need to update this from time to time. What does it look like on, say, Mojave or Catalina? Thanks, Michael [1] https://bugs.webkit.org/show_bug.cgi?id=199745 From mcatanzaro at igalia.com Fri Jul 12 12:04:50 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Fri, 12 Jul 2019 14:04:50 -0500 Subject: [webkit-dev] What's the current Safari UA? In-Reply-To: <1562942278.127379.0@igalia.com> References: <1562942278.127379.0@igalia.com> Message-ID: <1562958290.109292.1@igalia.com> I received a good answer and will upgrade our versions accordingly. Thanks. From jbedard at apple.com Fri Jul 12 12:18:50 2019 From: jbedard at apple.com (Jonathan Bedard) Date: Fri, 12 Jul 2019 12:18:50 -0700 Subject: [webkit-dev] Moving to Python 3 Message-ID: Hello WebKit developers, Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3. I propose that, over the next 9 months, we do the following: 1. Make any no-cost Python 3 compatibility changes, in particular - print foo -> print(foo) - import .foo -> import webkitpy.foo 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results) 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like: - dict.iteritems() -> dict.items() - dict.items() -> list(dict.items()) 4. Install Python 3 on macOS Sierra and Mojave bots 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch) 6. Convert testing scripts and webkitpy to Python 3 in a single change The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems(): ... In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language. There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy_horton at apple.com Fri Jul 12 12:37:43 2019 From: timothy_horton at apple.com (Tim Horton) Date: Fri, 12 Jul 2019 12:37:43 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: Message-ID: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard wrote: > > Hello WebKit developers, > > Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3. > > I propose that, over the next 9 months, we do the following: > > 1. Make any no-cost Python 3 compatibility changes, in particular > - print foo -> print(foo) > - import .foo -> import webkitpy.foo > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results) > 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like: > - dict.iteritems() -> dict.items() > - dict.items() -> list(dict.items()) > 4. Install Python 3 on macOS Sierra and Mojave bots > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch) > 6. Convert testing scripts and webkitpy to Python 3 in a single change > > The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: > > for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems(): > ... > > In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language. The official Python 3 transition documentation has a fairly elegant solution to this, actually?? https://legacy.python.org/dev/peps/pep-0469/ See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me. > There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3. > > Jonathan > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertma at chromium.org Fri Jul 12 12:45:54 2019 From: robertma at chromium.org (Robert Ma) Date: Fri, 12 Jul 2019 15:45:54 -0400 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> References: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> Message-ID: Any thoughts on bytes and Unicode strings, especially the string literals in the code base? On Fri, Jul 12, 2019 at 3:38 PM Tim Horton wrote: > > > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard wrote: > > Hello WebKit developers, > > Now that the Catalina developer seeds are available, it is official that > the new Mac developer tools come with Python 3. As a result, we need to > continue the ongoing discussion about migrating our Python 2.7 scripts to > Python 3. > > I propose that, over the next 9 months, we do the following: > > 1. Make any no-cost Python 3 compatibility changes, in particular > - print foo -> print(foo) > - import .foo -> import webkitpy.foo > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts > like bisect-builds, block-spammers, compare-results) > 3. Make most Python 3 compatibility changes which sacrifice efficiency, > subject to a case-by-case audit. These would be things like: > - dict.iteritems() -> dict.items() > - dict.items() -> list(dict.items()) > 4. Install Python 3 on macOS Sierra and Mojave bots > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like > clean-webkit, merge-results-json, webkit-patch) > 6. Convert testing scripts and webkitpy to Python 3 in a single change > > The trouble I foresee us encountering with any scheme which attempts a > conversion which retains both Python 2.7 and Python 3 compatibility is code > like this: > > for expectation_string, expectation_enum in > test_expectations.TestExpectations.EXPECTATIONS.iteritems(): > ... > > In this code, the EXPECTATIONS dictionary is thousands of elements long. > In Python 2.7, iteritems() gives us an iterator instead of creating a new > list, like items() would. In Python 3, iteritems() doesn’t exist, but > items() does, and now gives us an iterator instead of creating a new list. > The trouble here is that, in this case, creating a new list will be very > expensive, expensive enough that we might manage to impact the testing run. > There isn’t really an elegant way around this problem if we want to support > both Python 2.7 and Python 3, other than defining different code paths for > each language. > > > The official Python 3 transition documentation has a fairly elegant > solution to this, actually?? > > https://legacy.python.org/dev/peps/pep-0469/ > > See "Migrating to the common subset of Python 2 and 3” — you define > different iteritems() helpers in the two cases. Seems pretty reasonable to > me. > > There are other small gotchas as well. For example, ‘%’ is no longer a > protected character, which can actually change the behavior of regexes. > That’s why I think it’s better to just try and directly convert things > instead of attempting to be compatible with both Python 2.7 and Python 3. > > Jonathan > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Don.Olmstead at sony.com Fri Jul 12 12:46:46 2019 From: Don.Olmstead at sony.com (Don.Olmstead at sony.com) Date: Fri, 12 Jul 2019 19:46:46 +0000 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> References: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> Message-ID: <92CA7C118E7E1740A32EB4E6361DEFB2AE4DF200@USCULXMSG15.am.sony.com> Mentioned this on the other thread but here’s it again. https://python-modernize.readthedocs.io/en/latest/ and http://python-future.org/automatic_conversion.html might be of use considering the sheer amount of code. They’re both mentioned in https://docs.python.org/3/howto/pyporting.html for migration. You can run individual rules one by one over the code. That might be ok rather than running all the rules at once. Especially for review. Modernize seemed to work pretty well but it wasn’t smart enough to detect shebang lines. From: webkit-dev On Behalf Of Tim Horton Sent: Friday, July 12, 2019 12:38 PM To: Jonathan Bedard Cc: webkit-dev at lists.webkit.org Subject: Re: [webkit-dev] Moving to Python 3 On Jul 12, 2019, at 12:18 PM, Jonathan Bedard > wrote: Hello WebKit developers, Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3. I propose that, over the next 9 months, we do the following: 1. Make any no-cost Python 3 compatibility changes, in particular - print foo -> print(foo) - import .foo -> import webkitpy.foo 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results) 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like: - dict.iteritems() -> dict.items() - dict.items() -> list(dict.items()) 4. Install Python 3 on macOS Sierra and Mojave bots 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch) 6. Convert testing scripts and webkitpy to Python 3 in a single change The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems(): ... In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language. The official Python 3 transition documentation has a fairly elegant solution to this, actually?? https://legacy.python.org/dev/peps/pep-0469/ See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me. There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3. Jonathan _______________________________________________ webkit-dev mailing list webkit-dev at lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcatanzaro at igalia.com Fri Jul 12 12:49:22 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Fri, 12 Jul 2019 14:49:22 -0500 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: Message-ID: <1562960962.109292.2@igalia.com> On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard wrote: > The trouble I foresee us encountering with any scheme which attempts > a conversion which retains both Python 2.7 and Python 3 compatibility > is code like this: Is python2 support required for a well-motivated transitional purpose? I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already. Then million-dollar question is: what shebangs will we use for our scripts? Will #!/usr/bin/env python3 work for Apple? Michael From jbedard at apple.com Fri Jul 12 12:55:46 2019 From: jbedard at apple.com (Jonathan Bedard) Date: Fri, 12 Jul 2019 12:55:46 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> Message-ID: <21D61B81-808F-4899-8D93-5E91DCA87CDB@apple.com> > On Jul 12, 2019, at 12:45 PM, Robert Ma wrote: > > Any thoughts on bytes and Unicode strings, especially the string literals in the code base? My experience with this has been you mostly have to pay attention to where your code interfaces with other processes. In webkitpy, I suspect that most of the our work here will probably be where we are calling other commands or reading files. Because we call most commands through either the Executive class or the FileSystem class, I think there is a good chance we can make changes here that effect most of the codebase. Jonathan > > On Fri, Jul 12, 2019 at 3:38 PM Tim Horton > wrote: > > >> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard > wrote: >> >> Hello WebKit developers, >> >> Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3. >> >> I propose that, over the next 9 months, we do the following: >> >> 1. Make any no-cost Python 3 compatibility changes, in particular >> - print foo -> print(foo) >> - import .foo -> import webkitpy.foo >> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results) >> 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like: >> - dict.iteritems() -> dict.items() >> - dict.items() -> list(dict.items()) >> 4. Install Python 3 on macOS Sierra and Mojave bots >> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch) >> 6. Convert testing scripts and webkitpy to Python 3 in a single change >> >> The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: >> >> for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems(): >> ... >> >> In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language. > > The official Python 3 transition documentation has a fairly elegant solution to this, actually?? > > https://legacy.python.org/dev/peps/pep-0469/ > > See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me. > >> There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3. >> >> Jonathan >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev at lists.webkit.org >> https://lists.webkit.org/mailman/listinfo/webkit-dev > > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From aperez at igalia.com Fri Jul 12 12:58:02 2019 From: aperez at igalia.com (Adrian Perez de Castro) Date: Fri, 12 Jul 2019 22:58:02 +0300 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> References: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> Message-ID: <20190712225802.GB8153@momiji> Hello, On Fri, 12 Jul 2019 12:37:43 -0700, Tim Horton wrote: > > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard wrote: > > > > Hello WebKit developers, > > > > Now that the Catalina developer seeds are available, it is official that > > the new Mac developer tools come with Python 3. As a result, we need to > > continue the ongoing discussion about migrating our Python 2.7 scripts to > > Python 3. Given that GNU/Linux distributions have started already a while ago switching it is great that MacOS is now following suit as well \o/ We have a bug already for this: https://bugs.webkit.org/show_bug.cgi?id=184986 If one has to make code compatible with both Python 2.7 and 3.x, my advice would be to use the Six module, which provides ready-made helpers which behave consistently across both versions: https://six.readthedocs.io/ > > I propose that, over the next 9 months, we do the following: > > > > 1. Make any no-cost Python 3 compatibility changes, in particular > > - print foo -> print(foo) > > - import .foo -> import webkitpy.foo > > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results) > > 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like: > > - dict.iteritems() -> dict.items() > > - dict.items() -> list(dict.items()) > > 4. Install Python 3 on macOS Sierra and Mojave bots > > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch) > > 6. Convert testing scripts and webkitpy to Python 3 in a single change > > > > The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: > > > > for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems(): > > ... > > > > In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language. > > The official Python 3 transition documentation has a fairly elegant solution to this, actually?? > > https://legacy.python.org/dev/peps/pep-0469/ > > See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me. Instead of rolling our own, I would rather use Six (mentioned above). It covers all the differences with the different “.iter*()” methods: https://six.readthedocs.io/#six.iterkeys https://six.readthedocs.io/#six.itervalues https://six.readthedocs.io/#six.iteritems https://six.readthedocs.io/#six.iterlists ... > > There are other small gotchas as well. For example, ‘%’ is no longer a > > protected character, which can actually change the behavior of regexes. > > That’s why I think it’s better to just try and directly convert things > > instead of attempting to be compatible with both Python 2.7 and Python 3. In my experience some of the major troubles making a codebase compatible with both Pythons are the string types: - In Python 2, “str” is also usable for binary data (basically it's backed by a char[], while “unicode” is actual text which may contain any code point. - In Python 3, “string” is kind of the old “unicode” (textual data), and “bytes” is kind of “string” but only holds binary data and in general cannot be used where a string would have been used. The Six module helps a bit with the text types (see “six.text_type” and “six.binary_type”. Regards, —Adrián -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available URL: From jbedard at apple.com Fri Jul 12 13:04:40 2019 From: jbedard at apple.com (Jonathan Bedard) Date: Fri, 12 Jul 2019 13:04:40 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <1562960962.109292.2@igalia.com> References: <1562960962.109292.2@igalia.com> Message-ID: > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro wrote: > > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard wrote: >> The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: > > Is python2 support required for a well-motivated transitional purpose? > > I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already. We still need to figure out whether (and for how long) we intend to retain Python 2 support. Over the last few months, opinions on this have changed quite a bit, so I’m trying to determine what our path forward is going to be. In my opinion, a few months after Catalina GMs, we no longer need to maintain Python 2 support, assuming that we provide adequate automation for installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about shebangs. > > Then million-dollar question is: what shebangs will we use for our scripts? Will #!/usr/bin/env python3 work for Apple? If we move straight to Python 3, we would need to use the Python 3 shebang. Catalina has both Python 2.7 (name ‘python’) and Python 3 (named ‘python3’). I think that this is what we would want to do because it allows us to pretty explicitly convert scripts one at a time. > > Michael > > From jbedard at apple.com Fri Jul 12 13:09:24 2019 From: jbedard at apple.com (Jonathan Bedard) Date: Fri, 12 Jul 2019 13:09:24 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <7EC7E686-FBEF-4049-9ED2-DCD55C67BADF@apple.com> Message-ID: <0DCFF00D-11D8-43AB-9E2D-34ED31678275@apple.com> > On Jul 12, 2019, at 1:07 PM, Keith Rollin wrote: > >> On Jul 12, 2019, at 13:37, Tim Horton wrote: >> >> See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me. > > I did something like this in webkit-triage. Search for “viewitems” for the compatibility function I used, as well as for “PY3” to see some of the other thunks I put into place. > > — Keith > Actually, hadn’t seen those tricks yet, thanks Tim and Keith for calling them out! Those would allow us to keep scripts both Python 2.7 and Python 3 compatible for longer. Jonathan From rniwa at webkit.org Fri Jul 12 15:22:34 2019 From: rniwa at webkit.org (Ryosuke Niwa) Date: Fri, 12 Jul 2019 15:22:34 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <1562960962.109292.2@igalia.com> Message-ID: On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard wrote: > > > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro > wrote: > > > > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard > wrote: > >> The trouble I foresee us encountering with any scheme which attempts a > conversion which retains both Python 2.7 and Python 3 compatibility is code > like this: > > > > Is python2 support required for a well-motivated transitional purpose? > > > > I had previously proposed making all our scripts work with both python2 > and python3 only because I thought Apple was going to require python2 > indefinitely. Now that you're interested in this transition, there's > probably no need to continue python2 support. Anyone building WebKit on > older versions of macOS can reasonably be expected to manually install > python3, right? And it's clear that you're prepared to do this for > infrastructure/bots already. > > We still need to figure out whether (and for how long) we intend to retain > Python 2 support. Over the last few months, opinions on this have changed > quite a bit, so I’m trying to determine what our path forward is going to > be. > > In my opinion, a few months after Catalina GMs, we no longer need to > maintain Python 2 support, assuming that we provide adequate automation for > installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are > explicit about shebangs. > I don't think it's acceptable to require installation of Python 3 just to build & run tests on WebKit unless the installation itself is automated and compartmentalized to WebKit's development given how bad Python is with respect to having multiple versions of Python's and managing packages between them. I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble. - R. Niwa -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcatanzaro at igalia.com Fri Jul 12 19:02:45 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Fri, 12 Jul 2019 21:02:45 -0500 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <1562960962.109292.2@igalia.com> Message-ID: <1562983365.74108.1@igalia.com> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa wrote: > I frequently do WebKit development in older versions of macOS to > diagnose old OS specific regressions, and having to install Python 3 > each time I install an old OS is too much of a trouble. I understand it would be a hassle. :/ But please consider it relative to the additional difficulty of rewriting all of webkitpy to support multiple versions of python at the same time, or setting up a wrapper layer of bash scripts around all of our python scripts to enter a virtualenv before executing the real script. Michael From rniwa at webkit.org Fri Jul 12 19:34:13 2019 From: rniwa at webkit.org (Ryosuke Niwa) Date: Fri, 12 Jul 2019 19:34:13 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <1562983365.74108.1@igalia.com> References: <1562960962.109292.2@igalia.com> <1562983365.74108.1@igalia.com> Message-ID: On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro wrote: > On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa wrote: > > I frequently do WebKit development in older versions of macOS to > > diagnose old OS specific regressions, and having to install Python 3 > > each time I install an old OS is too much of a trouble. > > I understand it would be a hassle. :/ But please consider it relative > to the additional difficulty of rewriting all of webkitpy to support > multiple versions of python at the same time, or setting up a wrapper > layer of bash scripts around all of our python scripts to enter a > virtualenv before executing the real script. > Yeah, and it sounds strictly better that the trouble is handled by people who maintain Python code who presumably more about Python than a random WebKit contributor who may not know how to setup virtual environment in Python, etc... Again, the argument that the difficulty can be overcome and it's a minor inconvenience isn't convincing. I can, for example, suggest that we should just build WebKit using the latest version of clang. Anyone who uses a system that doesn't come with the latest release of clang should just build clang instead. There is a significant cost in having to support MSVC and GC++ so we should just use clang everywhere and only the latest version like Chromium does. Each team & person has a different preference & perspective when it comes to tooling. Please don't break someone else's working workflow based on your preference. - R. Niwa -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjs at apple.com Fri Jul 12 23:20:34 2019 From: mjs at apple.com (Maciej Stachowiak) Date: Fri, 12 Jul 2019 23:20:34 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: Message-ID: <1428CA02-215C-4B2B-9EFD-34F8D8665099@apple.com> > On Jul 12, 2019, at 3:23 PM, Ryosuke Niwa wrote: > > >> On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard wrote: > >> >> > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro wrote: >> > >> > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard wrote: >> >> The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this: >> > >> > Is python2 support required for a well-motivated transitional purpose? >> > >> > I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already. >> >> We still need to figure out whether (and for how long) we intend to retain Python 2 support. Over the last few months, opinions on this have changed quite a bit, so I’m trying to determine what our path forward is going to be. >> >> In my opinion, a few months after Catalina GMs, we no longer need to maintain Python 2 support, assuming that we provide adequate automation for installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about shebangs. > > I don't think it's acceptable to require installation of Python 3 just to build & run tests on WebKit unless the installation itself is automated and compartmentalized to WebKit's development given how bad Python is with respect to having multiple versions of Python's and managing packages between them. > > I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble. It’s possible install a python without installing it on the system, and install modules and any needed additions, using virtualenv: https://virtualenv.pypa.io/en/stable/ This is the pro way to use python without caring about what happens to be on the system. I suggest we proceed by gradually converting our scripts to use a Python 3 virtualenv. This will also spare us the need to install Python modules onto the system. The tricky part will be that webkitpy would have to work both ways until conversion is complete (or at the extreme I guess we could fork it). > - R. Niwa > > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbedard at apple.com Fri Jul 12 23:21:58 2019 From: jbedard at apple.com (Jonathan Bedard) Date: Fri, 12 Jul 2019 23:21:58 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: Message-ID: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com> I would agree that if we move to Python 3, we need a script which installs Python 3 in an impossible to mess-up way on Mojave and High Sierra. I don’t think the clang comparison is fair here. Python 2 is officially deprecated in 2020, we can’t expect security updates to the language or any libraries we depend on 6 months from now. It’s not really a question of if we stop supporting Python 2, but rather, when we stop supporting Python 2. It’s also worth noting that in Catalina, we will need some script to install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer included by default in the new Xcode. Given this, it doesn’t seem terrible if the script responsible for installing XCode CL tools also installs Python 3 for older Mac versions. Jonathan > On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa wrote: > > > >> On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro wrote: >> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa wrote: >> > I frequently do WebKit development in older versions of macOS to >> > diagnose old OS specific regressions, and having to install Python 3 >> > each time I install an old OS is too much of a trouble. >> >> I understand it would be a hassle. :/ But please consider it relative >> to the additional difficulty of rewriting all of webkitpy to support >> multiple versions of python at the same time, or setting up a wrapper >> layer of bash scripts around all of our python scripts to enter a >> virtualenv before executing the real script. > > Yeah, and it sounds strictly better that the trouble is handled by people who maintain Python code who presumably more about Python than a random WebKit contributor who may not know how to setup virtual environment in Python, etc... > > Again, the argument that the difficulty can be overcome and it's a minor inconvenience isn't convincing. I can, for example, suggest that we should just build WebKit using the latest version of clang. Anyone who uses a system that doesn't come with the latest release of clang should just build clang instead. There is a significant cost in having to support MSVC and GC++ so we should just use clang everywhere and only the latest version like Chromium does. > > Each team & person has a different preference & perspective when it comes to tooling. Please don't break someone else's working workflow based on your preference. > > - R. Niwa > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rniwa at webkit.org Sat Jul 13 15:43:40 2019 From: rniwa at webkit.org (Ryosuke Niwa) Date: Sat, 13 Jul 2019 15:43:40 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com> References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com> Message-ID: On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard wrote: > I would agree that if we move to Python 3, we need a script which installs > Python 3 in an impossible to mess-up way on Mojave and High Sierra. > > I don’t think the clang comparison is fair here. Python 2 is officially > deprecated in 2020, we can’t expect security updates to the language or any > libraries we depend on 6 months from now. It’s not really a question of if > we stop supporting Python 2, but rather, when we stop supporting Python 2. > I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system. > It’s also worth noting that in Catalina, we will need some script to > install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no > longer included by default in the new Xcode. Given this, it doesn’t seem > terrible if the script responsible for installing XCode CL tools also > installs Python 3 for older Mac versions. > Yes, as long as it doesn’t replace or break existing Python2.7 and/or other python scripts in the system. > On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa wrote: > > > > > On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro > wrote: > >> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa wrote: >> > I frequently do WebKit development in older versions of macOS to >> > diagnose old OS specific regressions, and having to install Python 3 >> > each time I install an old OS is too much of a trouble. >> >> I understand it would be a hassle. :/ But please consider it relative >> to the additional difficulty of rewriting all of webkitpy to support >> multiple versions of python at the same time, or setting up a wrapper >> layer of bash scripts around all of our python scripts to enter a >> virtualenv before executing the real script. >> > > Yeah, and it sounds strictly better that the trouble is handled by people > who maintain Python code who presumably more about Python than a random > WebKit contributor who may not know how to setup virtual environment in > Python, etc... > > Again, the argument that the difficulty can be overcome and it's a minor > inconvenience isn't convincing. I can, for example, suggest that we should > just build WebKit using the latest version of clang. Anyone who uses a > system that doesn't come with the latest release of clang should just build > clang instead. There is a significant cost in having to support MSVC and > GC++ so we should just use clang everywhere and only the latest version > like Chromium does. > > Each team & person has a different preference & perspective when it comes > to tooling. Please don't break someone else's working workflow based on > your preference. > > - R. Niwa > > -- - R. Niwa -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjs at apple.com Sat Jul 13 16:02:27 2019 From: mjs at apple.com (Maciej Stachowiak) Date: Sat, 13 Jul 2019 16:02:27 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: Message-ID: > On Jul 13, 2019, at 3:44 PM, Ryosuke Niwa wrote: > > > > >> On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard wrote: >> I would agree that if we move to Python 3, we need a script which installs Python 3 in an impossible to mess-up way on Mojave and High Sierra. >> >> I don’t think the clang comparison is fair here. Python 2 is officially deprecated in 2020, we can’t expect security updates to the language or any libraries we depend on 6 months from now. It’s not really a question of if we stop supporting Python 2, but rather, when we stop supporting Python 2. > > I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system. > >> >> It’s also worth noting that in Catalina, we will need some script to install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer included by default in the new Xcode. Given this, it doesn’t seem terrible if the script responsible for installing XCode CL tools also installs Python 3 for older Mac versions. > > Yes, as long as it doesn’t replace or break existing Python2.7 and/or other python scripts in the system. This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3. > >> >>>> On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa wrote: >>>> >>> >> >>> >>>> On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro wrote: >>>> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa wrote: >>>> > I frequently do WebKit development in older versions of macOS to >>>> > diagnose old OS specific regressions, and having to install Python 3 >>>> > each time I install an old OS is too much of a trouble. >>>> >>>> I understand it would be a hassle. :/ But please consider it relative >>>> to the additional difficulty of rewriting all of webkitpy to support >>>> multiple versions of python at the same time, or setting up a wrapper >>>> layer of bash scripts around all of our python scripts to enter a >>>> virtualenv before executing the real script. >>> >>> Yeah, and it sounds strictly better that the trouble is handled by people who maintain Python code who presumably more about Python than a random WebKit contributor who may not know how to setup virtual environment in Python, etc... >>> >>> Again, the argument that the difficulty can be overcome and it's a minor inconvenience isn't convincing. I can, for example, suggest that we should just build WebKit using the latest version of clang. Anyone who uses a system that doesn't come with the latest release of clang should just build clang instead. There is a significant cost in having to support MSVC and GC++ so we should just use clang everywhere and only the latest version like Chromium does. >>> >>> Each team & person has a different preference & perspective when it comes to tooling. Please don't break someone else's working workflow based on your preference. >>> >>> - R. Niwa >>> > -- > - R. Niwa > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcatanzaro at igalia.com Sat Jul 13 16:14:39 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Sat, 13 Jul 2019 18:14:39 -0500 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References:

Message-ID: <1563059679.128946.0@igalia.com> On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak wrote: > This is exactly what virtualenvs can do. And this is how we should do > it IMO, even for systems that natively have some version of Python 3. I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code. Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK? Michael From rniwa at webkit.org Sat Jul 13 18:17:38 2019 From: rniwa at webkit.org (Ryosuke Niwa) Date: Sat, 13 Jul 2019 18:17:38 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <1563059679.128946.0@igalia.com> References:

<1563059679.128946.0@igalia.com> Message-ID: On Sat, Jul 13, 2019 at 4:14 PM Michael Catanzaro wrote: > On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak > wrote: > > This is exactly what virtualenvs can do. And this is how we should do > > it IMO, even for systems that natively have some version of Python 3. > > I guess that's fine for everything not required by the CMake build, > e.g. build-webkit and webkitpy. That's probably 99% of our python code. > > Scripts required by the CMake build should use system python3 though, > not the virtualenv. I guess that's 1% or less of our python. OK? That seems okay given macOS / iOS ports don’t use CMake right now. Virtually all ports that use CMake as the build system would have python3 in the base installation or already require installations of a bunch of software (e.g. Windows port), right? - R. Niwa -- - R. Niwa -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjs at apple.com Sat Jul 13 19:26:24 2019 From: mjs at apple.com (Maciej Stachowiak) Date: Sat, 13 Jul 2019 19:26:24 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <1563059679.128946.0@igalia.com> References:

<1563059679.128946.0@igalia.com> Message-ID: <8C645CD2-1954-4C74-9813-68D0378A5A70@apple.com> > On Jul 13, 2019, at 4:14 PM, Michael Catanzaro wrote: > > On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak wrote: >> This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3. > > I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code. > > Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK? Can you clarify why this is needed? From mjs at apple.com Sat Jul 13 19:40:23 2019 From: mjs at apple.com (Maciej Stachowiak) Date: Sat, 13 Jul 2019 19:40:23 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <8C645CD2-1954-4C74-9813-68D0378A5A70@apple.com> References:

<1563059679.128946.0@igalia.com> <8C645CD2-1954-4C74-9813-68D0378A5A70@apple.com> Message-ID: <7D4B864F-2859-4ECA-8A55-48E50E2E94FF@apple.com> > On Jul 13, 2019, at 7:26 PM, Maciej Stachowiak wrote: > > > >> On Jul 13, 2019, at 4:14 PM, Michael Catanzaro wrote: >> >> On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak wrote: >>> This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3. >> >> I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code. >> >> Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK? > > Can you clarify why this is needed? (Also which specific Python scripts are needed by the CMake build?) From pulkomandy at gmail.com Sat Jul 13 22:48:40 2019 From: pulkomandy at gmail.com (Adrien Destugues) Date: Sun, 14 Jul 2019 07:48:40 +0200 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com> Message-ID: <98F929B0-705A-45F4-AD1D-AA537FE1E4B1@gmail.com> >I don’t think anyone is arguing that we’d eventually need to move to >Python3. I’m arguing that it’s not okay to require random WebKit >contributor to know some obscure python insanity to install Python 3, >or >have a script that installs Python 3 and breaks all other python >scripts in >the system. > > Python3 would normally install a "python3" executable that's independant from the "python" one from Python2? It's fairly common to have these side by side precisely because they are not compatible. -- Adrien. From mcatanzaro at igalia.com Sun Jul 14 06:37:26 2019 From: mcatanzaro at igalia.com (Michael Catanzaro) Date: Sun, 14 Jul 2019 08:37:26 -0500 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <8C645CD2-1954-4C74-9813-68D0378A5A70@apple.com> References:

<1563059679.128946.0@igalia.com> <8C645CD2-1954-4C74-9813-68D0378A5A70@apple.com> Message-ID: <1563111446.21674.0@igalia.com> On Sat, Jul 13, 2019 at 9:26 PM, Maciej Stachowiak wrote: > Can you clarify why this is needed? Well it just wouldn't seem very kosher to use a virtualenv for the serious work of performing real distro builds, right? In contrast to developer scripts for developer convenience, where I'd say it's perfectly fine to do whatever we want. But official builds should surely be performed using system dependencies only. Anyway, it doesn't seem like a problem. From searching for 'python' in my build.ninja, I find: JSC: ud_itab.py generateWasmOpsHeader.py generateWasmValidateInlinesHeader.py generateWasmB3IRGeneratorInlinesHeader.py create_regex_tables generateYarrUnicodePropertyTables.py generateIntlCanonicalizeLanguage.py KeywordLookupGenerator.py generate-inspector-protocol-bindings.py generate-js-builtins.py generateYarrCanonicalizeUnicode generate-combined-inspector-json.py jsmin.py cssmin.py make-js-file-arrays.py WebCore: create-html-entity-table create-http-header-name-table makeSelectorPseudoClassAndCompatibilityElementMap.py makeSelectorPseudoElementsMap.py WebKit: generate-message-receiver.py generate-messages-header.py WebInspectorUI: copy-user-interface-resources.pl (perl script that runs some python) Tools: generate-inspector-gresource-manifest.py It's possible I've missed some, but that's probably most of them. Basically all the scripts under Source/ -- the scripts that are really required to build -- and that one platform-specific exception under Tools/, shouldn't require a virtualenv IMO. That doesn't seem too difficult to ensure. We could either have them use the virtualenv only optionally if it's somehow already available (I'm not familiar with how it works) or only if the scripts detect that webkitpy exists, or just make this subset of scripts compatible with both python2 and python3 for the next couple years. In contrast, the vast majority of our python scripts are not required to build. They live under Tools/Scripts, are only used by developers, and are not included in release tarballs at all. That includes all of webkitpy, anything used by build-webkit, anything used by run-webkit-tests, etc. I'd say we can go crazy here. You get a virtualenv, you get a virtualenv, everybody gets a virtualenv! Michael From fujii.hironori at gmail.com Mon Jul 15 23:04:09 2019 From: fujii.hironori at gmail.com (Fujii Hironori) Date: Tue, 16 Jul 2019 15:04:09 +0900 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com> Message-ID: On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa wrote: > > I don’t think anyone is arguing that we’d eventually need to move to > Python3. I’m arguing that it’s not okay to require random WebKit > contributor to know some obscure python insanity to install Python 3, or > have a script that installs Python 3 and breaks all other python scripts in > the system. > > Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guijemont at igalia.com Tue Jul 16 04:28:26 2019 From: guijemont at igalia.com (Guillaume Emont) Date: Tue, 16 Jul 2019 13:28:26 +0200 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com>

Message-ID: <156327650628.13865.13819651565633583352@brendan> Quoting Fujii Hironori (2019-07-16 08:04:09) > > On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa wrote: > > > I don’t think anyone is arguing that we’d eventually need to move to > Python3. I’m arguing that it’s not okay to require random WebKit > contributor to know some obscure python insanity to install Python 3, or > have a script that installs Python 3 and breaks all other python scripts in > the system. > > > > Just out of curiosity. As far as I know, installing Python 3 breaks nothing. > What and why are they got broken? I suspect Ryosuke is talking about a case where python 3 has already been installed on the OS (but is not part of the original OS), and we install python 3 also, and the scripts that were using the first python 3 installed end up using WebKit's python 3 instead, which could lack a python module required by these scripts, hence breaking them. I think this situation should be easy to avoid with a virtualenv though. Best regards, Guillaume From annulen at yandex.ru Tue Jul 16 04:36:37 2019 From: annulen at yandex.ru (Konstantin Tokarev) Date: Tue, 16 Jul 2019 14:36:37 +0300 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <156327650628.13865.13819651565633583352@brendan> References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com>

<156327650628.13865.13819651565633583352@brendan> Message-ID: <36921351563276997@myt2-66bcb87429e6.qloud-c.yandex.net> 16.07.2019, 14:33, "Guillaume Emont" : > Quoting Fujii Hironori (2019-07-16 08:04:09) >> On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa wrote: >> >> I don’t think anyone is arguing that we’d eventually need to move to >> Python3. I’m arguing that it’s not okay to require random WebKit >> contributor to know some obscure python insanity to install Python 3, or >> have a script that installs Python 3 and breaks all other python scripts in >> the system. >> >> Just out of curiosity. As far as I know, installing Python 3 breaks nothing. >> What and why are they got broken? > > I suspect Ryosuke is talking about a case where python 3 has already > been installed on the OS (but is not part of the original OS), and we > install python 3 also, and the scripts that were using the first python > 3 installed end up using WebKit's python 3 instead, which could lack a > python module required by these scripts, hence breaking them. But we have autoinstaller which handles this situation. > > I think this situation should be easy to avoid with a virtualenv though. > > Best regards, > > Guillaume > _______________________________________________ > webkit-dev mailing list > webkit-dev at lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev -- Regards, Konstantin From ap at webkit.org Tue Jul 16 12:46:29 2019 From: ap at webkit.org (Alexey Proskuryakov) Date: Tue, 16 Jul 2019 12:46:29 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com>

Message-ID: <7BF6B0C8-B2FE-4C39-8C8E-27850487CE00@webkit.org> > 15 июля 2019 г., в 23:04, Fujii Hironori написал(а): > > > On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa > wrote: > > I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system. > > > Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken? As always, it will be up to the people doing the work to decide how it's done. The feedback is clear: - They shouldn't make it excessively difficult to do WebKit engineering on older versions of macOS. "Excessively" is not clearly defined, but it seems obvious that there is a tradeoff between tools work difficulty, and difficulty for engineers who go back to older macOS versions (and implicitly, difficulty for people who set up bots, QA engineers, and others involved). I don't think that anyone ever suggested that it will be up to each engineer to figure out the best way to install Python 3. - virtualenv works great for some projects. Definitely worth looking at it as one of the primary paths forward. It's not just Python itself, but we don't want to pollute the system with modules either. Although the latter already has a pretty nice solution in WebKit with autoinstall. - virtualenv shouldn't be required when building WebKit, and dynamic dependencies in general are not OK in this scenario. - Alexey -------------- next part -------------- An HTML attachment was scrubbed... URL: From darin at apple.com Tue Jul 16 15:02:34 2019 From: darin at apple.com (Darin Adler) Date: Tue, 16 Jul 2019 15:02:34 -0700 Subject: [webkit-dev] Moving to Python 3 In-Reply-To: <7BF6B0C8-B2FE-4C39-8C8E-27850487CE00@webkit.org> References: <4915657D-281D-471C-825C-D4AFD597FEFD@apple.com>

<7BF6B0C8-B2FE-4C39-8C8E-27850487CE00@webkit.org> Message-ID: > On Jul 16, 2019, at 12:46 PM, Alexey Proskuryakov wrote: > > - They shouldn't make it excessively difficult to do WebKit engineering on older versions of macOS. > > "Excessively" is not clearly defined, but it seems obvious that there is a tradeoff between tools work difficulty, and difficulty for engineers who go back to older macOS versions (and implicitly, difficulty for people who set up bots, QA engineers, and others involved). > > I don't think that anyone ever suggested that it will be up to each engineer to figure out the best way to install Python 3. Lets keep in mind our strategy to keep development of WebKit on macOS easy. We want to preserve this. The steps (assuming git) are: https://webkit.org/build-tools/ • download and install Xcode from Apple % xcode-select --install https://webkit.org/getting-the-code/ % git clone git://git.webkit.org/WebKit.git WebKit % Tools/Scripts/webkit-patch setup-git-clone https://webkit.org/building-webkit/ % build-webkit --debug https://webkit.org/running-webkit/ % run-safari We’d really like to keep it to a small number of steps. Having to download and install anything else would be a significant additional step. — Darin From g.raju2000 at gmail.com Tue Jul 16 23:22:21 2019 From: g.raju2000 at gmail.com (Rajagopalan Gangadharan) Date: Wed, 17 Jul 2019 11:52:21 +0530 Subject: [webkit-dev] Help regarding coordinated graphics Message-ID: Hello everybody As you guys know we have been working on porting webkit2 to haiku. https://github.com/haiku/webkit/tree/webkit2 So we have a working IPC, network process and minibrowser configured to test the api. Now the most important part is to get it rendering. Currently we would like to use coordinated graphics to acheive the same as it helps us to figure out which would be better for our platform and also to make sure all other things are in place. It would be helpful if you guys take time to clarify my queries :) 1) How does this coordinated graphics model work ( i read the trac still i couldnt get a good hold of it) ->i couldnt get the flow right im just confused 2) Which part of webprocess instructs or stores info about drawing onto to a sharable bitmap. ->where does the info regarding a layer is being parsed and created( I know it is webcore but where is it linked with webprocess) 3) What does a backing store , layer tree do? ->Is like backing store is where the sharable bitmap is created which is being passed onto uiprocess and layer tree stores info of each layer which then is being composited? 4) What are the important places i need to take care(in code) like eg. PageClient,DrawingAreaCoordinatedGraphics etc. ->I actually found my drawing area couldnt start drawing because the drawing area size was empty so I may have missed lot of implementation details it would help if you guys could point me to required places that are required. Thank you for the time :) Regards, G.Rajagopalan -------------- next part -------------- An HTML attachment was scrubbed... URL: From aakash_jain at apple.com Thu Jul 11 07:18:59 2019 From: aakash_jain at apple.com (Aakash Jain) Date: Thu, 11 Jul 2019 10:18:59 -0400 Subject: [webkit-dev] Recent EWS improvements In-Reply-To: References: <77F145E2-6ED8-4EDA-B76E-38FB74B1F225@apple.com>