[webkit-dev] Moving to Python 3

[Adrian Perez de Castro]

Hello,

On Fri, 12 Jul 2019 12:37:43 -0700, Tim Horton <timothy_horton at apple.com> wrote:

> > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:
> > 
> > Hello WebKit developers,
> > 
> > Now that the Catalina developer seeds are available, it is official that
> > the new Mac developer tools come with Python 3. As a result, we need to
> > continue the ongoing discussion about migrating our Python 2.7 scripts to
> > Python 3.

Given that GNU/Linux distributions have started already a while ago switching
it is great that MacOS is now following suit as well \o/

We have a bug already for this: https://bugs.webkit.org/show_bug.cgi?id=184986

If one has to make code compatible with both Python 2.7 and 3.x, my advice
would be to use the Six module, which provides ready-made helpers which
behave consistently across both versions: https://six.readthedocs.io/
 
> > I propose that, over the next 9 months, we do the following:
> > 
> > 1. Make any no-cost Python 3 compatibility changes, in particular
> >     - print foo -> print(foo)
> >     - import .foo -> import webkitpy.foo
> > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results)
> > 3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like:
> >     - dict.iteritems() -> dict.items()
> >     - dict.items() -> list(dict.items())
> > 4. Install Python 3 on macOS Sierra and Mojave bots
> > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch)
> > 6. Convert testing scripts and webkitpy to Python 3 in a single change
> > 
> > The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:
> > 
> >     for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> >         ...
> > 
> > In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language.
> 
> The official Python 3 transition documentation has a fairly elegant solution to this, actually??
> 
> https://legacy.python.org/dev/peps/pep-0469/
> 
> See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me.

Instead of rolling our own, I would rather use Six (mentioned above). It
covers all the differences with the different “.iter*()” methods:

  https://six.readthedocs.io/#six.iterkeys
  https://six.readthedocs.io/#six.itervalues
  https://six.readthedocs.io/#six.iteritems
  https://six.readthedocs.io/#six.iterlists
  ...

> > There are other small gotchas as well. For example, ‘%’ is no longer a
> > protected character, which can actually change the behavior of regexes.
> > That’s why I think it’s better to just try and directly convert things
> > instead of attempting to be compatible with both Python 2.7 and Python 3.

In my experience some of the major troubles making a codebase compatible
with both Pythons are the string types:

  - In Python 2, “str” is also usable for binary data (basically it's backed
    by a char[], while “unicode” is actual text which may contain any code
	point.
  - In Python 3, “string” is kind of the old “unicode” (textual data),
    and “bytes” is kind of “string” but only holds binary data and in
	general cannot be used where a string would have been used.

The Six module helps a bit with the text types (see “six.text_type” and
“six.binary_type”.

Regards,
—Adrián
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20190712/3d7a122f/attachment-0004.bin>