[Webkit-unassigned] [Bug 260877] New: Cannot run test-webkitpy: errors when installing many pypi dependencies with python 3.12, and various assertion functions removed

Tue Aug 29 14:11:44 PDT 2023

https://bugs.webkit.org/show_bug.cgi?id=260877

            Bug ID: 260877
           Summary: Cannot run test-webkitpy: errors when installing many
                    pypi dependencies with python 3.12, and various
                    assertion functions removed
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: mcatanzaro at redhat.com

I'd like to be able to run test-webkitpy locally using python 3.12, but can't because pylint fails to install:

Downloading pylint-2.6.0...
Installing pylint-2.6.0...
error in pylint setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '>=3.5.*'
Failed to install pylint-2.6.0!

Fortunately, this is already fixed in the latest version of pylint. Then, next problem:

Downloading beautifulsoup4-4.9.3...
Installing beautifulsoup4-4.9.3...
error in beautifulsoup4 setup command: use_2to3 is invalid.
Failed to install beautifulsoup4-4.9.3!

This is also conveniently already fixed in the latest version of beautifulsoup4. Next problem:

Downloading Genshi-0.7.3...
Installing Genshi-0.7.3...
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'features'
  warnings.warn(msg)
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'convert_2to3_doctests'
  warnings.warn(msg)
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'use_2to3_fixers'
  warnings.warn(msg)
error in Genshi setup command: use_2to3 is invalid.

If you guessed that it was already fixed in the latest version of Genshi, you'd unsurprisingly be correct.

Next problem occurs when compiling lupa. There are a bunch of GCC warnings and then many errors. The first error is:

lupa/_lupa.c: In function ‘__Pyx_PyInt_AddObjC’:
lupa/_lupa.c:34443:51: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
34443 |         const digit* digits = ((PyLongObject*)op1)->ob_digit;
      |                                                   ^~

All of the errors are fixed in the latest version, lupa 2.0. However, it's not fixed in lupa 1.14.1. I was a little nervous about the major version increase, so I tried to find any documentation of incompatibilities between 1.x and 2.0, but couldn't immediately find anything. Anyway, it seems to work, so let's assume it's good if EWS is happy.

Next problem is a very long backtrace that ends in:

    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/html5lib/_inputstream.py", line 4, in <module>
      from six.moves import http_client, urllib
      from six.moves import http_client, urllib
  ModuleNotFoundError: No module named 'six.moves'
  ModuleNotFoundError: No module named 'six.moves'

At first I feared this would be a bug that a simple version upgrade would not fix, but nope, just need to upgrade six and the problem goes away.

Next problem, a backtrace that ends in:

  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_legacy_unittest.py", line 31, in <module>
    from pyfakefs.fake_filesystem_unittest import TestCaseMixin
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/pyfakefs/fake_filesystem_unittest.py", line 73, in <module>
    from pyfakefs import fake_pathlib
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/pyfakefs/fake_pathlib.py", line 139, in <module>
    flavour = pathlib._Flavour if pathlib else object
              ^^^^^^^^^^^^^^^^
AttributeError: module 'pathlib' has no attribute '_Flavour

This is fixed by upgrading pyfakefs.

Finally, I am able to run test-webkitpy! 10 tests fail due to some assertion functions that were renamed. They all look like one or the other of these two examples:

[1231/2225] webkitpy.port.xvfbdriver_unittest.XvfbDriverTest.test_xvfb_not_replying erred:                            
  Traceback (most recent call last):
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/port/xvfbdriver_unittest.py", line 99, in test_xvfb_not_replying
      self.assertRaisesRegexp(RuntimeError, 'Unable to start Xvfb display server', driver.start, False, [])
      ^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'XvfbDriverTest' object has no attribute 'assertRaisesRegexp'. Did you mean: 'assertRaisesRegex'?

[1760/2225] webkitpy.w3c.test_converter_unittest.W3CTestConverterTest.test_convert_for_webkit_harness_only erred:
  Traceback (most recent call last):
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/w3c/test_converter_unittest.py", line 111, in test_convert_for_webkit_harness_only
      self.verify_test_harness_paths(converted[2], 1, 1)
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/w3c/test_converter_unittest.py", line 368, in verify_test_harness_paths
      self.assertEquals(len(converted.findAll(src=orig_path_pattern)), num_src_paths, 'testharness src path should not have been converted')
      ^^^^^^^^^^^^^^^^^
  AttributeError: 'W3CTestConverterTest' object has no attribute 'assertEquals'. Did you mean: 'assertEqual'?

According to https://docs.python.org/3.12/whatsnew/3.12.html these were removed from python itself since they were deprecated since python 3.2. Maybe we had deprecation warnings disabled? Easy enough to fix as it only requires renaming things.

Finally, I see some warnings caused by bug #260729, which do not cause any test failures and which we can address separately. I also see one remaining unfixed warning:

[0/2225] reporelaypy.tests.checkout_unittest.CheckoutUnittest.test_constructor_no_sentinal/usr/lib/python3.12/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
  warnings.warn(

But I don't want to investigate this one, because it's not an error and all the tests pass, and I'm tired now. :)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20230829/ec99828e/attachment.htm>