Bug 2175154 - pyflakes fails to build with Python 3.12: SyntaxError: source code string cannot contain null bytes
Summary: pyflakes fails to build with Python 3.12: SyntaxError: source code string can...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: pyflakes
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ali Erdinc Koroglu
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.12
TreeView+ depends on / blocked
 
Reported: 2023-03-03 11:32 UTC by Tomáš Hrnčiar
Modified: 2023-07-14 01:05 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-07-14 01:05:14 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tomáš Hrnčiar 2023-03-03 11:32:07 UTC
pyflakes fails to build with Python 3.12.0a5.

=================================== FAILURES ===================================
_____________________ CheckTests.test_misencodedFileUTF16 ______________________

codeString = b'\xff\xfe#\x00 \x00c\x00o\x00d\x00i\x00n\x00g\x00:\x00 \x00a\x00s\x00c\x00i\x00i\x00\n\x00x\x00 \x00=\x00 \x00"\x00\x03&"\x00\n\x00'
filename = '/tmp/tmpqqdezo1u'
reporter = <pyflakes.reporter.Reporter object at 0x7fb061306e40>

    def check(codeString, filename, reporter=None):
        """
        Check the Python source given by C{codeString} for flakes.
    
        @param codeString: The Python source to check.
        @type codeString: C{str}
    
        @param filename: The name of the file the source came from, used to report
            errors.
        @type filename: C{str}
    
        @param reporter: A L{Reporter} instance, where errors and warnings will be
            reported.
    
        @return: The number of warnings emitted.
        @rtype: C{int}
        """
        if reporter is None:
            reporter = modReporter._makeDefaultReporter()
        # First, compile into an AST and handle syntax errors.
        try:
>           tree = ast.parse(codeString, filename=filename)

pyflakes/api.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = b'\xff\xfe#\x00 \x00c\x00o\x00d\x00i\x00n\x00g\x00:\x00 \x00a\x00s\x00c\x00i\x00i\x00\n\x00x\x00 \x00=\x00 \x00"\x00\x03&"\x00\n\x00'
filename = '/tmp/tmpqqdezo1u', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E       SyntaxError: source code string cannot contain null bytes

/usr/lib64/python3.12/ast.py:51: SyntaxError

During handling of the above exception, another exception occurred:

self = <pyflakes.test.test_api.CheckTests testMethod=test_misencodedFileUTF16>

        def test_misencodedFileUTF16(self):
            """
            If a source file contains bytes which cannot be decoded, this is
            reported on stderr.
            """
            SNOWMAN = chr(0x2603)
            source = ("""\
    # coding: ascii
    x = "%s"
    """ % SNOWMAN).encode('utf-16')
            with self.makeTempFile(source) as sourcePath:
>               self.assertHasErrors(
                    sourcePath, [f"{sourcePath}: problem decoding source\n"])

pyflakes/test/test_api.py:631: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyflakes/test/test_api.py:340: in assertHasErrors
    count = withStderrTo(err, checkPath, path)
pyflakes/test/test_api.py:32: in withStderrTo
    return f(*args, **kwargs)
pyflakes/api.py:72: in checkPath
    return check(codestr, filename, reporter)
pyflakes/api.py:41: in check
    reporter.syntaxError(filename, e.args[0], e.lineno, e.offset, e.text)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyflakes.reporter.Reporter object at 0x7fb061306e40>
filename = '/tmp/tmpqqdezo1u'
msg = 'source code string cannot contain null bytes', lineno = None
offset = None, text = None

    def syntaxError(self, filename, msg, lineno, offset, text):
        """
        There was a syntax error in C{filename}.
    
        @param filename: The path to the file with the syntax error.
        @ptype filename: C{unicode}
        @param msg: An explanation of the syntax error.
        @ptype msg: C{unicode}
        @param lineno: The line number where the syntax error occurred.
        @ptype lineno: C{int}
        @param offset: The column on which the syntax error occurred, or None.
        @ptype offset: C{int}
        @param text: The source code containing the syntax error.
        @ptype text: C{unicode}
        """
        if text is None:
            line = None
        else:
            line = text.splitlines()[-1]
    
        # lineno might be 0 if the error came from stdin
>       lineno = max(lineno, 1)
E       TypeError: '>' not supported between instances of 'int' and 'NoneType'

pyflakes/reporter.py:60: TypeError
_____________ CheckTests.test_nonDefaultFollowsDefaultSyntaxError ______________

self = <pyflakes.test.test_api.CheckTests testMethod=test_nonDefaultFollowsDefaultSyntaxError>

        def test_nonDefaultFollowsDefaultSyntaxError(self):
            """
            Source which has a non-default argument following a default argument
            should include the line number of the syntax error.  However these
            exceptions do not include an offset.
            """
            source = """\
    def foo(bar=baz, bax):
        pass
    """
            with self.makeTempFile(source) as sourcePath:
                if PYPY and sys.version_info >= (3, 9):
                    column = 18
                elif PYPY:
                    column = 8
                elif sys.version_info >= (3, 10):
                    column = 18
                elif sys.version_info >= (3, 9):
                    column = 21
                elif sys.version_info >= (3, 8):
                    column = 9
                else:
                    column = 8
                last_line = ' ' * (column - 1) + '^\n'
                columnstr = '%d:' % column
>               self.assertHasErrors(
                    sourcePath,
                    ["""\
    {}:1:{} non-default argument follows default argument
    def foo(bar=baz, bax):
    {}""".format(sourcePath, columnstr, last_line)])

pyflakes/test/test_api.py:496: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyflakes/test/test_api.py:341: in assertHasErrors
    self.assertEqual(
E   AssertionError: Tuples differ: (1, '[18 chars]:18: parameter without a default follows param[62 chars]^\n') != (1, '[18 chars]:18: non-default argument follows default argu[47 chars]^\n')
E   
E   First differing element 1:
E   '/tmp[14 chars]:18: parameter without a default follows param[61 chars] ^\n'
E   '/tmp[14 chars]:18: non-default argument follows default argu[46 chars] ^\n'
E   
E     (1,
E   +  '/tmp/tmpel20desh:1:18: non-default argument follows default argument\n'
E   -  '/tmp/tmpel20desh:1:18: parameter without a default follows parameter with a '
E   -  'default\n'
E      'def foo(bar=baz, bax):\n'
E      '                 ^\n')
=========================== short test summary info ============================
FAILED pyflakes/test/test_api.py::CheckTests::test_misencodedFileUTF16 - Type...
FAILED pyflakes/test/test_api.py::CheckTests::test_nonDefaultFollowsDefaultSyntaxError
================== 2 failed, 727 passed, 17 skipped in 1.66s ===================

The parser now raises SyntaxError when parsing source code containing null bytes. (Contributed by Pablo Galindo in gh-96670.)
https://github.com/python/cpython/issues/96670
https://docs.python.org/3.12/whatsnew/3.12.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.12/fedora-rawhide-x86_64/05576757-pyflakes/

For all our attempts to build pyflakes with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/package/pyflakes/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Adam Williamson 2023-07-14 01:05:14 UTC
This is already resolved in https://koji.fedoraproject.org/koji/buildinfo?buildID=2214346 .


Note You need to log in before you can comment on or make changes to this bug.