Bug 2193391 - python-cffi fails to build with Python 3.12: ValueError: library 'libm.so.6' has already been closed
Summary: python-cffi fails to build with Python 3.12: ValueError: library 'libm.so.6' ...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: python-cffi
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Python Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.12
TreeView+ depends on / blocked
 
Reported: 2023-05-05 13:44 UTC by Tomáš Hrnčiar
Modified: 2023-08-02 12:16 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-02 12:16:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tomáš Hrnčiar 2023-05-05 13:44:15 UTC
python-cffi fails to build with Python 3.12.0a7.

=================================== FAILURES ===================================
_____________________________ TestFFI.test_dlclose _____________________________

self = <cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x7f23a8fc4e60>

>   lambda self: read_variable(BType, name),
    lambda self, value: write_variable(BType, name, value)))
E   ValueError: library 'libm.so.6' has already been closed

cffi/api.py:848: ValueError

The above exception was the direct cause of the following exception:

self = <testing.cffi0.test_ffi_backend.TestFFI object at 0x7f23aacd1760>

    def test_dlclose(self):
        if self.Backend is CTypesBackend:
            pytest.skip("not with the ctypes backend")
        ffi = FFI(backend=self.Backend())
        ffi.cdef("int foobar(void); extern int foobaz;")
        lib = ffi.dlopen(lib_m)
        ffi.dlclose(lib)
        e = pytest.raises(ValueError, getattr, lib, 'foobar')
        assert str(e.value).startswith("library '")
        assert str(e.value).endswith("' has already been closed")
>       e = pytest.raises(ValueError, getattr, lib, 'foobaz')

testing/cffi0/test_function.py:540: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cffi/api.py:913: in __getattr__
    return getattr(self, name)
cffi/api.py:912: in __getattr__
    make_accessor(name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'foobaz'

    def make_accessor(name):
>       with ffi._lock:
E       SystemError: <built-in method __enter__ of _thread.lock object at 0x7f23a83baf80> returned a result with an exception set

cffi/api.py:901: SystemError
___________________ TestThread.test_first_calls_in_parallel ____________________

self = <testing.embedding.test_thread.TestThread object at 0x7f23a8f9f7d0>

    def test_first_calls_in_parallel(self):
        add1_cffi = self.prepare_module('add1')
        self.compile('thread1-test', [add1_cffi], threads=True)
        for i in range(20):
>           output = self.execute('thread1-test')

testing/embedding/test_thread.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testing.embedding.test_thread.TestThread object at 0x7f23a8f9f7d0>
name = 'thread1-test'

    def execute(self, name):
        path = self.get_path()
        print('running %r in %r' % (name, path))
        executable_name = name
        if sys.platform == 'win32':
            executable_name = os.path.join(path, executable_name + '.exe')
        else:
            executable_name = os.path.join('.', executable_name)
        popen = self._run_base([executable_name], cwd=path,
                               stdout=subprocess.PIPE,
                               universal_newlines=True)
        result = popen.stdout.read()
        err = popen.wait()
        if err:
>           raise OSError("%r failed with exit code %r" % (
                os.path.join(path, executable_name), err))
E           OSError: '/tmp/ffi-0/embedding/test_first_calls_in_parallel/./thread1-test' failed with exit code -6

testing/embedding/test_basic.py:175: OSError
----------------------------- Captured stdout call -----------------------------
* setting env var 'LD_LIBRARY_PATH' to '/usr/bin:/tmp/ffi-0/embedding/test_first_calls_in_parallel'
compiling thread1-test with ['/tmp/ffi-0/embedding/test_basic/_add1_cffi.so']
running 'thread1-test' in '/tmp/ffi-0/embedding/test_first_calls_in_parallel'
RUNNING: ['./thread1-test'] {'cwd': '/tmp/ffi-0/embedding/test_first_calls_in_parallel', 'stdout': -1, 'universal_newlines': True}
----------------------------- Captured stderr call -----------------------------
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Python runtime state: initialized

Thread 0x00007f84057fa6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f8407fff6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f840e1616c0 (most recent call first):
  <no Python frame>

Extension modules: _cffi_backend (total: 1)
____________________ TestThread.test_load_in_parallel_more _____________________

self = <testing.embedding.test_thread.TestThread object at 0x7f23a8f9fc20>

    def test_load_in_parallel_more(self):
        add2_cffi = self.prepare_module('add2')
        add3_cffi = self.prepare_module('add3')
        self.compile('thread3-test', [add2_cffi, add3_cffi], threads=True)
        for i in range(150):
>           output = self.execute('thread3-test')

testing/embedding/test_thread.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testing.embedding.test_thread.TestThread object at 0x7f23a8f9fc20>
name = 'thread3-test'

    def execute(self, name):
        path = self.get_path()
        print('running %r in %r' % (name, path))
        executable_name = name
        if sys.platform == 'win32':
            executable_name = os.path.join(path, executable_name + '.exe')
        else:
            executable_name = os.path.join('.', executable_name)
        popen = self._run_base([executable_name], cwd=path,
                               stdout=subprocess.PIPE,
                               universal_newlines=True)
        result = popen.stdout.read()
        err = popen.wait()
        if err:
>           raise OSError("%r failed with exit code %r" % (
                os.path.join(path, executable_name), err))
E           OSError: '/tmp/ffi-0/embedding/test_load_in_parallel_more/./thread3-test' failed with exit code -6

testing/embedding/test_basic.py:175: OSError
----------------------------- Captured stdout call -----------------------------
* setting env var 'LD_LIBRARY_PATH' to '/usr/bin:/tmp/ffi-0/embedding/test_load_in_parallel_more'
RUNNING: ['/usr/bin/python3', '/tmp/ffi-0/embedding/test_load_in_parallel_more/add3.py'] {'cwd': '/tmp/ffi-0/embedding/test_load_in_parallel_more', 'stdout': -1, 'universal_newlines': True}
generating ./_add3_cffi.c
the current directory is '/tmp/ffi-0/embedding/test_load_in_parallel_more'
FILENAME: /tmp/ffi-0/embedding/test_load_in_parallel_more/_add3_cffi.so
compiling thread3-test with ['/tmp/ffi-0/embedding/test_two_modules/_add2_cffi.so', '/tmp/ffi-0/embedding/test_load_in_parallel_more/_add3_cffi.so']
running 'thread3-test' in '/tmp/ffi-0/embedding/test_load_in_parallel_more'
RUNNING: ['./thread3-test'] {'cwd': '/tmp/ffi-0/embedding/test_load_in_parallel_more', 'stdout': -1, 'universal_newlines': True}
running 'thread3-test' in '/tmp/ffi-0/embedding/test_load_in_parallel_more'
RUNNING: ['./thread3-test'] {'cwd': '/tmp/ffi-0/embedding/test_load_in_parallel_more', 'stdout': -1, 'universal_newlines': True}
running 'thread3-test' in '/tmp/ffi-0/embedding/test_load_in_parallel_more'
RUNNING: ['./thread3-test'] {'cwd': '/tmp/ffi-0/embedding/test_load_in_parallel_more', 'stdout': -1, 'universal_newlines': True}
----------------------------- Captured stderr call -----------------------------
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Python runtime state: initialized

Thread 0x00007f6ed57fa6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed77fe6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed27f46c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed17f26c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed5ffb6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed4ff96c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed7fff6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed6ffd6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed2ff56c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6edd70e6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed1ff36c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ed0ff16c0 (most recent call first):
  <no Python frame>

Thread 0x00007f6ede7106c0 (most recent call first):
  <no Python frame>

Extension modules: _cffi_backend (total: 1)
______________________ TestThreadLocal.test_thread_local _______________________

self = <testing.embedding.test_tlocal.TestThreadLocal object at 0x7f23a8f9e030>

    def test_thread_local(self):
        tlocal_cffi = self.prepare_module('tlocal')
        self.compile('tlocal-test', [tlocal_cffi], threads=True)
        for i in range(10):
>           output = self.execute('tlocal-test')

testing/embedding/test_tlocal.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testing.embedding.test_tlocal.TestThreadLocal object at 0x7f23a8f9e030>
name = 'tlocal-test'

    def execute(self, name):
        path = self.get_path()
        print('running %r in %r' % (name, path))
        executable_name = name
        if sys.platform == 'win32':
            executable_name = os.path.join(path, executable_name + '.exe')
        else:
            executable_name = os.path.join('.', executable_name)
        popen = self._run_base([executable_name], cwd=path,
                               stdout=subprocess.PIPE,
                               universal_newlines=True)
        result = popen.stdout.read()
        err = popen.wait()
        if err:
>           raise OSError("%r failed with exit code %r" % (
                os.path.join(path, executable_name), err))
E           OSError: '/tmp/ffi-0/embedding/test_thread_local/./tlocal-test' failed with exit code -6

testing/embedding/test_basic.py:175: OSError
----------------------------- Captured stdout call -----------------------------
* setting env var 'LD_LIBRARY_PATH' to '/usr/bin:/tmp/ffi-0/embedding/test_thread_local'
RUNNING: ['/usr/bin/python3', '/tmp/ffi-0/embedding/test_thread_local/tlocal.py'] {'cwd': '/tmp/ffi-0/embedding/test_thread_local', 'stdout': -1, 'universal_newlines': True}
generating ./_tlocal_cffi.c
the current directory is '/tmp/ffi-0/embedding/test_thread_local'
FILENAME: /tmp/ffi-0/embedding/test_thread_local/_tlocal_cffi.so
compiling tlocal-test with ['/tmp/ffi-0/embedding/test_thread_local/_tlocal_cffi.so']
running 'tlocal-test' in '/tmp/ffi-0/embedding/test_thread_local'
RUNNING: ['./tlocal-test'] {'cwd': '/tmp/ffi-0/embedding/test_thread_local', 'stdout': -1, 'universal_newlines': True}
----------------------------- Captured stderr call -----------------------------
Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread
Python runtime state: initialized

Thread 0x00007f7bfa59b6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bf27fc6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bf37fe6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bf3fff6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bf17fa6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bf2ffd6c0 (most recent call first):
  <no Python frame>

Thread 0x00007f7bfad9c6c0 (most recent call first):
  <no Python frame>

Extension modules: _cffi_backend (total: 1)
=========================== short test summary info ============================
FAILED testing/cffi0/test_ffi_backend.py::TestFFI::test_dlclose - SystemError...
FAILED testing/embedding/test_thread.py::TestThread::test_first_calls_in_parallel
FAILED testing/embedding/test_thread.py::TestThread::test_load_in_parallel_more
FAILED testing/embedding/test_tlocal.py::TestThreadLocal::test_thread_local
= 4 failed, 1939 passed, 96 skipped, 4 xfailed, 573 warnings in 285.26s (0:04:45) =

https://docs.python.org/3.12/whatsnew/3.12.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.12/fedora-rawhide-x86_64/05839144-python-cffi/

For all our attempts to build python-cffi with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/package/python-cffi/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2023-05-05 15:21:01 UTC
I am leaving for vacation and will be back in ~10 days. This is a pretty serious blocker for the Python 3.12 rebuild hence assigning it to Python Maint for somebody else to look at.

Comment 2 Tomáš Hrnčiar 2023-05-11 06:43:32 UTC
Upstream report: https://foss.heptapod.net/pypy/cffi/-/issues/562

I've tried to add it to "Links" field above, but I am getting:
    The external tracker URL you specified does not belong to a tracker known to Red Hat Bugzilla

Comment 3 Tomáš Hrnčiar 2023-05-24 13:18:52 UTC
I skipped the failing tests in Python3.12 COPR, and so far I haven't encountered any package failing because of it.

Comment 4 Tomas Orsava 2023-08-02 12:16:51 UTC
This has been fixed upstream, will get into Fedora in the next update.


Note You need to log in before you can comment on or make changes to this bug.