Bug 2323186

Summary: dnf fails to build with Python 3.14: TypeError: cannot pickle '_thread.RLock' object
Product: [Fedora] Fedora Reporter: Karolina Surma <ksurma>
Component: dnfAssignee: Petr Pisar <ppisar>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: high    
Version: rawhideCC: daniel.mach, jmracek, jrohel, ksurma, mblaha, mhroncok, packaging-team-maint, pkratoch, ppisar, rpm-software-management
Target Milestone: ---Keywords: Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: dnf-4.22.0-3.fc42 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-01-07 10:36:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2322407    

Description Karolina Surma 2024-11-01 15:10:07 UTC
dnf fails to build with Python 3.14.0a1.

1: ERROR: test_another_process (tests.test_lock.ProcessLockTest.test_another_process)
1: ----------------------------------------------------------------------
1: Traceback (most recent call last):
1:   File "/builddir/build/BUILD/dnf-4.21.1-build/dnf-4.21.1/tests/test_lock.py", line 113, in test_another_process
1:     process.start()
1:     ~~~~~~~~~~~~~^^
1:   File "/usr/lib64/python3.14/multiprocessing/process.py", line 121, in start
1:     self._popen = self._Popen(self)
1:                   ~~~~~~~~~~~^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 224, in _Popen
1:     return _default_context.get_context().Process._Popen(process_obj)
1:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 300, in _Popen
1:     return Popen(process_obj)
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
1:     super().__init__(process_obj)
1:     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
1:     self._launch(process_obj)
1:     ~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch
1:     reduction.dump(process_obj, buf)
1:     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/reduction.py", line 60, in dump
1:     ForkingPickler(file, protocol).dump(obj)
1:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
1: TypeError: cannot pickle '_thread.RLock' object
1: when serializing dict item 'thread_lock'
1: when serializing dnf.lock.ProcessLock state
1: when serializing dnf.lock.ProcessLock object
1: when serializing dict item 'lock'
1: when serializing tests.test_lock.OtherProcess state
1: when serializing tests.test_lock.OtherProcess object
1: 
1: ======================================================================
1: ERROR: test_another_process_blocking (tests.test_lock.ProcessLockTest.test_another_process_blocking)
1: ----------------------------------------------------------------------
1: Traceback (most recent call last):
1:   File "/builddir/build/BUILD/dnf-4.21.1-build/dnf-4.21.1/tests/test_lock.py", line 123, in test_another_process_blocking
1:     process.start()
1:     ~~~~~~~~~~~~~^^
1:   File "/usr/lib64/python3.14/multiprocessing/process.py", line 121, in start
1:     self._popen = self._Popen(self)
1:                   ~~~~~~~~~~~^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 224, in _Popen
1:     return _default_context.get_context().Process._Popen(process_obj)
1:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 300, in _Popen
1:     return Popen(process_obj)
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
1:     super().__init__(process_obj)
1:     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
1:     self._launch(process_obj)
1:     ~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch
1:     reduction.dump(process_obj, buf)
1:     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/reduction.py", line 60, in dump
1:     ForkingPickler(file, protocol).dump(obj)
1:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
1: TypeError: cannot pickle '_thread.RLock' object
1: when serializing dict item 'thread_lock'
1: when serializing dnf.lock.ProcessLock state
1: when serializing dnf.lock.ProcessLock object
1: when serializing dict item 'lock'
1: when serializing tests.test_lock.OtherProcess state
1: when serializing tests.test_lock.OtherProcess object
1: 
1: ----------------------------------------------------------------------
1: Ran 804 tests in 6.090s
1: 
1: FAILED (errors=2, skipped=1)
1: <sys>:0: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
1/1 Test #1: test .............................***Failed    6.56 sec

https://docs.python.org/3.14/whatsnew/3.14.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.14/fedora-rawhide-x86_64/08189199-dnf/

For all our attempts to build dnf with Python 3.14, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.14/package/dnf/

Testing and mass rebuild of packages is happening in copr.
You can follow these instructions to test locally in mock if your package builds with Python 3.14:
https://copr.fedorainfracloud.org/coprs/g/python/python3.14/

Let us know here if you have any questions.

Python 3.14 is planned to be included in Fedora 43.
To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.14.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires),
so if this package is required a lot, it's important for us to get it fixed soon.

We'd appreciate help from the people who know this package best,
but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2024-11-25 13:03:13 UTC
This patch makes the failures go away:

 diff --git a/tests/test_lock.py b/tests/test_lock.py
index ce9806b4..d45557e9 100644
--- a/tests/test_lock.py
+++ b/tests/test_lock.py
@@ -42,6 +42,15 @@ import tests.support
 from tests.support import mock
 
 
+# The tests here are not compatible with the forkserver method,
+# which is the default on Python 3.14+.
+# See https://github.com/python/cpython/issues/125714
+if multiprocessing.get_start_method() == 'forkserver':
+    mp_context = multiprocessing.get_context(method='fork')
+else:
+    mp_context = multiprocessing.get_context()
+
+
 class ConcurrencyMixin(object):
     def __init__(self, lock):
         self.lock = lock
@@ -61,11 +70,11 @@ class OtherThread(ConcurrencyMixin, threading.Thread):
         self.queue = dnf.pycomp.Queue(1)
 
 
-class OtherProcess(ConcurrencyMixin, multiprocessing.Process):
+class OtherProcess(ConcurrencyMixin, mp_context.Process):
     def __init__(self, lock):
         ConcurrencyMixin.__init__(self, lock)
-        multiprocessing.Process.__init__(self)
-        self.queue = multiprocessing.Queue(1)
+        mp_context.Process.__init__(self)
+        self.queue = mp_context.Queue(1)
 
 
 TARGET = os.path.join(tests.support.USER_RUNDIR, 'unit-test.pid')


However, I still get:

+ ctest -VV
UpdateCTestConfiguration  from :/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/build-py3/DartConfiguration.tcl
UpdateCTestConfiguration  from :/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/build-py3/DartConfiguration.tcl
Test project /builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/build-py3
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: test

1: Test command: /usr/bin/python3 "-m" "unittest" "discover" "-s" "tests" "-t" "/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0"
1: Working Directory: /builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0
1: Environment variables: 
1:  PYTHONPATH=/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/
1:  ASAN_OPTIONS=verify_asan_link_order=0,detect_leaks=0
1: Test timeout computed to be: 10000000
1: ....History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
1: ......History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
1: ...EHistory database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
1: .History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
1: ...EHistory database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
1: .
1/1 Test #1: test .............................***Exception: SegFault  0.54 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.54 sec

The following tests FAILED:
	  1 - test (SEGFAULT)
Errors while running CTest
Output from these tests are in: /builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/build-py3/Testing/Temporary/LastTest.log


That log has:


Start testing: Nov 25 14:01 CET
----------------------------------------------------------
1/1 Testing: test
1/1 Test: test
Command: "/usr/bin/python3" "-m" "unittest" "discover" "-s" "tests" "-t" "/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0"
Directory: /builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0
"test" start time: Nov 25 14:01 CET
Output:
----------------------------------------------------------
....History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
......History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
...EHistory database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
.History database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
...EHistory database cannot be created, using in-memory database instead: SQLite error on "/tmp/tests/history.sqlite": Open failed: unable to open database file
.
<end of output>
Test time =   0.54 sec
----------------------------------------------------------
Test Failed.
"test" end time: Nov 25 14:01 CET
"test" time elapsed: 00:00:00
----------------------------------------------------------

End testing: Nov 25 14:01 CET

Comment 2 Petr Pisar 2024-12-03 13:18:49 UTC
Has something relevant changed in Python 3.14 in the mean time? I tried to reproduce it with current Copr repository (python3-3.14.0~a2-1.fc42.x86_64) and I do not observe the _thread.RLock exception. Though I confirm the segfault.

Comment 3 Miro Hrončok 2024-12-03 13:24:15 UTC
Not that I know of, but perhaps the segfault just happens sooner?

Comment 4 Miro Hrončok 2024-12-10 14:24:51 UTC
I wonder if the segfault is bz2330562. Considering the failures in https://github.com/rpm-software-management/dnf/pull/2172 look very similar.

Comment 5 Miro Hrončok 2024-12-10 14:28:58 UTC
Nope, it faults even with python3-libdnf 0.73.4-2.fc42

Comment 6 Miro Hrončok 2024-12-11 16:29:55 UTC
With bz2331665 fixed, we are back at:

1: ======================================================================
1: ERROR: test_another_process_blocking (tests.test_lock.ProcessLockTest.test_another_process_blocking)
1: ----------------------------------------------------------------------
1: Traceback (most recent call last):
1:   File "/builddir/build/BUILD/dnf-4.22.0-build/dnf-4.22.0/tests/test_lock.py", line 123, in test_another_process_blocking
1:     process.start()
1:     ~~~~~~~~~~~~~^^
1:   File "/usr/lib64/python3.14/multiprocessing/process.py", line 121, in start
1:     self._popen = self._Popen(self)
1:                   ~~~~~~~~~~~^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 224, in _Popen
1:     return _default_context.get_context().Process._Popen(process_obj)
1:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/context.py", line 300, in _Popen
1:     return Popen(process_obj)
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
1:     super().__init__(process_obj)
1:     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
1:     self._launch(process_obj)
1:     ~~~~~~~~~~~~^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch
1:     reduction.dump(process_obj, buf)
1:     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
1:   File "/usr/lib64/python3.14/multiprocessing/reduction.py", line 60, in dump
1:     ForkingPickler(file, protocol).dump(obj)
1:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
1: TypeError: cannot pickle '_thread.RLock' object
1: when serializing dict item 'thread_lock'
1: when serializing dnf.lock.ProcessLock state
1: when serializing dnf.lock.ProcessLock object
1: when serializing dict item 'lock'
1: when serializing tests.test_lock.OtherProcess state
1: when serializing tests.test_lock.OtherProcess object
1: 
1: ----------------------------------------------------------------------
1: Ran 804 tests in 4.890s
1: 
1: FAILED (errors=2, skipped=1)
1: <sys>:0: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
1/1 Test #1: test .............................***Failed    5.30 sec

Comment 7 Miro Hrončok 2024-12-11 16:33:04 UTC
And my attempt from comment #1 works: https://github.com/rpm-software-management/dnf/pull/2173

Comment 8 Petr Pisar 2025-01-07 09:20:26 UTC
Merged upstream.

I guess you need this fix in Fedora 42 to unblock your upgrade of Python there. I will apply it to Fedora 42.

Comment 9 Miro Hrončok 2025-01-07 10:36:33 UTC
> I guess you need this fix in Fedora 42 to unblock your upgrade of Python there.

Dist git was fine, but no harm in shipping a rawhide update. Thanks.