Hide Forgot
This affects RHEL python on aarch64 as well: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11036517 +++ This bug was initially created as a clone of Bug #1289389 +++ python-pthreading tests fail on aarch64 with the output below. As there is no architecture-specific code in python-pthreading, I believe the issue is in python itself. *** Error in `/usr/bin/python': free(): invalid next size (fast): 0x000002ab0dd2a220 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x70e80)[0x3ffafe80e80] /lib64/libc.so.6(+0x7945c)[0x3ffafe8945c] /lib64/libc.so.6(cfree+0x48)[0x3ffafe8c928] /usr/lib64/python2.7/lib-dynload/_ctypes.so(+0x7738)[0x3ffaf977738] /usr/lib64/python2.7/lib-dynload/_ctypes.so(+0x779c)[0x3ffaf97779c] /lib64/libpython2.7.so.1.0(+0x9f4ec)[0x3ffb01af4ec] /lib64/libpython2.7.so.1.0(+0x9e714)[0x3ffb01ae714] /lib64/libpython2.7.so.1.0(+0x9f444)[0x3ffb01af444] /lib64/libpython2.7.so.1.0(+0x9e714)[0x3ffb01ae714] /lib64/libpython2.7.so.1.0(+0x9f444)[0x3ffb01af444] /lib64/libpython2.7.so.1.0(+0x81b1c)[0x3ffb0191b1c] /lib64/libpython2.7.so.1.0(+0x9f564)[0x3ffb01af564] /lib64/libpython2.7.so.1.0(+0x559bc)[0x3ffb01659bc] /lib64/libpython2.7.so.1.0(+0x6f110)[0x3ffb017f110] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x4cc)[0x3ffb01f69d4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58e0)[0x3ffb01f5fc0] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4] /lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58e0)[0x3ffb01f5fc0] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20] /lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x2c)[0x3ffb01f6e34] /lib64/libpython2.7.so.1.0(+0x100c80)[0x3ffb0210c80] /lib64/libpython2.7.so.1.0(PyRun_FileExFlags+0x8c)[0x3ffb0211ec4] /lib64/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xec)[0x3ffb0213144] ======= Memory map: ======== 2aad6eb0000-2aad6ec0000 r-xp 00000000 08:04 5774170 /usr/bin/python2.7 2aad6ec0000-2aad6ed0000 r--p 00000000 08:04 5774170 /usr/bin/python2.7 2aad6ed0000-2aad6ee0000 rw-p 00010000 08:04 5774170 /usr/bin/python2.7 2ab0dc30000-2ab0dda0000 rw-p 00000000 00:00 0 [heap] 3ffa8000000-3ffa8030000 rw-p 00000000 00:00 0 3ffa8030000-3ffac000000 ---p 00000000 00:00 0 3ffaefb0000-3ffaefd0000 r-xp 00000000 08:04 6029878 /usr/lib64/libgcc_s-5.1.1-20150618.so.1 3ffaefd0000-3ffaefe0000 rw-p 00010000 08:04 6029878 /usr/lib64/libgcc_s-5.1.1-20150618.so.1 3ffaeff0000-3ffaf000000 ---p 00000000 00:00 0 3ffaf000000-3ffaf910000 rw-p 00000000 00:00 0 3ffaf910000-3ffaf920000 r-xp 00000000 08:04 6685818 /usr/lib64/python2.7/lib-dynload/_struct.so 3ffaf920000-3ffaf930000 r--p 00000000 08:04 6685818 /usr/lib64/python2.7/lib-dynload/_struct.so 3ffaf930000-3ffaf940000 rw-p 00010000 08:04 6685818 /usr/lib64/python2.7/lib-dynload/_struct.so 3ffaf940000-3ffaf950000 r-xp 00000000 08:04 6030710 /usr/lib64/libffi.so.6.0.2 3ffaf950000-3ffaf960000 r--p 00000000 08:04 6030710 /usr/lib64/libffi.so.6.0.2 3ffaf960000-3ffaf970000 rw-p 00010000 08:04 6030710 /usr/lib64/libffi.so.6.0.2 3ffaf970000-3ffaf990000 r-xp 00000000 08:04 6685800 /usr/lib64/python2.7/lib-dynload/_ctypes.so 3ffaf990000-3ffaf9a0000 r--p 00010000 08:04 6685800 /usr/lib64/python2.7/lib-dynload/_ctypes.so 3ffaf9a0000-3ffaf9b0000 rw-p 00020000 08:04 6685800 /usr/lib64/python2.7/lib-dynload/_ctypes.so 3ffaf9b0000-3ffaf9c0000 r-xp 00000000 08:04 6685850 /usr/lib64/python2.7/lib-dynload/timemodule.so 3ffaf9c0000-3ffaf9d0000 r--p 00000000 08:04 6685850 /usr/lib64/python2.7/lib-dynload/timemodule.so 3ffaf9d0000-3ffaf9e0000 rw-p 00010000 08:04 6685850 /usr/lib64/python2.7/lib-dynload/timemodule.so 3ffaf9e0000-3ffafa20000 rw-p 00000000 00:00 0 3ffafa20000-3ffafa30000 r-xp 00000000 08:04 6685824 /usr/lib64/python2.7/lib-dynload/cStringIO.so 3ffafa30000-3ffafa40000 r--p 00000000 08:04 6685824 /usr/lib64/python2.7/lib-dynload/cStringIO.so 3ffafa40000-3ffafa50000 rw-p 00010000 08:04 6685824 /usr/lib64/python2.7/lib-dynload/cStringIO.so 3ffafa50000-3ffafa90000 rw-p 00000000 00:00 0 3ffafa90000-3ffafaa0000 r-xp 00000000 08:04 6685804 /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so 3ffafaa0000-3ffafab0000 r--p 00000000 08:04 6685804 /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so 3ffafab0000-3ffafac0000 rw-p 00010000 08:04 6685804 /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so 3ffafac0000-3ffafad0000 r-xp 00000000 08:04 6685806 /usr/lib64/python2.7/lib-dynload/_heapq.so 3ffafad0000-3ffafae0000 r--p 00000000 08:04 6685806 /usr/lib64/python2.7/lib-dynload/_heapq.so 3ffafae0000-3ffafaf0000 rw-p 00010000 08:04 6685806 /usr/lib64/python2.7/lib-dynload/_heapq.so 3ffafaf0000-3ffafb00000 r-xp 00000000 08:04 6685834 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so 3ffafb00000-3ffafb10000 r--p 00000000 08:04 6685834 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so 3ffafb10000-3ffafb20000 rw-p 00010000 08:04 6685834 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so 3ffafb20000-3ffafb30000 r-xp 00000000 08:04 6685839 /usr/lib64/python2.7/lib-dynload/operator.so 3ffafb30000-3ffafb40000 r--p 00000000 08:04 6685839 /usr/lib64/python2.7/lib-dynload/operator.so 3ffafb40000-3ffafb50000 rw-p 00010000 08:04 6685839 /usr/lib64/python2.7/lib-dynload/operator.so 3ffafb50000-3ffafb60000 r-xp 00000000 08:04 6685797 /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so 3ffafb60000-3ffafb70000 r--p 00000000 08:04 6685797 /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so 3ffafb70000-3ffafb80000 rw-p 00010000 08:04 6685797 /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so 3ffafb80000-3ffafbc0000 rw-p 00000000 00:00 0 3ffafbc0000-3ffafbd0000 r-xp 00000000 08:04 6685832 /usr/lib64/python2.7/lib-dynload/grpmodule.so 3ffafbd0000-3ffafbe0000 r--p 00000000 08:04 6685832 /usr/lib64/python2.7/lib-dynload/grpmodule.so 3ffafbe0000-3ffafbf0000 rw-p 00010000 08:04 6685832 /usr/lib64/python2.7/lib-dynload/grpmodule.so 3ffafbf0000-3ffafc00000 r-xp 00000000 08:04 6685847 /usr/lib64/python2.7/lib-dynload/stropmodule.so 3ffafc00000-3ffafc10000 r--p 00000000 08:04 6685847 /usr/lib64/python2.7/lib-dynload/stropmodule.so 3ffafc10000-3ffafc20000 rw-p 00010000 08:04 6685847 /usr/lib64/python2.7/lib-dynload/stropmodule.so 3ffafc20000-3ffafca0000 rw-p 00000000 00:00 0 3ffafca0000-3ffafcb0000 r-xp 00000000 08:04 6685810 /usr/lib64/python2.7/lib-dynload/_localemodule.so 3ffafcb0000-3ffafcc0000 r--p 00000000 08:04 6685810 /usr/lib64/python2.7/lib-dynload/_localemodule.so 3ffafcc0000-3ffafcd0000 rw-p 00010000 08:04 6685810 /usr/lib64/python2.7/lib-dynload/_localemodule.so 3ffafcd0000-3ffafe10000 rw-p 00000000 00:00 0 3ffafe10000-3ffaff80000 r-xp 00000000 08:04 6030499 /usr/lib64/libc-2.22.so 3ffaff80000-3ffaff90000 r--p 00160000 08:04 6030499 /usr/lib64/libc-2.22.so 3ffaff90000-3ffaffa0000 rw-p 00170000 08:04 6030499 /usr/lib64/libc-2.22.so 3ffaffa0000-3ffb0050000 r-xp 00000000 08:04 6030507 /usr/lib64/libm-2.22.so 3ffb0050000-3ffb0060000 r--p 000a0000 08:04 6030507 /usr/lib64/libm-2.22.so 3ffb0060000-3ffb0070000 rw-p 000b0000 08:04 6030507 /usr/lib64/libm-2.22.so 3ffb0070000-3ffb0080000 r-xp 00000000 08:04 6030533 /usr/lib64/libutil-2.22.so 3ffb0080000-3ffb0090000 r--p 00000000 08:04 6030533 /usr/lib64/libutil-2.22.so 3ffb0090000-3ffb00a0000 rw-p 00010000 08:04 6030533 /usr/lib64/libutil-2.22.so 3ffb00a0000-3ffb00b0000 r-xp 00000000 08:04 6030505 /usr/lib64/libdl-2.22.so 3ffb00b0000-3ffb00c0000 r--p 00000000 08:04 6030505 /usr/lib64/libdl-2.22.so 3ffb00c0000-3ffb00d0000 rw-p 00010000 08:04 6030505 /usr/lib64/libdl-2.22.so 3ffb00d0000-3ffb00f0000 r-xp 00000000 08:04 6030525 /usr/lib64/libpthread-2.22.so 3ffb00f0000-3ffb0100000 r--p 00010000 08:04 6030525 /usr/lib64/libpthread-2.22.so 3ffb0100000-3ffb0110000 rw-p 00020000 08:04 6030525 /usr/lib64/libpthread-2.22.so 3ffb0110000-3ffb02a0000 r-xp 00000000 08:04 6032178 /usr/lib64/libpython2.7.so.1.0 3ffb02a0000-3ffb02b0000 r--p 00180000 08:04 6032178 /usr/lib64/libpython2.7.so.1.0 3ffb02b0000-3ffb02f0000 rw-p 00190000 08:04 6032178 /usr/lib64/libpython2.7.so.1.0 3ffb02f0000-3ffb0310000 rw-p 00000000 00:00 0 3ffb0310000-3ffb0320000 r--p 00000000 00:00 0 [vvar] 3ffb0320000-3ffb0330000 r-xp 00000000 00:00 0 [vdso] 3ffb0330000-3ffb0350000 r-xp 00000000 08:04 6030492 /usr/lib64/ld-2.22.so 3ffb0350000-3ffb0360000 r--p 00010000 08:04 6030492 /usr/lib64/ld-2.22.so 3ffb0360000-3ffb0370000 rw-p 00020000 08:04 6030492 /usr/lib64/ld-2.22.so 3ffdd7c0000-3ffdd7f0000 rw-p 00000000 00:00 0 [stack] --- Additional comment from Fedora Admin XMLRPC Client on 2016-01-29 08:04:19 EST --- This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. --- Additional comment from Jan Kurik on 2016-02-24 09:06:50 EST --- This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle. Changing version to '24'. More information and reason for this action is here: https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
Since pthreading uses ctypes and monkeypatches the standard library, I don't think "no architecture-specific code in python-pthreading" is a good reason to automatically blame Python. Can you prepare a minimal reproducer?
No reproducer was posted so reassigning it to the component that it originates from.
I wonder if https://bugs.python.org/issue29804 might be related?
Any status on this for 7.5? Anything the ARM team can do to help?
Based on mock and koji scratch builds of the same SRPM, the tests pass in F27 but not in F26 or RHEL 7. Therefore this is most likely in either python or libffi.
We still don't have a good reproducer, and initial attempts to investigate didn't yield much. The original comment about "no architecture-specific code in python-pthreading" is misleading, since pthreading uses ctypes. We'll get to it when we have some free cycles, but it's low priority. Let us know if we need to allocate more time to it.
Still waiting for a reproducer.
I backported a change from master to Python 2.7 which may fix test_ctypes on arm64, but I don't have access to arm64 so I cannot ensure that it really fixes the test: https://github.com/python/cpython/commit/3243f8c1fb16b6de73f1d7a30f5d09047553bce3 https://bugs.python.org/issue29804#msg322964
Ok, I got access to a RHEL7/AArch64 VM on Beaker. "python-pthreading tests fail on aarch64 with the output below. As there is no architecture-specific code in python-pthreading, I believe the issue is in python itself." Well, there *is* architecture-specific code in python-pthreading: # These come from pthread.h (via bits/pthreadtypes.h) # We prefer to be on a safe side and use sizes for 64 bit implementation SIZEOF_MUTEX_T = 40 SIZEOF_COND_T = 48 SIZEOF_MUTEXATTR_T = 4 These sizes in bytes are hardcoded values of sizeof(pthread_mutex_t), sizeof(pthread_cond_t) and sizeof(pthread_mutexattr_t) and you know what? ... The sizes are platform specific :-) The tests pass with the following change: diff --git a/pthread.py b/pthread.py index 50d1e9a..a61c0c2 100644 --- a/pthread.py +++ b/pthread.py @@ -37,7 +37,7 @@ LIBPTHREAD = "libpthread.so.0" # These come from pthread.h (via bits/pthreadtypes.h) # We prefer to be on a safe side and use sizes for 64 bit implementation -SIZEOF_MUTEX_T = 40 +SIZEOF_MUTEX_T = 48 SIZEOF_COND_T = 48 SIZEOF_MUTEXATTR_T = 4 Notes: * the latest python-pthreading version (0.1.3-3) has been released 4 years ago: https://pypi.org/project/pthreading/ * On PyPI, the homepage is set to http://git.fedorahosted.org/git/?p=pthreading.git which now redirects to https://fedoraproject.org/wiki/Infrastructure/Fedorahosted-retirement?p=pthreading.git * Source of the RPM package: https://src.fedoraproject.org/rpms/python-pthreading/tree/master
I created a PR for Fedora Rawhide: https://src.fedoraproject.org/rpms/python-pthreading/pull-request/2
Sandro, is this the right component for pthreading?
It's not, RHEL7 has a python-pthreading component.
I proposed a fix upstream: https://gerrit.ovirt.org/#/c/94974/
fix merged upstream, pending a release
I'm not sure what's your question to me, Sandro. I believe that with the move to python 3 we can orphan this package.
Sandro, what kind of info do you need?
(In reply to Nir Soffer from comment #20) > Sandro, what kind of info do you need? Need a release including https://gerrit.ovirt.org/#/c/94974/ but since this is not going to be in 4.3 I think we can close this bug. 4.4 is not going to use python-pthreading on el8
Removing stale need infos.