Bug 1336557 - python-pthreading tests fail on AArch64
Summary: python-pthreading tests fail on AArch64
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python-pthreading
Version: 7.2
Hardware: aarch64
OS: Linux
medium
medium
Target Milestone: rc
: 7.6
Assignee: Marcin Sobczyk
QA Contact:
URL:
Whiteboard:
Depends On: 1289389
Blocks: epel7aarch64 1498541 1706775
TreeView+ depends on / blocked
 
Reported: 2016-05-16 21:12 UTC by Yaakov Selkowitz
Modified: 2020-05-20 23:43 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1289389
Environment:
Last Closed: 2019-12-12 23:15:29 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Yaakov Selkowitz 2016-05-16 21:12:12 UTC
This affects RHEL python on aarch64 as well:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11036517

+++ This bug was initially created as a clone of Bug #1289389 +++

python-pthreading tests fail on aarch64 with the output below.  As there is no architecture-specific code in python-pthreading, I believe the issue is in python itself.

*** Error in `/usr/bin/python': free(): invalid next size (fast): 0x000002ab0dd2a220 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x70e80)[0x3ffafe80e80]
/lib64/libc.so.6(+0x7945c)[0x3ffafe8945c]
/lib64/libc.so.6(cfree+0x48)[0x3ffafe8c928]
/usr/lib64/python2.7/lib-dynload/_ctypes.so(+0x7738)[0x3ffaf977738]
/usr/lib64/python2.7/lib-dynload/_ctypes.so(+0x779c)[0x3ffaf97779c]
/lib64/libpython2.7.so.1.0(+0x9f4ec)[0x3ffb01af4ec]
/lib64/libpython2.7.so.1.0(+0x9e714)[0x3ffb01ae714]
/lib64/libpython2.7.so.1.0(+0x9f444)[0x3ffb01af444]
/lib64/libpython2.7.so.1.0(+0x9e714)[0x3ffb01ae714]
/lib64/libpython2.7.so.1.0(+0x9f444)[0x3ffb01af444]
/lib64/libpython2.7.so.1.0(+0x81b1c)[0x3ffb0191b1c]
/lib64/libpython2.7.so.1.0(+0x9f564)[0x3ffb01af564]
/lib64/libpython2.7.so.1.0(+0x559bc)[0x3ffb01659bc]
/lib64/libpython2.7.so.1.0(+0x6f110)[0x3ffb017f110]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x4cc)[0x3ffb01f69d4]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58e0)[0x3ffb01f5fc0]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x709bc)[0x3ffb01809bc]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xc20)[0x3ffb01f1300]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(+0x708c8)[0x3ffb01808c8]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0x5a500)[0x3ffb016a500]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(+0xa50b4)[0x3ffb01b50b4]
/lib64/libpython2.7.so.1.0(PyObject_Call+0x68)[0x3ffb015afa0]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x39e8)[0x3ffb01f40c8]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a18)[0x3ffb01f60f8]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x58e0)[0x3ffb01f5fc0]
/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x818)[0x3ffb01f6d20]
/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x2c)[0x3ffb01f6e34]
/lib64/libpython2.7.so.1.0(+0x100c80)[0x3ffb0210c80]
/lib64/libpython2.7.so.1.0(PyRun_FileExFlags+0x8c)[0x3ffb0211ec4]
/lib64/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xec)[0x3ffb0213144]
======= Memory map: ========
2aad6eb0000-2aad6ec0000 r-xp 00000000 08:04 5774170                      /usr/bin/python2.7
2aad6ec0000-2aad6ed0000 r--p 00000000 08:04 5774170                      /usr/bin/python2.7
2aad6ed0000-2aad6ee0000 rw-p 00010000 08:04 5774170                      /usr/bin/python2.7
2ab0dc30000-2ab0dda0000 rw-p 00000000 00:00 0                            [heap]
3ffa8000000-3ffa8030000 rw-p 00000000 00:00 0 
3ffa8030000-3ffac000000 ---p 00000000 00:00 0 
3ffaefb0000-3ffaefd0000 r-xp 00000000 08:04 6029878                      /usr/lib64/libgcc_s-5.1.1-20150618.so.1
3ffaefd0000-3ffaefe0000 rw-p 00010000 08:04 6029878                      /usr/lib64/libgcc_s-5.1.1-20150618.so.1
3ffaeff0000-3ffaf000000 ---p 00000000 00:00 0 
3ffaf000000-3ffaf910000 rw-p 00000000 00:00 0 
3ffaf910000-3ffaf920000 r-xp 00000000 08:04 6685818                      /usr/lib64/python2.7/lib-dynload/_struct.so
3ffaf920000-3ffaf930000 r--p 00000000 08:04 6685818                      /usr/lib64/python2.7/lib-dynload/_struct.so
3ffaf930000-3ffaf940000 rw-p 00010000 08:04 6685818                      /usr/lib64/python2.7/lib-dynload/_struct.so
3ffaf940000-3ffaf950000 r-xp 00000000 08:04 6030710                      /usr/lib64/libffi.so.6.0.2
3ffaf950000-3ffaf960000 r--p 00000000 08:04 6030710                      /usr/lib64/libffi.so.6.0.2
3ffaf960000-3ffaf970000 rw-p 00010000 08:04 6030710                      /usr/lib64/libffi.so.6.0.2
3ffaf970000-3ffaf990000 r-xp 00000000 08:04 6685800                      /usr/lib64/python2.7/lib-dynload/_ctypes.so
3ffaf990000-3ffaf9a0000 r--p 00010000 08:04 6685800                      /usr/lib64/python2.7/lib-dynload/_ctypes.so
3ffaf9a0000-3ffaf9b0000 rw-p 00020000 08:04 6685800                      /usr/lib64/python2.7/lib-dynload/_ctypes.so
3ffaf9b0000-3ffaf9c0000 r-xp 00000000 08:04 6685850                      /usr/lib64/python2.7/lib-dynload/timemodule.so
3ffaf9c0000-3ffaf9d0000 r--p 00000000 08:04 6685850                      /usr/lib64/python2.7/lib-dynload/timemodule.so
3ffaf9d0000-3ffaf9e0000 rw-p 00010000 08:04 6685850                      /usr/lib64/python2.7/lib-dynload/timemodule.so
3ffaf9e0000-3ffafa20000 rw-p 00000000 00:00 0 
3ffafa20000-3ffafa30000 r-xp 00000000 08:04 6685824                      /usr/lib64/python2.7/lib-dynload/cStringIO.so
3ffafa30000-3ffafa40000 r--p 00000000 08:04 6685824                      /usr/lib64/python2.7/lib-dynload/cStringIO.so
3ffafa40000-3ffafa50000 rw-p 00010000 08:04 6685824                      /usr/lib64/python2.7/lib-dynload/cStringIO.so
3ffafa50000-3ffafa90000 rw-p 00000000 00:00 0 
3ffafa90000-3ffafaa0000 r-xp 00000000 08:04 6685804                      /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so
3ffafaa0000-3ffafab0000 r--p 00000000 08:04 6685804                      /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so
3ffafab0000-3ffafac0000 rw-p 00010000 08:04 6685804                      /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so
3ffafac0000-3ffafad0000 r-xp 00000000 08:04 6685806                      /usr/lib64/python2.7/lib-dynload/_heapq.so
3ffafad0000-3ffafae0000 r--p 00000000 08:04 6685806                      /usr/lib64/python2.7/lib-dynload/_heapq.so
3ffafae0000-3ffafaf0000 rw-p 00010000 08:04 6685806                      /usr/lib64/python2.7/lib-dynload/_heapq.so
3ffafaf0000-3ffafb00000 r-xp 00000000 08:04 6685834                      /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
3ffafb00000-3ffafb10000 r--p 00000000 08:04 6685834                      /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
3ffafb10000-3ffafb20000 rw-p 00010000 08:04 6685834                      /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
3ffafb20000-3ffafb30000 r-xp 00000000 08:04 6685839                      /usr/lib64/python2.7/lib-dynload/operator.so
3ffafb30000-3ffafb40000 r--p 00000000 08:04 6685839                      /usr/lib64/python2.7/lib-dynload/operator.so
3ffafb40000-3ffafb50000 rw-p 00010000 08:04 6685839                      /usr/lib64/python2.7/lib-dynload/operator.so
3ffafb50000-3ffafb60000 r-xp 00000000 08:04 6685797                      /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so
3ffafb60000-3ffafb70000 r--p 00000000 08:04 6685797                      /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so
3ffafb70000-3ffafb80000 rw-p 00010000 08:04 6685797                      /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so
3ffafb80000-3ffafbc0000 rw-p 00000000 00:00 0 
3ffafbc0000-3ffafbd0000 r-xp 00000000 08:04 6685832                      /usr/lib64/python2.7/lib-dynload/grpmodule.so
3ffafbd0000-3ffafbe0000 r--p 00000000 08:04 6685832                      /usr/lib64/python2.7/lib-dynload/grpmodule.so
3ffafbe0000-3ffafbf0000 rw-p 00010000 08:04 6685832                      /usr/lib64/python2.7/lib-dynload/grpmodule.so
3ffafbf0000-3ffafc00000 r-xp 00000000 08:04 6685847                      /usr/lib64/python2.7/lib-dynload/stropmodule.so
3ffafc00000-3ffafc10000 r--p 00000000 08:04 6685847                      /usr/lib64/python2.7/lib-dynload/stropmodule.so
3ffafc10000-3ffafc20000 rw-p 00010000 08:04 6685847                      /usr/lib64/python2.7/lib-dynload/stropmodule.so
3ffafc20000-3ffafca0000 rw-p 00000000 00:00 0 
3ffafca0000-3ffafcb0000 r-xp 00000000 08:04 6685810                      /usr/lib64/python2.7/lib-dynload/_localemodule.so
3ffafcb0000-3ffafcc0000 r--p 00000000 08:04 6685810                      /usr/lib64/python2.7/lib-dynload/_localemodule.so
3ffafcc0000-3ffafcd0000 rw-p 00010000 08:04 6685810                      /usr/lib64/python2.7/lib-dynload/_localemodule.so
3ffafcd0000-3ffafe10000 rw-p 00000000 00:00 0 
3ffafe10000-3ffaff80000 r-xp 00000000 08:04 6030499                      /usr/lib64/libc-2.22.so
3ffaff80000-3ffaff90000 r--p 00160000 08:04 6030499                      /usr/lib64/libc-2.22.so
3ffaff90000-3ffaffa0000 rw-p 00170000 08:04 6030499                      /usr/lib64/libc-2.22.so
3ffaffa0000-3ffb0050000 r-xp 00000000 08:04 6030507                      /usr/lib64/libm-2.22.so
3ffb0050000-3ffb0060000 r--p 000a0000 08:04 6030507                      /usr/lib64/libm-2.22.so
3ffb0060000-3ffb0070000 rw-p 000b0000 08:04 6030507                      /usr/lib64/libm-2.22.so
3ffb0070000-3ffb0080000 r-xp 00000000 08:04 6030533                      /usr/lib64/libutil-2.22.so
3ffb0080000-3ffb0090000 r--p 00000000 08:04 6030533                      /usr/lib64/libutil-2.22.so
3ffb0090000-3ffb00a0000 rw-p 00010000 08:04 6030533                      /usr/lib64/libutil-2.22.so
3ffb00a0000-3ffb00b0000 r-xp 00000000 08:04 6030505                      /usr/lib64/libdl-2.22.so
3ffb00b0000-3ffb00c0000 r--p 00000000 08:04 6030505                      /usr/lib64/libdl-2.22.so
3ffb00c0000-3ffb00d0000 rw-p 00010000 08:04 6030505                      /usr/lib64/libdl-2.22.so
3ffb00d0000-3ffb00f0000 r-xp 00000000 08:04 6030525                      /usr/lib64/libpthread-2.22.so
3ffb00f0000-3ffb0100000 r--p 00010000 08:04 6030525                      /usr/lib64/libpthread-2.22.so
3ffb0100000-3ffb0110000 rw-p 00020000 08:04 6030525                      /usr/lib64/libpthread-2.22.so
3ffb0110000-3ffb02a0000 r-xp 00000000 08:04 6032178                      /usr/lib64/libpython2.7.so.1.0
3ffb02a0000-3ffb02b0000 r--p 00180000 08:04 6032178                      /usr/lib64/libpython2.7.so.1.0
3ffb02b0000-3ffb02f0000 rw-p 00190000 08:04 6032178                      /usr/lib64/libpython2.7.so.1.0
3ffb02f0000-3ffb0310000 rw-p 00000000 00:00 0 
3ffb0310000-3ffb0320000 r--p 00000000 00:00 0                            [vvar]
3ffb0320000-3ffb0330000 r-xp 00000000 00:00 0                            [vdso]
3ffb0330000-3ffb0350000 r-xp 00000000 08:04 6030492                      /usr/lib64/ld-2.22.so
3ffb0350000-3ffb0360000 r--p 00010000 08:04 6030492                      /usr/lib64/ld-2.22.so
3ffb0360000-3ffb0370000 rw-p 00020000 08:04 6030492                      /usr/lib64/ld-2.22.so
3ffdd7c0000-3ffdd7f0000 rw-p 00000000 00:00 0                            [stack]

--- Additional comment from Fedora Admin XMLRPC Client on 2016-01-29 08:04:19 EST ---

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

--- Additional comment from Jan Kurik on 2016-02-24 09:06:50 EST ---

This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase

Comment 1 Petr Viktorin 2016-07-07 08:49:05 UTC
Since pthreading uses ctypes and monkeypatches the standard library, I don't think "no architecture-specific code in python-pthreading" is a good reason to automatically blame Python.

Can you prepare a minimal reproducer?

Comment 2 Charalampos Stratakis 2016-10-11 13:53:01 UTC
No reproducer was posted so reassigning it to the component that it originates from.

Comment 3 Yaakov Selkowitz 2017-03-17 20:58:16 UTC
I wonder if https://bugs.python.org/issue29804 might be related?

Comment 4 John Feeney 2017-10-18 21:36:32 UTC
Any status on this for 7.5? Anything the ARM team can do to help?

Comment 5 Yaakov Selkowitz 2018-03-21 03:16:57 UTC
Based on mock and koji scratch builds of the same SRPM, the tests pass in F27 but not in F26 or RHEL 7.  Therefore this is most likely in either python or libffi.

Comment 6 Petr Viktorin 2018-04-11 10:17:35 UTC
We still don't have a good reproducer, and initial attempts to investigate didn't yield much.
The original comment about "no architecture-specific code in python-pthreading" is misleading, since pthreading uses ctypes.

We'll get to it when we have some free cycles, but it's low priority. Let us know if we need to allocate more time to it.

Comment 7 Petr Viktorin 2018-06-07 14:15:04 UTC
Still waiting for a reproducer.

Comment 10 Victor Stinner 2018-08-02 14:51:06 UTC
I backported a change from master to Python 2.7 which may fix test_ctypes on arm64, but I don't have access to arm64 so I cannot ensure that it really fixes the test:

https://github.com/python/cpython/commit/3243f8c1fb16b6de73f1d7a30f5d09047553bce3
https://bugs.python.org/issue29804#msg322964

Comment 13 Victor Stinner 2018-10-15 15:44:26 UTC
Ok, I got access to a RHEL7/AArch64 VM on Beaker.

"python-pthreading tests fail on aarch64 with the output below.  As there is no architecture-specific code in python-pthreading, I believe the issue is in python itself."

Well, there *is* architecture-specific code in python-pthreading:

 # These come from pthread.h (via bits/pthreadtypes.h)
 # We prefer to be on a safe side and use sizes for 64 bit implementation
 
 SIZEOF_MUTEX_T = 40
 SIZEOF_COND_T = 48
 SIZEOF_MUTEXATTR_T = 4

These sizes in bytes are hardcoded values of sizeof(pthread_mutex_t), sizeof(pthread_cond_t) and sizeof(pthread_mutexattr_t) and you know what? ... The sizes are platform specific :-)

The tests pass with the following change:

diff --git a/pthread.py b/pthread.py
index 50d1e9a..a61c0c2 100644
--- a/pthread.py
+++ b/pthread.py
@@ -37,7 +37,7 @@ LIBPTHREAD = "libpthread.so.0"
 # These come from pthread.h (via bits/pthreadtypes.h)
 # We prefer to be on a safe side and use sizes for 64 bit implementation
 
-SIZEOF_MUTEX_T = 40
+SIZEOF_MUTEX_T = 48
 SIZEOF_COND_T = 48
 SIZEOF_MUTEXATTR_T = 4


Notes:

* the latest python-pthreading version (0.1.3-3) has been released 4 years ago: https://pypi.org/project/pthreading/
* On PyPI, the homepage is set to http://git.fedorahosted.org/git/?p=pthreading.git which now redirects to https://fedoraproject.org/wiki/Infrastructure/Fedorahosted-retirement?p=pthreading.git
* Source of the RPM package: https://src.fedoraproject.org/rpms/python-pthreading/tree/master

Comment 14 Victor Stinner 2018-10-16 11:03:55 UTC
I created a PR for Fedora Rawhide: https://src.fedoraproject.org/rpms/python-pthreading/pull-request/2

Comment 15 Nir Soffer 2018-10-16 11:50:40 UTC
Sandro, is this the right component for pthreading?

Comment 16 Petr Viktorin 2018-10-16 12:09:06 UTC
It's not, RHEL7 has a python-pthreading component.

Comment 17 Victor Stinner 2018-10-17 15:47:24 UTC
I proposed a fix upstream: https://gerrit.ovirt.org/#/c/94974/

Comment 18 Sandro Bonazzola 2019-05-06 09:48:29 UTC
fix merged upstream, pending a release

Comment 19 Dan Kenigsberg 2019-05-06 16:09:58 UTC
I'm not sure what's your question to me, Sandro.

I believe that with the move to python 3 we can orphan this package.

Comment 20 Nir Soffer 2019-08-02 14:31:03 UTC
Sandro, what kind of info do you need?

Comment 22 Sandro Bonazzola 2019-10-22 06:31:11 UTC
(In reply to Nir Soffer from comment #20)
> Sandro, what kind of info do you need?

Need a release including https://gerrit.ovirt.org/#/c/94974/ but since this is not going to be in 4.3 I think we can close this bug.
4.4 is not going to use python-pthreading on el8

Comment 25 Nir Soffer 2020-05-20 23:43:15 UTC
Removing stale need infos.


Note You need to log in before you can comment on or make changes to this bug.