Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionEdjunior Barbosa Machado
2022-03-23 13:33:51 UTC
Description of problem:
valgrind/memcheck is unable to debug go tests on rhel-9 ppc64le due to "VEX temporary storage exhausted" failure:
[root@ibm-p9z-23-lp10 ~]# go test crypto -v -count=1
=== RUN TestRC4OutOfBoundsWrite
--- PASS: TestRC4OutOfBoundsWrite (0.00s)
=== RUN TestCTROutOfBoundsWrite
--- PASS: TestCTROutOfBoundsWrite (0.00s)
=== RUN TestOFBOutOfBoundsWrite
--- PASS: TestOFBOutOfBoundsWrite (0.00s)
=== RUN TestCFBEncryptOutOfBoundsWrite
--- PASS: TestCFBEncryptOutOfBoundsWrite (0.00s)
=== RUN TestCFBDecryptOutOfBoundsWrite
--- PASS: TestCFBDecryptOutOfBoundsWrite (0.00s)
PASS
ok crypto 0.004s
[root@ibm-p9z-23-lp10 ~]# go test crypto -v -count=1 -c
[root@ibm-p9z-23-lp10 ~]# valgrind ./crypto.test
==39625== Memcheck, a memory error detector
==39625== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==39625== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==39625== Command: ./crypto.test
==39625==
==39625== Warning: set address range perms: large range [0x10a80000, 0x30a80000) (noaccess)
VEX temporary storage exhausted.
Pool = TEMP, start 0x5989d568 curr 0x59d57e70 end 0x59d620a7 (size 5000000)
vex: the `impossible' happened:
VEX temporary storage exhausted.
Increase N_{TEMPORARY,PERMANENT}_BYTES and recompile.
vex storage: T total 244479744 bytes allocated
vex storage: P total 192 bytes allocated
valgrind: the 'impossible' happened:
LibVEX called failure_exit().
host stacktrace:
==39625== at 0x58057790: show_sched_status_wrk (m_libcassert.c:406)
==39625== by 0x5805793F: report_and_quit (m_libcassert.c:477)
==39625== by 0x58057BCB: UnknownInlinedFun (m_libcassert.c:553)
==39625== by 0x58057BCB: vgPlain_core_panic_at (m_libcassert.c:558)
==39625== by 0x58057C1B: vgPlain_core_panic (m_libcassert.c:563)
==39625== by 0x580776C7: failure_exit (m_translate.c:761)
==39625== by 0x5818929B: vpanic (main_util.c:253)
==39625== by 0x5818934B: private_LibVEX_alloc_OOM (main_util.c:181)
==39625== by 0x582558EB: UnknownInlinedFun (main_util.h:176)
==39625== by 0x582558EB: doRegisterAllocation_v3 (host_generic_reg_alloc3.c:493)
==39625== by 0x581867C3: libvex_BackEnd (main_main.c:1133)
==39625== by 0x581867C3: LibVEX_Translate (main_main.c:1236)
==39625== by 0x5807B1F7: vgPlain_translate (m_translate.c:1830)
==39625== by 0x580CB4CF: handle_chain_me (scheduler.c:1169)
==39625== by 0x580CEB27: vgPlain_scheduler (scheduler.c:1514)
==39625== by 0x5812476F: thread_wrapper (syswrap-linux.c:101)
==39625== by 0x5812476F: run_a_thread_NORETURN (syswrap-linux.c:154)
sched status:
running_tid=1
Thread 1: status = VgTs_Runnable (lwpid 39625)
==39625== at 0x1000651C: indexbytebody (indexbyte_ppc64x.s:198)
client stack range: [0x1FFEFF0000 0x1FFF00FFFF] client SP: 0x1FFF00E758
valgrind stack range: [0x1002FB0000 0x10030AFFFF] top usage: 10128 of 1048576
Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.
If that doesn't help, please report this bug to: www.valgrind.org
In the bug report, send all the above text, the valgrind
version, and what OS and version you are using. Thanks.
[root@ibm-p9z-23-lp10 ~]# objdump -d crypto.test | grep -iC3 1000651C:
10006510: ff ff 60 38 li r3,-1
10006514: 00 00 6e f8 std r3,0(r14)
10006518: 20 00 80 4e blr
1000651c: 4c 55 c6 10 vbpermq v6,v6,v10
10006520: 4c 55 e7 10 vbpermq v7,v7,v10
10006524: 4c 55 08 11 vbpermq v8,v8,v10
10006528: 4c 55 29 11 vbpermq v9,v9,v10
[root@ibm-p9z-23-lp10 ~]#
This issue was not seen on other rhel-9 supported arches.
Version-Release number of selected component (if applicable):
golang-1.17.7-1.el9_0.ppc64le
valgrind-3.18.1-9.el9.ppc64le
RHEL-9.0.0-20220322.0
Steps to Reproduce:
1. yum install go-toolset golang-tests -y
2. go test crypto -c
3. valgrind ./crypto.test
Looks like the issue is directly related to the vbpermq instruction. The problematic source is https://github.com/golang/go/blob/master/src/internal/bytealg/indexbyte_ppc64x.s#L208. I wrote a simple C reproducer from that code, which triggers the issue:
---
int main(){
asm("vbpermq 6,6,10");
asm("vbpermq 7,7,10");
asm("vbpermq 8,8,10");
return 0;
}
I've also noticed that if you remove one of the asm lines, so you only have two vbpermq instruction, the failure does not occur.
Similar upstream bug (related to x86_64 vfmadd231ps) https://bugs.kde.org/show_bug.cgi?id=375839
I agree with Jesus analysis as reported upstream: https://bugs.kde.org/show_bug.cgi?id=451827
This is an issue with how the vbpermq is implemented in valgrind/VEX. Which also explains why this only shows up on ppc64le.
The upstream bug has been resolved and already tested by Jesus. It will appear in the next upstream valgrind 3.19.0 release currently scheduled for Friday April 8.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (valgrind bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:8066
Description of problem: valgrind/memcheck is unable to debug go tests on rhel-9 ppc64le due to "VEX temporary storage exhausted" failure: [root@ibm-p9z-23-lp10 ~]# go test crypto -v -count=1 === RUN TestRC4OutOfBoundsWrite --- PASS: TestRC4OutOfBoundsWrite (0.00s) === RUN TestCTROutOfBoundsWrite --- PASS: TestCTROutOfBoundsWrite (0.00s) === RUN TestOFBOutOfBoundsWrite --- PASS: TestOFBOutOfBoundsWrite (0.00s) === RUN TestCFBEncryptOutOfBoundsWrite --- PASS: TestCFBEncryptOutOfBoundsWrite (0.00s) === RUN TestCFBDecryptOutOfBoundsWrite --- PASS: TestCFBDecryptOutOfBoundsWrite (0.00s) PASS ok crypto 0.004s [root@ibm-p9z-23-lp10 ~]# go test crypto -v -count=1 -c [root@ibm-p9z-23-lp10 ~]# valgrind ./crypto.test ==39625== Memcheck, a memory error detector ==39625== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==39625== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info ==39625== Command: ./crypto.test ==39625== ==39625== Warning: set address range perms: large range [0x10a80000, 0x30a80000) (noaccess) VEX temporary storage exhausted. Pool = TEMP, start 0x5989d568 curr 0x59d57e70 end 0x59d620a7 (size 5000000) vex: the `impossible' happened: VEX temporary storage exhausted. Increase N_{TEMPORARY,PERMANENT}_BYTES and recompile. vex storage: T total 244479744 bytes allocated vex storage: P total 192 bytes allocated valgrind: the 'impossible' happened: LibVEX called failure_exit(). host stacktrace: ==39625== at 0x58057790: show_sched_status_wrk (m_libcassert.c:406) ==39625== by 0x5805793F: report_and_quit (m_libcassert.c:477) ==39625== by 0x58057BCB: UnknownInlinedFun (m_libcassert.c:553) ==39625== by 0x58057BCB: vgPlain_core_panic_at (m_libcassert.c:558) ==39625== by 0x58057C1B: vgPlain_core_panic (m_libcassert.c:563) ==39625== by 0x580776C7: failure_exit (m_translate.c:761) ==39625== by 0x5818929B: vpanic (main_util.c:253) ==39625== by 0x5818934B: private_LibVEX_alloc_OOM (main_util.c:181) ==39625== by 0x582558EB: UnknownInlinedFun (main_util.h:176) ==39625== by 0x582558EB: doRegisterAllocation_v3 (host_generic_reg_alloc3.c:493) ==39625== by 0x581867C3: libvex_BackEnd (main_main.c:1133) ==39625== by 0x581867C3: LibVEX_Translate (main_main.c:1236) ==39625== by 0x5807B1F7: vgPlain_translate (m_translate.c:1830) ==39625== by 0x580CB4CF: handle_chain_me (scheduler.c:1169) ==39625== by 0x580CEB27: vgPlain_scheduler (scheduler.c:1514) ==39625== by 0x5812476F: thread_wrapper (syswrap-linux.c:101) ==39625== by 0x5812476F: run_a_thread_NORETURN (syswrap-linux.c:154) sched status: running_tid=1 Thread 1: status = VgTs_Runnable (lwpid 39625) ==39625== at 0x1000651C: indexbytebody (indexbyte_ppc64x.s:198) client stack range: [0x1FFEFF0000 0x1FFF00FFFF] client SP: 0x1FFF00E758 valgrind stack range: [0x1002FB0000 0x10030AFFFF] top usage: 10128 of 1048576 Note: see also the FAQ in the source distribution. It contains workarounds to several common problems. In particular, if Valgrind aborted or crashed after identifying problems in your program, there's a good chance that fixing those problems will prevent Valgrind aborting or crashing, especially if it happened in m_mallocfree.c. If that doesn't help, please report this bug to: www.valgrind.org In the bug report, send all the above text, the valgrind version, and what OS and version you are using. Thanks. [root@ibm-p9z-23-lp10 ~]# objdump -d crypto.test | grep -iC3 1000651C: 10006510: ff ff 60 38 li r3,-1 10006514: 00 00 6e f8 std r3,0(r14) 10006518: 20 00 80 4e blr 1000651c: 4c 55 c6 10 vbpermq v6,v6,v10 10006520: 4c 55 e7 10 vbpermq v7,v7,v10 10006524: 4c 55 08 11 vbpermq v8,v8,v10 10006528: 4c 55 29 11 vbpermq v9,v9,v10 [root@ibm-p9z-23-lp10 ~]# This issue was not seen on other rhel-9 supported arches. Version-Release number of selected component (if applicable): golang-1.17.7-1.el9_0.ppc64le valgrind-3.18.1-9.el9.ppc64le RHEL-9.0.0-20220322.0 Steps to Reproduce: 1. yum install go-toolset golang-tests -y 2. go test crypto -c 3. valgrind ./crypto.test