Bug 810992

Created attachment 576300 [details]
the relevant part of the log

This seems to be a valgrind issue.  It panics on execution of OPENSSL_cpuid_setup (in /usr/lib/libcrypto.so.1.0.1).

Same issue occurs against valgrind-3.8.1:
http://ppc.koji.fedoraproject.org/koji/getfile?taskID=730100&name=build.log

../libtool --mode=execute /usr/bin/valgrind --tool=memcheck --leak-check=yes --num-callers=16 --log-file=log/valgrind582 ./libtest/lib582 Sftp://127.0.0.1:3299/builddir/build/BUILD/curl-7.27.0/tests/log/upload582.txt /builddir/build/BUILD/curl-7.27.0/tests/log/file582.txt mockbuild: >log/stdout582 2>log/stderr582
lib582 returned 1, when expecting 0
 exit FAILED
== Contents of files in the log/ dir after test 582
=== Start of file file582.txt
 Moooooooooooo
  upload this
=== End of file file582.txt
=== Start of file stderr582
 URL: Sftp://127.0.0.1:3299/builddir/build/BUILD/curl-7.27.0/tests/log/upload582.txt
 Set to upload 27 bytes
 * About to connect() to 127.0.0.1 port 3299 (#0)
 *   Trying 127.0.0.1...
 * connected
 * Connected to 127.0.0.1 (127.0.0.1) port 3299 (#0)
=== End of file stderr582
=== Start of file valgrind582
 ==44318== Memcheck, a memory error detector
 ==44318== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
 ==44318== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
 ==44318== Command: /builddir/build/BUILD/curl-7.27.0/tests/libtest/.libs/lt-lib582 Sftp://127.0.0.1:3299/builddir/build/BUILD/curl-7.27.0/tests/log/upload582.txt /builddir/build/BUILD/curl-7.27.0/tests/log/file582.txt mockbuild:
 ==44318== Parent PID: 44317
 ==44318== 
  op name: And64
  op type is (I64,I64) -> I64
 arg tys are (I32,I64)
 IR SANITY CHECK FAILURE
 IRSB {
    t0:I32   t1:I32   t2:I32   t3:I32   t4:F64   t5:F64   t6:I32   t7:I64
    t8:I32   t9:I32   t10:I32   t11:I32   t12:I32   t13:I32   t14:I32   t15:I32
    t16:I32   t17:I32   t18:I32   t19:I32   t20:I32   
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    IR-NoOp
    ------ IMark(0xF01D790, 4, 0) ------
    PUT(1172) = 0xF01D794:I32
    PUT(1168) = 0xF01D360:I32
    ------ IMark(0xF01D360, 4, 0) ------
    t9 = 8Uto32(GET:I8(1200))
    t8 = And32(t9,0x3:I32)
    t5 = GET:F64(160)
    t7 = ReinterpF64asI64(t5)
    t4 = I64StoF64(Xor32(t8,And32(Shl32(t8,0x1:I8),0x2:I32)),t7)
    PUT(160) = t4
    PUT(1168) = 0xF01D364:I32
    ------ IMark(0xF01D364, 4, 0) ------
    t10 = GET:I32(16)
    t12 = GET:I32(16)
    t11 = And64(Mux0X(And8(0x20:I8,0x1F:I8),t10,Or32(Shl32(t10,And8(0x20:I8,0x1F:I8)),Shr32(t10,Sub8(0x20:I8,And8(0x20:I8,0x1F:I8))))),0xFFFFFFFF:I64)
    PUT(16) = t11
    PUT(1168) = 0xF01D368:I32
    ------ IMark(0xF01D368, 4, 0) ------
    t18 = 0xFFFFFFFF:I32
    t15 = t18
    t19 = 0x1:I32
    t16 = t19
    t14 = And32(t16,t15)
    t17 = And32(GET:I32(1172),0xFFFFFFFC:I32)
    if (CmpEQ32(t14,0x0:I32)) { PUT(1168) = 0xF01D36C:I32; exit-Boring } 
    PUT(1168) = t17
    PUT(1168) = GET:I32(1168); exit-Return
 }
 IN STATEMENT:
 t11 = And64(Mux0X(And8(0x20:I8,0x1F:I8),t10,Or32(Shl32(t10,And8(0x20:I8,0x1F:I8)),Shr32(t10,Sub8(0x20:I8,And8(0x20:I8,0x1F:I8))))),0xFFFFFFFF:I64)
 ERROR = Iex.Binop: arg tys don't match op tys
 ... additional details precede BB printout
 vex: the `impossible' happened:
    sanityCheckFail: exiting due to bad IR
 vex storage: T total 163342056 bytes allocated
 vex storage: P total 96 bytes allocated
 valgrind: the 'impossible' happened:
    LibVEX called failure_exit().
 ==44318==    at 0x380649E8: report_and_quit (m_libcassert.c:235)
 ==44318==    by 0x38064C0F: vgPlain_core_panic_at (m_libcassert.c:319)
 ==44318==    by 0x38064C43: vgPlain_core_panic (m_libcassert.c:329)
 ==44318==    by 0x38081AAB: failure_exit (m_translate.c:708)
 ==44318==    by 0x3812101F: vpanic (main_util.c:226)
 ==44318==    by 0x38129EDB: sanityCheckFail (ir_defs.c:3232)
 ==44318==    by 0x3812CDDB: tcExpr (ir_defs.c:3544)
 ==44318==    by 0x3812DFE3: sanityCheckIRSB (ir_defs.c:3649)
 ==44318==    by 0x3811EE9B: LibVEX_Translate (main_main.c:655)
 ==44318==    by 0x3808419F: vgPlain_translate (m_translate.c:1559)
 ==44318==    by 0x380B4C2F: handle_chain_me (scheduler.c:1019)
 ==44318==    by 0x380B695F: vgPlain_scheduler (scheduler.c:1323)
 ==44318==    by 0x380CB3EF: run_a_thread_NORETURN (syswrap-linux.c:103)
 sched status:
   running_tid=1
 Thread 1: status = VgTs_Runnable
 ==44318==    at 0xF01D790: OPENSSL_cpuid_setup (in /usr/lib/libcrypto.so.1.0.1c)
 ==44318==    by 0xF0A2E23: OPENSSL_add_all_algorithms_noconf (in /usr/lib/libcrypto.so.1.0.1c)
 ==44318==    by 0xF402227: libssh2_init (in /usr/lib/libssh2.so.1.0.1)
 ==44318==    by 0xF4022F7: ??? (in /usr/lib/libssh2.so.1.0.1)
 ==44318==    by 0xF3F35F7: libssh2_session_init_ex (in /usr/lib/libssh2.so.1.0.1)
 ==44318==    by 0xFED9927: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFEB6BA3: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFEB6F27: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFEB6FF3: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFEC95FF: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFECA427: ??? (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0xFECA6D7: curl_multi_socket_action (in /builddir/build/BUILDROOT/curl-7.27.0-3.fc18.ppc/usr/lib/libcurl.so.4.2.0)
 ==44318==    by 0x10000AEB: notifyCurl (lib582.c:205)
 ==44318==    by 0x10001AF7: test (lib582.c:336)
 ==44318==    by 0xFC014B3: generic_start_main.isra.0 (in /usr/lib/libc-2.16.so)
 ==44318==    by 0xFC0166F: (below main) (in /usr/lib/libc-2.16.so)
 Note: see also the FAQ in the source distribution.
 It contains workarounds to several common problems.
 In particular, if Valgrind aborted or crashed after
 identifying problems in your program, there's a good chance
 that fixing those problems will prevent Valgrind aborting or
 crashing, especially if it happened in m_mallocfree.c.
 If that doesn't help, please report this bug to: www.valgrind.org
 In the bug report, send all the above text, the valgrind
 version, and what OS and version you are using.  Thanks.
=== End of file valgrind582

Simple reproducer:

# cat testppc.c 
#include <openssl/evp.h>

int
main (int argc, char **argv)
{
  OpenSSL_add_all_algorithms();
  // Do nothing
  EVP_cleanup();
  return 0;
}

# gcc -m32 -lssl -lcrypto -o testppc testppc.c 

# valgrind ./testppc
==27547== Memcheck, a memory error detector
==27547== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==27547== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==27547== Command: ./testppc
==27547== 
 op name: And64
 op type is (I64,I64) -> I64
arg tys are (I32,I64)

IR SANITY CHECK FAILURE

IRSB {
   t0:I32   t1:I32   t2:I32   t3:I32   t4:F64   t5:F64   t6:I32   t7:I64
   t8:I32   t9:I32   t10:I32   t11:I32   t12:I32   t13:I32   t14:I32   t15:I32
   t16:I32   t17:I32   t18:I32   t19:I32   t20:I32   

   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   IR-NoOp
   ------ IMark(0xFCDD860, 4, 0) ------
   PUT(1156) = 0xFCDD864:I32
   ------ IMark(0xFCDD430, 4, 0) ------
   PUT(1152) = 0xFCDD430:I32
   t9 = GET:I32(1184)
   t8 = And32(t9,0x3:I32)
   t5 = GET:F64(144)
   t7 = ReinterpF64asI64(t5)
   t4 = I64StoF64(Xor32(t8,And32(Shl32(t8,0x1:I8),0x2:I32)),t7)
   PUT(144) = t4
   ------ IMark(0xFCDD434, 4, 0) ------
   PUT(1152) = 0xFCDD434:I32
   t10 = GET:I32(0)
   t12 = GET:I32(0)
   t11 = And64(Mux0X(And8(0x20:I8,0x1F:I8),t10,Or32(Shl32(t10,And8(0x20:I8,0x1F:I8)),Shr32(t10,Sub8(0x20:I8,And8(0x20:I8,0x1F:I8))))),0xFFFFFFFF:I64)
   PUT(0) = t11
   ------ IMark(0xFCDD438, 4, 0) ------
   PUT(1152) = 0xFCDD438:I32
   t18 = 0xFFFFFFFF:I32
   t15 = t18
   t19 = 0x1:I32
   t16 = t19
   t14 = And32(t16,t15)
   t17 = And32(GET:I32(1156),0xFFFFFFFC:I32)
   if (CmpEQ32(t14,0x0:I32)) goto {Boring} 0xFCDD43C:I32
   goto {Return} t17
}

IN STATEMENT:

t11 = And64(Mux0X(And8(0x20:I8,0x1F:I8),t10,Or32(Shl32(t10,And8(0x20:I8,0x1F:I8)),Shr32(t10,Sub8(0x20:I8,And8(0x20:I8,0x1F:I8))))),0xFFFFFFFF:I64)

ERROR = Iex.Binop: arg tys don't match op tys
... additional details precede BB printout



vex: the `impossible' happened:
   sanityCheckFail: exiting due to bad IR
vex storage: T total 69105992 bytes allocated
vex storage: P total 144 bytes allocated

valgrind: the 'impossible' happened:
   LibVEX called failure_exit().
==27547==    at 0x38053D68: report_and_quit (m_libcassert.c:210)
==27547==    by 0x38053F8F: vgPlain_core_panic_at (m_libcassert.c:294)
==27547==    by 0x38053FC3: vgPlain_core_panic (m_libcassert.c:304)
==27547==    by 0x3807056B: failure_exit (m_translate.c:700)
==27547==    by 0x38107AEF: vpanic (main_util.c:226)
==27547==    by 0x3810F72B: sanityCheckFail (ir_defs.c:2881)
==27547==    by 0x3811249B: tcExpr (ir_defs.c:3185)
==27547==    by 0x381136A3: sanityCheckIRSB (ir_defs.c:3288)
==27547==    by 0x38105F97: LibVEX_Translate (main_main.c:561)
==27547==    by 0x38072BB3: vgPlain_translate (m_translate.c:1544)
==27547==    by 0x380A2FEB: vgPlain_scheduler (scheduler.c:901)
==27547==    by 0x380B7C13: run_a_thread_NORETURN (syswrap-linux.c:98)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==27547==    at 0xFCDD860: OPENSSL_cpuid_setup (in /usr/lib/libcrypto.so.1.0.1c)
==27547==    by 0xFD64FF3: OPENSSL_add_all_algorithms_noconf (in /usr/lib/libcrypto.so.1.0.1c)
==27547==    by 0x10000687: main (in /root/testppc)


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

With yum install openssl-debuginfo.ppc we see:

Thread 1: status = VgTs_Runnable
==28481==    at 0xFCDD860: OPENSSL_cpuid_setup (ppccap.c:96)
==28481==    by 0xFD64FF3: OPENSSL_add_all_algorithms_noconf (c_all.c:82)
==28481==    by 0x10000687: main (in /root/testppc)

Which is /usr/src/debug/openssl-1.0.1c/crypto/ppccap.c:

        if (sizeof(size_t)==4)
                {
                if (sigsetjmp(ill_jmp,1) == 0)
                        {
line 96 ===>            OPENSSL_ppc64_probe();
                        OPENSSL_ppccap_P |= PPC_FPU64;
                        }
                }

And ppc64_probe is defined in /usr/src/debug/openssl-1.0.1c/crypto/ppccpuid.s as:

.globl  OPENSSL_ppc64_probe
.type   OPENSSL_ppc64_probe,@function
.align  4
OPENSSL_ppc64_probe:
        fcfid   1,1
        rldicl  0,0,32,32
        blr
.long   0
.byte   0,12,0x14,0,0,0,0,0

There is a nice analysis of this issue in upstream bugzilla:
https://bugs.kde.org/show_bug.cgi?id=308573

See also this discussion in the openssl bug tracker about the probing method used: https://rt.openssl.org/Ticket/Display.html?id=2897&user=guest&pass=guest

I think the question is what the hw really does with the instruction in 32-bit mode.  Does it take 32-bit register, zero extend, do the rotation in 64-bit and truncate to 32-bit?  Or does it take the whole 64-bit register (what is the content of the upper bits?).  If the former, it isn't hard for the 64-bit only dis_rot_int instructions to do mode64 ? mkexpr(rS) : unop(Iop_32Uto64, mkexpr(rS))
as argument to ROTL etc., and finally at the end do unop(Iop_64to32, ...) to truncate it back.

My team member, Carl Love, and I have been investigating this issue, mostly from the Valgrind perspective since we have responsibility for supporting Valgrind on the ppc64 architecture.  While we realized that Valgrind was misbehaving in this circumstance and needed fixing, we also thought there might be an issue with how openssl was using (or possibly mis-using) a certain instruction that is from the 64-bit category.  This bug occurs only when the programs involved are compiled as 32-bit.  We opened bug #865124 to address the possibility of an openssl bug, but that bug has now been closed as NOTABUG. As noted in the closing comment, a workaround for this valgrind issue (i.e., choking on the rldicl instruction) is to set the OPENSSL_ppccap environment variable to 0 prior running the application under valgrind's control.

We are working on a patch to fix Valgrind so that it will not die when encountering rldicl (and other 64-bit cateogry instructions) when in 32-bit mode.  When this patch is accepted upstream, we'll update this bug.

Backported patches for upstream bugs KDE#308573 and KDE#309425 to valgrind rawhide package. When valgrind-3.8.1-5 hits the ppc build roots could you try enabling valgrind in the curl package again and do a new build to see if everything now works as expected?

confirmed that a curl build with valgrind enabled works now

valgrind-3.8.1-9.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/valgrind-3.8.1-9.fc18

valgrind-3.8.1-9.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.