Bug 1022346 - openssl-1.0.1e-28.fc19.i686 broken on Cyrix CPU
openssl-1.0.1e-28.fc19.i686 broken on Cyrix CPU
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: openssl (Show other bugs)
19
i686 Linux
unspecified Severity high
: ---
: ---
Assigned To: Tomas Mraz
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-23 02:02 EDT by Andre Robatino
Modified: 2013-11-13 22:36 EST (History)
2 users (show)

See Also:
Fixed In Version: openssl-1.0.1e-30.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1024274 (view as bug list)
Environment:
Last Closed: 2013-10-30 23:00:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andre Robatino 2013-10-23 02:02:52 EDT
Description of problem:
On an old i686 machine which does not support the NOPL instruction, the latest version of openssl breaks both incoming and outgoing ssh. For example, attempting outgoing ssh gives an error to the effect of "Illegal instruction (core dumped)", and starting sshd fails. Downgrading to 1:openssl-1.0.1e-4.fc19.i686 fixes it.

Version-Release number of selected component (if applicable):
openssl-1.0.1e-28.fc19.i686

How reproducible:
always

Additional info:
[root@emachines-pc ~]# cat /proc/cpuinfo 
processor	: 0
vendor_id	: CyrixInstead
cpu family	: 6
model		: 0
model name	: 06/00
stepping	: 1
cpu MHz		: 250.049
fdiv_bug	: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu de tsc msr cx8 pge cmov mmx
bogomips	: 500.09
clflush size	: 32
cache_alignment	: 32
address sizes	: 32 bits physical, 32 bits virtual
power management:

[root@emachines-pc ~]#
Comment 1 Andre Robatino 2013-10-24 03:24:40 EDT
Here is a stack trace from the core dump corresponding to the "Illegal instruction (core dumped)" message. I just installed openssl-debuginfo. I'm not sure if this is related to the machine not supporting NOPL. I have another, more modern 32-bit machine and this bug does not occur on it, or on my 64-bit machine.

[root@emachines-pc ~]# gdb ssh core.2100
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ssh...Reading symbols from /usr/bin/ssh...(no debugging symbols found)...done.
(no debugging symbols found)...done.
[New LWP 2100]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `ssh 192.168.0.2'.
Program terminated with signal 4, Illegal instruction.
#0  0xb7534955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
332		movl	$8,%ecx
Missing separate debuginfos, use: debuginfo-install openssh-clients-6.2p2-5.fc19.i686
(gdb)
Comment 2 Andre Robatino 2013-10-24 12:27:43 EDT
With all debuginfo packages installed (which took hours, BTW), basically the same output.

[root@emachines-pc ~]# gdb ssh core.5739 
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done.
done.
[New LWP 5739]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `ssh 192.168.0.2'.
Program terminated with signal 4, Illegal instruction.
#0  0xb7584955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
332		movl	$8,%ecx
(gdb)
Comment 3 Andre Robatino 2013-10-25 06:58:06 EDT
[root@emachines-pc ~]# ssh 192.168.0.2
Illegal instruction (core dumped)
[root@emachines-pc ~]# gdb ssh core.1597 
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done.
done.
[New LWP 1597]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `ssh 192.168.0.2'.
Program terminated with signal 4, Illegal instruction.
#0  0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
332		movl	$8,%ecx
(gdb) thread apply all bt full

Thread 1 (Thread 0xb6def9c0 (LWP 1597)):
#0  0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
No locals.
#1  0xb75abaca in get_random_bytes (buf=0xbf8e56a8 "\004", num=20)
    at eng_rdrand.c:69
        rnd = 3073238888
#2  0xb75b89f1 in RAND_bytes (buf=buf@entry=0xbf8e56a8 "\004", 
    num=num@entry=20) at rand_lib.c:164
        meth = <optimized out>
#3  0xb7795a80 in arc4random_stir () at bsd-arc4random.c:60
        rand_buf = "\004\000\000\000\000\247ö \344-\267\350\323緉\300v\267"
#4  0xb7795af3 in arc4random () at bsd-arc4random.c:48
        r = 0
        rp = 0xbf8e56e8
#5  0xb7786f94 in kex_send_kexinit (kex=0xb7e7d3c0) at kex.c:219
        rnd = <optimized out>
        cookie = 0xb7e7fa28 ""
        i = 0
        kex = 0xb7e7d3c0
#6  0xb7787a1b in kex_setup (proposal=proposal@entry=0xb77b70c0 <myproposal>)
    at kex.c:278
        kex = 0xb7e7d3c0
---Type <return> to continue, or q <return> to quit---
Comment 4 Andre Robatino 2013-10-25 07:01:20 EDT
Sorry for the incomplete output in the previous post.

[root@emachines-pc ~]# ssh 192.168.0.2
Illegal instruction (core dumped)
[root@emachines-pc ~]# gdb ssh core.1597 
GNU gdb (GDB) Fedora 7.6.1-42.fc19
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done.
done.
[New LWP 1597]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `ssh 192.168.0.2'.
Program terminated with signal 4, Illegal instruction.
#0  0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
332		movl	$8,%ecx
(gdb) thread apply all bt full

Thread 1 (Thread 0xb6def9c0 (LWP 1597)):
#0  0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332
No locals.
#1  0xb75abaca in get_random_bytes (buf=0xbf8e56a8 "\004", num=20)
    at eng_rdrand.c:69
        rnd = 3073238888
#2  0xb75b89f1 in RAND_bytes (buf=buf@entry=0xbf8e56a8 "\004", 
    num=num@entry=20) at rand_lib.c:164
        meth = <optimized out>
#3  0xb7795a80 in arc4random_stir () at bsd-arc4random.c:60
        rand_buf = "\004\000\000\000\000\247ö \344-\267\350\323緉\300v\267"
#4  0xb7795af3 in arc4random () at bsd-arc4random.c:48
        r = 0
        rp = 0xbf8e56e8
#5  0xb7786f94 in kex_send_kexinit (kex=0xb7e7d3c0) at kex.c:219
        rnd = <optimized out>
        cookie = 0xb7e7fa28 ""
        i = 0
        kex = 0xb7e7d3c0
#6  0xb7787a1b in kex_setup (proposal=proposal@entry=0xb77b70c0 <myproposal>)
    at kex.c:278
        kex = 0xb7e7d3c0
---Type <return> to continue, or q <return> to quit---
#7  0xb77604d5 in ssh_kex2 (host=host@entry=0xb7e7d308 "192.168.0.2", 
    hostaddr=hostaddr@entry=0xb77b8640 <hostaddr>, port=port@entry=22)
    at sshconnect2.c:250
        kex = <optimized out>
        orig = <optimized out>
        gss = 0x0
        gss_host = 0x0
#8  0xb775c745 in ssh_login (
    sensitive=sensitive@entry=0xb77b8620 <sensitive_data>, 
    orighost=0xb7e6d9d0 "192.168.0.2", 
    hostaddr=hostaddr@entry=0xb77b8640 <hostaddr>, port=22, 
    pw=pw@entry=0xb7e6c320, timeout_ms=-1000) at sshconnect.c:1203
        host = 0xb7e7d308 "192.168.0.2"
        cp = <optimized out>
        server_user = 0xb7e72158 "root"
        local_user = 0xb7e7d208 "root"
#9  0xb774fa01 in main (ac=<optimized out>, av=<optimized out>) at ssh.c:930
        i = <optimized out>
        opt = <optimized out>
        exit_status = <optimized out>
        use_syslog = <optimized out>
        p = <optimized out>
        cp = <optimized out>
---Type <return> to continue, or q <return> to quit---
        line = <optimized out>
        argv0 = <optimized out>
        buf = "/root/.ssh\000config", '\000' <repeats 703 times>...
        host_arg = <optimized out>
        thishost = "emachines-pc", '\000' <repeats 1012 times>
        shorthost = "emachines-pc", '\000' <repeats 1012 times>
        portstr = "22", '\000' <repeats 29 times>
        st = {st_dev = 64768, __pad1 = 0, __st_ino = 6422539, st_mode = 16832, 
          st_nlink = 2, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, 
          st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {
            tv_sec = 1382438789, tv_nsec = 805595048}, st_mtim = {
            tv_sec = 1381742119, tv_nsec = 413687496}, st_ctim = {
            tv_sec = 1381742119, tv_nsec = 413687496}, st_ino = 6422539}
        pw = 0xb7e6c320
        dummy = 0
        timeout_ms = -1000
        sp = <optimized out>
        fwd = {listen_host = 0x0, listen_port = 0, connect_host = 0x0, 
          connect_port = 0, allocated_port = 0, handle = 0}
(gdb)
Comment 5 Tomas Mraz 2013-10-25 08:09:16 EDT
I wonder whether we support such old CPU at all. Although it might be possible that the base/core packages until now worked.

What's weird that the OPENSSL_ia32_rdrand() is called at all because this function should be called only in case the RDRAND instruction is detected. So it seems the detection gives false positive on your CPU for some reason.

exporting OPENSSL_ia32cap=~0x4000000000000000 environment variable should workaround it.
Comment 6 Andre Robatino 2013-10-25 08:59:35 EDT
(In reply to Tomas Mraz from comment #5)
> I wonder whether we support such old CPU at all. Although it might be
> possible that the base/core packages until now worked.

As I understand it, this CPU should be supported (barely), but please confirm. (I can't find the exact CPU requirements at the moment, but offhand I know that cmov is required, NOPL is not, and up to now the only version of Fedora which I couldn't run was one which was known to be broken due to requiring NOPL.)

> What's weird that the OPENSSL_ia32_rdrand() is called at all because this
> function should be called only in case the RDRAND instruction is detected.
> So it seems the detection gives false positive on your CPU for some reason.
> 
> exporting OPENSSL_ia32cap=~0x4000000000000000 environment variable should
> workaround it.

Yes, it works for allowing outgoing ssh. I just have to figure out now where to set the environment for sshd for when it starts automatically, so I can use incoming as well.
Comment 7 Tomas Mraz 2013-10-25 09:26:13 EDT
You can try to set the environment variable in /etc/sysconfig/sshd.

It is clear that the problem is actually with the OPENSSL_ia32_cpuid() function misdetecting on your CPU. Could you try to work it out with upstream developers by creating issue on upstream issue tracker? Just send e-mail to rt@openssl.org describing your cpu and that it misdetects that rdrand instruction is present.
Comment 8 Andre Robatino 2013-10-26 04:59:57 EDT
Same bug (and workaround) in openssl-1.0.1e-29.fc19.i686. I sent a "Bug report" email to rt@openssl.org but have not received any acknowledgement. Someone else in https://lists.fedoraproject.org/pipermail/users/2013-October/441959.html described what might be the same problem on different hardware, I asked them to report here.
Comment 9 Andre Robatino 2013-10-28 04:43:47 EDT
Took a while, but I was assigned an openssl.org ticket:

https://rt.openssl.org/Ticket/Display.html?id=3151
Comment 10 Andre Robatino 2013-10-28 11:17:01 EDT
From the above ticket:

This was reported in
http://rt.openssl.org/Ticket/Display.html?id=3005&user=guest&pass=guest
and already fixed.

There is a patch http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=5702e965d759dde8a098d8108660721ba2b93a7d in this ticket which is supposed to fix it. I'm willing to test a patched build.
Comment 11 Andre Robatino 2013-10-28 12:02:47 EDT
From looking at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=699692 which was referenced in ticket #3005, the patch was tested on a Debian build and worked for the original reporter. Do I need to do anything for this patch to get into Fedora? (I can't reply on the openssl.org tickets, only here.)
Comment 12 Fedora Update System 2013-10-29 12:42:09 EDT
openssl-1.0.1e-30.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc18
Comment 13 Fedora Update System 2013-10-29 12:42:37 EDT
openssl-1.0.1e-30.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc19
Comment 14 Fedora Update System 2013-10-29 12:43:02 EDT
openssl-1.0.1e-30.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc20
Comment 15 Andre Robatino 2013-10-29 13:27:03 EDT
Thanks! Confirmed that openssl-1.0.1e-30.fc19 fixes incoming and outgoing ssh on my machine. Since it's running F19, that's the only one I can test, but the patch looks simple, so that should be enough.
Comment 16 Fedora Update System 2013-10-30 23:00:12 EDT
openssl-1.0.1e-30.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2013-11-10 01:52:37 EST
openssl-1.0.1e-30.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 18 Fedora Update System 2013-11-13 22:36:39 EST
openssl-1.0.1e-30.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.