Hide Forgot
Description of problem: On an old i686 machine which does not support the NOPL instruction, the latest version of openssl breaks both incoming and outgoing ssh. For example, attempting outgoing ssh gives an error to the effect of "Illegal instruction (core dumped)", and starting sshd fails. Downgrading to 1:openssl-1.0.1e-4.fc19.i686 fixes it. Version-Release number of selected component (if applicable): openssl-1.0.1e-28.fc19.i686 How reproducible: always Additional info: [root@emachines-pc ~]# cat /proc/cpuinfo processor : 0 vendor_id : CyrixInstead cpu family : 6 model : 0 model name : 06/00 stepping : 1 cpu MHz : 250.049 fdiv_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr cx8 pge cmov mmx bogomips : 500.09 clflush size : 32 cache_alignment : 32 address sizes : 32 bits physical, 32 bits virtual power management: [root@emachines-pc ~]#
Here is a stack trace from the core dump corresponding to the "Illegal instruction (core dumped)" message. I just installed openssl-debuginfo. I'm not sure if this is related to the machine not supporting NOPL. I have another, more modern 32-bit machine and this bug does not occur on it, or on my 64-bit machine. [root@emachines-pc ~]# gdb ssh core.2100 GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ssh...Reading symbols from /usr/bin/ssh...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 2100] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". Core was generated by `ssh 192.168.0.2'. Program terminated with signal 4, Illegal instruction. #0 0xb7534955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 332 movl $8,%ecx Missing separate debuginfos, use: debuginfo-install openssh-clients-6.2p2-5.fc19.i686 (gdb)
With all debuginfo packages installed (which took hours, BTW), basically the same output. [root@emachines-pc ~]# gdb ssh core.5739 GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done. done. [New LWP 5739] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". Core was generated by `ssh 192.168.0.2'. Program terminated with signal 4, Illegal instruction. #0 0xb7584955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 332 movl $8,%ecx (gdb)
[root@emachines-pc ~]# ssh 192.168.0.2 Illegal instruction (core dumped) [root@emachines-pc ~]# gdb ssh core.1597 GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done. done. [New LWP 1597] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". Core was generated by `ssh 192.168.0.2'. Program terminated with signal 4, Illegal instruction. #0 0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 332 movl $8,%ecx (gdb) thread apply all bt full Thread 1 (Thread 0xb6def9c0 (LWP 1597)): #0 0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 No locals. #1 0xb75abaca in get_random_bytes (buf=0xbf8e56a8 "\004", num=20) at eng_rdrand.c:69 rnd = 3073238888 #2 0xb75b89f1 in RAND_bytes (buf=buf@entry=0xbf8e56a8 "\004", num=num@entry=20) at rand_lib.c:164 meth = <optimized out> #3 0xb7795a80 in arc4random_stir () at bsd-arc4random.c:60 rand_buf = "\004\000\000\000\000\247ö \344-\267\350\323緉\300v\267" #4 0xb7795af3 in arc4random () at bsd-arc4random.c:48 r = 0 rp = 0xbf8e56e8 #5 0xb7786f94 in kex_send_kexinit (kex=0xb7e7d3c0) at kex.c:219 rnd = <optimized out> cookie = 0xb7e7fa28 "" i = 0 kex = 0xb7e7d3c0 #6 0xb7787a1b in kex_setup (proposal=proposal@entry=0xb77b70c0 <myproposal>) at kex.c:278 kex = 0xb7e7d3c0 ---Type <return> to continue, or q <return> to quit---
Sorry for the incomplete output in the previous post. [root@emachines-pc ~]# ssh 192.168.0.2 Illegal instruction (core dumped) [root@emachines-pc ~]# gdb ssh core.1597 GNU gdb (GDB) Fedora 7.6.1-42.fc19 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ssh...Reading symbols from /usr/lib/debug/usr/bin/ssh.debug...done. done. [New LWP 1597] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/libthread_db.so.1". Core was generated by `ssh 192.168.0.2'. Program terminated with signal 4, Illegal instruction. #0 0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 332 movl $8,%ecx (gdb) thread apply all bt full Thread 1 (Thread 0xb6def9c0 (LWP 1597)): #0 0xb7541955 in OPENSSL_ia32_rdrand () at x86cpuid.s:332 No locals. #1 0xb75abaca in get_random_bytes (buf=0xbf8e56a8 "\004", num=20) at eng_rdrand.c:69 rnd = 3073238888 #2 0xb75b89f1 in RAND_bytes (buf=buf@entry=0xbf8e56a8 "\004", num=num@entry=20) at rand_lib.c:164 meth = <optimized out> #3 0xb7795a80 in arc4random_stir () at bsd-arc4random.c:60 rand_buf = "\004\000\000\000\000\247ö \344-\267\350\323緉\300v\267" #4 0xb7795af3 in arc4random () at bsd-arc4random.c:48 r = 0 rp = 0xbf8e56e8 #5 0xb7786f94 in kex_send_kexinit (kex=0xb7e7d3c0) at kex.c:219 rnd = <optimized out> cookie = 0xb7e7fa28 "" i = 0 kex = 0xb7e7d3c0 #6 0xb7787a1b in kex_setup (proposal=proposal@entry=0xb77b70c0 <myproposal>) at kex.c:278 kex = 0xb7e7d3c0 ---Type <return> to continue, or q <return> to quit--- #7 0xb77604d5 in ssh_kex2 (host=host@entry=0xb7e7d308 "192.168.0.2", hostaddr=hostaddr@entry=0xb77b8640 <hostaddr>, port=port@entry=22) at sshconnect2.c:250 kex = <optimized out> orig = <optimized out> gss = 0x0 gss_host = 0x0 #8 0xb775c745 in ssh_login ( sensitive=sensitive@entry=0xb77b8620 <sensitive_data>, orighost=0xb7e6d9d0 "192.168.0.2", hostaddr=hostaddr@entry=0xb77b8640 <hostaddr>, port=22, pw=pw@entry=0xb7e6c320, timeout_ms=-1000) at sshconnect.c:1203 host = 0xb7e7d308 "192.168.0.2" cp = <optimized out> server_user = 0xb7e72158 "root" local_user = 0xb7e7d208 "root" #9 0xb774fa01 in main (ac=<optimized out>, av=<optimized out>) at ssh.c:930 i = <optimized out> opt = <optimized out> exit_status = <optimized out> use_syslog = <optimized out> p = <optimized out> cp = <optimized out> ---Type <return> to continue, or q <return> to quit--- line = <optimized out> argv0 = <optimized out> buf = "/root/.ssh\000config", '\000' <repeats 703 times>... host_arg = <optimized out> thishost = "emachines-pc", '\000' <repeats 1012 times> shorthost = "emachines-pc", '\000' <repeats 1012 times> portstr = "22", '\000' <repeats 29 times> st = {st_dev = 64768, __pad1 = 0, __st_ino = 6422539, st_mode = 16832, st_nlink = 2, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = { tv_sec = 1382438789, tv_nsec = 805595048}, st_mtim = { tv_sec = 1381742119, tv_nsec = 413687496}, st_ctim = { tv_sec = 1381742119, tv_nsec = 413687496}, st_ino = 6422539} pw = 0xb7e6c320 dummy = 0 timeout_ms = -1000 sp = <optimized out> fwd = {listen_host = 0x0, listen_port = 0, connect_host = 0x0, connect_port = 0, allocated_port = 0, handle = 0} (gdb)
I wonder whether we support such old CPU at all. Although it might be possible that the base/core packages until now worked. What's weird that the OPENSSL_ia32_rdrand() is called at all because this function should be called only in case the RDRAND instruction is detected. So it seems the detection gives false positive on your CPU for some reason. exporting OPENSSL_ia32cap=~0x4000000000000000 environment variable should workaround it.
(In reply to Tomas Mraz from comment #5) > I wonder whether we support such old CPU at all. Although it might be > possible that the base/core packages until now worked. As I understand it, this CPU should be supported (barely), but please confirm. (I can't find the exact CPU requirements at the moment, but offhand I know that cmov is required, NOPL is not, and up to now the only version of Fedora which I couldn't run was one which was known to be broken due to requiring NOPL.) > What's weird that the OPENSSL_ia32_rdrand() is called at all because this > function should be called only in case the RDRAND instruction is detected. > So it seems the detection gives false positive on your CPU for some reason. > > exporting OPENSSL_ia32cap=~0x4000000000000000 environment variable should > workaround it. Yes, it works for allowing outgoing ssh. I just have to figure out now where to set the environment for sshd for when it starts automatically, so I can use incoming as well.
You can try to set the environment variable in /etc/sysconfig/sshd. It is clear that the problem is actually with the OPENSSL_ia32_cpuid() function misdetecting on your CPU. Could you try to work it out with upstream developers by creating issue on upstream issue tracker? Just send e-mail to rt@openssl.org describing your cpu and that it misdetects that rdrand instruction is present.
Same bug (and workaround) in openssl-1.0.1e-29.fc19.i686. I sent a "Bug report" email to rt@openssl.org but have not received any acknowledgement. Someone else in https://lists.fedoraproject.org/pipermail/users/2013-October/441959.html described what might be the same problem on different hardware, I asked them to report here.
Took a while, but I was assigned an openssl.org ticket: https://rt.openssl.org/Ticket/Display.html?id=3151
From the above ticket: This was reported in http://rt.openssl.org/Ticket/Display.html?id=3005&user=guest&pass=guest and already fixed. There is a patch http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=5702e965d759dde8a098d8108660721ba2b93a7d in this ticket which is supposed to fix it. I'm willing to test a patched build.
From looking at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=699692 which was referenced in ticket #3005, the patch was tested on a Debian build and worked for the original reporter. Do I need to do anything for this patch to get into Fedora? (I can't reply on the openssl.org tickets, only here.)
openssl-1.0.1e-30.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc18
openssl-1.0.1e-30.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc19
openssl-1.0.1e-30.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/openssl-1.0.1e-30.fc20
Thanks! Confirmed that openssl-1.0.1e-30.fc19 fixes incoming and outgoing ssh on my machine. Since it's running F19, that's the only one I can test, but the patch looks simple, so that should be enough.
openssl-1.0.1e-30.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
openssl-1.0.1e-30.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
openssl-1.0.1e-30.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.