That sounds like OpenSSL isn't built appropriately for your CPU. Although I'm slightly confused because I don't see why we end up in OpenSSL code at all. We use GnuTLS. Can you show a backtrace?

The question is whether the SIGILL you're seeing in the gdb is the SIGILL that is causing the crash. The problem is that openssl library on ARM uses SIGILL to test for the CPU features supported. If the instruction is not supported the SIGILL is properly handled by the library and the instruction is later not used. The SIGILL in _armv7_tick should be ignored when debugging.

Here is a backtrace

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0xb671fbe8 in _armv7_tick () from /lib/libcrypto.so.10
(gdb) backtrace
#0  0xb671fbe8 in _armv7_tick () from /lib/libcrypto.so.10
#1  0xb671bf9c in OPENSSL_cpuid_setup () from /lib/libcrypto.so.10
#2  0xb6fdf71c in call_init.part () from /lib/ld-linux-armhf.so.3
#3  0xb6fdf880 in _dl_init () from /lib/ld-linux-armhf.so.3
#4  0xb6fcfb04 in _dl_start_user () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)


By the way, openconnect worked in fedora 22 on this same arm box

Ok, but this is the backtrace which is not useful. Either the SIGILL handler gets somehow overridden by openconnect or there is another place where the SIGILL is happening. GDB will however catch the SIGILL in the OPENSSL_cpuid_setup although it is handled by a handler. If you continue at this point, what happens?

(gdb) continue
Continuing.
No server specified
Usage:  openconnect [options] <server>
Open client for Cisco AnyConnect VPN, version v7.06-3.fc23

Which is odd to me, it is as if it did not parse the command line?

The command line needs to be specified in the run command in gdb.

oops, sorry, I am a newbie at this

(gdb) run -b https://xxx.xxx.xxx.xxx
Starting program: /usr/sbin/openconnect -b https://xxx.xxx.xxx.xxx
Cannot parse expression `.L967 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0xb671fbe8 in _armv7_tick () from /lib/libcrypto.so.10
(gdb) continue
Continuing.
POST https://xxx.xxx.xxx.xxx/
Attempting to connect to server xxx.xxx.xxx.xxx:443
SSL negotiation with xxx.xxx.xxx.xxx

Program received signal SIGILL, Illegal instruction.
0xb6647af8 in _nettle_salsa20_core () from /lib/libnettle.so.6

(gdb) backtrace
#0  0xb6647af8 in _nettle_salsa20_core () from /lib/libnettle.so.6
#1  0xb6647db4 in nettle_salsa20r12_crypt () from /lib/libnettle.so.6
#2  0xb6c9b1e0 in wrap_nettle_rnd () from /lib/libgnutls.so.30
#3  0xb6bcdd2c in create_tls_random () from /lib/libgnutls.so.30
#4  0xb6bce0e0 in _gnutls_set_client_random () from /lib/libgnutls.so.30
#5  0xb6bcf460 in send_hello () from /lib/libgnutls.so.30
#6  0xb6bd2ca4 in gnutls_handshake () from /lib/libgnutls.so.30
#7  0xb6f9a9d8 in cstp_handshake () from /lib/libopenconnect.so.5
#8  0xb6f9aff8 in openconnect_open_https () from /lib/libopenconnect.so.5
#9  0xb6f7cc70 in do_https_request () from /lib/libopenconnect.so.5
#10 0xb6f8d1a0 in cstp_obtain_cookie () from /lib/libopenconnect.so.5
#11 0x7f559508 in main ()

So the real crash is in nettle.

(In reply to Tim Krantz from comment #7)

> Program received signal SIGILL, Illegal instruction.
> 0xb6647af8 in _nettle_salsa20_core () from /lib/libnettle.so.6
> (gdb) backtrace
> #0  0xb6647af8 in _nettle_salsa20_core () from /lib/libnettle.so.6
> #1  0xb6647db4 in nettle_salsa20r12_crypt () from /lib/libnettle.so.6
> #2  0xb6c9b1e0 in wrap_nettle_rnd () from /lib/libgnutls.so.30
> #3  0xb6bcdd2c in create_tls_random () from /lib/libgnutls.so.30
> #4  0xb6bce0e0 in _gnutls_set_client_random () from /lib/libgnutls.so.30
> #5  0xb6bcf460 in send_hello () from /lib/libgnutls.so.30
> #6  0xb6bd2ca4 in gnutls_handshake () from /lib/libgnutls.so.30
> #7  0xb6f9a9d8 in cstp_handshake () from /lib/libopenconnect.so.5
> #8  0xb6f9aff8 in openconnect_open_https () from /lib/libopenconnect.so.5
> #9  0xb6f7cc70 in do_https_request () from /lib/libopenconnect.so.5
> #10 0xb6f8d1a0 in cstp_obtain_cookie () from /lib/libopenconnect.so.5
> #11 0x7f559508 in main ()

Hi Tim,
 Could you do:
dnf debuginfo-install nettle

and run after you get the SIGILL and the backtrace, "x/i $pc"

Nikos, here you go!

(gdb) backtrace
#0  _nettle_salsa20_core () at salsa20-core-internal.s:58
#1  0xb6647db4 in nettle_salsa20r12_crypt (ctx=ctx@entry=0xb6cc7f40 <nonce_ctx>,
    length=<optimized out>, length@entry=29, c=c@entry=0x7f67ce83 "", m=m@entry=0x7f67ce83 "")
    at salsa20r12-crypt.c:65
#2  0xb6c9b1e0 in wrap_nettle_rnd_nonce (_ctx=<optimized out>, datasize=29, data=0x7f67ce83)
    at rnd.c:302
#3  wrap_nettle_rnd (_ctx=<optimized out>, level=level@entry=0, data=0x7f67ce83,
    datasize=datasize@entry=29) at rnd.c:321
#4  0xb6bcdd2c in _gnutls_rnd (len=29, data=<optimized out>, level=GNUTLS_RND_NONCE)
    at ./random.h:37
#5  create_tls_random (dst=<optimized out>) at gnutls_handshake.c:205
#6  0xb6bce0e0 in _gnutls_set_client_random (session=<optimized out>, rnd=<optimized out>)
    at gnutls_handshake.c:230
#7  0xb6bcf460 in send_client_hello (again=<optimized out>, session=0x7f67ce38)
    at gnutls_handshake.c:2039
#8  send_hello (session=session@entry=0x7f67ce38, again=<optimized out>)
    at gnutls_handshake.c:2249
#9  0xb6bd2ca4 in handshake_client (session=0x7f67ce38) at gnutls_handshake.c:2732
#10 gnutls_handshake (session=0x7f67ce38) at gnutls_handshake.c:2583
#11 0xb6f9a9d8 in cstp_handshake (vpninfo=vpninfo@entry=0x7f58ccf8, init=64, init@entry=1)
    at gnutls.c:2250
#12 0xb6f9aff8 in openconnect_open_https (vpninfo=vpninfo@entry=0x7f58ccf8) at gnutls.c:2227
#13 0xb6f7cc70 in do_https_request (vpninfo=vpninfo@entry=0x7f58ccf8, method=0xb6f9f700 "POST",
    method@entry=0xb6ffec80 <__stack_chk_guard> "",
    request_body_type=request_body_type@entry=0xb6f9fa28 "application/x-www-form-urlencoded",
    request_body=request_body@entry=0x7f58d480, form_buf=0xbefff25c, form_buf@entry=0x1bb,
    fetch_redirect=fetch_redirect@entry=0) at http.c:875
#14 0xb6f8d1a0 in cstp_obtain_cookie (vpninfo=<optimized out>) at auth.c:1238
#15 0x7f559508 in main (argc=0, argv=0x0) at main.c:1388
(gdb) x/i$pc
=> 0xb6647af8 <_nettle_salsa20_core+8>: vldmia  r12, {d20-d25}

Thank you. One more thing, what is the actual system this crash happens? (e.g., output of /proc/cpuinfo)

My pleasure!

in human terms it is a globalscale mirabox.
(I have 4 or 5 other arm devices I can try this on if you need that)

cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
BogoMIPS        : 37.39
Features        : half thumb fastmult vfp edsp thumbee vfpv3 vfpv3d16 tls idivt
CPU implementer : 0x56
CPU architecture: 7
CPU variant     : 0x1
CPU part        : 0x581
CPU revision    : 1

Hardware        : Marvell Armada 370/XP (Device Tree)
Revision        : 0000
Serial          : 0000000000000000

I suspect this is about the neon instruction set. Do the rpms in that scratch build work? It disables neon support unconditionally:

http://koji.fedoraproject.org/koji/taskinfo?taskID=12031034

Yes, it appears to work correctly with the scratch build.

nettle-3.1.1-5.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-63e4a08916

nettle-3.1.1-5.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update nettle'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-63e4a08916

nettle-3.1.1-5.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.