Note that the condition is hit also by simply running

  adb start-server

while there is no adbkey file in neither ~/.android nor
/var/lib/adb/.android.

#0  BN_div (dv=dv@entry=0x0, rm=rm@entry=0x6b0680, num=0x0,
divisor=divisor@entry=0x6b1c90, ctx=ctx@entry=0x6b01e0)
at crypto/bn/bn_div.c:155 <-- this is openssl

150│     /*
151│      * Invalid zero-padding would have particularly bad consequences so don't
152│      * just rely on bn_check_top() here (bn_check_top() works only for
153│      * BN_DEBUG builds)
154│      */
155├>    if ((num->top > 0 && num->d[num->top - 1] == 0) ||
156│         (divisor->top > 0 && divisor->d[divisor->top - 1] == 0)) {
157│         BNerr(BN_F_BN_DIV, BN_R_NOT_INITIALIZED);
158│         return 0;
159│     }

(gdb) p	num
$1 = (const BIGNUM *) 0x

That is, NULL pointer dereference here.


#1  0x000000000041a39b in android_pubkey_encode	(key=key@entry=0x6b10e0,
key_buffer=key_buffer@entry=0x7fffffffb750 "@", size=size@entry=524)
at libcrypto_utils/android_pubkey.c:140 <-- android-tools already

138│   // Compute and store n0inv = -1 / N[0] mod 2^32.
139│   if (!ctx || !r32 || !n0inv || !BN_set_bit(r32, 32) ||
140├>      !BN_mod(n0inv, key->n, r32, ctx) ||
141│       !BN_mod_inverse(n0inv, n0inv, r32, ctx) || !BN_sub(n0inv, r32, n0inv)) {
142│     goto cleanup;
143│   }

That means, modulus component of the key (key->n) is absent, although
it was present at least at some key initialization steps earlier.

In my case, this was with:
android-tools-20170311gite7195be7725a-1.fc27.x86_64
openssl-1.1.0e-1.fc27.x86_64

Maybe this has something to do with new OpenSSL 1.1.0 API,
i.e., [bug 1383740] (?)

re [comment 4]:  Indeed.

https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes#Compatibility_Layer

> If you want to access rsa->n you now have to do something like:
> 
>     const BIGNUM *n;
> 
>     RSA_get0_key(rsa, &n, NULL, NULL);

This bug hence blocks mentioned [bug 1383740].

I've tried rebuilding android-tools with this change:

--- a/android-tools.spec
+++ b/android-tools.spec
@@ -49,7 +49,9 @@ Requires(post): systemd
 Requires(preun): systemd
 Requires(postun): systemd
 BuildRequires: zlib-devel
-BuildRequires: openssl-devel
+# Not OpenSSL 1.1.0 ready yet (rhbz 1441234)
+#BuildRequires: openssl-devel
+BuildRequires: compat-openssl10-devel
 BuildRequires: libselinux-devel
 BuildRequires: f2fs-tools-devel
 BuildRequires: gtest-devel

ldd for adb shows:

> libcrypto.so.10 => /lib64/libcrypto.so.10

but that did not help in any way :-/

Maybe the issue is not in system openssl, but in bundled boringssl.

Created attachment 1274033 [details]
Pkgs repo patch to solve adb crash when generating a key (due to changed OpenSSL 1.1.0 API)

Nope, gdb explicitly referred to openssl sources.

I am attaching a patch that worked for me when generating a local key.

That doesn't mean all issues with OpenSSL 1.1.0 are solved, it's more
likely a matter of future trial & error (or careful review, which I don't
have enough time for).

In that patch, I've missed to swap also "key->e" usage.

Anyway, it would be vital to have tests enabled during build,
as this very occurrence would most likely be caught with
libcrypto_utils/tests/android_pubkey_test.cpp test.
Good news is that gtest is packaged, but still having the suite
up and usable does not look exactly easy.

android-tools-20170311gite7195be7725a-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9da6979158

android-tools-20170311gite7195be7725a-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9da6979158

android-tools-20170311gite7195be7725a-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.