Bug 1829790

Summary: libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
Product: [Fedora] Fedora Reporter: Petr <petamail>
Component: krb5Assignee: Robbie Harwood <rharwood>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: abokovoy, associations, j, npmccallum, rharwood, rkudyba, sahana, sbose, ssorce, tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-02 11:51:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Petr 2020-04-30 11:37:02 UTC 
Description of problem:
After upgrade to Fedora 32, Matlab 2020a complain about:
"symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b"


Version-Release number of selected component (if applicable):
krb5-libs-1.18-1.fc32.x86_64


Additional info:
I checked version of this library for Fedora31 (krb5-libs-1.17-45.fc31.x86_64.rpm), it doesn't contain mentioned symbol and it works well;
Comment 1 Simo Sorce 2020-04-30 17:46:56 UTC 
Can you please provide the NVR of your installed openssl package?
As that symbol comes from there and should have appeared in our OpenSSL 1.1.1c package IIRC (it is a backport from upstream master, so I wonder if we mistakenly broke the symbol versioning).
Comment 2 Simo Sorce 2020-04-30 17:53:26 UTC 
You may also want to run:
$ objdump -TC /usr/lib/libcrypto.so.1.1 |grep EVP_KDF
and paste the result here
Comment 3 Robbie Harwood 2020-04-30 19:11:46 UTC 
Additional data point for later reference: krb5-libs has `Requires: openssl-libs >= 1:1.1.1d-4`.

I think that's not the right file - on my machines it would be /usr/lib64/libcrypto.so.1.  These are what my test VMs look like, for later comparison:

On rawhide (nvr openssl-libs-1.1.1g-1.fc33.x86_64):

[root@kerberos ~]# objdump -TC /usr/lib64/libcrypto.so.1.1 | grep EVP_KDF
0000000000176080 g    DF .text	00000000000000f0  OPENSSL_1_1_1b EVP_KDF_ctrl
0000000000176170 g    DF .text	000000000000008e  OPENSSL_1_1_1b EVP_KDF_ctrl_str
0000000000176020 g    DF .text	0000000000000021  OPENSSL_1_1_1b EVP_KDF_reset
0000000000176200 g    DF .text	0000000000000030  OPENSSL_1_1_1b EVP_KDF_size
0000000000176050 g    DF .text	0000000000000023  OPENSSL_1_1_1b EVP_KDF_vctrl
0000000000175f00 g    DF .text	000000000000011d  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000175ec0 g    DF .text	0000000000000031  OPENSSL_1_1_1b EVP_KDF_CTX_free
0000000000176230 g    DF .text	0000000000000023  OPENSSL_1_1_1b EVP_KDF_derive
[root@kerberos ~]# 

On f32 (nvr openssl-libs-1.1.1g-1.fc32.x86_64):

[root@kerberos vagrant]# objdump -TC /usr/lib64/libcrypto.so.1.1 | grep EVP_KDF
0000000000176000 g    DF .text	00000000000000f0  OPENSSL_1_1_1b EVP_KDF_ctrl
00000000001760f0 g    DF .text	000000000000008e  OPENSSL_1_1_1b EVP_KDF_ctrl_str
0000000000175fa0 g    DF .text	0000000000000021  OPENSSL_1_1_1b EVP_KDF_reset
0000000000176180 g    DF .text	0000000000000030  OPENSSL_1_1_1b EVP_KDF_size
0000000000175fd0 g    DF .text	0000000000000023  OPENSSL_1_1_1b EVP_KDF_vctrl
0000000000175e80 g    DF .text	000000000000011d  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000175e40 g    DF .text	0000000000000031  OPENSSL_1_1_1b EVP_KDF_CTX_free
00000000001761b0 g    DF .text	0000000000000023  OPENSSL_1_1_1b EVP_KDF_derive
[root@kerberos vagrant]#
Comment 4 Petr 2020-04-30 19:22:41 UTC 
Hi, I have installed these openssl packages:

openssl-pkcs11-0.4.10-5.fc32.i686
openssl-libs-1.1.1g-1.fc32.i686
openssl-1.1.1g-1.fc32.x86_64
openssl-pkcs11-0.4.10-5.fc32.x86_64
openssl-libs-1.1.1g-1.fc32.x86_64


[~]$ objdump -TC /usr/lib/libcrypto.so.1.1 | grep EVP_KDF
0012b990 g    DF .text  00000065  OPENSSL_1_1_1b EVP_KDF_ctrl
0012ba00 g    DF .text  000000aa  OPENSSL_1_1_1b EVP_KDF_ctrl_str
0012b930 g    DF .text  00000021  OPENSSL_1_1_1b EVP_KDF_reset
0012bab0 g    DF .text  0000002e  OPENSSL_1_1_1b EVP_KDF_size
0012b960 g    DF .text  00000023  OPENSSL_1_1_1b EVP_KDF_vctrl
0012b830 g    DF .text  000000fe  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0012b7e0 g    DF .text  00000044  OPENSSL_1_1_1b EVP_KDF_CTX_free
0012bae0 g    DF .text  00000023  OPENSSL_1_1_1b EVP_KDF_derive

[~]$ objdump -TC /usr/lib64/libcrypto.so.1.1 | grep EVP_KDF
0000000000176000 g    DF .text  00000000000000f0  OPENSSL_1_1_1b EVP_KDF_ctrl
00000000001760f0 g    DF .text  000000000000008e  OPENSSL_1_1_1b EVP_KDF_ctrl_str
0000000000175fa0 g    DF .text  0000000000000021  OPENSSL_1_1_1b EVP_KDF_reset
0000000000176180 g    DF .text  0000000000000030  OPENSSL_1_1_1b EVP_KDF_size
0000000000175fd0 g    DF .text  0000000000000023  OPENSSL_1_1_1b EVP_KDF_vctrl
0000000000175e80 g    DF .text  000000000000011d  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000175e40 g    DF .text  0000000000000031  OPENSSL_1_1_1b EVP_KDF_CTX_free
00000000001761b0 g    DF .text  0000000000000023  OPENSSL_1_1_1b EVP_KDF_derive
Comment 5 Robbie Harwood 2020-05-01 15:57:44 UTC 
Simo, any other thoughts?
Comment 6 Simo Sorce 2020-05-01 17:35:51 UTC 
As far as I can tell I do nost see any symbol issues, so I am at a loss why libkrb5 would complain.
Does it reproduce on rawhide for you Robbie ?
Comment 7 Simo Sorce 2020-05-01 17:36:54 UTC 
Also CCing Tomáŝ in case he has any ideas, but the openssl side looks right to me.
Comment 8 Petr 2020-05-01 18:00:12 UTC 
I don't know if it is helpfull for you, but I figured out that scilab report the same problem,
https://www.scilab.org/download/6.1.0/scilab-6.1.0.bin.linux-x86_64.tar.gz

[~]$ ./scilab-6.1.0/bin/scilab
scilab-bin: symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
Comment 9 Petr 2020-05-02 11:51:52 UTC 
I learned something more about shared libraries and I finally found out where the problem is.

Both products (Matlab & Scilab) contains their own shared library libcrypto.so.1.1, but uses system library libk5crypto.so.3.
System library libk5crypto.so.3 uses new symbols, which system library libcrypto.so.1.1 has, but Matlab's library libcrypto.so.1.1 hasn't.

So I deleted libcrypto.so.1.1 from Matlab directory and it works now (same as scilab).

I'm so sorry for wasting your time ...

Best regards
Petr
Comment 10 Simo Sorce 2020-05-04 13:55:44 UTC 
Petr,
this means those products are using most probably outdated libraries w/o getting CVE bugfixes when the system gets them.
I would open a bug report upstream to stop doing this stupdi library interposing on all systems and do it only where the proper library version is missing (arguably they do this to handle RHEL/CentOS 6 which are stuck on openssl 1.0.2).
That said at least Matlab is a proprietary product so ... good luck, any number of things can break when they play fast and loose with critical libararies like openssl.
Comment 11 RobbieTheK 2020-08-17 01:39:45 UTC 
I'm seeing this on Fedora 32, whilst installing ROS 2, and running colcon build --symlink-install. We do not have Matlab installed. We do have Anaconda Python 3.7 and Fedora Python 3.8. When I tried to use flags for colcon build, the same errors occur.

/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_derive@OPENSSL_1_1_1b'
collect2: error: ld returned 1 exit status
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_unparse_lower'
gmake[2]: *** [CMakeFiles/point_cloud_renderable_test_target.dir/build.make:130: point_cloud_renderable_test_target] Error 1
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_unparse_lower'
/usr/bin/ld: /usr/lib64/libcurl.so: undefined reference to `SSLv3_client_method@OPENSSL_1_1_0'
gmake[1]: *** [CMakeFiles/Makefile2:280: CMakeFiles/point_cloud_renderable_test_target.dir/all] Error 2
/usr/bin/ld: /usr/lib64/libcurl.so: undefined reference to `SSLv3_client_method@OPENSSL_1_1_0'
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/wrench_visual_test_target.dir/build.make:130: wrench_visual_test_target] Error 1
gmake[2]: *** [CMakeFiles/billboard_line_test_target.dir/build.make:130: billboard_line_test_target] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:194: CMakeFiles/wrench_visual_test_target.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:337: CMakeFiles/billboard_line_test_target.dir/all] Error 2
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_ctrl@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_CTX_free@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_ctrl@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_CTX_new_id@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_generate'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_CTX_free@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_derive@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_CTX_new_id@OPENSSL_1_1_1b'
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_unparse_lower'
/usr/bin/ld: /usr/lib64/libcurl.so: undefined reference to `SSLv3_client_method@OPENSSL_1_1_0'
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_generate'
/usr/bin/ld: /usr/lib64/libssh.so.4: undefined reference to `EVP_KDF_derive@OPENSSL_1_1_1b'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/point_cloud_test_target.dir/build.make:130: point_cloud_test_target] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:164: CMakeFiles/point_cloud_test_target.dir/all] Error 2
/usr/bin/ld: /usr/lib64/libSM.so: undefined reference to `uuid_unparse_lower'
/usr/bin/ld: /usr/lib64/libcurl.so: undefined reference to `SSLv3_client_method@OPENSSL_1_1_0'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/string_helper_test.dir/build.make:128: string_helper_test] Error 1

here's a objdump:
objdump -TC /usr/lib64/libcrypto.so.1.1 |grep EVP_KDF
0000000000176000 g    DF .text  00000000000000f0  OPENSSL_1_1_1b EVP_KDF_ctrl
00000000001760f0 g    DF .text  000000000000008e  OPENSSL_1_1_1b EVP_KDF_ctrl_str
0000000000175fa0 g    DF .text  0000000000000021  OPENSSL_1_1_1b EVP_KDF_reset
0000000000176180 g    DF .text  0000000000000030  OPENSSL_1_1_1b EVP_KDF_size
0000000000175fd0 g    DF .text  0000000000000023  OPENSSL_1_1_1b EVP_KDF_vctrl
0000000000175e80 g    DF .text  000000000000011d  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000175e40 g    DF .text  0000000000000031  OPENSSL_1_1_1b EVP_KDF_CTX_free
00000000001761b0 g    DF .text  0000000000000023  OPENSSL_1_1_1b EVP_KDF_derive

5.7.10-201.fc32.x86_64
rpm -qa|grep ssl
openssl-perl-1.1.1g-1.fc32.x86_64
openssl-libs-1.1.1g-1.fc32.x86_64
openssl-pkcs11-0.4.10-6.fc32.x86_64
openssl-1.1.1g-1.fc32.x86_64
rubygem-openssl-2.1.2-132.fc32.x86_64
poco-netssl-1.9.4-2.fc32.x86_64
openssl-devel-1.1.1g-1.fc32.x86_64
compat-openssl10-1.0.2o-9.fc32.x86_64
openssl-gost-engine-1.1.0.3-6.fc32.x86_64
apr-util-openssl-1.6.1-12.fc32.x86_64
openssl-libs-debuginfo-1.1.1g-1.fc32.x86_64
openssl-debuginfo-1.1.1g-1.fc32.x86_64
openssl-ibmpkcs11-1.0.2-5.fc32.x86_64
NetworkManager-fortisslvpn-1.3.90-7.fc32.x86_64
openssl-debugsource-1.1.1g-1.fc32.x86_64
xmlsec1-openssl-1.2.29-1.fc32.x86_64
mod_ssl-2.4.43-5.fc32.x86_64
openssl-static-1.1.1g-1.fc32.x86_64
Comment 12 Tomas Mraz 2020-08-17 08:55:19 UTC 
This is result of having some third party (or upstream) openssl library installed somewhere and trying to link the libssh and curl against it. It simply won't work.

You can also see that you have another problem with uuid_unparse_lower - so you have also a third party libuuid somewhere.

You have to find these libraries and remove/avoid them when linking.

There is nothing to be fixed on Fedora side.
Comment 13 RobbieTheK 2020-08-17 13:18:32 UTC 
(In reply to Tomas Mraz from comment #12)
> This is result of having some third party (or upstream) openssl library
> installed somewhere and trying to link the libssh and curl against it. It
> simply won't work.
> 
> You can also see that you have another problem with
> uuid_unparse_lower - so you have also a third party libuuid
> somewhere.
> 
> You have to find these libraries and remove/avoid them when linking.

Could it be Anaconda Python that is the culprit?
 ls -l /usr/local/bin/anaconda3/pkgs/libuuid-1.0.3-h1bed415_2/lib/libuuid.so.1.0.0
-rwxrwxr-x 1 root root 18472 Jan 11  2018 /usr/local/bin/anaconda3/pkgs/libuuid-1.0.3-h1bed415_2/lib/libuuid.so.1.0.0


I was specifying to use Fedora's Python 3.8 but perhaps I am not using the correct options during building.
Comment 14 Tomas Mraz 2020-08-17 14:02:20 UTC 
It looks like so (at least for the libuuid).