Bug 1473756 - Fail to start server: incompatible OpenSSL version
Fail to start server: incompatible OpenSSL version
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: mariadb (Show other bugs)
27
Unspecified Linux
unspecified Severity medium
: ---
: ---
Assigned To: Michal Schorm
Fedora Extras Quality Assurance
:
Depends On: 1544986
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-21 10:39 EDT by Anton Guda
Modified: 2018-03-30 08:45 EDT (History)
9 users (show)

See Also:
Fixed In Version: mariadb-10.2.13-2.fc27 mariadb-10.2.13-2.fc28
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-03-13 19:15:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Anton Guda 2017-07-21 10:39:00 EDT
Description of problem:
After upgrade to mariadb-10.2.7-4.fc27, it fails to start.
Error message:
[ERROR] Incompatible OpenSSL version. Cannot continue

Version-Release number of selected component (if applicable):

mariadb-10.2.7-4.fc27.x86_64
openssl-1.1.0f-7.fc27.x86_64
compat-openssl10-1.0.2j-6.fc26.x86_64

How reproducible:
Always


Steps to Reproduce:
1. update mariaddb
2. try to restart server

Actual results:
Fail to start, error message

Expected results:
Just work.
Comment 1 Anton Guda 2017-07-24 03:51:28 EDT
After update to mariadb-10.2.7-5.fc27.x86_64 - the same result.
Comment 2 Michal Schorm 2017-07-26 04:36:11 EDT
Hello,
I tried to reproduce, but so far I wasn't succesful.

I have the exact set of the packages - mariadb, openssl, compat-openssl.

--

Do you know from which version did you updated?
Could you try to uninstall and re-install mariadb completely?
Which Fedora version do you use? (I suppose you use 25 or 26 with Rawhide repository)
Comment 3 Anton Guda 2017-07-27 15:35:56 EDT
last working was mariadb-10.2.6-4.fc27

now used is mariadb-10.2.7-6.fc27 - not work too.
rpm -V mariadb shows nothing.
Used rawhide - rolling for long time.
Some packages was rebuilded myself - but seems not closely realted.

I suppose to wait untill FC27 mass rebuild finish
before investigating.
Comment 4 Anton Guda 2017-08-08 06:04:13 EDT
After update (the same result):
rpm -q mariadb
mariadb-10.2.7-8.fc27.x86_64
rpm -q openssl
openssl-1.1.0f-9.fc27.x86_64

ldd /usr/libexec/mysqld
        linux-vdso.so.1 (0x00007ffcea6c6000)
        libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f8a840cc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a83ead000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f8a83cab000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f8a83a94000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a83821000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8a835eb000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8a8337f000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8a82ef8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a82cf4000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f8a8296e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8a82619000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8a82234000)
        libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f8a81f27000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f8a81d1f000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8a81af9000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8a818e5000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a816bc000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a814a5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8a86048000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f8a812a2000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f8a8108d000)

I can also send strace and ltrace logs, or other debug info,
but I failed to detect come criminal behaviour for now.
Comment 5 Jan Kurik 2017-08-15 03:03:53 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.
Comment 6 Anton Guda 2017-08-15 09:44:46 EDT
gdb trace shows, that error occurs on function
CRYPTO_set_mem_functions called from
check_openssl_compatibility ()

ltrace part:
CRYPTO_set_mem_functions(0x916f6ba4f0, 0x7f7817924aa0, 0x7f7817924a30, 0x9170ca8698) = 0
Comment 7 Honza Horak 2017-08-20 06:25:00 EDT
Anton, do you by any chance use some server-side plugins? Especially not coming from mariadb Fedora build?
Comment 8 Anton Guda 2017-08-20 09:03:59 EDT
I carefully checked directories
/etc/mysql/
  fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch

/etc/my.cnf.d/
  auth_gssapi.cnf
  cracklib_password_check.cnf
  enable_encryption.preset
  mariadb-server.cnf
  mysql-clients.cnf
  rocksdb.cnf
  tokudb.cnf

(all from mariadb-server-10.2.7-8.fc27.x86_64, mariadb-config-10.2.7-8.fc27.x86_64, no difference)

local config have only password - removed too.

/usr/lib64/mysql (9 files) (+plugin: 45 files)

Nothing external

Sometimes i used percona-toolkit, but
 1. Was removed during investigation
 2. It have no plugins - only executables.

Now I have suspicion about OpneSSL usage, hope to verify soon.
Comment 9 Honza Horak 2017-08-21 05:59:37 EDT
(In reply to Anton Guda from comment #8)
>   fabric.cfg - mysql-utilities-1.5.6-5.fc27.noarch

mysql-utilities uses mysql-connector-python, which is Python-only implementation of the MySQL protocol, so shouldn't cause troubles here.
Comment 10 Anton Guda 2017-08-31 13:32:21 EDT
I rebuilt the mariadb package, with patch, where chech_openssl_compatibility always returns 0.
All works for me. It seems, that CRYPTO_set_mem_functions
sometimes fails w/o known reasons.
Comment 11 Sunil P 2017-11-21 13:19:51 EST
I was able to reproduce mariadb startup failure issue. Apparently it's caused by dracut-fips-046-5.fc27.x86_64 package. Reproduce steps...

##### Start mariadb check status
# systemctl start mariadb
# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-21 10:01:01 PST; 8s ago
# systemctl stop mariadb

##### Install dracut-fips-046-5.fc27.x86_64 package.

# dnf install dracut-fips-046-5.fc27.x86_64
Last metadata expiration check: 0:06:00 ago on Tue 21 Nov 2017 09:56:48 AM PST.
Dependencies resolved.
=============================================================================================================================================
 Package                            Arch                          Version                                Repository                     Size
=============================================================================================================================================
Installing:
 dracut-fips                        x86_64                        046-5.fc27                             fedora                         48 k
Installing dependencies:
 hmaccalc                           x86_64                        0.9.14-8.fc27                          fedora                         32 k

Transaction Summary
=============================================================================================================================================
Install  2 Packages

Total download size: 81 k
Installed size: 104 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): hmaccalc-0.9.14-8.fc27.x86_64.rpm                                                                     113 kB/s |  32 kB     00:00
(2/2): dracut-fips-046-5.fc27.x86_64.rpm                                                                     168 kB/s |  48 kB     00:00
---------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                        114 kB/s |  81 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                     1/1
  Installing       : hmaccalc-0.9.14-8.fc27.x86_64                                                                                       1/2
  Installing       : dracut-fips-046-5.fc27.x86_64                                                                                       2/2
  Running scriptlet: dracut-fips-046-5.fc27.x86_64                                                                                       2/2
  Verifying        : dracut-fips-046-5.fc27.x86_64                                                                                       1/2
  Verifying        : hmaccalc-0.9.14-8.fc27.x86_64                                                                                       2/2

Installed:
  dracut-fips.x86_64 046-5.fc27                                         hmaccalc.x86_64 0.9.14-8.fc27

Complete!

##### Start Mariadb again. It fail with Incompatible OpenSSL version. Cannot continue...

# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl  status mariadb.service" and "journalctl  -xe" for details.

# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-11-21 10:04:34 PST; 4min 11s ago
  Process: 20748 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
  Process: 21077 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE)
  Process: 21040 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 21016 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 21077 (code=exited, status=1/FAILURE)

Nov 21 10:04:33 systemd[1]: Starting MariaDB 10.2 database server...
Nov 21 10:04:34 mysql-prepare-db-dir[21040]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Nov 21 10:04:34 mysql-prepare-db-dir[21040]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
Nov 21 10:04:34 mysqld[21077]: 2017-11-21 10:04:34 140737353538816 [ERROR] Incompatible OpenSSL version. Cannot continue...

##### Remove dracut-fips-046-5.fc27.x86_64, mariadb start successfully again.

# dnf remove dracut-fips-046-5.fc27.x86_64
# systemctl start mariadb
# systemctl status mariadb
● mariadb.service - MariaDB 10.2 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-21 10:11:57 PST; 6s ago
Comment 12 Anton Guda 2017-12-02 07:06:25 EST
Confirm, removing dracut-fips really helps.
May be add this package to conflicts?
Comment 13 Michal Schorm 2017-12-04 10:30:00 EST
The package conflict is used only for packages that can't be *installed* together.

I went through this issue today, and I found the cause and reported it to the upstream.
https://jira.mariadb.org/browse/MDEV-14567

I can't help you much, because the 'dracut-fips' package actually alter the behaviour of openssl and the MariaDB won't start as expected when such behavious occurs.
However MariaDB should be able to run in FIPS mode, so I think upstream will be interested in that issue.
Comment 14 Fedora Update System 2018-03-06 07:20:06 EST
mariadb-10.2.13-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4a4dc47afe
Comment 15 Fedora Update System 2018-03-07 15:02:45 EST
mariadb-10.2.13-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f
Comment 16 Fedora Update System 2018-03-07 15:03:17 EST
mariadb-10.2.13-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5
Comment 17 Fedora Update System 2018-03-08 10:28:31 EST
mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-83bbd0c22f
Comment 18 Fedora Update System 2018-03-08 11:22:32 EST
mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-00647ae0d5
Comment 19 Fedora Update System 2018-03-13 19:15:08 EDT
mariadb-10.2.13-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 20 Fedora Update System 2018-03-30 08:45:13 EDT
mariadb-10.2.13-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.