Bug 2050541
| Summary: | mariadb cannot bootstrap initial DB when FIPS is enabled | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Damien Ciabrini <dciabrin> |
| Component: | mariadb | Assignee: | Michal Schorm <mschorm> |
| Status: | CLOSED ERRATA | QA Contact: | Jakub Heger <jheger> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.0 | CC: | alee, databases-maint, dciabrin, hhorak, jheger, ljavorsk, lmiccini |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | mariadb-10.5.13-2.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-05-17 12:42:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Heads up, bug 4531 in percona has been marked as a DUP of 3840 [1], whose fix unfortunately only consists in disabling FIPS. [1] https://github.com/percona/percona-server/commit/86e2187f88214703568643817374832a534a342a The patch to allow md5 in FIPS mode is included in MariaDB as well: https://github.com/MariaDB/server/blob/c0f5fd27549c84607defa64c5b651343dd29e0ee/mysys_ssl/my_md5.cc#L58 ... static void md5_init(EVP_MD_CTX *context) { EVP_MD_CTX_init(context); #ifdef EVP_MD_CTX_FLAG_NON_FIPS_ALLOW /* Ok to ignore FIPS: MD5 is not used for crypto here */ EVP_MD_CTX_set_flags(context, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW); #endif EVP_DigestInit_ex(context, EVP_md5(), NULL); } ... However, the problem is related to OpenSSL 3.0.0 which does not support EVP_MD_CTX_FLAG_NON_FIPS_ALLOW any longer. In OpenSSL 1.1.1 the non FIPS allowed flag is context specific, while in 3.0.0 it is a different EVP_MD provider. If there is no other issue with FIPS, this should be possible to fix hopefully easily, maybe with something like this: ... static void md5_init(EVP_MD_CTX *context) { + EVP_MD *md5; + md5 = EVP_MD_fetch(NULL, "MD5", "fips=no"); EVP_MD_CTX_init(context); #ifdef EVP_MD_CTX_FLAG_NON_FIPS_ALLOW /* Ok to ignore FIPS: MD5 is not used for crypto here */ EVP_MD_CTX_set_flags(context, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW); #endif - EVP_DigestInit_ex(context, EVP_md5(), NULL); + EVP_DigestInit_ex(context, md5, NULL); + EVP_MD_free(md5); } ... (I'm currently testing this) The change looks good so far, basic tests work fine in FIPS environment with the following change: https://gitlab.com/redhat/centos-stream/rpms/mariadb/-/merge_requests/19 In order to successfully work in FIPS in CentOS Stream 9/RHEL-9, we also need a fix for the python3.9 from this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1942527 Also created a patch for Fedora: https://src.fedoraproject.org/rpms/mariadb/pull-request/47 Reported to upstream as well: https://jira.mariadb.org/browse/MDEV-27778 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (new packages: mariadb), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:2381 |
Description of problem: When FIPS is enabled on the system with "fips-mode-setup --enable", some hash/crypto functions become unavailable in openssl, such as for instance MD5. Unfortunately, mariadb makes use of MD5 internally to create VIEWs in the DB. This causes mariadb to crash whenever MD5 is used internally. The issue materializes for instance when one wants to bootstrap a mysql database on disk with mysql_install_db: # mysql_install_db 2022-02-03 16:52:19 0 [Warning] You need to use --log-bin to make --binlog-format work. 220203 16:52:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.12-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467871 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x55bc84575098 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffda2bbe048 thread_stack 0x49000 /usr/libexec/mysqld(my_print_stacktrace+0x41)[0x55bc8230df81] Printing to addr2line failed /usr/libexec/mysqld(handle_fatal_signal+0x498)[0x55bc81de0ea8] /lib64/libc.so.6(+0x42720)[0x7fc0ba566720] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x55bc84ba2750): CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT Host, User, IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password, IF(JSON_VALUE(Priv, '$.access') & 1, 'Y', 'N') AS Select_priv, IF(JSON_VALUE(Priv, '$.access') & 2, 'Y', 'N') AS Insert_priv, IF(JSON_VALUE(Priv, '$.access') & 4, 'Y', 'N') AS Update_priv, IF(JSON_VALUE(Priv, '$.access') & 8, 'Y', 'N') AS Delete_priv, IF(JSON_VALUE(Priv, '$.access') & 16, 'Y', 'N') AS Create_priv, IF(JSON_VALUE(Priv, '$.access') & 32, 'Y', 'N') AS Drop_priv, IF(JSON_VALUE(Priv, '$.access') & 64, 'Y', 'N') AS Reload_priv, IF(JSON_VALUE(Priv, '$.access') & 128, 'Y', 'N') AS Shutdown_priv, IF(JSON_VALUE(Priv, '$.access') & 256, 'Y', 'N') AS Process_priv, IF(JSON_VALUE(Priv, '$.access') & 512, 'Y', 'N') AS File_priv, IF(JSON_VALUE(Priv, '$.access') & 1024, 'Y', 'N') AS Grant_priv, IF(JSON_VALUE(Priv, '$.access') & 2048, 'Y', 'N') AS References_priv, IF(JSON_VALUE(Priv, '$.access') & 4096, 'Y', 'N') AS Index_priv, IF(JSON_VALUE(Priv, '$.access') & 8192, 'Y', 'N') AS Alter_priv, IF(JSON_VALUE(Priv, '$.access') & 16384, 'Y', 'N') AS Show_db_priv, IF(JSON_VALUE(Priv, '$.access') & 32768, 'Y', 'N') AS Super_priv, IF(JSON_VALUE(Priv, '$.access') & 65536, 'Y', 'N') AS Create_tmp_table_priv, IF(JSON_VALUE(Priv, '$.access') & 131072, 'Y', 'N') AS Lock_tables_priv, IF(JSON_VALUE(Priv, '$.access') & 262144, 'Y', 'N') AS Execute_priv, IF(JSON_VALUE(Priv, '$.access') & 524288, 'Y', 'N') AS Repl_slave_priv, IF(JSON_VALUE(Priv, '$.access') & 1048576, 'Y', 'N') AS Repl_client_priv, IF(JSON_VALUE(Priv, '$.access') & 2097152, 'Y', 'N') AS Create_view_priv, IF(JSON_VALUE(Priv, '$.access') & 4194304, 'Y', 'N') AS Show_view_priv, IF(JSON_VALUE(Priv, '$.access') & 8388608, 'Y', 'N') AS Create_routine_priv, IF(JSON_VALUE(Priv, '$.access') & 16777216, 'Y', 'N') AS Alter_routine_priv, IF(JSON_VALUE(Priv, '$.access') & 33554432, 'Y', 'N') AS Create_user_priv, IF(JSON_VALUE(Priv, '$.access') & 67108864, 'Y', 'N') AS Event_priv, IF(JSON_VALUE(Priv, '$.access') & 134217728, 'Y', 'N') AS Trigger_priv, IF(JSON_VALUE(Priv, '$.access') & 268435456, 'Y', 'N') AS Create_tablespace_priv, IF(JSON_VALUE(Priv, '$.access') & 536870912, 'Y', 'N') AS Delete_history_priv, ELT(IFNULL(JSON_VALUE(Priv, '$.ssl_type'), 0) + 1, '', 'ANY','X509', 'SPECIFIED') AS ssl_type, IFNULL(JSON_VALUE(Priv, '$.ssl_cipher'), '') AS ssl_cipher, IFNULL(JSON_VALUE(Priv, '$.x509_issuer'), '') AS x509_issuer, IFNULL(JSON_VALUE(Priv, '$.x509_subject'), '') AS x509_subject, CAST(IFNULL(JSON_VALUE(Priv, '$.max_questions'), 0) AS UNSIGNED) AS max_questions, CAST(IFNULL(JSON_VALUE(Priv, '$.max_updates'), 0) AS UNSIGNED) AS max_updates, CAST(IFNULL(JSON_VALUE(Priv, '$.max_connections'), 0) AS UNSIGNED) AS max_connections, CAST(IFNULL(JSON_VALUE(Priv, '$.max_user_connections'), 0) AS SIGNED) AS max_user_connections, IFNULL(JSON_VALUE(Priv, '$.plugin'), '') AS plugin, IFNULL(JSON_VALUE(Priv, '$.authentication_string'), '') AS authentication_string, IF(IFNULL(JSON_VALUE(Priv, '$.password_last_changed'), 1) = 0, 'Y', 'N') AS password_expired, ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role, IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role, CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time FROM global_priv; Connection ID (thread ID): 1 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_ with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_ca che=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_f or_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1048576 1048576 processes Max open files 1048576 1048576 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 79845 79845 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h Segmentation fault (core dumped) Tracing the core with GDB confirms the segfault occurs while attempting to use MD5: Thread 1 (Thread 0x7f063e566000 (LWP 916738)): #0 0x00007f063e85595b in kill () at ../sysdeps/unix/syscall-template.S:120 #1 0x000055a56d657afd in handle_fatal_signal (sig=<optimized out>) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/signal_handler.cc:356 #2 <signal handler called> #3 0x0000000000000000 in ?? () #4 0x000055a56d85930f in md5_input (len=4455, buf=0x55a570804f18 "select `mysql`.`global_priv`.`Host` AS `Host`,`mysql`.`global_priv`.`User` AS `User`,if(json_value(`mysql`.`global_priv`.`Priv`,'$.plugin') in ('mysql_native_password','mysql_old_password'),ifnull(jso"..., context=0x7ffc7442b470) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/mysys_ssl/my_md5.cc:65 #5 my_md5 (digest=0x7ffc7442b4f0 "", buf=0x55a570804f18 "select `mysql`.`global_priv`.`Host` AS `Host`,`mysql`.`global_priv`.`User` AS `User`,if(json_value(`mysql`.`global_priv`.`Priv`,'$.plugin') in ('mysql_native_password','mysql_old_password'),ifnull(jso"..., len=4455) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/mysys_ssl/my_md5.cc:90 #6 0x000055a56d4e1fc7 in TABLE_LIST::calc_md5 (this=<optimized out>, buffer=0x7ffc7442b570 "") at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/table.cc:5556 #7 0x000055a56d4d6474 in mysql_register_view (thd=0x55a57018ca48, view=0x55a5707db340, mode=VIEW_CREATE_NEW) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/sql_view.cc:968 #8 0x000055a56d4d74ce in mysql_create_view (thd=<optimized out>, views=0x55a5707db340, mode=VIEW_CREATE_NEW) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/sql_view.cc:643 #9 0x000055a56d412010 in mysql_execute_command (thd=0x55a57018ca48) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/sql_parse.cc:5858 #10 0x000055a56d414d0b in mysql_parse (thd=0x55a57018ca48, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/sql_parse.cc:8100 #11 0x000055a56d402906 in bootstrap (file=<optimized out>) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/sql_class.h:232 #12 0x000055a56d2e0ba6 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/mariadb-10.5.13-1.el9.x86_64/sql/mysqld.cc:5574 #13 0x00007f063e840560 in __libc_start_call_main (main=main@entry=0x55a56d296660 <main(int, char**)>, argc=argc@entry=11, argv=argv@entry=0x7ffc74433c38) at ../sysdeps/nptl/libc_start_call_main.h:58 #14 0x00007f063e84060c in __libc_start_main_impl (main=0x55a56d296660 <main(int, char**)>, argc=11, argv=0x7ffc74433c38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc74433c28) at ../csu/libc-start.c:409 #15 0x000055a56d2cb865 in _start () Version-Release number of selected component (if applicable): mariadb-10.5.13-1.el9.x86_64 How reproducible: Always Steps to Reproduce: 1. fips-mode-setup --enable 2. reboot 3. dnf install -y mariadb-server 4. mysql_install_db Actual results: The mariadb server crashes while executing an initial bootstrasp SQL statement Expected results: The server shouldn't crash Additional info: It seems the very same issue has been reported upstream for Percona's fork of MySQL [1], and is claimed to be fixed, although I haven't looked at it so far. [1] https://jira.percona.com/browse/PS-4531