Bug 1352946

Summary: Mariadb server crashes when galera replication is enabled
Product: [Fedora] Fedora Reporter: Damien Ciabrini <dciabrin>
Component: mariadbAssignee: Damien Ciabrini <dciabrin>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 24CC: dciabrin, hhorak, jdornak, jstanek, lordfoom, mbayer, mmuzila, tobias.jungel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mariadb-10.1.17-1.fc24 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-15 22:52:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Damien Ciabrini 2016-07-05 14:48:34 UTC
Description of problem:
Mariadb server segfaults when it is configured to enable Galera replication.


Version-Release number of selected component (if applicable):
mariadb-10.1.14-1.fc24.x86_64

How reproducible:
always

Steps to Reproduce:
1. Ensure that mysql has created files on disk, create it otherwise

   mysql_install_db
   chmod -R mysql. /var/lib/mysql
   
2. Enable galera by ensuring the following variable are set in /etc/my.cnf.d/mariadb-server.cnf

    [galera]
    # Mandatory settings
    wsrep_on=ON
    wsrep_provider=/usr/lib64/galera/libgalera_smm.so
    wsrep_cluster_address=gcomm://
    wsrep_cluster_name=galera
    binlog_format=row
    default_storage_engine=InnoDB
    innodb_autoinc_lock_mode=2

3. Start a new galera cluster

    [root@fedora ~]# mysqld_safe --wsrep-new-cluster

Actual results:
mysqld segfaults and mysqld_safe exit in error

    160705 12:34:43 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
    160705 12:34:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    160705 12:34:43 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.X8ZxeH' --pid-file='/var/lib/mysql/fedora2.localdomain-recover.pid'
    160705 12:34:46 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    /usr/bin/mysqld_safe: line 188:  1799 Segmentation fault      (core dumped) nohup /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep-new-cluster --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1 < /dev/null >> /var/log/mariadb/mariadb.log 2>&1
    160705 12:34:46 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

A stack trace is dumped in /var/log/mariadb/mariadb.log:

   Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    2016-07-05 12:34:46 139974166951680 [Note] WSREP: Member 0.0 () synced with group.
    2016-07-05 12:34:46 139974166951680 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
    stack_bottom = 0x7f4e54072d18 thread_stack 0x48400
    /usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x565322e6f39e]
    /usr/libexec/mysqld(handle_fatal_signal+0x3f5)[0x5653229b4d25]
    /lib64/libpthread.so.0(+0x10c10)[0x7f4e5a210c10]
    /usr/libexec/mysqld(+0x53928e)[0x56532295328e]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM19process_conf_changeEPvRK15wsrep_view_infoiNS_10Replicator5StateEl+0x38a)[0x7f4e553e178a]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x54b)[0x7f4e553baa7b]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x76)[0x7f4e553bc166]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x86)[0x7f4e553e0f46]
    /usr/lib64/galera/libgalera_smm.so(galera_recv+0x2e)[0x7f4e553fb61e]
    /usr/libexec/mysqld(+0x5483f7)[0x5653229623f7]
    /usr/libexec/mysqld(start_wsrep_THD+0x484)[0x565322951ef4]
    /lib64/libpthread.so.0(+0x75ba)[0x7f4e5a2075ba]
    /lib64/libc.so.6(clone+0x6d)[0x7f4e5871a7cd]

Expected results:
mysqld should keep running, a new galera cluster should be available.
Checking that galera is started would require running the following:

[root@fedora ~]# mysql -e "show status like 'wsrep_cluster%';"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 1                                    |
| wsrep_cluster_size       | 1                                    |
| wsrep_cluster_state_uuid | c2f94f44-3fce-11e6-a869-17918323b495 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
 

Additional info:
As reported upstream [1,2] the segfault seems to be related to a bug which is triggered when compiling with gcc 6.0 (or gcc5.x --std=c++11)

[1] https://jira.mariadb.org/browse/MDEV-10335
[2] https://github.com/codership/mysql-wsrep/issues/267

Comment 1 Damien Ciabrini 2016-08-12 14:22:02 UTC
Fixed in https://github.com/codership/mysql-wsrep/pull/274

Comment 2 Tobias Jungel 2016-08-29 14:01:12 UTC
running into the same issue here and just found this report here. Any chance of seeing a new packet anytime soon?

Comment 3 Honza Horak 2016-08-29 14:37:30 UTC
It will be fixed with 10.1.17 (to be released every day, so we won't probably manage to go through testing repo much sooner):
https://github.com/MariaDB/server/commit/fec296cc10f0d1319e032b72e92e3c824b7fc390

Sorry for the delay, I admin we should have go with downstream patch sooner.

Comment 5 Tobias Jungel 2016-09-13 07:54:23 UTC
yeah, it would be nice to go with the patch sooner. At least in F24 Galera is not working at all. But maybe I am sort of insane to use Fedora as a Server ;)

But since there is now as well a CVE reported (1375201) I have a bit of a hope, that this will be done faster *sncr*.

Comment 6 Fedora Update System 2016-09-13 14:46:59 UTC
mariadb-10.1.17-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7411ec62f

Comment 7 Fedora Update System 2016-09-14 02:52:06 UTC
mariadb-10.1.17-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7411ec62f

Comment 8 Fedora Update System 2016-09-15 22:52:47 UTC
mariadb-10.1.17-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 lordfoom 2016-09-19 09:07:40 UTC
Hi there,

I'm a newbie, so please bear with me if I'm doing something stupid.

This morning I attempted to update my Fedora 24 system (running KDE plasma), and got the following error:

Error: Transaction check error:
  file /usr/lib64/mysql/plugin/dialog.so from install of mariadb-common-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-server-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysql from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqladmin from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlbinlog from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlcheck from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqldump from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlimport from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlshow from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlslap from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysql.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysqladmin.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysqlbinlog.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
<SNIP>

and continues in that vein for a while.

Now, I installed Percona 5.7 from rpms because I couldn't do it from the repos. I don't have mariadb installed and don't want it installed (want to match production). 

Is there anything I can do to stop fedora from installing mariadb when I run a system update? I've tried unselecting the packages in Apper, but it says the packages are required.

Long story short: is there a bug introduced by the bugfix, or have I done something silly?

Thank you.

Comment 10 lordfoom 2016-09-19 09:08:32 UTC
And the reason I am commenting on this issue is because the update note said this is the issue the mariadb install relates to.