Bug 1352946 - Mariadb server crashes when galera replication is enabled
Summary: Mariadb server crashes when galera replication is enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mariadb
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Damien Ciabrini
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-05 14:48 UTC by Damien Ciabrini
Modified: 2016-10-27 06:00 UTC (History)
8 users (show)

Fixed In Version: mariadb-10.1.17-1.fc24
Clone Of:
Environment:
Last Closed: 2016-09-15 22:52:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/codership mysql-wsrep issues 267 0 None None None 2016-08-12 14:22:01 UTC

Internal Links: 1377242

Description Damien Ciabrini 2016-07-05 14:48:34 UTC
Description of problem:
Mariadb server segfaults when it is configured to enable Galera replication.


Version-Release number of selected component (if applicable):
mariadb-10.1.14-1.fc24.x86_64

How reproducible:
always

Steps to Reproduce:
1. Ensure that mysql has created files on disk, create it otherwise

   mysql_install_db
   chmod -R mysql. /var/lib/mysql
   
2. Enable galera by ensuring the following variable are set in /etc/my.cnf.d/mariadb-server.cnf

    [galera]
    # Mandatory settings
    wsrep_on=ON
    wsrep_provider=/usr/lib64/galera/libgalera_smm.so
    wsrep_cluster_address=gcomm://
    wsrep_cluster_name=galera
    binlog_format=row
    default_storage_engine=InnoDB
    innodb_autoinc_lock_mode=2

3. Start a new galera cluster

    [root@fedora ~]# mysqld_safe --wsrep-new-cluster

Actual results:
mysqld segfaults and mysqld_safe exit in error

    160705 12:34:43 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
    160705 12:34:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    160705 12:34:43 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.X8ZxeH' --pid-file='/var/lib/mysql/fedora2.localdomain-recover.pid'
    160705 12:34:46 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
    /usr/bin/mysqld_safe: line 188:  1799 Segmentation fault      (core dumped) nohup /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep-new-cluster --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1 < /dev/null >> /var/log/mariadb/mariadb.log 2>&1
    160705 12:34:46 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

A stack trace is dumped in /var/log/mariadb/mariadb.log:

   Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    2016-07-05 12:34:46 139974166951680 [Note] WSREP: Member 0.0 () synced with group.
    2016-07-05 12:34:46 139974166951680 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
    stack_bottom = 0x7f4e54072d18 thread_stack 0x48400
    /usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x565322e6f39e]
    /usr/libexec/mysqld(handle_fatal_signal+0x3f5)[0x5653229b4d25]
    /lib64/libpthread.so.0(+0x10c10)[0x7f4e5a210c10]
    /usr/libexec/mysqld(+0x53928e)[0x56532295328e]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM19process_conf_changeEPvRK15wsrep_view_infoiNS_10Replicator5StateEl+0x38a)[0x7f4e553e178a]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x54b)[0x7f4e553baa7b]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x76)[0x7f4e553bc166]
    /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x86)[0x7f4e553e0f46]
    /usr/lib64/galera/libgalera_smm.so(galera_recv+0x2e)[0x7f4e553fb61e]
    /usr/libexec/mysqld(+0x5483f7)[0x5653229623f7]
    /usr/libexec/mysqld(start_wsrep_THD+0x484)[0x565322951ef4]
    /lib64/libpthread.so.0(+0x75ba)[0x7f4e5a2075ba]
    /lib64/libc.so.6(clone+0x6d)[0x7f4e5871a7cd]

Expected results:
mysqld should keep running, a new galera cluster should be available.
Checking that galera is started would require running the following:

[root@fedora ~]# mysql -e "show status like 'wsrep_cluster%';"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 1                                    |
| wsrep_cluster_size       | 1                                    |
| wsrep_cluster_state_uuid | c2f94f44-3fce-11e6-a869-17918323b495 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
 

Additional info:
As reported upstream [1,2] the segfault seems to be related to a bug which is triggered when compiling with gcc 6.0 (or gcc5.x --std=c++11)

[1] https://jira.mariadb.org/browse/MDEV-10335
[2] https://github.com/codership/mysql-wsrep/issues/267

Comment 1 Damien Ciabrini 2016-08-12 14:22:02 UTC
Fixed in https://github.com/codership/mysql-wsrep/pull/274

Comment 2 Tobias Jungel 2016-08-29 14:01:12 UTC
running into the same issue here and just found this report here. Any chance of seeing a new packet anytime soon?

Comment 3 Honza Horak 2016-08-29 14:37:30 UTC
It will be fixed with 10.1.17 (to be released every day, so we won't probably manage to go through testing repo much sooner):
https://github.com/MariaDB/server/commit/fec296cc10f0d1319e032b72e92e3c824b7fc390

Sorry for the delay, I admin we should have go with downstream patch sooner.

Comment 5 Tobias Jungel 2016-09-13 07:54:23 UTC
yeah, it would be nice to go with the patch sooner. At least in F24 Galera is not working at all. But maybe I am sort of insane to use Fedora as a Server ;)

But since there is now as well a CVE reported (1375201) I have a bit of a hope, that this will be done faster *sncr*.

Comment 6 Fedora Update System 2016-09-13 14:46:59 UTC
mariadb-10.1.17-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7411ec62f

Comment 7 Fedora Update System 2016-09-14 02:52:06 UTC
mariadb-10.1.17-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b7411ec62f

Comment 8 Fedora Update System 2016-09-15 22:52:47 UTC
mariadb-10.1.17-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 lordfoom 2016-09-19 09:07:40 UTC
Hi there,

I'm a newbie, so please bear with me if I'm doing something stupid.

This morning I attempted to update my Fedora 24 system (running KDE plasma), and got the following error:

Error: Transaction check error:
  file /usr/lib64/mysql/plugin/dialog.so from install of mariadb-common-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-server-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysql from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqladmin from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlbinlog from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlcheck from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqldump from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlimport from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlshow from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/bin/mysqlslap from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysql.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysqladmin.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
  file /usr/share/man/man1/mysqlbinlog.1.gz from install of mariadb-3:10.1.17-1.fc24.x86_64 conflicts with file from package Percona-Server-client-57-5.7.10-3.1.el7.x86_64
<SNIP>

and continues in that vein for a while.

Now, I installed Percona 5.7 from rpms because I couldn't do it from the repos. I don't have mariadb installed and don't want it installed (want to match production). 

Is there anything I can do to stop fedora from installing mariadb when I run a system update? I've tried unselecting the packages in Apper, but it says the packages are required.

Long story short: is there a bug introduced by the bugfix, or have I done something silly?

Thank you.

Comment 10 lordfoom 2016-09-19 09:08:32 UTC
And the reason I am commenting on this issue is because the update note said this is the issue the mariadb install relates to.


Note You need to log in before you can comment on or make changes to this bug.