Bug 1065676

Summary: [abrt] mariadb-server: _ma_read_block_record(): mysqld killed by SIGSEGV
Product: [Fedora] Fedora Reporter: rng <rgasch>
Component: mariadbAssignee: Honza Horak <hhorak>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: hhorak, jdornak, rgasch
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/cb6192d3c367d247a26a3279a65c444eb7fcc6b2
Whiteboard: abrt_hash:ee8b8b16384f17fa76d0998ffdb9223cdd04365c
Fixed In Version: mariadb-5.5.37-1.fc19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-19 10:06:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: core_backtrace
none
File: dso_list
none
File: environ
none
File: exploitable
none
File: limits
none
File: maps
none
File: open_fds
none
File: proc_pid_status
none
File: var_log_messages
none
The SQL schema and data needed to run the sql statement causing the crash none

Description rng 2014-02-15 18:03:03 UTC
Description of problem:
The following SQL statement generates a "SQLSTATE[HY000]: General error: 2006 MySQL server has gone away" error: 

SELECT COUNT(*) , a.spoo_value AS "1__autojoinUser__name_first" , b.spoo_value AS "2__autojoinUser__name" , c.spoo_value AS "2__autojoinUser__city" , d.spoo_value AS "1__autojoinUser__name_last" , e.spoo_value AS "2__autojoinUser__image" , f.spoo_value AS "3__autojoinUser__image" , g.spoo_value AS "1__autojoinUser__image" , h.spoo_value AS "3__autojoinUser__city" , i.spoo_value AS "1__autojoinUser__coverimage" , j.spoo_value AS "1__autojoinUser__city" , k.spoo_value AS "1__autojoinUser__musicstyle" , l.spoo_value AS "3__autojoinUser__name" , m.spoo_value AS "2__autojoinUser__coverimage" , n.spoo_value AS "3__autojoinUser__coverimage" , o.ssco_value AS "__autojoinUser__userTypeID" , p.ssco_value AS "__autojoinUser__profileAlbumID" FROM bpsocial_post AS tbl LEFT JOIN bpsocial_profile_object_option a ON a.spoo_uid = sspo_uid AND a.spoo_option_id = 1 LEFT JOIN bpsocial_profile_object_option b ON b.spoo_uid = sspo_uid AND b.spoo_option_id = 2 LEFT JOIN bpsocial_profile_object_option c ON c.spoo_uid = sspo_uid AND c.spoo_option_id = 3 LEFT JOIN bpsocial_profile_object_option d ON d.spoo_uid = sspo_uid AND d.spoo_option_id = 5 LEFT JOIN bpsocial_profile_object_option e ON e.spoo_uid = sspo_uid AND e.spoo_option_id = 4 LEFT JOIN bpsocial_profile_object_option f ON f.spoo_uid = sspo_uid AND f.spoo_option_id = 11 LEFT JOIN bpsocial_profile_object_option g ON g.spoo_uid = sspo_uid AND g.spoo_option_id = 7 LEFT JOIN bpsocial_profile_object_option h ON h.spoo_uid = sspo_uid AND h.spoo_option_id = 10 LEFT JOIN bpsocial_profile_object_option i ON i.spoo_uid = sspo_uid AND i.spoo_option_id = 18 LEFT JOIN bpsocial_profile_object_option j ON j.spoo_uid = sspo_uid AND j.spoo_option_id = 6 LEFT JOIN bpsocial_profile_object_option k ON k.spoo_uid = sspo_uid AND k.spoo_option_id = 8 LEFT JOIN bpsocial_profile_object_option l ON l.spoo_uid = sspo_uid AND l.spoo_option_id = 9 LEFT JOIN bpsocial_profile_object_option m ON m.spoo_uid = sspo_uid AND m.spoo_option_id = 19 LEFT JOIN bpsocial_profile_object_option n ON n.spoo_uid = sspo_uid AND n.spoo_option_id = 20 LEFT JOIN bpsocial_config o ON o.ssco_domain = CONCAT('user_', sspo_uid) AND o.ssco_name = 'userTypeID' LEFT JOIN bpsocial_config p ON p.ssco_domain = CONCAT('user_', sspo_uid) AND p.ssco_name = 'profileAlbumID' WHERE tbl.sspo_uid IN (SELECT ssli_oid FROM bpsocial_like WHERE ssli_ot = 'bpsocial_profile' AND ssli_uid = 5 UNION ALL SELECT 5) GROUP BY a.spoo_value, b.spoo_value, c.spoo_value, d.spoo_value, e.spoo_value, f.spoo_value, g.spoo_value, h.spoo_value, i.spoo_value, j.spoo_value, k.spoo_value, l.spoo_value, m.spoo_value, n.spoo_value, o.ssco_value, p.ssco_value

I can provide you with the schema and data which resulted in this crash.

Version-Release number of selected component:
mariadb-server-5.5.34-3.fc20

Additional info:
reporter:       libreport-2.1.12
backtrace_rating: 4
cmdline:        /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
crash_function: _ma_read_block_record
executable:     /usr/libexec/mysqld
kernel:         3.12.10-300.fc20.x86_64
runlevel:       N 5
type:           CCpp
uid:            27

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 _ma_read_block_record at /usr/src/debug/mariadb-5.5.34/storage/maria/ma_blockrec.c:5119
 #1 _ma_cmp_block_unique at /usr/src/debug/mariadb-5.5.34/storage/maria/ma_blockrec.c:5159
 #2 _ma_check_unique at /usr/src/debug/mariadb-5.5.34/storage/maria/ma_unique.c:66
 #3 maria_write at /usr/src/debug/mariadb-5.5.34/storage/maria/ma_write.c:135
 #4 ha_write_tmp_row at /usr/src/debug/mariadb-5.5.34/sql/sql_class.h:4358
 #5 end_unique_update at /usr/src/debug/mariadb-5.5.34/sql/sql_select.cc:18149
 #6 evaluate_join_record at /usr/src/debug/mariadb-5.5.34/sql/sql_select.cc:16922
 #7 sub_select at /usr/src/debug/mariadb-5.5.34/sql/sql_select.cc:16703
 #8 evaluate_join_record at /usr/src/debug/mariadb-5.5.34/sql/sql_select.cc:16922
 #9 sub_select at /usr/src/debug/mariadb-5.5.34/sql/sql_select.cc:16703

Comment 1 rng 2014-02-15 18:03:11 UTC
Created attachment 863592 [details]
File: backtrace

Comment 2 rng 2014-02-15 18:03:14 UTC
Created attachment 863593 [details]
File: cgroup

Comment 3 rng 2014-02-15 18:03:17 UTC
Created attachment 863594 [details]
File: core_backtrace

Comment 4 rng 2014-02-15 18:03:19 UTC
Created attachment 863595 [details]
File: dso_list

Comment 5 rng 2014-02-15 18:03:22 UTC
Created attachment 863596 [details]
File: environ

Comment 6 rng 2014-02-15 18:03:24 UTC
Created attachment 863597 [details]
File: exploitable

Comment 7 rng 2014-02-15 18:03:27 UTC
Created attachment 863598 [details]
File: limits

Comment 8 rng 2014-02-15 18:03:30 UTC
Created attachment 863599 [details]
File: maps

Comment 9 rng 2014-02-15 18:03:37 UTC
Created attachment 863600 [details]
File: open_fds

Comment 10 rng 2014-02-15 18:03:40 UTC
Created attachment 863601 [details]
File: proc_pid_status

Comment 11 rng 2014-02-15 18:03:42 UTC
Created attachment 863602 [details]
File: var_log_messages

Comment 12 Honza Horak 2014-02-17 11:32:17 UTC
(In reply to rng from comment #0)
> Description of problem:
> The following SQL statement generates a "SQLSTATE[HY000]: General error:
> 2006 MySQL server has gone away" error: 
> 
> SELECT COUNT(*) ...
> I can provide you with the schema and data which resulted in this crash.

If you can, that would be great. Do I understand correctly, that you're able to reproduce repeatedly?

> Version-Release number of selected component:
> mariadb-server-5.5.34-3.fc20

What about new version, mariadb-5.5.35-3.fc20 -- does it also suffer from this issue?

Comment 13 rng 2014-02-17 16:40:12 UTC
Let me test mariadb-5.5.35-3.fc20 tonight (this is a bit complicated because I've switched to MySql in order to be able to work, so I need to switch back again). I'll get back to you within 24 hours.

Comment 14 rng 2014-02-18 23:50:56 UTC
Sorry, I didn't have time to check this today ... I will do so tomorrow.

Comment 15 rng 2014-02-20 01:04:40 UTC
Created attachment 865305 [details]
The SQL schema and data needed to run the sql statement causing the crash

Simply load this file into your database and run the SQL statement given in the bug report against it. You will get the "server has gone away" error.

Comment 16 rng 2014-02-20 01:06:00 UTC
MariaDB 5.5.35-3.fc20 still has the same problem. Please see my previous message (comment 15) for a reproducible case.

Comment 17 Honza Horak 2014-02-24 17:38:45 UTC
Thanks, it's easily reproducible on my machine as well with SQL command from comment #15 and a command from comment #0; also crashes with a bit simplified following SQL command:

SELECT Count(*)
FROM   bpsocial_post AS tbl
       LEFT JOIN bpsocial_profile_object_option a
              ON a.spoo_uid = sspo_uid
                 AND a.spoo_option_id = 1
       LEFT JOIN bpsocial_profile_object_option b
              ON b.spoo_uid = sspo_uid
                 AND b.spoo_option_id = 2
       LEFT JOIN bpsocial_profile_object_option c
              ON c.spoo_uid = sspo_uid
                 AND c.spoo_option_id = 3
       LEFT JOIN bpsocial_profile_object_option d
              ON d.spoo_uid = sspo_uid
                 AND d.spoo_option_id = 5
       LEFT JOIN bpsocial_profile_object_option e
              ON e.spoo_uid = sspo_uid
                 AND e.spoo_option_id = 4
       LEFT JOIN bpsocial_profile_object_option f
              ON f.spoo_uid = sspo_uid
                 AND f.spoo_option_id = 11
       LEFT JOIN bpsocial_profile_object_option g
              ON g.spoo_uid = sspo_uid
                 AND g.spoo_option_id = 7
       LEFT JOIN bpsocial_profile_object_option h
              ON h.spoo_uid = sspo_uid
                 AND h.spoo_option_id = 10
       LEFT JOIN bpsocial_profile_object_option i
              ON i.spoo_uid = sspo_uid
                 AND i.spoo_option_id = 18
       LEFT JOIN bpsocial_profile_object_option j
              ON j.spoo_uid = sspo_uid
                 AND j.spoo_option_id = 6
GROUP  BY a.spoo_value,
          b.spoo_value,
          c.spoo_value,
          d.spoo_value,
          e.spoo_value,
          f.spoo_value,
          g.spoo_value,
          h.spoo_value,
          i.spoo_value,
          j.spoo_value;


Surprisingly, it doesn't crash with community mysql-5.5.35, so it is probably a MariaDB-only bug.

Reported upstream as: https://mariadb.atlassian.net/browse/MDEV-5724

Comment 18 Honza Horak 2014-03-10 13:11:12 UTC
Relevant comment from the upstream bug report (Michael Widenius commented on MDEV-5724):
----------------------------------------

The problem is that the temporary table we have to create to store the interim result has a row length of 300K.  We try to allocate on row on the stack (which works) but next call causes a segmentation fault.

I will fix this by not allocating bigger rows than 16K on the stack for rows.

To avoid this bug, do one of the following:
- Increase size of the thread-stack startup options (288K by default). 300K was needed for the above case.
- Change some of the big VARCHAR to BLOB as BLOB only takes 16 bytes in the record buffer

Comment 19 Honza Horak 2014-05-19 10:06:30 UTC
As mariadb-5.5.37-1.fc19 is already in the stable repo, this bug should also be already fixed, as upstream states this is fixed in 5.5.37. Please, re-open if you are still able to reproduce this.