Bug 842052

Summary: mysqld crash during mysqldump
Product: Red Hat Enterprise Linux 6 Reporter: Levente Farkas <lfarkas>
Component: mysqlAssignee: Honza Horak <hhorak>
Status: CLOSED ERRATA QA Contact: qe-baseos-daemons
Severity: low Docs Contact:
Priority: low    
Version: 6.3CC: azelinka, byte, hhorak, kvolny, ohudlick, praiskup
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mysql-5.1.71-1.el6 Doc Type: Bug Fix
Doc Text:
Cause: MySQL daemon worked with uninitialized memory when accessing non-nullable GEOMETRY types. Consequence: Daemon could crash when mysqldump was performed. Fix: Memory is initialized properly. Result: mysqldump doesn't make daemon crashed any more.
Story Points: ---
Clone Of:
: 1035868 (view as bug list) Environment:
Last Closed: 2013-11-21 20:40:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1035868    
Attachments:
Description Flags
my.cnf
none
backtrace none

Description Levente Farkas 2012-07-21 11:21:29 UTC
when i try to dump our current database i always got errors and crash in mysqld log files after a simple:
mysqldump --all-databases
where one of the database is google maps (provides by google).

mysql-5.1.61-4.el6.x86_64

upstream bug: http://bugs.mysql.com/bug.php?id=65971

here is the log from mysql:
------------------------------------------------
120721 13:13:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120721 13:13:51  InnoDB: Initializing buffer pool, size = 8.0M
120721 13:13:51  InnoDB: Completed initialization of buffer pool
120721 13:13:51  InnoDB: Started; log sequence number 0 2417296880
120721 13:13:52 [Note] Event Scheduler: Loaded 0 events
120721 13:13:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
11:15:39 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=6
max_threads=151
thread_count=5
connection_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 572953 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x20a2150
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f7778654d98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x84c049]
/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a0fa3]
/lib64/libpthread.so.0() [0x341940f500]
/lib64/libc.so.6(memcpy+0xa0) [0x3418c88ba0]
/usr/libexec/mysqld(String::copy(char const*, unsigned int, charset_info_st*)+0x91) [0x5b7d11]
/usr/libexec/mysqld(String::copy(char const*, unsigned int, charset_info_st*, charset_info_st*, unsigned int*)+0xfa) [0x5b7e1a]
/usr/libexec/mysqld() [0x6c02ec]
/usr/libexec/mysqld() [0x6c0733]
/usr/libexec/mysqld() [0x6bf0b4]
/usr/libexec/mysqld(get_all_tables(THD*, TABLE_LIST*, Item*)+0x147) [0x6c83e7]
/usr/libexec/mysqld(get_schema_tables_result(JOIN*, enum_schema_table_state)+0x1f7) [0x6b8907]
/usr/libexec/mysqld(JOIN::exec()+0x4ed) [0x631a9d]
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x15c) [0x62dfbc]
/usr/libexec/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x633734]
/usr/libexec/mysqld() [0x5c747a]
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x4ce) [0x5c84ae]
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2d3) [0x5cda33]
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd05) [0x5cf865]
/usr/libexec/mysqld(do_command(THD*)+0xea) [0x5d03ba]
/usr/libexec/mysqld(handle_one_connection+0x23d) [0x5c39ed]
/lib64/libpthread.so.0() [0x3419407851]
/lib64/libc.so.6(clone+0x6d) [0x3418ce76dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f774c0049e0): is an invalid pointer
Connection ID (thread ID): 5
Status: NOT_KILLED
------------------------------------------------

Comment 2 Honza Horak 2012-07-27 11:23:45 UTC
Thank you for reporting. I've gone through the discussion at [1], but I'm still not able to reproduce such failure. Are you able to reproduce it with the test case attached at [1] on a fresh datadir? If so, are there any other steps required or does it fail even on a fresh and empty DB on your machine?

[1] http://bugs.mysql.com/bug.php?id=65971

Comment 3 Levente Farkas 2012-07-27 11:33:25 UTC
it's always happened on our server. and even mysql (ie. oracle) can reproduce it (as you can see in the above bz). unfortunately the database is the commercial google maps api so i assume i can't send it to you.

anyway we try to drop the database and import it then the same happened:-(

so currently either mysql fix it and gives a patch or we'd have to update from 5.1 to 5.5 in which this's already fixed.

Comment 4 Honza Horak 2012-07-27 12:33:54 UTC
Is it possible to provide your my.cnf file, please?

Comment 5 Levente Farkas 2012-07-27 12:37:15 UTC
Created attachment 600765 [details]
my.cnf

Comment 6 Honza Horak 2012-07-31 16:04:32 UTC
Thank you for the my.cnf file. Unfortunately, I'm still not able to reproduce the failure. I believe a better backtrace could help, so I'd like to ask you for the following:

1. install mysql-debuginfo package if not done already
2. run mysqld with "--core-file" option (otherwise no coredump is generated)
3. generate a backtrace using the coredump file and e.g. gdb's "bt full" command 

Please, be aware that the coredump file is usually very large and check the backtrace and hide all sensitive data if needed.

Comment 7 Honza Horak 2012-08-01 15:35:44 UTC
Created attachment 601772 [details]
backtrace

I've finally reproduced it, using:
$ valgrind --malloc-fill=ff /usr/libexec/mysqld --core-file

and it is also possible using:
MALLOC_PERTURB_=250 /usr/libexec/mysqld --core-file

The attached file is a coredump file.

Comment 8 Levente Farkas 2012-08-03 20:38:38 UTC
mysql has a fix for it. dare i ask for an src.rpm from rh?

Comment 9 Tom Lane 2012-08-04 18:16:55 UTC
(In reply to comment #8)
> mysql has a fix for it. dare i ask for an src.rpm from rh?

The way to get the priority of this bug raised to the point where it might get fixed in RHEL6 (without any upstream fix) is to open an issue through your regular customer support contact.

Comment 11 Honza Horak 2013-04-02 07:46:11 UTC
*** Bug 880104 has been marked as a duplicate of this bug. ***

Comment 18 errata-xmlrpc 2013-11-21 20:40:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1647.html