Bug 1385720

Summary: mysql server crashed after VM restart
Product: Red Hat Enterprise Linux 6 Reporter: Tomas Pelka <tpelka>
Component: mysqlAssignee: Jakub Dorňák <jdornak>
Status: CLOSED INSUFFICIENT_DATA QA Contact: qe-baseos-daemons
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.8CC: byte, databases-maint, hhorak, tpelka
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-25 12:25:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
my.cnf none

Description Tomas Pelka 2016-10-17 15:00:38 UTC 
Description of problem:
61017 09:52:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161017  9:52:18  InnoDB: Initializing buffer pool, size = 8.0M
161017  9:52:18  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 13 1572000176
161017  9:52:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 13 1572005480
161017  9:52:18  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 13:52:18 UTC - mysqld got signal 11 ;
53 This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
54 We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=0
max_threads=300
55 thread_count=0
connection_count=0
56 It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 664112 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
57 58 59 60 61 62 63 64 65 66 67 stack_bottom = 0 thread_stack 0x40000
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 /usr/libexec/mysqld(my_print_stacktrace+0x29) [0x8509f9]
/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a3f13]
/lib64/libpthread.so.0() [0x3532a0f7e0]
/usr/libexec/mysqld(page_cur_insert_rec_low+0x288) [0x7999b8]
/usr/libexec/mysqld(page_cur_parse_insert_rec+0x4eb) [0x79a55b]
/usr/libexec/mysqld() [0x787a0d]
/usr/libexec/mysqld(recv_recover_page+0x34f) [0x7894cf]
/usr/libexec/mysqld(buf_page_io_complete+0x548) [0x74a888]
/usr/libexec/mysqld(fil_aio_wait+0xfa) [0x762a3a]
/usr/libexec/mysqld() [0x7c8ab0]
/lib64/libpthread.so.0() [0x3532a07aa1]
/lib64/libc.so.6(clone+0x6d) [0x35326e8aad]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
161017 09:52:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
161017 10:52:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161017 10:52:26  InnoDB: Initializing buffer pool, size = 8.0M
161017 10:52:26  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 13 1572000176
161017 10:52:26  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 13 1572005480
161017 10:52:26  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 14:52:26 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

58 key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=0
59 max_threads=300
thread_count=0
connection_count=0
60 It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 664112 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
61 Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
62 63 64 65 66 67 68 69 70 stack_bottom = 0 thread_stack 0x40000
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 /usr/libexec/mysqld(my_print_stacktrace+0x29) [0x8509f9]
/usr/libexec/mysqld(handle_fatal_signal+0x483) [0x6a3f13]
/lib64/libpthread.so.0() [0x3532a0f7e0]
/usr/libexec/mysqld(page_cur_insert_rec_low+0x288) [0x7999b8]
/usr/libexec/mysqld(page_cur_parse_insert_rec+0x4eb) [0x79a55b]
/usr/libexec/mysqld() [0x787a0d]
/usr/libexec/mysqld(recv_recover_page+0x34f) [0x7894cf]
/usr/libexec/mysqld(buf_page_io_complete+0x548) [0x74a888]
/usr/libexec/mysqld(fil_aio_wait+0xfa) [0x762a3a]
/usr/libexec/mysqld() [0x7c8ab0]
/lib64/libpthread.so.0() [0x3532a07aa1]
/lib64/libc.so.6(clone+0x6d) [0x35326e8aad]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
161017 10:52:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Version-Release number of selected component (if applicable):
mysql-5.1.73-7.el6

How reproducible:
100% for me

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
I would like to get data from there if possible.
Comment 3 Tomas Pelka 2016-10-17 16:42:55 UTC 
Created attachment 1211473 [details]
my.cnf
Comment 4 Honza Horak 2016-10-25 12:25:08 UTC 
Tomas, we talked together offline, which didn't help much. Anyway, as for this bug, I think we cannot do much, unless we will have a reproducer that would "break" the data directory from working state to broken state, the same way as it happened to you. Closing this bug for now, but feel free to re-open if there are some more information.