Bug 620593 - Kernel Panic When Starting MySQL 5.5.5-m3-1
Kernel Panic When Starting MySQL 5.5.5-m3-1
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.3
x86_64 Linux
low Severity medium
: rc
: ---
Assigned To: Red Hat Kernel Manager
Red Hat Kernel QE team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-02 19:00 EDT by Chris
Modified: 2010-08-09 15:24 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-09 15:24:37 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Chris 2010-08-02 19:00:36 EDT
Description of problem:
I was testing an upgrade to 5.5.5-m3 on two servers and received a kernel panic in both cases.

-- Mount options
LABEL=mojo              /local/mysql/mojo       xfs   noatime,nobarrier,allocsize=1g 1 2
LABEL=data0             /local/mysql/data       xfs   noatime,allocsize=1g 1 2

Binary Logs are going to mojo.  MYSQL data (i.e. .frm, .idb, etc) are going to data.

Sun hardware:
4140

2xINTEL SSD 160GB g2

Using LVM

OS:
Kernel: Linux m100452.ningops.com 2.6.18-128.1.10.0.1.el5 #1 SMP Thu May 7 17:49:20 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

From MySQL version:
MySQL-server-community-5.5.3_m3-1.rhel5.x86_64.rpm
TO MySQL version:
MySQL-server-5.5.5_m3-1.rhel5.x86_64.rpm 


How reproducible:


Steps to Reproduce:
1. Load the following RPMS:
MySQL-client-community-5.5.3_m3-1.rhel5
MySQL-devel-community-5.5.3_m3-1.rhel5
MySQL-shared-community-5.5.3_m3-1.rhel5
MySQL-server-community-5.5.3_m3-1.rhel5

2.Start mysql
3.Load some data
4. Stop mysql
5. remove these rpms
MySQL-client-community-5.5.3_m3-1.rhel5
MySQL-devel-community-5.5.3_m3-1.rhel5
MySQL-shared-community-5.5.3_m3-1.rhel5
MySQL-server-community-5.5.3_m3-1.rhel5

6. load the following:
MySQL-devel-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-client-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-shared-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-server-5.5.5_m3-1.rhel5.x86_64.rpm

7. start mysql /etc/init.d/mysql start
8. try to run mysql_upgrade
9. wait for the system to panic
  
Actual results:
Kernel Panic:
Starting mysql: Starting MySQL.......Access to block zero: fs <dm-0> inode: 11745961 start_block : 0 start_off : 200 blkcnt : 0 extent-state : 0
.----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at ...ld/BUILD/xfs-kmod-0.4/_kmod_build_/support/debug.c:57
invalid opcode: 0000 [1] SMP
last sysfs file: /devices/pci0000:80/0000:80:0f.0/0000:83:00.0/0000:84:04.0/0000:86:00.1/irq
CPU 6
Modules linked in: nfs lockd fscache nfs_acl sunrpc bonding xfs(U) dm_round_robin dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi
acpi_memhotplug ac parport_pc lp parport joydev e1000e i2c_nforce2 sg pcspkr i2c_core dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log
dm_mod sata_nv libata shpchp mptsas mptscsih mptbase scsi_transport_sas ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
Pid: 2863, comm: xfsdatad/6 Tainted: G 2.6.18-128.1.10.0.1.el5 #1
RIP: 0010:[<ffffffff884d26b3>] [<ffffffff884d26b3>] :xfs:cmn_err+0xda/0x11f
RSP: 0018:ffff810826569c00 EFLAGS: 00010246
RAX: 0000000000000071 RBX: ffffffff884d59cf RCX: ffffffff802f8aa8
.RDX: ffffffff802f8aa8 RSI: 0000000000000297 RDI: ffffffff884ece4c
RBP: 0000000000000000 R08: ffffffff802f8aa8 R09: 0000000000000046
R10: ffff8108265698a0 R11: 0000000000000280 R12: 0000000000000297
R13: ffff81041f90f800 R14: 0000000000000000 R15: ffff810826569d30
FS: 000000004ae9d940(0000) GS:ffff8104362d98c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000009c1ee2 CR3: 0000000827e7c000 CR4: 00000000000006e0
Process xfsdatad/6 (pid: 2863, threadinfo ffff810826568000, task ffff810422c7c080)
Stack: 0000003000000030 ffff810826569d00 ffff810826569c20 0000000000000250
ffff810827f43500 ffff810827f43500 ffff81042732fa20 0000000000b33aa9
0000000000000000 0000000000000200 0000000000000250 ffffffff884c8e8a
Call Trace:
[<ffffffff884c8e8a>] :xfs:kmem_zone_alloc+0x56/0xa3
[<ffffffff8002fd81>] __up_write+0x27/0xf2
[<ffffffff884b1b7e>] :xfs:xfs_iomap_write_unwritten+0x1cf/0x221
. [<ffffffff884b1700>] :xfs:xfs_iomap+0x320/0x4eb
[<ffffffff884c949a>] :xfs:xfs_end_bio_unwritten+0x0/0x38
[<ffffffff884c94c9>] :xfs:xfs_end_bio_unwritten+0x2f/0x38
[<ffffffff8004d15c>] run_workqueue+0x94/0xe4
[<ffffffff800499dd>] worker_thread+0x0/0x122
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80049acd>] worker_thread+0xf0/0x122
[<ffffffff8008a4cf>] default_wake_function+0x0/0xe
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032383>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032285>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11


Code: 0f 0b 68 54 82 4d 88 c2 39 00 eb 2b 48 c7 c6 a3 82 4d 88 48
RIP [<ffffffff884d26b3>] :xfs:cmn_err+0xda/0x11f
RSP <ffff810826569c00>
<0>Kernel panic - not syncing: Fatal exception


Expected results:
mysql starts

Additional info:
Comment 1 Chris 2010-08-03 12:19:22 EDT
looks like starting the mysql server with innodb_use_native_aio=0 fixed the issue.
Comment 2 Eric Sandeen 2010-08-09 15:24:37 EDT
> Kernel BUG at ...ld/BUILD/xfs-kmod-0.4/_kmod_build_/support/debug.c:57

You hit an assert in an old unsupported external kernel module, built with debugging turned on; this is not a RHEL bug.

(RHEL5 does now have real xfs support, talk to RH support folks if you need it)

Note You need to log in before you can comment on or make changes to this bug.