Bug 620593

Summary:	Kernel Panic When Starting MySQL 5.5.5-m3-1
Product:	Red Hat Enterprise Linux 5	Reporter:	Chris <chris>
Component:	kernel	Assignee:	Red Hat Kernel Manager <kernel-mgr>
Status:	CLOSED WONTFIX	QA Contact:	Red Hat Kernel QE team <kernel-qe>
Severity:	medium	Docs Contact:
Priority:	low
Version:	5.3	CC:	esandeen
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-08-09 19:24:37 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chris 2010-08-02 23:00:36 UTC

Description of problem:
I was testing an upgrade to 5.5.5-m3 on two servers and received a kernel panic in both cases.

-- Mount options
LABEL=mojo              /local/mysql/mojo       xfs   noatime,nobarrier,allocsize=1g 1 2
LABEL=data0             /local/mysql/data       xfs   noatime,allocsize=1g 1 2

Binary Logs are going to mojo.  MYSQL data (i.e. .frm, .idb, etc) are going to data.

Sun hardware:
4140

2xINTEL SSD 160GB g2

Using LVM

OS:
Kernel: Linux m100452.ningops.com 2.6.18-128.1.10.0.1.el5 #1 SMP Thu May 7 17:49:20 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

From MySQL version:
MySQL-server-community-5.5.3_m3-1.rhel5.x86_64.rpm
TO MySQL version:
MySQL-server-5.5.5_m3-1.rhel5.x86_64.rpm 


How reproducible:


Steps to Reproduce:
1. Load the following RPMS:
MySQL-client-community-5.5.3_m3-1.rhel5
MySQL-devel-community-5.5.3_m3-1.rhel5
MySQL-shared-community-5.5.3_m3-1.rhel5
MySQL-server-community-5.5.3_m3-1.rhel5

2.Start mysql
3.Load some data
4. Stop mysql
5. remove these rpms
MySQL-client-community-5.5.3_m3-1.rhel5
MySQL-devel-community-5.5.3_m3-1.rhel5
MySQL-shared-community-5.5.3_m3-1.rhel5
MySQL-server-community-5.5.3_m3-1.rhel5

6. load the following:
MySQL-devel-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-client-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-shared-5.5.5_m3-1.rhel5.x86_64.rpm
MySQL-server-5.5.5_m3-1.rhel5.x86_64.rpm

7. start mysql /etc/init.d/mysql start
8. try to run mysql_upgrade
9. wait for the system to panic
  
Actual results:
Kernel Panic:
Starting mysql: Starting MySQL.......Access to block zero: fs <dm-0> inode: 11745961 start_block : 0 start_off : 200 blkcnt : 0 extent-state : 0
.----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at ...ld/BUILD/xfs-kmod-0.4/_kmod_build_/support/debug.c:57
invalid opcode: 0000 [1] SMP
last sysfs file: /devices/pci0000:80/0000:80:0f.0/0000:83:00.0/0000:84:04.0/0000:86:00.1/irq
CPU 6
Modules linked in: nfs lockd fscache nfs_acl sunrpc bonding xfs(U) dm_round_robin dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi
acpi_memhotplug ac parport_pc lp parport joydev e1000e i2c_nforce2 sg pcspkr i2c_core dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log
dm_mod sata_nv libata shpchp mptsas mptscsih mptbase scsi_transport_sas ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
Pid: 2863, comm: xfsdatad/6 Tainted: G 2.6.18-128.1.10.0.1.el5 #1
RIP: 0010:[<ffffffff884d26b3>] [<ffffffff884d26b3>] :xfs:cmn_err+0xda/0x11f
RSP: 0018:ffff810826569c00 EFLAGS: 00010246
RAX: 0000000000000071 RBX: ffffffff884d59cf RCX: ffffffff802f8aa8
.RDX: ffffffff802f8aa8 RSI: 0000000000000297 RDI: ffffffff884ece4c
RBP: 0000000000000000 R08: ffffffff802f8aa8 R09: 0000000000000046
R10: ffff8108265698a0 R11: 0000000000000280 R12: 0000000000000297
R13: ffff81041f90f800 R14: 0000000000000000 R15: ffff810826569d30
FS: 000000004ae9d940(0000) GS:ffff8104362d98c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000009c1ee2 CR3: 0000000827e7c000 CR4: 00000000000006e0
Process xfsdatad/6 (pid: 2863, threadinfo ffff810826568000, task ffff810422c7c080)
Stack: 0000003000000030 ffff810826569d00 ffff810826569c20 0000000000000250
ffff810827f43500 ffff810827f43500 ffff81042732fa20 0000000000b33aa9
0000000000000000 0000000000000200 0000000000000250 ffffffff884c8e8a
Call Trace:
[<ffffffff884c8e8a>] :xfs:kmem_zone_alloc+0x56/0xa3
[<ffffffff8002fd81>] __up_write+0x27/0xf2
[<ffffffff884b1b7e>] :xfs:xfs_iomap_write_unwritten+0x1cf/0x221
. [<ffffffff884b1700>] :xfs:xfs_iomap+0x320/0x4eb
[<ffffffff884c949a>] :xfs:xfs_end_bio_unwritten+0x0/0x38
[<ffffffff884c94c9>] :xfs:xfs_end_bio_unwritten+0x2f/0x38
[<ffffffff8004d15c>] run_workqueue+0x94/0xe4
[<ffffffff800499dd>] worker_thread+0x0/0x122
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80049acd>] worker_thread+0xf0/0x122
[<ffffffff8008a4cf>] default_wake_function+0x0/0xe
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032383>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff8009d9e3>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032285>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11


Code: 0f 0b 68 54 82 4d 88 c2 39 00 eb 2b 48 c7 c6 a3 82 4d 88 48
RIP [<ffffffff884d26b3>] :xfs:cmn_err+0xda/0x11f
RSP <ffff810826569c00>
<0>Kernel panic - not syncing: Fatal exception


Expected results:
mysql starts

Additional info:

Comment 1 Chris 2010-08-03 16:19:22 UTC

looks like starting the mysql server with innodb_use_native_aio=0 fixed the issue.

Comment 2 Eric Sandeen 2010-08-09 19:24:37 UTC

> Kernel BUG at ...ld/BUILD/xfs-kmod-0.4/_kmod_build_/support/debug.c:57

You hit an assert in an old unsupported external kernel module, built with debugging turned on; this is not a RHEL bug.

(RHEL5 does now have real xfs support, talk to RH support folks if you need it)