Bug 175728

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051111 Firefox/1.5

Description of problem:
One of our developers reported that the system had hung when she was trying to shutdown the Oracle databases we have installed on the server. Searching the logfiles I came over the following error: kernel BUG at mm/prio_tree.c:527! (invalid operand: 0000 [#1])

I think the version of Oracle is 9i, the latest database release, with the 10g web interface loaded in front of this. As the developer is unavailable as I write this, I can't confirm the exact versions. All I know is that it's the latest available Oracle packages. No confirmation on how up to date the patches are either. Sorry.

What happens is that when the scripts are run to start cleanly shutting down the databases, the server drops the remote X console session. Attempts to reconnect fail, and there is no response from the console either. All that I could see that was happening was that the caps lock and scroll lock lights were flashing on and off at a frequency of once per second. Searching for information on any diagnostics lights available for this type of server proved fruitless. I'm unsure if this is the OS or the hardware warning of a problem.

Serching the logfiles showed up the error that's attached later on in this report, to which I have dropped back to the previous kernel release (kernel-smp-2.6.9-11.EL) to see if this resolves the problem. 

Lastly, the system was only running for a rough time of 28 days since the last reboot, and hasn't had this problem before then, or until now. I don't know if the databases were stopped and started before last night or not. Since the last reboot, the only package added to the system was the Veritas Netbackup Enterprise client software (V4.6(FP6) - No agents are installed that allow backups to be made of live and running databases), a few days previous to this happening. I'm unsure if this has anything to do with this or not, but I do know that the backups are running fine, and another two standby RHL ES4 servers with the smp-2.6.9-22.EL kernel are working fine with no problems, although they have no Oracle software installed on them at present.

Version-Release number of selected component (if applicable):
kernel-smp-2.6.9-22.EL

How reproducible:
Didn't try

Steps to Reproduce:
1. Invoke shutdown script to close the Oracle databases running on the server.


Actual Results:  Have not attempted this yet as the developer knows the script command, and I don't. Will fill in more information here when I get the commands and attempt to recreate this on both versions of the kernel that I'm running on the server.

Expected Results:  Oracle databases should have shutdown cleanly, allowing Vertias to back them up (No agent running to allow backup of live databases).

Additional info:

From the Kernel Log File:

Dec 13 18:01:58 itssoracleweb2 kernel: ------------[ cut here ]------------

Dec 13 18:01:58 itssoracleweb2 kernel: kernel BUG at mm/prio_tree.c:527!

Dec 13 18:01:58 itssoracleweb2 kernel: invalid operand: 0000 [#1]

Dec 13 18:01:58 itssoracleweb2 kernel: SMP 

Dec 13 18:01:58 itssoracleweb2 kernel: Modules linked in: 8021q sg cpqci(U) parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc button battery ac md5 ipv6 ohci_hcd tg3 bcm5700(U) floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod cciss aic7xxx sd_mod scsi_mod

Dec 13 18:01:58 itssoracleweb2 kernel: CPU:    2

Dec 13 18:01:58 itssoracleweb2 kernel: EIP:    0060:[<c0145093>]    Tainted: P      VLI

Dec 13 18:01:58 itssoracleweb2 kernel: EFLAGS: 00210206   (2.6.9-22.0.1.ELsmp) 

Dec 13 18:01:58 itssoracleweb2 kernel: EIP is at vma_prio_tree_add+0x10/0x95

Dec 13 18:01:58 itssoracleweb2 kernel: eax: d55b938c   ebx: d55b938c   ecx: 00000000   edx: 000001f3

Dec 13 18:01:58 itssoracleweb2 kernel: esi: ec69a90c   edi: b6f92000   ebp: d6e8dc80   esp: c9c79ee0

Dec 13 18:01:58 itssoracleweb2 kernel: Untitled 1ds: 007b   es: 007b   ss: 0068

Dec 13 18:01:58 itssoracleweb2 kernel: Process oracle (pid: 15520, threadinfo=c9c79000 task=e795a830)

Dec 13 18:01:58 itssoracleweb2 kernel: Stack: f01a9ddc d55b9124 c014e386 dc399c80 00000000 00000000 00000000 f7295380 

Dec 13 18:01:58 itssoracleweb2 kernel:        f72a015c f72a0140 00000000 b6ea2000 d55b938c d55b9124 b6ea2000 b6f92000 

Dec 13 18:01:58 itssoracleweb2 kernel:        c014f6f2 00000203 d55b938c d6e8dc80 b6ea2000 d55b9124 b6e92000 d6e8dc80 

Dec 13 18:01:58 itssoracleweb2 kernel: Call Trace:

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c014e386>] vma_adjust+0x166/0x2d6

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c014f6f2>] split_vma+0xa2/0xd2

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c014f7f9>] do_munmap+0xd7/0x137

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c014eaed>] do_mmap_pgoff+0x311/0x666

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c010b67f>] sys_mmap2+0x7e/0xaf

Dec 13 18:01:58 itssoracleweb2 kernel:  [<c02d0fb7>] syscall_call+0x7/0xb

Dec 13 18:01:58 itssoracleweb2 kernel: Code: ff ff 8d 4c 24 04 89 e2 89 d8 e8 fa fd ff ff 85 c0 74 d4 eb a2 8b 03 5a 59 5b c3 56 89 d6 8b 4e 4c 53 8b 50 4c 89 c3 39 ca 74 08 <0f> 0b 0f 02 04 4c 2e c0 8b 43 08 2b 43 04 c1 e8 0c 8d 54 02 ff 

Dec 13 18:01:58 itssoracleweb2 kernel:  <0>Fatal exception: panic in 5 seconds

Dec 13 18:01:59 itssoracleweb2 dbus: Can't send to audit system: USER_AVC pid=3360 uid=81 loginuid=-1 message=avc:  denied  { send_msg } for  scontext=user_u:system_r:unconfined_t tcontext=user_u:system_r:initrc_t tclass=dbus