Description of problem: After upgrading to kernel 2.6.9-55.0.2.ELsmp from 2.6.9-42.0.10.ELsmp, system hard locks, sometimes during boot as early as the INIT line and sometimes after being up and running for a while. Downgrading back to 2.6.9-42.0.10.ELsmp has fixed the issue. Version-Release number of selected component (if applicable): someplace between 2.6.9-42.0.10.ELsmp and 2.6.9-55.0.2.ELsmp How reproducible: While I cannot reproduce when it happens, with the 2.6.9-55.0.2.ELsmp kernel, I have been able to reproduce it on every boot. Steps to Reproduce: 1. Upgrade kernel 2. Reboot Actual results: Computer hard locks. Display shows whatever was last disaplyed. Computer no longer pings. Keyboard no longer functions (even the led keys do not light their leds). Have to press reset button on computer. Expected results: The system is stable as it is with 2.6.9-42.0.10.ELsmp Additional info: Computer has dual dual-core Opteron 2216s (socket F). 4 GB of RAM forcedeth network driver Tyan S2915a2nrf motherboard as mentioned above, the locks happen at different points during different boots. I've seen them as early as printing out the INIT version line, and then the computer locking. I've seen them happen during the starting of the HAL daemon. I've seem them happen while the computer is running. Nothing is left in any of the log files. 32-bit version of rhel4 with all other updates applied.
ok, i'd like to figure out what patch broke this...via a binary search of the kernels b/w 42.0.10 and 55.0.2. I've posted the following 4 kernels to http://people.redhat.com/~jbaron/bz245941/ kernel-smp-2.6.9-42.12.EL.i686.rpm kernel-smp-2.6.9-42.25.EL.i686.rpm kernel-smp-2.6.9-42.37.EL.i686.rpm kernel-smp-2.6.9-55.EL.i686.rpm Pleae start with 42.25, if that works then go to 42.37, else try 42.12. After you post results I will post more kernels to narrow this down. thanks.
Ok. It is a production server, so, I'll have to schedule time to do this (and then leave it up and running for a while to see if it stays up). Also, will need kernel-smp-devel packages for each of those kernels to build a couple of extra kernel modules needed to boot the system completely. thanks.
Hi. Still need appropriate kernel-devel packages for the below kernels. thnx. (In reply to comment #1) > ok, i'd like to figure out what patch broke this...via a binary search of the > kernels b/w 42.0.10 and 55.0.2. I've posted the following 4 kernels to > http://people.redhat.com/~jbaron/bz245941/ > > kernel-smp-2.6.9-42.12.EL.i686.rpm > kernel-smp-2.6.9-42.25.EL.i686.rpm > kernel-smp-2.6.9-42.37.EL.i686.rpm > kernel-smp-2.6.9-55.EL.i686.rpm > > > Pleae start with 42.25, if that works then go to 42.37, else try 42.12. After > you post results I will post more kernels to narrow this down. thanks.
ok, i've updated that link with kernel-devel pkgs. thanks.
any test results yet here? thanks.
This week and next week are my scheduled testing weeks :) Just done w. the first kernel... 2.6.9-42.25.ELsmp ... seems to be ok. Moving on and up to the next one.
2.6.9-42.37.ELsmp appears to be ok. Will move up to the next one on Monday.
With kernel-smp-2.6.9-55.EL.i686.rpm, the system hard locked after being up for about 5 minutes. So, kernel-smp-2.6.9-55.EL.i686.rpm does not work, while 2.6.9-42.37.ELsmp works just fine.
ok, thanks for the update. I've put all the kernels b/w 42.37 and 55 at: http://people.redhat.com/~jbaron/bz245941/ Please binary search these. thanks.
Also just to be clear, 42.38, 42.39, and 42.40, followed by 43, 44....is the sequential ordering.
kernel-smp-2.6.9-47.EL is broke. Moving on to the next kernel to test.
kernel-smp-2.6.9-43.EL fails. Moving on.
2.6.9-42.39.ELsmp seems to be running ok. Which leaves just kernel-smp-2.6.9-42.40.EL.i686.rpm to try. Hopefully tomorrow.
ok, if you look at the changelog for -43 there was only 1 minor change. Thus, if 42.39 is ok, the problematic change was very likely introduced in 42.40. Unfortunately, 42.40 has a large number of changes...i'm posting it below in case anything stands out. A futher description of the h/w might help us guess. Is there anything in logs at all? thanks. * Tue Jan 9 2007 Jason Baron <jbaron> [2.6.9-42.40] -add PCIe power management quirk (Geoff Gustafson) [197009] -MCE Thresholding support for family 0x10 AMD processors (Bhavana Nagendra) [196894] -i386: acpi_skip_timer_override for NVIDIA (Brian Maly) [193937 200412] -fix aic79xx module removal path (David Milburn) [212388] -fix stack overflow due to recursion between scsi_request_fn and blk_requeue_request (Neil Horman) [202848] -Fix JBD race in t_forget list handling (Eric Sandeen) [176738] -Prevent userspace EOVERFLOW errors from increasing i_ino value (Jeff Layton) [213652 215549] -minimal changes to allow Lustre client support (Eric Sandeen) [196637] -Efficient translation of intel binaries to ppc (Janice Girouard) [196794] -Update MPTSAS driver to v3.02.73rh (Konrad Rzeszutek) [180936 196789 193760 196782] -Support the LSI MegaRAID driver (Konrad Rzeszutek) [196790 193056] -avoid ESB2 IDE interrupt storm (Kimball Murray) [218498] -new ppc Host Ethernet Adapter Device Driver (Janice Girouard) [196774] -USB equipment cannot be used on certain Toshiba systems (Brian Maly) [170134] -Support for p-state transitions on Intel and support for ACPI 3.0 _PSD method (Brian Maly) [196798] -Backport the 'IBM eBus patch' from kernel 2.6.16 and 2.6.17 (Janice Girouard) [196799] -ext3: fix READA failures cause "directory hole" (Chip Coldwell, Stephen Tweedie) [213921]-Audit: 196233: correct/update audit filter rule comparison (Eric Paris) [196233] -fix bre channel cable-pull bug (Chip Coldwell) [163818] -Fix page_is_ram on x86_64 to always return 1 (Chris Lalancette) [221273] -proc: fix readdir race fix (Nobuhiro Tachino) [212631] -Add netdump support to 8139cp driver (Chris Lalancette) [217932] -powernow: remove __initdata from tscsync (Prarit Bhargava) [221975] -fix incorrect cache_decay_ticks calculation on x86_64 and i386 (Jason Baron) [192395] -aio: fix kernel panic in aio_free_ring (Jeff Moyer) [220971] {CVE-2006-5754} -Fix for missing headers in kernel-xenU-devel (Don Dutile) [219538] -fix ext2_check_page denial of service (Eric Sandeen) [217021] {CVE-2006-6054} -PPC64: Return correct value from request_irq() (David Howells) [172436] -dm: stalls on resume if noflush is used (Milan Broz) [221386] -NFSv4 client doesn't return a delegation before removing a file (Steve Dickson) [155929] -fix listxattr syscall can corrupt user space programs (Eric Sandeen) [220677] {CVE-2006-5753}
Also what are the extra kernel modules that you are using? Perhaps the problem lies there...
Also it would be interesting to see if the problem still exits with the current U6 beta candidate...found at: http://people.redhat.com/~jbaron/rhel4/
42.40 gave me a lock during the boot process. With respect to kernel modules, the two custom modules I am using are: 1. openafs 2. forcedeth from nvidia since the forcedeth in the kernel does not support gigabit (it sorta thinks it does, so, you end up with a network that doesn't work) That said, some of the locks occurred when booting into single user mode so that I could compile the gigabit enabled forcedeth. So, the forcedeth from the redhat kernel was used and the openafs module was not loaded as oafs doesn't start in single user mode. This would seem to rule out either of the two kernel modules causing issues.
One difference I've just noted in the startup. With kernel -39 and earlier, it would appear that the following is printed out: Sep 11 11:41:58 smoke kernel: Total of 4 processors activated (19202.47 BogoMIPS ). Sep 11 11:41:58 smoke kernel: ENABLING IO-APIC IRQs Sep 11 11:41:58 smoke kernel: ..TIMER: vector=0x31 pin1=2 pin2=-1 Sep 11 11:41:58 smoke kernel: ..MP-BIOS bug: 8254 timer not connected to IO-APIC Sep 11 11:41:58 smoke kernel: ...trying to set up timer (IRQ0) through the 8259A ... failed. Sep 11 11:41:58 smoke kernel: ...trying to set up timer as Virtual Wire IRQ... f ailed. Sep 11 11:41:58 smoke kernel: ...trying to set up timer as ExtINT IRQ... works. Sep 11 11:41:58 smoke kernel: checking TSC synchronization across 4 CPUs: Sep 11 11:41:58 smoke kernel: CPU#0 had 1344057 usecs TSC skew, fixed it up. Sep 11 11:41:58 smoke kernel: CPU#1 had 0 usecs TSC skew, fixed it up. Sep 11 11:41:58 smoke kernel: CPU#2 had 0 usecs TSC skew, fixed it up. Sep 11 11:41:58 smoke kernel: CPU#3 had 0 usecs TSC skew, fixed it up. Sep 11 11:41:58 smoke mdmonitor: mdadm startup succeeded Sep 11 11:41:58 smoke kernel: Brought up 4 CPUs Sep 11 11:41:58 smoke kernel: zapping low mappings. Sep 11 11:41:58 smoke kernel: checking if image is initramfs... it is Sep 11 11:41:58 smoke kernel: Freeing initrd memory: 760k freed Sep 11 11:41:58 smoke kernel: NET: Registered protocol family 16 Sep 11 11:41:58 smoke kernel: PCI: BIOS BUG #81[00000282] found Starting with -40, the output differs: Sep 11 11:26:21 smoke kernel: Total of 4 processors activated (19202.51 BogoMIPS ). Sep 11 11:26:21 smoke kernel: ENABLING IO-APIC IRQs Sep 11 11:26:21 smoke kernel: ..TIMER: vector=0x31 pin1=0 pin2=-1 Sep 11 11:26:21 smoke kernel: checking TSC synchronization across 4 CPUs: Sep 11 11:26:21 smoke kernel: CPU#0 had 1344058 usecs TSC skew, fixed it up. Sep 11 11:26:21 smoke kernel: CPU#1 had 0 usecs TSC skew, fixed it up. Sep 11 11:26:21 smoke kernel: CPU#2 had 0 usecs TSC skew, fixed it up. Sep 11 11:26:21 smoke kernel: CPU#3 had 0 usecs TSC skew, fixed it up. Sep 11 11:26:21 smoke kernel: Brought up 4 CPUs Sep 11 11:26:21 smoke kernel: zapping low mappings. Sep 11 11:26:21 smoke kernel: checking if image is initramfs... it is Sep 11 11:26:21 smoke kernel: Freeing initrd memory: 714k freed Sep 11 11:26:21 smoke kernel: NET: Registered protocol family 16 Sep 11 11:26:21 smoke kernel: PCI: BIOS BUG #81[00000282] found
Created attachment 210831 [details] output of lspci -v
Created attachment 210841 [details] /var/log/messages - only shows successful boots
Created attachment 210851 [details] /proc/cpuinfo
ok, thanks for the info. Looks to me that this is likely caused by: -i386: acpi_skip_timer_override for NVIDIA (Brian Maly) [193937 200412] We can confirm the theory by backing that patch out...but i believe that if you pass 'noapic' with a problematic kernel you may be able to workaround the problem. Also, there should be a message like: "BIOS IRQ0 pin2 override ignored." in the problematic boot case. Do you see that? I will likely build a kernel without that patch on Monday to confirm my suspicions.
I only have one /var/log/messages with a recorded boot of a kernel that later locked. Version 2.6.9-55 And, yes, that boot does have the line you mention. If you missed it, note that I also attached my lspci, cpuinfo, and the /var/log/messages showing the recent good kernel boots.
ok, i've placed a 60.1.EL.apic.1 kernel at: http://people.redhat.com/~jbaron/bz245941/ This kernel is our latest beta kernel built without the apic patch which is causing this problem. If you can confirm that this kernel is fine, we'll be sure that we have found the source of this regression. thanks.
With the new kernel, the message "MP-BIOS bug: 8254 timer not connected to IO-APIC" is now again in the output. Running the new kernel, now, to see if it holds and does not lock up.
At this point, it's been up for 6 hours running on 2.6.9-60.1.EL.apic.1smp -- since that is longer than it ever stayed up when the problem existed (usually, never finishing booting or crashing within 15 minutes), I'm going to call the problem fixed in this kernel.
ok, thanks for testing this...do have all the latest BIOS updates installed on this box?
My understanding is that there is a newer bios, which I have not yet installed.
well it might be worth tyring the latest update before we consider changing this patch. thanks.
Ok. Certainly worth a shot, though, it doesn't look promising. About the only thing in the changelog that comes close is: Performance tune recommandation from AMD/nVidia Updated AMD Opteron CPU support
Just tested with the bios update. Still locks. In fact, the hardware became unhappy enough after one boot that I had to remove power completely before it would come back. Of note, on my first attempt, I happened to try 2.6.9-47 and for the first time actually got a Kernel panic error message: MP-BIOS Bug: 8254 timer not connected to IO-APIC (why this is there under -47 is interesting since it wasn't there with most of the "later" kernels) Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'no-apic' option. So, this would seem to further confirm the behavior we are seeing with the patch removed. If I did not mention it before, the motherboard is a Tyan S2915A2NRF
This may be related to Bug #432405 Can we get an 'lspci' output to determine which chipset is being used?
Created attachment 295479 [details] lspci -v output Here's the lspci -v output for the affected machine. thanks.
Hello? It's been over two months... what's going on with this bug? thanks.
Re: Comment #26, This problem is resolved in the newest kernel? Or this is still a problem in the newest kernel? Just trying to clarify.
Which patch do you think should fix it? I have not been able to test the newest kernel yet... I am working on moving user data to an alternate server so that I can schedule downtime to test. thanks!
I was trying to clarify mention in Comment #26 that the problem is fixed in the 2.6.9-60.1.EL.apic.1smp (and later) kernels. I read this to mean the problem is now fixed, but Im trying to verify this is the case. We could try testing the latest kernel to be sure if needed, but we probably dont need to do that unless there is still uncertainty as to this being fixed or not.
Nothing later than the specially modified 2.6.9-60.1.EL.apic.1smp has been tried. In fact, the particular machine is still running 2.6.9-60.1.EL.apic.1smp (since I hadn't heard anything back after verifying that kernel and doing the bios update test). So, it's unclear if some patch or removal of a patch later is supposed to fix/work around this issue.
Could the reporter test the newest kernel? The patch that caused this regression had been reworked in a later kernel revision. The new code now checks to see if hpet is enabled first, otherwise any timer override is invalid and not used.
The customer has tested kernel-2.6.9-60.1.EL.apic.1smp which was provided to him earlier and it is working fine for him. Now his server is in production and he can not test anymore even with the latest available kernel for RHEL 4. The customer wants to know which RHEL 4 kernel has fixed his issue so that he can update to that kernel. He wants definitive answer for this as his server is in production and running a test kernel provided earlier. This event sent from IssueTracker by streeter issue 248090
Im to establish that this was fixed but cant do so without having someone test this on the affected hardware to verify this. So we are stuck unless someone can test this or we can get access to affected hardware. I would suggest trying a 4.7 kernel as far as testing fixes is concerned.
Presumably 2.6.9-78.0.8.ELsmp would be sufficient to test with? I can try and schedule sometime in January or February of 2009 to test this if that kernel has the above mentioned reworked patch.
Yes, 2.6.9-78.0.8.ELsmp should be fine for testing. If this kernel does not work, I have a feeling acpi_skip_timer_override should not be set for this hardware. In this is the case I would like to debug this further so as to take advantage of this testing window. So I would like to work out a test patch to test as well. Are you able to test a kernel patch (i.e. patch and build a kernel from source)? If not I can try and spin a test kernel with the patch already applied instead. Please let me know. Thanks.
Either way works.
Since it is fixed in 2.6.9-78.0.8.ELsmp I'm closing this bug.