Bug 249563
| Summary: | Deadlocks on high disk I/O | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Matthias Hensler <adv> | ||||||||||
| Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> | ||||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||
| Severity: | medium | Docs Contact: | |||||||||||
| Priority: | medium | ||||||||||||
| Version: | 7 | CC: | bruno, bugzilla.redhat.com, chris.brown, davej, mishu, rrauenza, sputhenp | ||||||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | All | ||||||||||||
| OS: | Linux | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | 2.6.24 | Doc Type: | Bug Fix | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2008-02-03 19:44:57 UTC | Type: | --- | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Matthias Hensler
2007-07-25 15:24:57 UTC
Created attachment 159939 [details]
Processlist showing the bug
Created attachment 159940 [details]
Calltrace from echo t >/proc/sysrq-trigger
This is the full calltrace (after killing the crond processes shown in the
processlisting) resulting from "echo t > /proc/sysrq-trigger"
Try booting with kernel options nohz=off highres=off to disable the new timer code. Also try 2.6.22.1-33 (In reply to comment #3) > Try booting with kernel options > > nohz=off highres=off > > to disable the new timer code. > > Also try 2.6.22.1-33 OK, I will do. However I do not think that it is related to the new timer code, since the problem was already there with a lot older kernel. In the meantime I have mounted all partitions with default values and had no deadlock so far (have to watch it further). Might it be possible that "noatime" is causing the problem? The CFQ scheduler might expect more dirty pages from doing high I/O like rsync. If noatime is set there won't be any, at least not on the partition reading from. > OK, I will do. However I do not think that it is related to the new timer
> code, since the problem was already there with a lot older kernel.
I would like to give an update on this as I think that I found the root cause of
the problem.
So far we have not changed anything (still running 2.6.22.1-27 with the new
timer code). The only thing that was changed were the mount options.
Both systems now run stable for over a week, so the problem is not longer existent.
Old fstab:
/dev/hd2/varspoolimap /var/spool/imap ext3
async,auto,data=ordered,exec,noacl,noatime,nodev,nosuid,nouser,rw 1 2
New fstab:
/dev/hd2/varspoolimap /var/spool/imap ext3 defaults
1 2
Since "defaults" implies rw,suid,dev,exec,auto,nouser,async the only differences
here are:
data=ordered (but that is a ext3-default, so also no difference)
noacl (should no make a difference, SELinux is off, no security
contextes are used here)
noatime (which I suspect is the rootcause)
nodev (should not affect rsync)
nosuid (likewise should not affect rsync)
I am pretty sure that mounting our imap-spool with noatime seems to cause
deadlocks on high I/O load. So there is mostlikely some bug within the kernel
and the schedular. At least since 2.6.19.
I provided the stacktrace of such a situation with the original bugreport.
Anything more I could provide to narrow the issue down?
(In reply to comment #5) > Anything more I could provide to narrow the issue down? In the meantime I recompiled 2.6.22.1-27.fc7 with just the twoline patch from Richard Kennedy (http://lkml.org/lkml/2007/8/2/89) which seems to have fixed this issue for now. I am still watching the systems closely to be sure that the patch indeed has the indended effect. Regards, Matthias Created attachment 160963 [details] Patch to fix the issue Posted by Richard Kennedy on LKML: http://lkml.org/lkml/2007/8/2/89 Hello, I'm reviewing this bug as part of the kernel bug triage project, an attempt to isolate current bugs in the fedora kernel. http://fedoraproject.org/wiki/KernelBugTriage I am CC'ing myself to this bug and will try and assist you in resolving it if I can. There hasn't been much activity on this bug for a while. Could you tell me if you are still having problems with the latest kernel? Could you also test 2.6.23 from rawhide as this should have the fix indicated in comment #7. If the problem no longer exists then please close this bug or I'll do so in a few days if there is no additional information lodged. Cheers Chris > There hasn't been much activity on this bug for a while. Could you tell me if > you are still having problems with the latest kernel? Yes, the bug is still present. > Could you also test 2.6.23 > from rawhide as this should have the fix indicated in comment #7. The last activity was my mail from august 9th to LKML (see http://lkml.org/lkml/2007/8/9/346). There was no further answer to that. The patch from comment #7 has resolved the issue (the affected system has now a uptime of nearly 50 days). However, that patch is neither present upstream (checked 2.6.23-rc7), nor in rawhide (checked kernel-2.6.23-0.189.rc6.git8.fc8). So, the problem still exists. Regards, Matthias Okay, will re-assign and hopefully get this included soon. Chuck? This sounds very similar to a problem I have been seeing with FC5 and F7. I am holding one machine (that I can't afford to have hangs happen on every couple of days) at 2.6.19-1.2288.2.4.fc5 and it is working fine. (In reply to comment #11) > This sounds very similar to a problem I have been seeing with FC5 and F7. I am > holding one machine (that I can't afford to have hangs happen on every couple of > days) at 2.6.19-1.2288.2.4.fc5 and it is working fine. If you can post some details to confirm the dupe that would be helpful - similar output to Matthias' for example. I submitted 235043 which refers to this issue and probably has some of what you want. The two systems I saw the problem on (though one I kept at FC5 and never tried F7) have only 512MB of memory and use ext3 file systems with journal=data and write barriers running on top of software raid 1 on pata disks. Hi, any news here? I'm running the system with the problem mentioned above together with Matthias Hensler. We currently using 2.6.22.1-27.fc7 (last ChangeLog entry * Tue Jul 17 2007 John W. Linville <linville>) with patch form comment #7, noatime option enabled and haven't faced the problem again. Is this patch or another patch trying to fix this problem included in RawHide kernel kernel-2.6.23.1-30.fc8. If yes, we would offer to try it on our system. We just go round and round on linux-kernel and nobody seems to have a safe short-term fix. Supposedly 2.6.25 will have the real fix but it's hundreds of lines of code changes. FWIW this patch should be much safer: http://lkml.org/lkml/diff/2007/9/20/415/1 Is anybody else using patch from comment #16 ? Two days ago I've compiled kernel-2.6.22.9-91.bz249563.fc7 including the fix from comment #16 Yesterday the server got slow and somebody from first level support rebooted the server without further investigating the situation. Today the server got stuck a bit (means some static html pages from webmail frontend still worked, but login (ssh and serial) timed ou)t. Unfortunately I had no open session to investigate it further and I had to do hard reset. For now I have reverted back to kernel-2.6.22.1-27.bz249563.fc7 with fix from comment #7. I didn't try any of the patches, but I did collect some vmstat and meminfo data and post a pointer to it on lmkl yesterday. I don't know if the right people have looked at it, but during the thread someone mentioned they were interested in seeing that data. Created attachment 246931 [details]
vmstat and memifo data from while processes were stuck
This is the mixed vmstat and meminfo data I posted a link to previously. Since
it didn't get any attention on lmkl, I wanted to put a copy somewhere where
anyone who might want to look at it later can find it.
Any news regarding this problem? I'm in the process to update to FC8, but I don't want to take the risk that I'm stuck without a running kernel. Was this problem addressed in FC8 kernel? If you are already using the latest FC5 or later without problem, then you probably won't run into what I am seeing. I have seen the problem happening continuously since late FC5 through current F8. In fact I just rebooted the machine where this problem is triggered to clear things up again about an hour ago. I may also be having the same deadlock. I had /var/spool/squid mounted noatime (just changed it after reading this bug report) Symptoms were Login prompt at console lets you type something in, password prompt doesn't come up System is pingable ssh attempts timeout Wasn't able to do any sysrq logging as the person who resets the machine doesn't have enough CS background. And I've never had an active login when it has happened. It happens every 3-7 days. The system is an older 650MHZ PIII with RAID1 and only 512MB of RAM. So it all sounds very similar to reports above. I wouldn't call my disk I/O unusually high, though. Currently running FC8 with 2.6.23. I'm not running plain kernel. I'm running kernel 2.6.22.1-27.bz249563.fc7 This is local compiled 2.6.22.1-27 with additional patch from comment #7 Without the patch I'm facing the problem described in this bug report. I need to know if the problem is fixed in FC8 kernel, since in comment #17 I've tried a updated FC7 kernel with the patch from comment #7 and faced the problem again. The problem should be fixed properly in 2.6.24. Setting to NEEDINFO then - could all reporters test with a 2.6.24-based kernel and report their results. Thanks in advance. OK, will do so in a few days and post the result here. I turned off noatime, rebooted, and have had the hang/lockup since then with 2.6.23 on FC8. I have upgraded the machine I have been having problems on to rawhide and am using a 2.6.24 kernel. So far the problem hasn't reoccurred, but it is still too early for me to say the problem has been fixed. I have had 2.6.24-2.fc9 running continuously for 3.5 days, which considering that I have been doing significant yum updates in that time is longer that I would have expected to go without a lockup under the old kernels. I probably need to go a few more days to be really sure, but if things still look good on Friday I will try an upgrade of my system I have held on FC5 because of this problem. After a week I rebooted the machine to start using a kernel update. It hadn't made a week in a long time (about 6 months), so I think there is a very good chance the problem is fixed in the 2.6.24 kernel. Okay, I'm closing to reflect that then. As the original reporter hasn't commented in a while and there's plenty of evidence to believe things are resolved. Please re-open if the problem re-occurs. Cheers Chris I'm still getting deadlocks of some sort. ssh half responds (sock connects, no communication..), syslog will stop logging. System is FC8, with a 2.6.24 kernel (can't look at the subversion right now since the system is locked up and at another location.) sysrq keys work for unmounting, killing, and rebooting, which is the SOP when the system locks up like this, which is about once a week. How do people get the state of the system when this happens? (In reply to comment #32) > > How do people get the state of the system when this happens? sysrq-t, but you've got to have some way to capture it. So I've set up a silly scheme.. might work.. I've read that for some people, an active login still works mostly. So, I ssh in and while there, every 60 seconds, echo 't' to the sysrq /proc file. Meanwhile, I use klogctl to read from the kernel ring buffer to stdout... Hopefully while in this state, existing sockets remain working, sshd's and klogctl remains working... The output gets logged on the host performing the ssh. We'll see if it works.... If it doesn't I guess I'd need to boot with a serial port configured as console and take a dump with a null modem and laptop? I don't see any other way. |