Bug 531347 - abrt eating 99% cpu
abrt eating 99% cpu
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: abrt (Show other bugs)
rawhide
All Linux
low Severity high
: ---
: ---
Assigned To: Jiri Moskovcak
Fedora Extras Quality Assurance
:
Depends On:
Blocks: F12Blocker/F12FinalBlocker
  Show dependency treegraph
 
Reported: 2009-10-27 15:57 EDT by Matthias Clasen
Modified: 2015-02-01 17:49 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-11-03 15:57:01 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Matthias Clasen 2009-10-27 15:57:05 EDT
In recent rawhide, I regularly see abrtd go into a spastic mode where it starts consuming all my cpu...

strace says:

[...]

open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640) = -1 EEXIST (File exists)
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_RDONLY) = 7
read(7, ""..., 15)                      = 0
close(7)                                = 0
open("/var/cache/abrt/ccpp-1255924492-2006.lock", O_WRONLY|O_CREAT|O_EXCL, 0640^C <unfinished ...>
Comment 1 Denys Vlasenko 2009-10-29 11:01:04 EDT
Something left an empty lock file there, and this is exactly what we hoped never happen:

    while ((fd = open(pLockFile, O_WRONLY | O_CREAT | O_EXCL, 0640)) < 0)
    {
...
        int r = read(fd, pid_buf, sizeof(pid_buf) - 1);
        close(fd);
        if (r == 0)
        {
            /* Other process did not write out PID yet.
             * We HOPE it did not crash... */
            continue;
        }
...
    }
    write(fd, pPID, len);

One question, how in the hell it happened, but now it would not be easy to find out.

I have a patch which uses symlinks instead of short ordinary files.
Symlinks can be created atomically, avoiding the possibility of "empty file".

Will apply in to abrt git now.
Comment 2 Adam Williamson 2009-10-29 14:32:32 EDT
for blocker evaluation purposes: denys, do you think it'd be safe/wise to apply this fix in F12 final?

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 3 Adam Williamson 2009-10-30 11:09:05 EDT
for the record, the patch for this is:

http://git.fedorahosted.org/git/abrt.git?p=abrt.git;a=commit;h=01057ae36d686d8202547b9ff45bd1635415d13c

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 4 Denys Vlasenko 2009-10-30 12:38:50 EDT
(In reply to comment #2)
> for blocker evaluation purposes: denys, do you think it'd be safe/wise to apply
> this fix in F12 final?

I think at this point abrt git should go to repository almost daily. It is not stable yet. IOW: it has so many bugs that git almost always works better (has less bugs) than anything from three days ago. There is almost no "stability" to be gained by holding back abrt releases.
Comment 5 Jiri Moskovcak 2009-10-30 12:42:09 EDT
Anyone interested in testing this can try the latest abrt build from my repo:

You can find the repo file at:
http://jmoskovc.fedorapeople.org/abrt-rawhide.repo

Thanks,
Jirka
Comment 6 Adam Williamson 2009-10-30 18:20:37 EDT
comment #4 makes me uncomfortable about shipping abrt by default, but point taken. Matthias, can you test the build from Jiri's repo and confirm it fixes this for you? If Matthias confirms, Jiri + Denys please send a build to Koji and a tag request for final ASAP. Thanks!

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 7 Matthias Clasen 2009-10-31 01:35:32 EDT
I have installed the build from the abrt-rawhide repo, and have not seen the 100% cpu problem since. Will keep monitoring it over the weekend.
Comment 8 Jiri Moskovcak 2009-11-02 04:49:26 EST
Matthias, 
was there any selinux warning/denials, when this happened?

Thanks,
Jirka
Comment 9 Matthias Clasen 2009-11-02 08:51:25 EST
Sorry, I can't say. And the 100% cpu eating hasn't reoccurred since updating to the abrt-rawhide repo.
Comment 10 Adam Williamson 2009-11-02 18:59:23 EST
I've been running the updated abrt for a few hours and have seen no problems with it.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 11 Bill Nottingham 2009-11-03 15:46:43 EST
Given that abrt-0.11 has been tagged, should we close this? I'm at least marking this as modified.
Comment 12 Adam Williamson 2009-11-03 15:57:01 EST
it only just got tagged. that's what I was waiting on before changing it.

I think the current feedback is enough to close it, yes. If anyone sees the CPU usage bug come back they can re-open.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Note You need to log in before you can comment on or make changes to this bug.