Bug 1764835 - clamd at 100% CPU and SystemD keeps restarting clamd
Summary: clamd at 100% CPU and SystemD keeps restarting clamd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: clamav
Version: 29
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Sergio Basto
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-23 19:53 UTC by John Griffiths
Modified: 2019-12-10 02:17 UTC (History)
12 users (show)

Fixed In Version: clamav-0.101.5-1.fc30 clamav-0.101.5-1.fc31 clamav-0.101.5-1.el8 clamav-0.101.5-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-27 18:06:46 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Screenshot of top showing clamd and clamscan processes. (62.64 KB, image/png)
2019-10-23 19:53 UTC, John Griffiths
no flags Details

Description John Griffiths 2019-10-23 19:53:46 UTC
Created attachment 1628612 [details]
Screenshot of top showing clamd and clamscan processes.

Description of problem:
When the postfix/amavizd/spamassassing email system started, clamd would be at 100% CPU. This may happen only on slower systems. 

The clamd@.service (clamd) would keep restarting and never complete starting. the PID file (/var/run/clamd.amavisd/clamd.sock) was never created.

SystemD would time out and restart clamd. The default timeout in /etc/systemd/system.conf is DefaultTimeoutStopSec=90s

I found on Google https://exoticbaryon.anset.org/2019/08/30/fixing-clamd-stuck-at-100-cpu/ and tried the fix:
Edit the /lib/systemd/system/clamd@.service file adding the TimeoutStartSec = 10min. Excessive, but it worked.
[Service]
Type = forking
TimeoutStartSec = 10min
ExecStart = /usr/sbin/clamd -c /etc/clamd.d/%i.conf
Restart = on-failure

I suggest the default /lib/systemd/system/clamd@.service file be modified to add the TimeoutStartSec value.

I used time systemctl start clamd@amavisd and got:

real    1m41.368s
user    0m0.007s
sys     0m0.022s

so 120sec may be plenty.

Version-Release number of selected component (if applicable):
clamd-0.101.4-1.fc29.x86_64
clamav-0.101.4-1.fc29.x86_64
systemd-239-14.git33ccd62.fc29.x86_64

How reproducible:
Depends on speed of system. 
Happened every time on AMD Phenom(tm) II X4 955 Processor (Yes, I know it is old.)
Did not happen on Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz


Steps to Reproduce:
1. systemctl start clamd (this starts as part of amavisd)
2. CPU goes to 100%
3. If is stays at 100% for longer than 90 seconds, clamd is restarting. If not, it will not be a problem on that system.
4. If it happens, edit /lib/systemd/system/clamd@.service file as above.
5. systemctl stop amavisd
6. systemctl stop clamd
7. systemctl daemon-reload
8. systemctl start amavisd

Actual results:
If it is a slow system, the CPU will be at 100% for clamd and stay there.
If it is a fairly fast system, the CPU will be at 100% for clamd and then go back to normal after clamd starts.

Expected results:
When clamd starts either as part of amavisd or as a service of systemctl, it should not timeout and keep restarting on slower systems.

Additional info:
amavisd will start clamscan for emails as the come in since clamd is not running. The clamscan processes also run at 100% CPU until they finish.

Bugzilla does not have clamd as a component.

Comment 1 Orion Poplawski 2019-10-24 01:32:36 UTC
Upstream has changed to 420s, so that will be coming down the pipe eventually.  You might want to check that it takes less than that to start.

BTW - bugzilla deals with source packages, not the installed components.  rpm -qi clamd shows the source component.

Comment 2 Sergio Basto 2019-10-25 00:33:19 UTC
is not duplicated of #1754064 ?

Comment 3 Orion Poplawski 2019-10-25 00:39:32 UTC
Same cause, but it's a different package really.

Comment 4 Ben Cotton 2019-10-31 18:44:31 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 5 Dimitris 2019-11-18 12:49:42 UTC
Same problem in CentOS 8 and epel-8 packages

clamav-0.101.4-1.el8.x86_64
clamav-filesystem-0.101.4-1.el8.noarch
clamav-lib-0.101.4-1.el8.x86_64
clamav-milter-0.101.4-1.el8.x86_64
clamav-update-0.101.4-1.el8.x86_64
clamd-0.101.4-1.el8.x86_64

The problem appears in various types of systems, from bare metal (Intel i7-6500) to VirtualBox machines (AMD Ryzen 7 3700X) and to cloud vms offered by digitalocean and linode.

Instead of raising the timeout value, would it be possible to just ignore the timeout and just expect the clamd process to finish loading at some future time? If there is a problem, the clamd process will kill itself anyway.

Thank you.

Comment 6 Orion Poplawski 2019-11-19 03:50:27 UTC
I'm bumping the timeout to 420 seconds to match upstream.  Disabling it is a possibility but please take it up with upstream.

Comment 7 Fedora Update System 2019-11-19 04:08:35 UTC
FEDORA-2019-42b5351ae8 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-42b5351ae8

Comment 8 Fedora Update System 2019-11-19 04:08:36 UTC
FEDORA-2019-e9cb0041e6 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e9cb0041e6

Comment 9 Fedora Update System 2019-11-20 01:27:36 UTC
clamav-0.101.4-2.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-525261067e

Comment 10 Fedora Update System 2019-11-20 02:19:23 UTC
clamav-0.101.4-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e9cb0041e6

Comment 11 Fedora Update System 2019-11-20 02:31:51 UTC
clamav-0.101.4-2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-42b5351ae8

Comment 12 Fedora Update System 2019-11-20 04:00:24 UTC
clamav-0.101.4-2.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-1815f56431

Comment 13 Fedora Update System 2019-11-20 04:03:35 UTC
clamav-0.101.4-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-0b0bad1b8f

Comment 14 Fedora Update System 2019-11-25 02:04:11 UTC
clamav-0.101.5-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1543eae191

Comment 15 Fedora Update System 2019-11-25 02:27:57 UTC
clamav-0.101.5-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-dcbfe89e39

Comment 16 Fedora Update System 2019-11-25 02:38:34 UTC
clamav-0.101.5-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-d6b0a398c2

Comment 17 Fedora Update System 2019-11-25 03:10:15 UTC
clamav-0.101.5-1.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-52445f11c2

Comment 18 Fedora Update System 2019-11-25 04:15:24 UTC
clamav-0.101.5-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-ce438e7857

Comment 19 Ben Cotton 2019-11-27 18:06:46 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 20 Fedora Update System 2019-12-01 01:04:05 UTC
clamav-0.101.5-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2019-12-04 01:15:20 UTC
clamav-0.101.5-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2019-12-10 02:01:13 UTC
clamav-0.101.5-1.el8 has been pushed to the Fedora EPEL 8 stable repository. If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2019-12-10 02:17:35 UTC
clamav-0.101.5-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.