on grep _NAME /etc/os-release PRETTY_NAME="Fedora 32 (Server Edition)" CPE_NAME="cpe:/o:fedoraproject:fedora:32" after recent update to uname -rm 5.8.14-200.fc32.x86_64 x86_64 with dnf info clamav-milter Last metadata expiration check: 0:10:11 ago on Wed 14 Oct 2020 08:39:41 PM PDT. Installed Packages Name : clamav-milter Version : 0.103.0 Release : 1.fc32 Architecture : x86_64 Size : 183 k Source : clamav-0.103.0-1.fc32.src.rpm Repository : @System From repo : updates Summary : Milter module for the Clam Antivirus scanner URL : https://www.clamav.net/ License : GPLv2 Description : This package contains files which are needed to run the clamav-milter. rpm -qa | grep -i clam | sort clamav-0.103.0-1.fc32.x86_64 clamav-data-0.103.0-1.fc32.noarch clamav-filesystem-0.103.0-1.fc32.noarch clamav-lib-0.103.0-1.fc32.x86_64 clamav-milter-0.103.0-1.fc32.x86_64 clamav-update-0.103.0-1.fc32.x86_64 clamd-0.103.0-1.fc32.x86_64 i notice in boot logs dmesg ... [ 72.552389] clamd[1014]: segfault at 0 ip 00007fede6a7801e sp 00007ffd8cdd50f8 error 4 in libc-2.31.so[7fede6939000+150000] [ 72.552414] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [ 72.552422] potentially unexpected fatal signal 11. [ 72.552425] CPU: 3 PID: 1014 Comm: clamd Not tainted 5.8.14-200.fc32.x86_64 #1 [ 72.552425] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [ 72.552430] RIP: 0033:0x7fede6a7801e [ 72.552432] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [ 72.552433] RSP: 002b:00007ffd8cdd50f8 EFLAGS: 00010287 [ 72.552435] RAX: 00007ffd8cdd5122 RBX: 0000000000000000 RCX: 0000000000000000 [ 72.552435] RDX: 000000000000006b RSI: 0000000000000000 RDI: 00007ffd8cdd5122 [ 72.552436] RBP: 00007ffd8cdd5122 R08: 000000000000006b R09: 0000000000000000 [ 72.552436] R10: 00007fede6a8aac0 R11: 00007fede6a8b3c0 R12: 0000000000000000 [ 72.552437] R13: 00005596b864b020 R14: 00007ffd8cdd5120 R15: 000000000000221a [ 72.552438] FS: 00007fede5a29040 GS: 0000000000000000 ... it's not fatal system continues to boot my services are running systemctl status clamav-daemon ● clamav-daemon.service - clamd scanner daemon Loaded: loaded (/etc/systemd/system/clamav-daemon.service; enabled; vendor preset: disabled) Active: activating (start) since Wed 2020-10-14 20:58:39 PDT; 41s ago TriggeredBy: ● clamav-daemon.socket Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Cntrl PID: 6026 (clamd) Tasks: 1 (limit: 9497) Memory: 218.0M CPU: 29.372s CGroup: /system.slice/clamav-daemon.service └─6026 /usr/sbin/clamd --config-file=/usr/local/etc/clamav/clamd.conf systemctl status clamav-milter -ln0 ● clamav-milter.service - Milter module for the Clam Antivirus scanner Loaded: loaded (/usr/lib/systemd/system/clamav-milter.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamav-milter.service.d └─override.conf Active: active (running) since Wed 2020-10-14 20:43:29 PDT; 12min ago Process: 1007 ExecStart=/usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf (code=exited, status=0/SUCCESS) Main PID: 1026 (clamav-milter) Tasks: 3 (limit: 9497) Memory: 2.0M CPU: 24ms CGroup: /system.slice/clamav-milter.service └─1026 /usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf so far, I've noticed no *runtime* problems. investigating further ...
was a core file collected by abrtd or systemd (coredumpctl)?
atm, i don't have abrt* installed there's nothing from sysctl -a | grep dumpable fs.suid_dumpable = 2 coredumpctl list No coredumps found. checking this morning, I note that it keeps repeating ... dmesg ... [41052.864211] systemd[1]: Starting clamd scanner daemon... ... [41092.516095] clamd[35439]: segfault at 0 ip 00007f83da9ca01e sp 00007fff8f1b9998 error 4 in libc-2.31.so[7f83da88b000+150000] [41092.516119] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41092.516123] potentially unexpected fatal signal 11. [41092.516126] CPU: 0 PID: 35439 Comm: clamd Not tainted 5.8.15-201.fc32.x86_64 #1 [41092.516126] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [41092.516130] RIP: 0033:0x7f83da9ca01e [41092.516132] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41092.516133] RSP: 002b:00007fff8f1b9998 EFLAGS: 00010287 [41092.516134] RAX: 00007fff8f1b99c2 RBX: 0000000000000000 RCX: 0000000000000000 [41092.516135] RDX: 000000000000006b RSI: 0000000000000000 RDI: 00007fff8f1b99c2 [41092.516136] RBP: 00007fff8f1b99c2 R08: 000000000000006b R09: 0000000000000000 [41092.516136] R10: 00007f83da9dcac0 R11: 00007f83da9dd3c0 R12: 0000000000000000 [41092.516137] R13: 000055ced08e5020 R14: 00007fff8f1b99c0 R15: 000000000000221a [41092.516138] FS: 00007f83d997b040 GS: 0000000000000000 ... [41355.871185] systemd[1]: Starting clamd scanner daemon... [41395.940384] clamd[37491]: segfault at 0 ip 00007f1ab1c7d01e sp 00007ffec5c9d328 error 4 in libc-2.31.so[7f1ab1b3e000+150000] [41395.940398] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41395.940402] potentially unexpected fatal signal 11. [41395.940404] CPU: 2 PID: 37491 Comm: clamd Not tainted 5.8.15-201.fc32.x86_64 #1 [41395.940405] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [41395.940407] RIP: 0033:0x7f1ab1c7d01e [41395.940409] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41395.940410] RSP: 002b:00007ffec5c9d328 EFLAGS: 00010287 [41395.940411] RAX: 00007ffec5c9d352 RBX: 0000000000000000 RCX: 0000000000000000 [41395.940412] RDX: 000000000000006b RSI: 0000000000000000 RDI: 00007ffec5c9d352 [41395.940413] RBP: 00007ffec5c9d352 R08: 000000000000006b R09: 0000000000000000 [41395.940413] R10: 00007f1ab1c8fac0 R11: 00007f1ab1c903c0 R12: 0000000000000000 [41395.940414] R13: 0000563036355020 R14: 00007ffec5c9d350 R15: 000000000000221a [41395.940415] FS: 00007f1ab0c2e040 GS: 0000000000000000 [41656.124192] systemd[1]: clamav-daemon.service: start operation timed out. Terminating. [41658.837370] systemd[1]: clamav-daemon.service: Failed with result 'timeout'. [41658.837598] systemd[1]: Failed to start clamd scanner daemon. [41658.837734] systemd[1]: clamav-daemon.service: Consumed 57.029s CPU time. [41659.124901] systemd[1]: clamav-daemon.service: Scheduled restart job, restart counter is at 137. [41659.125102] systemd[1]: Stopped clamd scanner daemon. [41659.125113] systemd[1]: clamav-daemon.service: Consumed 57.029s CPU time. ... [41659.126434] systemd[1]: Starting clamd scanner daemon... [41696.702127] clamd[37506]: segfault at 0 ip 00007fb0494d001e sp 00007ffebc8fa458 error 4 in libc-2.31.so[7fb049391000+150000] [41696.702146] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41696.702150] potentially unexpected fatal signal 11. [41696.702152] CPU: 1 PID: 37506 Comm: clamd Not tainted 5.8.15-201.fc32.x86_64 #1 [41696.702153] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [41696.702156] RIP: 0033:0x7fb0494d001e [41696.702158] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [41696.702159] RSP: 002b:00007ffebc8fa458 EFLAGS: 00010287 [41696.702161] RAX: 00007ffebc8fa482 RBX: 0000000000000000 RCX: 0000000000000000 [41696.702162] RDX: 000000000000006b RSI: 0000000000000000 RDI: 00007ffebc8fa482 [41696.702162] RBP: 00007ffebc8fa482 R08: 000000000000006b R09: 0000000000000000 [41696.702163] R10: 00007fb0494e2ac0 R11: 00007fb0494e33c0 R12: 0000000000000000 [41696.702164] R13: 0000561876869020 R14: 00007ffebc8fa480 R15: 000000000000221a [41696.702165] FS: 00007fb048481040 GS: 0000000000000000 [41776.378650] systemd[1]: Starting system activity accounting tool... [41776.387374] systemd[1]: sysstat-collect.service: Succeeded. [41776.387609] systemd[1]: Finished system activity accounting tool. [41959.380362] systemd[1]: clamav-daemon.service: start operation timed out. Terminating. [41962.056952] systemd[1]: clamav-daemon.service: Failed with result 'timeout'. [41962.057168] systemd[1]: Failed to start clamd scanner daemon. [41962.057324] systemd[1]: clamav-daemon.service: Consumed 54.484s CPU time. [41962.381017] systemd[1]: clamav-daemon.service: Scheduled restart job, restart counter is at 138. [41962.381214] systemd[1]: Stopped clamd scanner daemon. [41962.381226] systemd[1]: clamav-daemon.service: Consumed 54.484s CPU time. ... [41962.382485] systemd[1]: Starting clamd scanner daemon... it correlates with every inbound mail receipt; on this system, clam's invoked by clamav-milter on each mail. checking, i don't see _any_ errors in clamav* logs, milter or otherwise. still poking around ...
What's the output of: cat /proc/sys/kernel/core_pattern You also seem to be running a custom clamav-daemon.service. What's the content of that? Also: grep -Fi core /proc/3044/limits for the PID of your clamd process. coredumpctl shows a coredump if I kill clamd with SIGILL on my system so I don't know why it doesn't on yours. Make sure clamav-debuginfo is installed then perhaps try to attach to the clamd process with gdb. Since your coredumps come regularly you shouldn't have to wait long then you could capture a backtrace.
> What's the output of: cat /proc/sys/kernel/core_pattern cat /proc/sys/kernel/core_pattern /var/lib/systemd/coredump/core > You also seem to be running a custom clamav-daemon.service. What's the content of that? cat /etc/systemd/system/clamav-daemon.service [Unit] Description = clamd scanner daemon Documentation=man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Requires=clamav-daemon.socket After = syslog.target nss-lookup.target network.target clamav-freshclam.target Before=clamav-daemon.target OnFailure=unit-status-mail@%n.service ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc} ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} [Service] User = clamav Group = clamav Type = forking ExecStart = /usr/sbin/clamd \ --config-file=/usr/local/etc/clamav/clamd.conf Restart = on-failure PrivateTmp = true CPUQuota=70% LimitNICE=35 Nice=19 TimeoutSec=300 [Install] WantedBy = multi-user.target Also=clamav-daemon.socket > Also: > grep -Fi core /proc/3044/limits > for the PID of your clamd process. grep -Fi core /proc/`pidof clamd`/limits Max core file size unlimited unlimited bytes > coredumpctl shows a coredump if I kill clamd with SIGILL on my system so I don't know why it doesn't on yours. coredumpctl No coredumps found. -> 4 - SIGILL kill -4 `pidof clamd` coredumpctl No coredumps found. > Make sure clamav-debuginfo is installed dnf install clamav-debuginfo > then perhaps try to attach to the clamd process with gdb. Since your coredumps come regularly you shouldn't have to wait long then you could capture a backtrace. systemctl start clamd ... gdb clamd 38621 wait dmesg -w ... [31468.462625] clamd[38625]: segfault at 0 ip 00007fdb4c07501e sp 00007ffdc7fb1c48 error 4 in libc-2.31.so[7fdb4bf36000+150000] [31468.462639] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [31468.462643] potentially unexpected fatal signal 11. [31468.462645] CPU: 1 PID: 38625 Comm: clamd Not tainted 5.8.15-201.fc32.x86_64 #1 [31468.462646] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [31468.462650] RIP: 0033:0x7fdb4c07501e [31468.462651] Code: fd d7 c9 0f bc d1 c5 fe 7f 27 c5 fe 7f 6f 20 c5 fe 7f 77 40 c5 fe 7f 7f 60 49 83 c0 1f 49 29 d0 48 8d 7c 17 61 e9 c2 04 00 00 <c5> fe 6f 1e c5 fe 6f 56 20 c5 fd 74 cb c5 fd d7 d1 49 83 f8 21 0f [31468.462652] RSP: 002b:00007ffdc7fb1c48 EFLAGS: 00010287 [31468.462654] RAX: 00007ffdc7fb1c72 RBX: 0000000000000000 RCX: 0000000000000000 [31468.462655] RDX: 000000000000006b RSI: 0000000000000000 RDI: 00007ffdc7fb1c72 [31468.462655] RBP: 00007ffdc7fb1c72 R08: 000000000000006b R09: 0000000000000000 [31468.462656] R10: 00007fdb4c087ac0 R11: 00007fdb4c0883c0 R12: 0000000000000000 [31468.462657] R13: 000055a8f3307020 R14: 00007ffdc7fb1c70 R15: 000000000000221a [31468.462657] FS: 00007fdb4b026040 GS: 0000000000000000 ... (gdb) bt #0 0x00007fdb4bfdeeca in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffdc7fb1d8c, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27 #1 0x00007fdb4bfdee8b in __GI___waitpid (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffdc7fb1d8c, options=options@entry=0) at waitpid.c:38 #2 0x00007fdb4bfdee73 in __wait (stat_loc=stat_loc@entry=0x7ffdc7fb1d8c) at wait.c:25 #3 0x000055a8f25a78d4 in daemonize_parent_wait (user=0x55a8f32efc90 "clamav", log_file=0x55a8f330d600 "/var/log/clamav/clamav-daemon.log") at misc.c:362 #4 0x000055a8f259be1b in main (argc=<optimized out>, argv=<optimized out>) at clamd.c:280
Thanks. It would be helpful to have the gdb output from when it captures the segfault - that should show what thread segfaults and also "thread apply all bt" - as the trace you show doesn't seem to be the thread that segfaulted.
reboot; on launch ps -eL | grep clamd 1592 1592 ? 00:00:00 clamd 1596 1596 ? 00:00:10 clamd systemctl status clamav-daemon.socket clamav-daemon.service clamav-freshclam.service clamav-milter.service -ln0 ● clamav-daemon.socket - Socket for Clam AntiVirus userspace daemon Loaded: loaded (/etc/systemd/system/clamav-daemon.socket; enabled; vendor preset: disabled) Active: active (running) since Fri 2020-10-23 09:56:42 PDT; 11min ago Triggers: ● clamav-daemon.service Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Listen: 127.0.0.1:3310 (Stream) Tasks: 0 (limit: 9497) Memory: 24.0K CPU: 1ms CGroup: /system.slice/clamav-daemon.socket ● clamav-daemon.service - clamd scanner daemon Loaded: loaded (/etc/systemd/system/clamav-daemon.service; enabled; vendor preset: disabled) Active: deactivating (stop-sigterm) (Result: timeout) TriggeredBy: ● clamav-daemon.socket Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Cntrl PID: 1592 (clamd) Tasks: 2 (limit: 9497) Memory: 1.2G CPU: 11.012s CGroup: /system.slice/clamav-daemon.service ├─1592 /usr/sbin/clamd --config-file=/usr/local/etc/clamav/clamd.conf └─1596 /usr/sbin/clamd --config-file=/usr/local/etc/clamav/clamd.conf ● clamav-freshclam.service - ClamAV virus database updater Loaded: loaded (/usr/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamav-freshclam.service.d └─override.conf Active: active (running) since Fri 2020-10-23 09:56:54 PDT; 11min ago Docs: man:freshclam(1) man:freshclam.conf(5) https://www.clamav.net/documents Process: 883 ExecStart=/usr/bin/freshclam -d --config-file=/usr/local/etc/clamav/freshclam.conf (code=exited, status=0/SUCCESS) Main PID: 906 (freshclam) Tasks: 1 (limit: 9497) Memory: 4.5M CPU: 80ms CGroup: /system.slice/clamav-freshclam.service └─906 /usr/bin/freshclam -d --config-file=/usr/local/etc/clamav/freshclam.conf ● clamav-milter.service - Milter module for the Clam Antivirus scanner Loaded: loaded (/usr/lib/systemd/system/clamav-milter.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamav-milter.service.d └─override.conf Active: active (running) since Fri 2020-10-23 09:57:12 PDT; 11min ago Process: 1588 ExecStart=/usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf (code=exited, status=0/SUCCESS) Main PID: 1604 (clamav-milter) Tasks: 3 (limit: 9497) Memory: 2.1M CPU: 32ms CGroup: /system.slice/clamav-milter.service └─1604 /usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf attach each clamd gdb /usr/sbin/clamd 1592 gdb /usr/sbin/clamd 1596 wait; the *1st* issue that appears, fairly early after a boot dmesg -w ... [ 874.876248] systemd[1]: clamav-daemon.service: State 'stop-sigterm' timed out. Killing. [ 874.876311] systemd[1]: clamav-daemon.service: Killing process 1592 (clamd) with signal SIGKILL. [ 874.876394] systemd[1]: clamav-daemon.service: Killing process 1596 (clamd) with signal SIGKILL. ... check ps -eL | grep clamd 1592 1592 ? 00:00:00 clamd <defunct> 1596 1596 ? 00:00:11 clamd <defunct> appears to be a timeout @ gdb /usr/sbin/clamd 1592 ... (gdb) thread apply all bt full Thread 1 (Thread 0x7f9ffa8c0040 (LWP 1592)): #0 0x00007f9ffb878eca in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7fff245838cc, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27 resultvar = 18446744073709551104 sc_ret = <optimized out> #1 0x00007f9ffb878e8b in __GI___waitpid (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7fff245838cc, options=options@entry=0) at waitpid.c:38 No locals. #2 0x00007f9ffb878e73 in __wait (stat_loc=stat_loc@entry=0x7fff245838cc) at wait.c:25 No locals. #3 0x000055fa0bc378d4 in daemonize_parent_wait (user=0x55fa0bd48c90 <error: Cannot access memory at address 0x55fa0bd48c90>, log_file=0x55fa0bd66600 <error: Cannot access memory at address 0x55fa0bd66600>) at misc.c:362 sig = <error reading variable sig (Cannot access memory at address 0x7fff245838d0)> exitStatus = <error reading variable exitStatus (Cannot access memory at address 0x7fff245838cc)> daemonizePid = <optimized out> #4 0x000055fa0bc2be1b in main (argc=<optimized out>, argv=<optimized out>) at clamd.c:280 daemonizeRet = 0 engine = <error reading variable engine (Cannot access memory at address 0x55fa0bc54058)> opt = <optimized out> sa = <error reading variable sa (Cannot access memory at address 0x7fff24583a80)> rlim = <error reading variable rlim (Cannot access memory at address 0x7fff245839e0)> dropPrivRet = 0 currtime = <error reading variable currtime (Cannot access memory at address 0x7fff245839c8)> dbdir = <optimized out> cfgfile = <optimized out> pua_cats = 0x0 pt = 0x7fff245839c8 <error: Cannot access memory at address 0x7fff245839c8> ret = <optimized out> tcpsock = 0 localsock = 0 min_port = <optimized out> max_port = <optimized out> sigs = <error reading variable sigs (Cannot access memory at address 0x7fff245839c0)> lsockets = <error reading variable lsockets (Cannot access memory at address 0x7fff245839d0)> nlsockets = <error reading variable nlsockets (Cannot access memory at address 0x7fff245839c4)> dboptions = 0 i = <optimized out> j = <optimized out> num_fd = 1 parentPid = <error reading variable parentPid (Cannot access memory at address 0x7fff24583970)> sb = <error reading variable sb (Cannot access memory at address 0x7fff245839f0)> mainpid = 0 old_umask = 0 user_name = <error reading variable user_name (Cannot access memory at address 0x7fff24583978)> Backtrace stopped: Cannot access memory at address 0x7fff24583b88 @ gdb /usr/sbin/clamd 1596 ... (gdb) thread apply all bt full Thread 1 (Thread 0x7f9ffa8c0040 (LWP 1596)): #0 0x00007f9ffbc74b62 in cli_dbgets (buff=buff@entry=0x7fff2457b420 <error: Cannot access memory at address 0x7fff2457b420>, size=size@entry=8192, fs=fs@entry=0x0, dbio=dbio@entry=0x7fff24580820) at readdb.c:723 bread = <optimized out> nl = <optimized out> #1 0x00007f9ffbc74f03 in cli_loadhash (fs=0x0, engine=0x55fa0bd680f0, signo=<error reading variable: Cannot access memory at address 0x7fff2457b3d8>, mode=<error reading variable: Cannot access memory at address 0x7fff2457b3c8>, options=27226, dbio=0x7fff24580820, dbname=<error reading variable: Cannot access memory at address 0x7fff2457d470>) at readdb.c:2490 tokens = <error reading variable tokens (Cannot access memory at address 0x7fff2457b3f0)> buffer = <error reading variable buffer (Cannot access memory at address 0x7fff2457b420)> buffer_cpy = 0x55fa0bee4ae0 <error: Cannot access memory at address 0x55fa0bee4ae0> pt = <error reading variable pt (Cannot access memory at address 0x7fff2457b3e8)> virname = <optimized out> ret = 0 size_field = 1 md5_field = <error reading variable md5_field (Cannot access memory at address 0x7fff2457b3d4)> line = 250549 sigs = <error reading variable sigs (Cannot access memory at address 0x7fff2457b3d0)> tokens_count = <optimized out> req_fl = 73 db = <error reading variable db (Cannot access memory at address 0x7fff2457b3c0)> size = <optimized out> Backtrace stopped: Cannot access memory at address 0x7fff2457d468 at this point systemctl status clamav-daemon.socket clamav-daemon.service clamav-freshclam.service clamav-milter.service -ln0 ● clamav-daemon.socket - Socket for Clam AntiVirus userspace daemon Loaded: loaded (/etc/systemd/system/clamav-daemon.socket; enabled; vendor preset: disabled) Active: active (running) since Fri 2020-10-23 09:56:42 PDT; 20min ago Triggers: ● clamav-daemon.service Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Listen: 127.0.0.1:3310 (Stream) Tasks: 0 (limit: 9497) Memory: 24.0K CPU: 1ms CGroup: /system.slice/clamav-daemon.socket ● clamav-daemon.service - clamd scanner daemon Loaded: loaded (/etc/systemd/system/clamav-daemon.service; enabled; vendor preset: disabled) !! Active: deactivating (stop-sigkill) (Result: timeout) !! TriggeredBy: ● clamav-daemon.socket Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Cntrl PID: 1592 (clamd) Tasks: 2 (limit: 9497) Memory: 448.0M CPU: 11.064s CGroup: /system.slice/clamav-daemon.service └─1592 [clamd] ● clamav-freshclam.service - ClamAV virus database updater Loaded: loaded (/usr/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamav-freshclam.service.d └─override.conf Active: active (running) since Fri 2020-10-23 09:56:54 PDT; 20min ago Docs: man:freshclam(1) man:freshclam.conf(5) https://www.clamav.net/documents Process: 883 ExecStart=/usr/bin/freshclam -d --config-file=/usr/local/etc/clamav/freshclam.conf (code=exited, status=0/SUCCESS) Main PID: 906 (freshclam) Tasks: 1 (limit: 9497) Memory: 4.5M CPU: 80ms CGroup: /system.slice/clamav-freshclam.service └─906 /usr/bin/freshclam -d --config-file=/usr/local/etc/clamav/freshclam.conf ● clamav-milter.service - Milter module for the Clam Antivirus scanner Loaded: loaded (/usr/lib/systemd/system/clamav-milter.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamav-milter.service.d └─override.conf Active: active (running) since Fri 2020-10-23 09:57:12 PDT; 19min ago Process: 1588 ExecStart=/usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf (code=exited, status=0/SUCCESS) Main PID: 1604 (clamav-milter) Tasks: 3 (limit: 9497) Memory: 2.1M CPU: 42ms CGroup: /system.slice/clamav-milter.service └─1604 /usr/sbin/clamav-milter --config-file /usr/local/etc/clamav/clamav-milter.conf i see this^ *before* i see subsequent OOPS as reported above. entirely possible that this is related -- so probly should clear it up _first_. looking into what/why. not immediately obvious to me which timeout is causing the kill.
i disabled/masked the pkg'd systemd units etc; now using full replacements in my /etc/systemd/system/ path i removed any/all .socket exec/deps so ONLY clamav-daemon.service clamav-freshclam.service clamav-milter.service are enabled & active. INCREASED milter/daemon read/write timeouts DECREASED in clamd conf, MaxThreads 1 MaxDirectoryRecursion 8 MaxRecursion 4 MaxQueue 15 and switched clamd to unix:, rather than TCP socket listener. restarted everything ... ... so far, after reboot, and just a couple hours of usage, no OOPS. I _suspect_ load/contention @ the TCP listener socket may have been the actual issue. need to keep an eye on it for awhile, & see if I spoke too soon!
seems to be behaving. closing this until if/when i can find a reproducible trigger.