Hide Forgot
Description of problem: System "Last Scan" date/time is not updating immediately after running the "# insights-client --collector malware-detection" command. Steps to Reproduce: 1. Performs a test scan 2. check the Last Scan time in Insights > malware > System 3. Perform a full filesystem scan. Edit /etc/insights-client/malware-detection-config.yml and set the test_scan option to false. test_scan: false 4. Run # insights-client --collector malware-detection 5. It will not update the Last Scan Date and it will show the test scan date/time. Expected results: full filesystem scan should update the last scan date immediately after completion of command # insights-client --collector malware-detection
1.) It's not about being updated immediately. The Red Hat Insights Console is not updated when test_scan is set to false. The scan runs but Insights never picks up the results and/or update the last scan date. 2.) Only test scans are logged in Insight console although 'test results are not uploaded to insights' it shows as if it was a full scan but it was not. 3.) Tested on RHEL 7.9 (EOL 2024) -- malware detection test scan matched against a false-positive file which was a core-dump from another vendor. If I delete the core-dump re-run a test scan (full scan doesn't work see point 1) it updates the signatures to not matched but the main system entry in Insights console still shows as matched. (Should revert to Not Matched..)
Unfortunately the customer case is closed now, but the output that would be helpful to help resolving this is all the log files from /var/log/insights-client/*. That is, reproduce the issue with insights-client --collector malware-detection, then zip up all the files in /var/log/insights-client/* and attach them to this ticket. The insights-client logs are collected with debugging enabled and that would be very helpful in trying to identify the problem. If there is anyone CC'd on this BZ that is encountering the problem, could they please reproduce the problem and attach all the /var/log/insights-client/* log files. Without them its hard to identify the problem.
Created attachment 1939022 [details] insights-client.log* Hi Mark, I have proceeded as follows: sudo rm -r /var/log/insights-client sudo insights-client sudo insights-client --compliance sudo insights-client --collector malware-detection See attached the new log-files. According to Red Hat Insights Console, no analysis was run across my system. Hope this helps to find a solution. regards, Dirk
Hi Dirk, Unfortunately the debug logs don't reveal what the problem is either. As a test, can you try excluding the /usr directory from being scan and try performing another scan. That is, in malware-detection-config.yml, add /usr under the filesystem_scan_exclude section, like so: filesystem_scan_exclude: - /usr Then perform another scan. I'm wondering if there is something about the /usr directory, or something in it, that is causing the scan to be aborted prematurely and the results not uploaded. Please zip up and attach the insights-client logs files again after doing this. Regards, Mark
Hi Dirk, Another thing to try. If it works with /usr excluded then edit malware-detection-config.yml again and do the following: - Remove /usr from the filesystem_scan_exclude option (ie undo the change made in my last post) - Set the option: add_metadata: false - Add this line at the end of the file: string_match_limit: 0 Then save it and try performing another scan. The /usr directory should show up again in the output but these options will skip getting extra metadata about any matches found in there, but matches will still show up (if any). Again, zip up all the insights-client log files and attach them to the ticket. I am looking at adding more logging to the internals of the malware-detection app to hopefully display more debugging information when it fails. Cheers, Mark
Created attachment 1939188 [details] insights-client.log -/usr* Hi Mark, see attached the logs after exclusion /usr and running steps 1 -4 like yesterday. According to Red Hat Insights Console, no analysis was run across my system. I'll start with the second idea in a moment...
Created attachment 1939196 [details] insights-client.log add_metadata* second try - see attached the logs after next edit of malware-detection-config.yml add_metadata: false add string_match_limit: 0 again run through the command sequence 1-4 ... no abort! at the end message: .. Scanning files in /usr ... Uploading Insights data. Successfully uploaded report for * According to Red Hat Insights Console, no analysis was run across my system. Dirk
The malware-detection app seems to be silently crashing and aborting the scan. I'm not yet able to identify what's causing it to crash and I'm adding some more logging into the app to help identify the cause of the problem. Hopefully these changes will be in the next egg to be released. Some interesting points from the logs ... In previous logs the scan would abort when scanning /usr. However when excluding /usr, it aborted when scanning /home, yet in other runs when /usr wasn't excluded it completed the scan of /home just fine. Scanning /home fine ... 2023-01-19 18:11:06,607 INFO insights.client.apps.malware_detection Scanning files in /home ... 2023-01-19 18:11:06,608 DEBUG insights.client.apps.malware_detection Yara command: ['nice', '-n', '19', '/bin/yara', '-s', '-N', '-a', '3600', '-p', '1', '-r', '-f', '-C', '/tmp/.tmpmdsigs6r965tv2', '/home'] 2023-01-19 18:11:06,608 DEBUG insights.util.subproc Executing: [['nice', '-n', '19', '/bin/yara', '-s', '-N', '-a', '3600', '-p', '1', '-r', '-f', '-C', '/tmp/.tmpmdsigs6r965tv2', '/home']] 2023-01-19 18:11:35,869 INFO insights.client.apps.malware_detection Scan time for /home: 29 seconds Aborting the scan whilst scanning /home ... ... 2023-01-19 17:32:21,623 INFO insights.client.apps.malware_detection Scanning files in /home ... 2023-01-19 17:32:21,623 DEBUG insights.client.apps.malware_detection Yara command: ['nice', '-n', '19', '/bin/yara', '-s', '-N', '-a', '3600', '-p', '1', '-r', '-f', '-C', '/tmp/.tmpmdsigsduaqs8t9', '/home'] 2023-01-19 17:32:21,623 DEBUG insights.util.subproc Executing: [['nice', '-n', '19', '/bin/yara', '-s', '-N', '-a', '3600', '-p', '1', '-r', '-f', '-C', '/tmp/.tmpmdsigsduaqs8t9', '/home']] 2023-01-19 17:34:11,036 DEBUG insights.client.core_collector Collection finished. ... 2023-01-19 17:34:11,053 INFO insights.client.client Uploading Insights data. Strange indeed. I expected that by excluding /usr, the scan would complete without error, but not so :/ I'll keep looking into it.
An update for insights-client has been made available today for RHEL 9.1 : https://access.redhat.com/downloads/content/insights-client/3.1.7-10.el9_1/noarch/fd431d51/package Customers being affected by the bug may want to check if (for whatever reason) the new version has resolved the problem - it's not very likely, but I think it's worth a try. sudo rm -r /var/cache/insights sudo rm -r /var/log/insights-client sudo insights-client --collector malware-detection
Created attachment 1940308 [details] Test insights-core egg with extra debugging Hi all, For those people encountering problems with malware-detection not functioning correctly, please try using the attached insights.zip to (hopefully) collect more debugging information as to why malware-detection is silently failing. Download insights.zip and put it in /tmp and run these commands, all as root: # export INSIGHTS_GPG=false # export BYPASS_GPG=false # export EGG=/tmp/insights.zip # insights-client --collector malware-detection It will still fail but hopefully now we'll have a better idea why it is failing. Then zip up the files in /var/log/insights-client/* files and attach them to this BZ. Thanks, Mark
Hi all, Many thanks for the support! I have just applied the steps outlined hereabove by Mark Huth and uploaded the log files. As a kind of disclaimer, please not that I am no sysadmin or linux expert, so I hope there is no gross misconfiguration on my end :-/, but the issue could be reproduced today. Please also note that the initial system where these issues were recorded was my computer running Red Hat 9.0 updated into 9.1 ; this time, it is the same computer but I did a fresh Red Hat 9.1 installation in Gnome-Boxes. Standard configurations are used, with the standard CIS hardening workstation 2 security profile applied during the installation. For more information, here is the link to the thread on the Red Hat community website where Christian Labisch has been helping on this issue since the beginning: https://access.redhat.com/discussions/6993162 Having had a quick look at the logs from today, it seems there are some network errors - the command test-connection was successfully completed though - and wonder if there could be any links with firmware-level issues at least contributing to this. I also just noted on the Red Hat community forum that the last update appears as done in 2017 (!) while both my computer and the VM's time are correct. Please feel free to let me know if any additional information could be helpful. Best regards, Alexandre
Sorry to write again, but it seems not possible for me to add an attachment - I don't know if additional permissions, I am using the version of Firefox provided with Red Hat 9.1 (102.7.0esr (64-bit). If that's ok, I can send these log files by email instead. Many thanks, Alexandre
Hi Alexandre, As Mark said, "It will still fail ..." What you can try is adding "timeout=3600" to the (extracted) /tmp/insights/insights/specs/datasources/malware_detection.py file. Add "timeout=3600" behind the line "@datasource(HostContext)", so that the modified line reads "@datasource(HostContext, timeout=3600)" (without the quotes of course). Save the file and (re-)compress the /tmp/insights directory to insights.zip. Repeat the test (as root user) and check if it worked as expected this time. rm -rf /var/cache/insights rm -rf /var/log/insights-client export INSIGHTS_GPG=false export BYPASS_GPG=false export EGG=/tmp/insights.zip insights-client --collector malware-detection Regards, Christian
Hi Christian, Many thanks for your follow up and new message. Sorry for the late response, but I am on a trip with limited access to my emails and to my Red Hat system. I have just noticed some recent change in my system, maybe at firmware level, as a live linux usb distribution that could be booted on my laptop can't be booted not anymore. That makes me think of something like this update on Fedora that, unless a mistake on my side, wasn't applied yet but I'll have a look at that too. https://ask.fedoraproject.org/t/secure-boot-dbx-update/26626 I am going to be back on my Red Hat OS next Wednesday, and will get back to you asap! Many thanks again everyone and speak soon, Alexandre
Hi all, A new insights-core egg has been released that contains the fix for this issue. Make sure you see version 3.1.5-1, like so: # insights-client --version Client: 3.1.7 Core: 3.1.5-1 Then run malware-detection as usual and hopefully now it will correctly upload an archive and you will see an updated timestamp for your system in the Insights WebUI. Cheers, Mark
Please feel free to reopen this BZ if it seems this issue has not been resolved.
Hi Mark, thx - works fine. now it also runs on my weakly performant system. regards. Dirk
Hi Mark, hi Christian and all, I have reinstalled Insights-client today and the full scans work - many thanks for the solution ! :-) Best regards, Alexandre