RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2022191 - Insights-client getting continuously spun up and consumes considerable CPU time in RHEL 9 Beta.
Summary: Insights-client getting continuously spun up and consumes considerable CPU ti...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: insights-client
Version: 9.0
Hardware: Unspecified
OS: Unspecified
urgent
low
Target Milestone: rc
: ---
Assignee: Link Dupont
QA Contact: Pavol Kotvan
URL:
Whiteboard:
: 2027263 2062153 (view as bug list)
Depends On: 2027263 2062136 2063195
Blocks: 2055036 2061967 2070323 2109244
TreeView+ depends on / blocked
 
Reported: 2021-11-11 02:57 UTC by Nikhil Gupta
Modified: 2022-07-20 18:23 UTC (History)
13 users (show)

Fixed In Version: insights-client-3.1.7-3.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2055036 2061967 2070323 2109244 (view as bug list)
Environment:
Last Closed: 2022-05-17 14:05:59 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ESSNTL-1583 0 None None None 2021-11-11 12:55:29 UTC
Red Hat Issue Tracker RHELPLAN-102465 0 None None None 2021-11-11 02:58:07 UTC

Description Nikhil Gupta 2021-11-11 02:57:13 UTC 
Description of problem:
Insights-client getting continuously spun up and consumes considerable CPU time in RHEL 9 Beta.

~~~
Nov 10 13:58:16 rhel9d audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=insights-client-results comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 10 13:58:16 rhel9d audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=insights-client-results comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 10 13:58:16 rhel9d systemd[1]: insights-client-results.service: Deactivated successfully.
Nov 10 13:58:16 rhel9d systemd[1]: insights-client-results.service: Consumed 2.793s CPU time.
Nov 10 13:58:23 rhel9d audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=insights-client-results comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 10 13:58:23 rhel9d audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=insights-client-results comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 10 13:58:23 rhel9d systemd[1]: insights-client-results.service: Deactivated successfully.
Nov 10 13:58:23 rhel9d systemd[1]: insights-client-results.service: Consumed 3.195s CPU time.
~~~

Version-Release number of selected component (if applicable):
insights-client-3.1.6-2.el9.noarch

How reproducible:
Always

Steps to Reproduce:
1. Install RHEL 9 BETA (minimal-install, nothing added)
2. subscription-manager register --auto-attach
3. dnf install insights-client
4. insights-client --register
5. Review insights activity in journal:  journalctl -f
6. Review CPU consumption: top 

Actual results:
- insights-client is getting started over and over again constantly.
- platform-python and insights-client are top offenders.

Expected results:
- insights-client should be started once a day.
Comment 1 Nikhil Gupta 2021-11-11 03:00:10 UTC 
Additional info:
Also, after having tried out the metrics rhel-system-role on RHEL 9 system, insights-client started getting continuously spun up and consumes considerable CPU time.

Reproducer:
```
1. Install RHEL 9 BETA
2. Install system roles and ansible:
dnf install ansible-core ansible-pcp rhel-system-roles 

3. Create playbook:
cat << 'EOF' >playbook.yml
---
- hosts: the-system
  vars:
    metrics_graph_service: yes
  roles:
    - rhel-system-roles.metrics
EOF

4. Run playbook:
ansible-playbook ./playbook.yml
```
Comment 4 Magnus Glantz 2021-11-11 12:47:40 UTC 
Ignore the additional info, it's unrelated.
Comment 7 Christian Labisch 2022-01-13 13:00:05 UTC 
After having registered RHEL 9 Beta to Insights, I experience performance issues on my host. CPU usage and network traffic rise up drastically.
I disabled all Insights related services and rebooted the VM, unfortunately the same occurs again, until I unregister the system from Insights.
Strange side effect is that even though the system isn't registered anymore, it re-appears after a restart on console.redhat.com again. Ideas ?
Comment 8 Ryan Blakley 2022-01-27 16:12:49 UTC 
(In reply to Christian Labisch from comment #7)
> After having registered RHEL 9 Beta to Insights, I experience performance
> issues on my host. CPU usage and network traffic rise up drastically.
> I disabled all Insights related services and rebooted the VM, unfortunately
> the same occurs again, until I unregister the system from Insights.
> Strange side effect is that even though the system isn't registered anymore,
> it re-appears after a restart on console.redhat.com again. Ideas ?

You can mask the insights-client-results.service to stop this issue from happening anymore until a fix is released.
Comment 9 Link Dupont 2022-01-28 20:18:31 UTC 
This issue is caused by this line in /usr/lib/systemd/system/insights-client-results.path

[Unit]
Description=Monitor @pkgsysconfdir@/.lastupload for modifications
Documentation=man:insights-client(8)
PartOf=insights-client.timer

[Path]
PathExists=@pkgsysconfdir@/.lastupload      # <<<
PathModified=@pkgsysconfdir@/.lastupload

[Install]
WantedBy=insights-client.timer

https://github.com/RedHatInsights/insights-client/blob/master/data/systemd/insights-client-results.path.in#L16

Because the file /etc/insights-client/.lastupload exists, the service is being triggered continuously.
Comment 10 Christian Labisch 2022-01-31 11:10:31 UTC 
Thanks @Link ! I have checked it and removed the .lastupload and the .last-upload.results files on January 29th.
Unfortunately that did not change the behavior ... the system re-appeared today after I had booted it yesterday.
I have removed it manually again today from console.redhat.com - below you you can see the facts of the system :

$ cat /usr/lib/systemd/system/insights-client-results.path
[Unit]
Description=Monitor /etc/insights-client/.lastupload for modifications
Documentation=man:insights-client(8)
PartOf=insights-client.timer

[Path]
PathExists=/etc/insights-client/.lastupload
PathModified=/etc/insights-client/.lastupload

[Install]
WantedBy=insights-client.timer

$ ls -la /etc/insights-client
total 1556
drwxr-xr-x.   2 root root    4096 29. Jan 16:04 .
drwxr-xr-x. 105 root root    8192 31. Jan 11:43 ..
-rw-------.   1 root root  145114 29. Jan 16:02 .cache.json
-rw-------.   1 root root     811 29. Jan 16:02 .cache.json.asc
-r--r--r--.   1 root root    4586  2. Mär 2021  cert-api.access.redhat.com.pem
-rw-r--r--.   1 root root     185  2. Mär 2021  .exp.sed
-rw-r--r--.   1 root root  143458 18. Okt 16:31 .fallback.json
-rw-r--r--.   1 root root     811 18. Okt 16:31 .fallback.json.asc
-rw-r--r--.   1 root root    1490 24. Aug 16:00 insights-client.conf
-rw-r--r--.   1 root root     151 20. Okt 16:34 insights-client.motd
-rw-r--r--.   1 root root      52 29. Jan 16:00 .insights-core.etag
-rw-r--r--.   1 root root      52 29. Jan 16:00 .insights-core-gpg-sig.etag
-rw-r--r--.   1 root root      36  2. Nov 19:17 machine-id
-r--r--r--.   1 root root    1147  2. Mär 2021  redhattools.pub.gpg
-rw-r--r--.   1 root root 1224762 18. Okt 16:31 rpm.egg
-rw-r--r--.   1 root root     811 18. Okt 16:31 rpm.egg.asc
-rw-r--r--.   1 root root      26 29. Jan 16:02 .unregistered
Comment 11 Link Dupont 2022-01-31 12:47:05 UTC 
You need to remove line 16 from /usr/lib/systemd/system/insights-client-results.path and run `systemctl daemon-reload` in order to fix it. Removing /etc/insights-client/.lastupload will not change the behavior of the systemd unit file.

Another, better workaround would be to create an override unit file that redefines insights-client-results.path until we push a bug fix. Run `systemctl edit --full`, edit the text buffer by deleting line 16, and then save and quit.
Comment 12 Christian Labisch 2022-01-31 13:15:19 UTC 
Thank you @Link ! I'm gonna check it out ... more important though will be that the major issue (high CPU usage and network traffic) will get resolved when one registers a RHEL 9 system to Insights as soon as possible.
Comment 13 Gaël Chamoulaud 2022-02-01 15:00:00 UTC 
*** Bug 2027263 has been marked as a duplicate of this bug. ***
Comment 14 Christian Labisch 2022-02-02 11:35:27 UTC 
FYI @Link : I removed line 16 from /usr/lib/systemd/system/insights-client-results.path two days ago as you suggested - but still the system re-appears.
I have removed the system manually from console.redhat.com again yesterday and rebooted - well, today I found it being back on console.redhat.com again.
Comment 15 Link Dupont 2022-02-02 14:57:54 UTC 
Removing line 16 from /usr/lib/systemd/system/insights-client-results.path will not remove the system from Insights Inventory. It only stops the insights-client-results.service unit from continuously restarting.
Comment 16 Christian Labisch 2022-02-02 15:05:17 UTC 
Thank you @Link ! Seems I misunderstood ... I have re-registered to Insights, and you are right - continuously restarting stopped. Thanks again Link ! :)
Comment 22 Gaël Chamoulaud 2022-03-09 13:42:36 UTC 
*** Bug 2062153 has been marked as a duplicate of this bug. ***
Comment 24 Christian Labisch 2022-04-06 09:40:07 UTC 
I just upgraded to RHEL 9.0 Beta 5 and it seems that the package insights-client 3.1.7-5.el9 has resolved the problem.
Comment 27 errata-xmlrpc 2022-05-17 14:05:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: insights-client), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2645
Note You need to log in before you can comment on or make changes to this bug.