RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1825843 - virt-who is connecting too much and eating cpu time
Summary: virt-who is connecting too much and eating cpu time
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-who
Version: 7.7
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: William Poteat
QA Contact: Eko
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-20 11:37 UTC by Suraj Patil
Modified: 2023-10-06 19:42 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-29 19:22:29 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:3865 0 None None None 2020-09-29 19:22:34 UTC

Description Suraj Patil 2020-04-20 11:37:22 UTC 
Description of problem:
virt-who eating up the CPU and /var/log/message filled up with logs.

Virt-who is executing send_data and getHostGuestMapping function repeatedly and eating up CPU. 

~~~
Apr 14 10:06:27 rhn01 /usr/bin/virt-who: 2020-04-14 10:06:27,770 [virtwho.main INFO] MainProcess(5418):Thread-2 @virt.py:_send_data:981 - Report for config "virt-who-config-3" gathered, placing in datastore
Apr 14 10:06:27 rhn01 /usr/bin/virt-who: 2020-04-14 10:06:27,949 [virtwho.main INFO] MainProcess(5418):Thread-2 @virt.py:_send_data:981 - Report for config "virt-who-config-3" gathered, placing in datastore
Apr 14 10:06:28 rhn01 /usr/bin/virt-who: 2020-04-14 10:06:28,457 [virtwho.main INFO] MainProcess(5418):Thread-2 @virt.py:_send_data:981 - Report for config "virt-who-config-3" gathered, placing in datastore
Apr 14 10:06:30 rhn01 /usr/bin/virt-who: 2020-04-14 10:06:30,805 [virtwho.main INFO] MainProcess(5418):Thread-2 @virt.py:_send_data:981 - Report for config "virt-who-config-3" gathered, placing in datastore
Apr 14 10:06:34 rhn01 /usr/bin/virt-who: 2020-04-14 10:06:34,375 [virtwho.main INFO] MainProcess(5418):Thread-2 @virt.py:_send_data:981 - Report for config "virt-who-config-3" gathered, placing in datastore
~~~

- It looks like config "virt-who-config-3" gets stuck in an infinite loop. Above logs are continuously generated until we stop the virt-who service. 

- Tried to reinstall and downgrade the virt-who package but no success. Also, confirmed that none of the config files is modified # rpm -V virt-who

- This issue is only with the virt-who-config-3, If we remove the virt-who-config-3 then virt-who services work properly with virt-who-config-2(which contains different vcenter). 

- virt-who -o (oneshot) working fine with "virt-who-config-3"

- There is nothing special or any extra parameter in virt-who-config-3.conf which can cause this.

Not sure but I think virt-who get stuck in this while loop - https://github.com/candlepin/virt-who/blob/e56d841f742f46eed853c8c106c7ed85eeb4b3e4/virtwho/virt/esx/esx.py#L153 and because of this self._send_data the method is executed and above logs are printed https://github.com/candlepin/virt-who/blob/e56d841f742f46eed853c8c106c7ed85eeb4b3e4/virtwho/virt/virt.py#L977

Version-Release number of selected component (if applicable):
- virt-who-0.24.7-1.el7.noarch
- Red Hat Enterprise Linux Server release 7.7 (Maipo)


Actual results:
"virt-who-config-3" gets stuck in an infinite loop. This works with oneshot, the problem is with a daemon.

Expected results:
virt-who service should work properly without repeatedly executing send_data and getHostGuestMapping function https://github.com/candlepin/virt-who/blob/6e8616350212b2d694e2e6a637ca4f7ff4cac2ef/virtwho/virt/esx/esx.py#L206



- Can it be the issue related to specific Vcenter?
Comment 7 asamad 2020-04-23 06:24:58 UTC 
Hello  yuefliu ,

Thanks for the update.

Please configure virt-who using virt-who configuration from satellite itself to reproduce the issue on test ENV.

https://bootp-73-3-198.eng.pek2.redhat.com/users/login >> Infrastructure >> Virt-who Configurations >> create a new config and then proceed with testing.
Comment 15 Sudhir Mallamprabhakara 2020-04-29 12:32:08 UTC 
@Rehana - Reading thru the comments, this sounds like a load/performance issue. Can we have someone from the perf team try to reproduce this?

-Sudhir
Comment 16 Rehana 2020-04-29 14:52:47 UTC 
(In reply to Sudhir Mallamprabhakara from comment #15)
> @Rehana - Reading thru the comments, this sounds like a load/performance
> issue. Can we have someone from the perf team try to reproduce this?
> 
> -Sudhir

Hi Sudhir, 

Thanks for reaching out , our dev team will be looking in this issue in the current sprint. Couple of our engineers are out on training this week , so I see this card mentioned in comment 14 picking up later this week. It would really helpful if you can help us with a test team who could helps us with the investigation . 

thanks,
Rehana
Comment 19 asamad 2020-05-07 08:09:53 UTC 
Hello William,

Thanks for the update.

Please help me with  virt-who build 0.28.5-1 package to test it and post you with result.

Awaiting for the attachments.
Comment 23 asamad 2020-06-01 03:25:11 UTC 
Hi Rehana,

Yes  virt-who build 0.28.5-1 package resolved the issue. Please proceed with errata process.

Thanks & Regards,
Abdul samad
Comment 28 errata-xmlrpc 2020-09-29 19:22:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt-who bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3865
Note You need to log in before you can comment on or make changes to this bug.