Bug 1999089

Summary:	New Host-guest mapping is not getting populated due to an existing blocked Hypervisor Heartbeat Update job
Product:	Red Hat Satellite	Reporter:	Sayan Das <saydas>
Component:	Candlepin	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED DUPLICATE	QA Contact:	Danny Synk <dsynk>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.9.0	CC:	ahumbe, bcourt, momran, msunil, nmoumoul, pmendezh, redakkan, satellite6-bugs
Target Milestone:	Unspecified	Keywords:	PrioBumpGSS, Triaged
Target Release:	Unused
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-25 14:53:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sayan Das 2021-08-30 12:03:17 UTC

Description of problem:

After virt-who restart, the Hypervisor update task (Actions::Katello::Host::Hypervisors) in UI shows successfully completed but the information about new ESX is missing or host-guest-mapping has not been updated. 


Version-Release number of selected component (if applicable):

Satellite 6.8 and 6.9 [ possibly 6.10 as well ]

How reproducible:

Under the situation where the actual hypervisor update job is blocked by some existing job 

Steps to Reproduce:
1. Configure virt-who in Satellite
2. Restart virt-who
3. Remove some ESX from the satellite
4. Restart virt-who

Actual results:

The removed ESX is not added back in Satellite UI

Following error noticed in candlepin.log

~~
Unable to queue job: Hypervisor Heartbeat Update; blocked by the following existing jobs: 8af3b8eb794a03aa017a46b4b156554a
~~



Expected results:


Either the "Actions::Katello::Host::Hypervisors" will fail with appropriate errors or candlepin should not have any existing jobs affecting the new job processing.


Additional info:


The virt-who service is unable to report back a hypervisor that was deleted from Satellite UI in Red Hat Satellite 6 - Red Hat Customer Portal
https://access.redhat.com/solutions/6177582

Bug #31412: virt-who can't update the VMs-to-ESXi Host mapping since the last update - Katello - Foreman
https://projects.theforeman.org/issues/31412#note-6

The issue was reported at upstream and two satellite customers have also faced the same issue. 

Even though the upstream issue was closed after suggesting a workaround, this needs to be fixed at product level if possible.

Comment 10 momran 2021-12-13 14:53:01 UTC

Hello,
In case # 03095475, we encountered an issue where a candlepin race condition occurs when satellite running virt-who using the default virt-who config (i.e. one different virt-who-reporter user per virt-who config file) tries to upload the whole report at once. We would like to address this issue and have a permanent fix in place.
Would you kindly advise?
Thanks in advance.

Comment 15 Nikos Moumoulidis 2022-03-25 14:53:10 UTC

Closing this. This issue (hypervisor job being blocked by another job) can have 2 known causes:

1. Either the CRL file has grown too large, causing an OutOfMemory error, resulting in the job processing subsystem shutting down which is a separate, essentially duplicate bug: 1927532
or
2. Misconfiguration of virt-who: multiple virt-who config files running on the same machine, reporting against the same Owner, that use different credentials (username/password). For this,
please create a bug in either of:

- Product: Red Hat Enterprise Linux (7/8), component: virt-who: virt-who should either detect the fact that 2 or more configurations against the same
organization are using the same username/password and fail, or perhaps, once having detected that they
are destined for the same organization just make a single request to the server (instead of one per-username).

- The Satellite Virt-Who configure plugin: it should disallow the generation of multiple virt-who configs
that use the same credentials (username/password), when those configs are for the same organization.

*** This bug has been marked as a duplicate of bug 1927532 ***

Comment 16 Nikos Moumoulidis 2022-04-01 14:08:33 UTC

(In reply to Nikos Moumoulidis from comment #15)

I realized that I made some errors here; instead of 'same username/password' / 'same credentials', it these should say 'different username/password' / 'different credentials'.

> - Product: Red Hat Enterprise Linux (7/8), component: virt-who: virt-who
> should either detect the fact that 2 or more configurations against the same
> organization are using the same username/password and fail, or perhaps, once
> having detected that they
> are destined for the same organization just make a single request to the
> server (instead of one per-username).
> 
> - The Satellite Virt-Who configure plugin: it should disallow the generation
> of multiple virt-who configs
> that use the same credentials (username/password), when those configs are
> for the same organization.
> 
> *** This bug has been marked as a duplicate of bug 1927532 ***

Comment 17 Red Hat Bugzilla 2023-09-15 01:14:24 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days