Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1937361

Summary: Engine is trying to refresh the capabilities of a rebooted host
Product: Red Hat Enterprise Virtualization Manager Reporter: Roni <reliezer>
Component: ovirt-ansible-collectionAssignee: Dana <delfassy>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Guilherme Santos <gdeolive>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4.5CC: ahadas, delfassy, mburman, mperina
Target Milestone: ---Keywords: Automation
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-21 09:38:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roni 2021-03-10 13:21:54 UTC
Created attachment 1762314 [details]
engine logs

Description of problem:
Engine is trying to refresh the capabilities of a rebooted host

Version-Release number of selected component (if applicable):
4.4.5.9-0.1.el8ev

How reproducible:
100%

Steps to Reproduce:
1. Install hosted-engine environment
2. Add new host to Engine (by default reboot host is checked/enable)
3. Wait until host will be added

Actual results:
Errors appear at the event-logs when host is still not in UP state.
e.g.:
Host host_mixed_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster golden_env_mixed_1 which is compatible with VDSM versions [4.20, 4.40, 4.30].

Expected results:
Host should be added successfully, no event error is expected

Additional info:
- Note that this makes the 'ovirt-ansible-collection' to fail
  when calling to "infra.roles.hosts" role to add additional hosts to the engine
- See attached engine logs

Comment 1 Dana 2021-03-31 12:46:43 UTC
What was the version of the host's cluster?

Comment 2 Dana 2021-04-01 12:36:26 UTC
From the log, the host was installed and rebooted successfully 
The error happens after
vds_id='b6cd4f0a-b969-4115-8ad5-100ab3613645', vds_name='host_mixed_3', vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error: VDSGenericException: VDSNetworkException: Connection issue Connection timeout

which is followed by 
2021-03-10 12:39:32,359+02 ERROR [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-13) [3cbef269] Could not find server cpu for server 'host_mixed_3' (b6cd4f0a-b969-4115-8ad5-100ab3613645), flags: 'null'

2021-03-10 12:39:32,379+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-162) [] EVENT_ID: VDS_FAILURE(12), Host host_mixed_3 is non responsive.

Could this be related to changes done around CPU?

Comment 3 Arik 2021-04-04 17:28:03 UTC
(In reply to Dana from comment #2)
> Could this be related to changes done around CPU?

That's unlikely.
It seems like we're missing information that should have been reported by the host - not only CPU flags but also the VDSM version ("Host host_mixed_2 is installed with VDSM version (<UNKNOWN>)")

Comment 4 Roni 2021-04-07 13:34:44 UTC
(In reply to Dana from comment #1)
> What was the version of the host's cluster?

It should be 4.5

Comment 5 Roni 2021-04-07 13:37:08 UTC
maybe the problem is that Engine is trying to refresh-capabilities of rebooted hosts

Comment 6 Roni 2021-04-07 13:43:33 UTC
Note that the problem occurs only when added hosts are rebooted, to WA the issue we disable the reboot

Comment 7 Dana 2021-04-08 12:10:48 UTC
Can you increase the reboot timeout in engine's config values to 15min. and see if it solves the issue?

Comment 8 Martin Perina 2021-04-08 12:30:52 UTC
(In reply to Dana from comment #7)
> Can you increase the reboot timeout in engine's config values to 15min. and
> see if it solves the issue?

See details in Doc Text of BZ1947403

Comment 9 Martin Perina 2021-04-21 09:38:09 UTC
Closing, we have increased default reboot timeout to 10 minutes in BZ1947403. If even 10 minutes is not enough for your host, please increase the reboot timeout as described in BZ1947403.

Comment 10 Roni 2021-04-22 06:58:47 UTC
FYI: did not reproduced with 4.4.6.5-447.gd80dda7.9.el8ev