Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1766556

Summary: [4.4.0-4] hosted engine deploy failed with connection error from engine VM to the host.
Product: [oVirt] ovirt-ansible-collection Reporter: Kobi Hakimi <khakimi>
Component: hosted-engine-setupAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: aoconnor, bugs, dagur, khakimi, lsvaty, mburman
Target Milestone: ovirt-4.4.0Keywords: AutomationBlocker, Triaged
Target Release: 1.0.30Flags: sbonazzo: ovirt-4.4?
sbonazzo: planning_ack?
sbonazzo: devel_ack+
mavital: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-ansible-hosted-engine-setup-1.0.30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 20:00:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine-logs-2019-10-28T15:42:09Z none

Description Kobi Hakimi 2019-10-29 11:27:04 UTC 
Created attachment 1630115 [details]
engine-logs-2019-10-28T15:42:09Z

Description of problem:
failed to deploy the hosted engine 4.4 with connection error from engine VM to the host.

Version-Release number of selected component (if applicable):
ovirt-ansible-hosted-engine-setup-1.0.29-1.el8ev.noarch
ovirt-hosted-engine-setup-2.4.0-0.1.master.20190918120243.gitd016c11.el8ev.noarch
rhel-8.1

How reproducible:
100%

Steps to Reproduce:
1. Run "hosted-engine --deploy"


Actual results:
we failed to deploy the hosted engine in the following task:
TASK [ovirt.hosted_engine_setup : Wait for the host to be up] ******************

we tried to update the virt module with a newer version 8.1 with [1]
and run the hosted-engine deploy but still failed to connect from the local hosted-engine-vm to host with the following error in engine log:
2019-10-28 11:57:42,435-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [6bd8af4e] EVENT_ID: VDS_ANSIBLE_INSTALL_FINISHED(561), Ansible host-deploy playbook execution has successfully finished on host lynx01.lab.eng.tlv2.redhat.com.
2019-10-28 11:57:42,477-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [6f1f8d72] Connecting to lynx01.lab.eng.tlv2.redhat.com/10.46.16.20
2019-10-28 11:57:42,480-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [6bd8af4e] Error: org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection issues during send request

vdsm.log:
2019-10-28 11:55:47,989-0400 ERROR (vm/0248f436) [virt.vm] (vmId='0248f436-53ce-4f7b-b821-f3339c8e9eda') Failed to connect to guest agent channel (vm:2266)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2264, in _vmDependentInit
self.guestAgent.start()
File "/usr/lib/python3.6/site-packages/vdsm/virt/guestagent.py", line 257, in start
self._prepare_socket()
File "/usr/lib/python3.6/site-packages/vdsm/virt/guestagent.py", line 299, in _prepare_socket
supervdsm.getProxy().prepareVmChannel(self._socketName)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in _call_
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>
**kwargs)
File "<string>", line 2, in prepareVmChannel
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
raise convert_to_error(kind, result)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/libvirt/qemu/channels/0248f436-53ce-4f7b-b821-f3339c8e9eda.com.redhat.rhevm.vdsm'

[1] - http://download.eng.tlv.redhat.com/rhel-8/nightly/ADVANCED-VIRT-8/latest-ADVANCED-VIRT-8.1.0-RHEL-8/compose/Advanced-virt/x86_64/os/


Expected results:
Deploy as expected

Additional info:
Comment 1 Simone Tiraboschi 2019-10-29 14:42:25 UTC 
Looking at host-deploy logs for sure the bug started from here:
2019-10-28 11:55:21,364-0400 DEBUG otopi.context context.dumpEnvironment:741 ENV NETWORK/firewalldEnable=bool:'False'
2019-10-28 11:55:21,364-0400 DEBUG otopi.context context.dumpEnvironment:741 ENV NETWORK/iptablesEnable=bool:'False'


host-deploy didn't tried to configure the firewall on the host and so the issue
Comment 2 Simone Tiraboschi 2019-10-29 15:05:44 UTC 
In ovirt-host-deploy-ansible-20191028115548-lynx01.lab.eng.tlv2.redhat.com-6bd8af4e.log we see:
2019-10-28 11:56:24,065 p=32280 u=ovirt |  ok: [lynx01.lab.eng.tlv2.redhat.com] => (item={u'service': u'vdsm'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": {
        "service": "vdsm"
    }
}

MSG:

Permanent and Non-Permanent(immediate) operation
Comment 3 Petr Matyáš 2020-03-10 15:46:13 UTC 
I don't think this is good to verify before we get RHEL 8 appliance.
Comment 4 Sandro Bonazzola 2020-03-11 13:48:50 UTC 
Can be verified with oVirt appliance which is already el8 based.
Comment 5 Nikolai Sednev 2020-03-31 08:58:45 UTC 
Deployment of HE 4.4 on NFS has succeeded.

Tested on host with these components:
rhvm-appliance.x86_64 2:4.4-20200326.0.el8ev
ovirt-hosted-engine-setup-2.4.4-1.el8ev.noarch
ovirt-hosted-engine-ha-2.4.2-1.el8ev.noarch
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
Linux 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Engine:
ovirt-engine-setup-base-4.4.0-0.26.master.el8ev.noarch
ovirt-engine-4.4.0-0.26.master.el8ev.noarch
openvswitch2.11-2.11.0-48.el8fdp.x86_64
Linux 4.18.0-192.el8.x86_64 #1 SMP Tue Mar 24 14:06:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
Comment 6 Sandro Bonazzola 2020-05-20 20:00:46 UTC 
This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020.

Since the problem described in this bug report should be
resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.