Bug 1786928 - Failed to add host to the environment using ansible runner
Summary: Failed to add host to the environment using ansible runner
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: ovirt-host-deploy-ansible
Version: 4.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Dana
QA Contact: Lucie Leistnerova
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-29 12:23 UTC by Eyal Shenitzky
Modified: 2020-01-16 14:26 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-16 14:26:25 UTC
oVirt Team: Infra
Embargoed:


Attachments (Terms of Use)

Description Eyal Shenitzky 2019-12-29 12:23:57 UTC
Description of problem:

Failed to add host using ansible runner.
An error message appears in the engine logs that refer to the host-deploy log file but the file doesn't exist.

Engine log:
2019-12-26 16:57:39,504+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [cc646d23-54fc-4e24-939d-27009c5
52272] Host installation failed for host '7cfcac59-88be-4473-8b77-089377aa59dd', 'test': Failed to execute Ansible host-deploy role: null. Please check logs for more detail
s: /home/engine/ovirt-engine/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20191226165736-10.35.0.150-cc646d23-54fc-4e24-939d-27009c552272.log


No other log is available only a message in the journalctl:

Dec 25 15:14:53 dhcp-1-80.tlv.redhat.com gunicorn[27902]: cb_event_handler event_data={'uuid': '46ace4a9-b93c-4488-90ed-895c2b077a79', 'stdout': u"ERROR! Unexpected Except>
Dec 25 15:14:53 dhcp-1-80.tlv.redhat.com gunicorn[27902]: cb_event_handler event_data={'uuid': '752d0e43-62e5-4378-b779-bc86c3fafae4', 'stdout': u'to see the full tracebac>
but the playbooks itself runs on another UUID (which I assume is the host's UUID0

/etc/ansible-runner-service/config.yaml:
---
version: 1

target_user: root

playbooks_root_dir: '/home/engine/ovirt-engine/share/ovirt-engine/ansible-runner-service-project'
ssh_private_key: '/home/engine/ovirt-engine/etc/pki/ovirt-engine/keys/engine_id_rsa'
port: 50001
target_user: root


Version-Release number of selected component (if applicable):
engine OS - fedora 30
host OS - fedora 30
engine - 4.4.0_master
python2-ansible-runner - 1.3.4-2.el7.noarch
ansible-runner - 1.3.4
ansible - 2.9.2-1.fc30.noarch

How reproducible:
100% 

Steps to Reproduce:
1. Create new host 

Actual results:
Ansible runner starts and failed after a few seconds with the above errors

Expected results:
the host should ve added to the environment

Additional info:

Comment 1 Martin Perina 2019-12-31 10:57:41 UTC
(In reply to Eyal Shenitzky from comment #0)
> Description of problem:
> 
> Failed to add host using ansible runner.
> An error message appears in the engine logs that refer to the host-deploy
> log file but the file doesn't exist.
> 
> Engine log:
> 2019-12-26 16:57:39,504+02 ERROR
> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
> (EE-ManagedThreadFactory-engine-Thread-3) [cc646d23-54fc-4e24-939d-27009c5
> 52272] Host installation failed for host
> '7cfcac59-88be-4473-8b77-089377aa59dd', 'test': Failed to execute Ansible
> host-deploy role: null. Please check logs for more detail
> s:
> /home/engine/ovirt-engine/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-
> ansible-20191226165736-10.35.0.150-cc646d23-54fc-4e24-939d-27009c552272.log
> 
> 
> No other log is available only a message in the journalctl:
> 
> Dec 25 15:14:53 dhcp-1-80.tlv.redhat.com gunicorn[27902]: cb_event_handler
> event_data={'uuid': '46ace4a9-b93c-4488-90ed-895c2b077a79', 'stdout':
> u"ERROR! Unexpected Except>
> Dec 25 15:14:53 dhcp-1-80.tlv.redhat.com gunicorn[27902]: cb_event_handler
> event_data={'uuid': '752d0e43-62e5-4378-b779-bc86c3fafae4', 'stdout': u'to
> see the full tracebac>
> but the playbooks itself runs on another UUID (which I assume is the host's
> UUID0
> 
> /etc/ansible-runner-service/config.yaml:
> ---
> version: 1
> 
> target_user: root
> 
> playbooks_root_dir:
> '/home/engine/ovirt-engine/share/ovirt-engine/ansible-runner-service-project'
> ssh_private_key:
> '/home/engine/ovirt-engine/etc/pki/ovirt-engine/keys/engine_id_rsa'
> port: 50001
> target_user: root
> 
> 
> Version-Release number of selected component (if applicable):
> engine OS - fedora 30
> host OS - fedora 30
> engine - 4.4.0_master
> python2-ansible-runner - 1.3.4-2.el7.noarch
> ansible-runner - 1.3.4
> ansible - 2.9.2-1.fc30.noarch

Eyal, most probably you have a mess in your installed packages, you are using Fedora 30, but you have installed python2-ansible-runner - 1.3.4-2.el7.noarch. On FC30 runs everything on Python3, so you should have installed:

python3-ansible-runner-1.3.4-2.fc30.noarch
ansible-runner-service-dev-1.0.1-1.fc30.noarch

Please check your installed packages, remove all packages which are not directly for FC30 and then install correct packages

Comment 2 Eyal Shenitzky 2020-01-01 12:13:05 UTC
Right, I will fix this and check again.
I all works well I will close the bug.

Comment 3 Martin Perina 2020-01-07 13:26:29 UTC
Any updates?

Comment 4 Eyal Shenitzky 2020-01-08 05:47:14 UTC
(In reply to Martin Perina from comment #3)
> Any updates?

Yes,
I fixed all the needed packages and now ansible runner started to work, 
but host installation still fails and no logs were created.

I am currently focusing on a different issue, I hope that I will be able to supply 
new logs/information about the problem during the week.

Comment 5 Martin Perina 2020-01-16 09:37:20 UTC
Eyal, any news?

Comment 6 Eyal Shenitzky 2020-01-16 11:19:40 UTC
Yes, ansible-runner managed to run but I am still failing to add host due to - 
failed to install yum-utils.

Probably local issue.
We cab close this bug for now, I will re-open if a real issue found.

Comment 7 Martin Perina 2020-01-16 14:26:25 UTC
Feel free to reopen when reproduced


Note You need to log in before you can comment on or make changes to this bug.