Bug 1596260

Summary: Undercloud is not reachable by mistral: "Authentication or permission failure" [osp13]
Product: Red Hat OpenStack Reporter: Alex Schultz <aschultz>
Component: openstack-tripleo-commonAssignee: Alex Schultz <aschultz>
Status: CLOSED ERRATA QA Contact: Artem Hrechanychenko <ahrechan>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: ahrechan, aschultz, dsavinea, fhubik, jschluet, jslagle, mburns, psedlak, rbrady, sasha, slinaber
Target Milestone: z2Keywords: Automation, AutomationBlocker, Rebase, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-8.6.3-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1593345 Environment:
Last Closed: 2018-08-29 16:37:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1593345    
Bug Blocks:    

Description Alex Schultz 2018-06-28 13:36:52 UTC
+++ This bug was initially created as a clone of Bug #1593345 +++

Description of problem:

018-06-20 14:43:47Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE  state changed
2018-06-20 14:43:47Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-06-20 14:43:48Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2018-06-20 14:43:48Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud/daa56963-8ba4-49b9-8242-d6b5c74f2dc4 CREATE_COMPLETE 

Deploying overcloud configuration
Enabling ssh admin (tripleo-admin) for hosts:
192.168.24.18 192.168.24.11 192.168.24.15
Using ssh user heat-admin for initial connection.
Using ssh key at /home/stack/.ssh/id_rsa for initial connection.
Inserting TripleO short term key for 192.168.24.18
Inserting TripleO short term key for 192.168.24.11
Inserting TripleO short term key for 192.168.24.15
Starting ssh admin enablement workflow
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - COMPLETE.
Removing TripleO short term key from 192.168.24.18
Removing TripleO short term key from 192.168.24.11
Removing TripleO short term key from 192.168.24.15
Removing short term keys locally
Enabling ssh admin - COMPLETE.
Config downloaded at /var/lib/mistral/9c5ad74e-3c88-4367-8502-f9f22fb86a49
Inventory generated at /var/lib/mistral/9c5ad74e-3c88-4367-8502-f9f22fb86a49/tripleo-ansible-inventory.yaml
Running ansible playbook at /var/lib/misOvercloud configuration failed.
tral/9c5ad74e-3c88-4367-8502-f9f22fb86a49/deploy_steps_playbook.yaml. See log file at /var/lib/mistral/9c5ad74e-3c88-4367-8502-f9f22fb86a49/ansible.log for progress. ...

Using /var/lib/mistral/9c5ad74e-3c88-4367-8502-f9f22fb86a49/ansible.cfg as config file

PLAY [Gather facts from undercloud] ********************************************

TASK [Gathering Facts] *********************************************************
fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /home/mistral/.ansible/tmp/ansible-tmp-1529505902.44-148383122247259 `\" && echo ansible-tmp-1529505902.44-148383122247259=\"` echo /home/mistral/.ansible/tmp/ansible-tmp-1529505902.44-148383122247259 `\" ), exited with result 1", "unreachable": true}

PLAY RECAP *********************************************************************
undercloud                 : ok=0    changed=0    unreachable=1    failed=0


Version-Release number of selected component (if applicable):
OSPd14

How reproducible:
always

Steps to Reproduce:
1. Deploy any OSPd14 topology using InfraRed and puddle 2018-06-19.4

Actual results:
Overcloud deploy stage fails with mentioned error

Additional info:
overcloud stack is created successfully, post-deployment mistral step fails

--- Additional comment from Alex Schultz on 2018-06-21 14:27:36 EDT ---

Raised the issue with ansible. Current workaround is to downgrade to ansible 2.5.2. I've confirmed it is an issue with 2.5.4+ but should work in 2.4.2

--- Additional comment from Filip HubĂ­k on 2018-06-22 05:56:42 EDT ---

I can confirm that home for mistral user manually created before overcloud deployment workarounds this specific issue.

--- Additional comment from Alex Schultz on 2018-06-22 17:00:58 EDT ---

Comment 11 Artem Hrechanychenko 2018-08-07 18:24:20 UTC
VERIFIED

[stack@undercloud-0 ~]$ cat core_puddle_version 
2018-08-03.3

openstack-tripleo-common-8.6.3-7.el7ost.noarch


PLAY [Gather facts undercloud] *************************************************

TASK [Gathering Facts] *********************************************************
Tuesday 07 August 2018  14:10:26 -0400 (0:00:00.059)       0:00:00.059 ******** 
ok: [localhost]

Comment 12 Joanne O'Flynn 2018-08-15 07:51:49 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 14 errata-xmlrpc 2018-08-29 16:37:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574