Bug 2067170

Summary:	Compute replacement (Scaledown&up) fails on: Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
Product:	Red Hat OpenStack	Reporter:	Marian Krcmarik <mkrcmari>
Component:	openstack-tripleo	Assignee:	Brendan Shephard <bshephar>
Status:	CLOSED DUPLICATE	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	17.0 (Wallaby)	CC:	bshephar, mburns
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-29 09:52:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marian Krcmarik 2022-03-23 13:09:08 UTC

Description of problem:
The compute node replacement (Scale down and up) fails during the overcloud scale up redeploy on following message:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'

I am testing it on DCN topology which is a multistack deployment but probably It does not have any impact on it.

The following steps are executed:
1. Remove one overcloud compute node:
openstack overcloud node delete --stack dcn2 --baremetal-deployment /home/stack/dcn2/baremetal_deployment_scaledown.yaml --yes
The baremetal_deployment_scaledown.yaml has decremented Count by 1 and Provisioned: false parameter for one of the compute nodes.
2. Delete the node from ironic:
openstack baremetal node delete $NODE
3. Import a new node:
openstack overcloud node import /home/stack/instackenv.json
4. Do some settings on the node(proper profile, provide it)
5. Provision the nodes:
openstack overcloud node provision --network-config --stack dcn2 -o /home/stack/templates/overcloud-baremetal-deployed-dcn2.yaml  /home/stack/dcn2/network/baremetal_deployment.yaml
6. and redeploy the overcloud with original overcloud deploy command line:

The last step fails on:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1365, in take_action
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     working_dir=self.working_dir)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1306, in take_action
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     heat_type=parsed_args.heat_type
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 178, in get_hosts_and_enable_ssh_admin
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     heat_type=heat_type
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 242, in enable_ssh_admin
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     ansible_timeout=timeout
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 640, in run_ansible_playbook
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     'inventory': _inventory(inventory),
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 377, in _inventory
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     constants.ANSIBLE_HOSTS_FILENAME
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/ansible_runner/utils.py", line 123, in dump_artifact
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     with open(fn, 'w') as f:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-03-22 16:00:59.483 494666 ERROR openstack [-] [Errno 13] Permission denied

If I do:
sudo chmod +w /home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml
then the overcloud redeploy is successful.

Version-Release number of selected component (if applicable):
tripleo-ansible-3.3.1-0.20220307002209.130185a.el8ost.noarch
openstack-tripleo-common-15.4.1-0.20220307005139.3db8093.el8ost.noarch
puppet-tripleo-14.2.3-0.20220307022326.30377d8.el8ost.noarch
ansible-role-tripleo-modify-image-1.3.1-0.20220215231337.30d23d5.el8ost.noarch
openstack-tripleo-heat-templates-14.3.1-0.20220307004703.09c1ce2.el8ost.noarch
ansible-tripleo-ipsec-11.0.1-0.20210910002917.b5559c8.el8ost.noarch
openstack-tripleo-common-containers-15.4.1-0.20220307005139.3db8093.el8ost.noarch
python3-tripleo-common-15.4.1-0.20220307005139.3db8093.el8ost.noarch
python3-tripleoclient-16.4.1-0.20220307002209.423daff.el8ost.noarch
openstack-tripleo-validations-14.2.2-0.20220302002301.aa52abe.el8ost.noarch
ansible-tripleo-ipa-0.2.3-0.20220301182436.6b0ed82.el8ost.noarch

How reproducible:
Always

Comment 1 Brendan Shephard 2022-03-27 22:51:18 UTC

Hey Marian,

Did you happen to capture what the permissions were before you did the chmod?

Is this deployment accessible at all?

Comment 2 Brendan Shephard 2022-03-29 09:32:32 UTC

Hey Marian,

I believe this is the problem:
https://review.opendev.org/c/openstack/python-tripleoclient/+/835500

Comment 3 Brendan Shephard 2022-03-29 09:52:45 UTC


*** This bug has been marked as a duplicate of bug 2069313 ***

Comment 4 Marian Krcmarik 2022-03-29 11:57:42 UTC

(In reply to Brendan Shephard from comment #2)
> Hey Marian,
> 
> I believe this is the problem:
> https://review.opendev.org/c/openstack/python-tripleoclient/+/835500

Hi,
Sorry about late reply, Yeah I think that's the problem, I probably try the fix, thanks.