Bug 2067170 - Compute replacement (Scaledown&up) fails on: Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
Summary: Compute replacement (Scaledown&up) fails on: Permission denied: '/home/stack/...
Keywords:
Status: CLOSED DUPLICATE of bug 2069313
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Brendan Shephard
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-23 13:09 UTC by Marian Krcmarik
Modified: 2022-03-29 11:57 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-29 09:52:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 835500 0 None NEW Make inventory file writable 2022-03-29 09:33:26 UTC
Red Hat Issue Tracker OSP-14191 0 None None None 2022-03-23 13:12:16 UTC

Description Marian Krcmarik 2022-03-23 13:09:08 UTC
Description of problem:
The compute node replacement (Scale down and up) fails during the overcloud scale up redeploy on following message:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'

I am testing it on DCN topology which is a multistack deployment but probably It does not have any impact on it.

The following steps are executed:
1. Remove one overcloud compute node:
openstack overcloud node delete --stack dcn2 --baremetal-deployment /home/stack/dcn2/baremetal_deployment_scaledown.yaml --yes
The baremetal_deployment_scaledown.yaml has decremented Count by 1 and Provisioned: false parameter for one of the compute nodes.
2. Delete the node from ironic:
openstack baremetal node delete $NODE
3. Import a new node:
openstack overcloud node import /home/stack/instackenv.json
4. Do some settings on the node(proper profile, provide it)
5. Provision the nodes:
openstack overcloud node provision --network-config --stack dcn2 -o /home/stack/templates/overcloud-baremetal-deployed-dcn2.yaml  /home/stack/dcn2/network/baremetal_deployment.yaml
6. and redeploy the overcloud with original overcloud deploy command line:

The last step fails on:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1365, in take_action
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     working_dir=self.working_dir)
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1306, in take_action
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     heat_type=parsed_args.heat_type
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 178, in get_hosts_and_enable_ssh_admin
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     heat_type=heat_type
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 242, in enable_ssh_admin
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     ansible_timeout=timeout
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 640, in run_ansible_playbook
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     'inventory': _inventory(inventory),
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 377, in _inventory
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     constants.ANSIBLE_HOSTS_FILENAME
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.6/site-packages/ansible_runner/utils.py", line 123, in dump_artifact
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     with open(fn, 'w') as f:
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud PermissionError: [Errno 13] Permission denied: '/home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml'
2022-03-22 16:00:59.481 494666 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-03-22 16:00:59.483 494666 ERROR openstack [-] [Errno 13] Permission denied

If I do:
sudo chmod +w /home/stack/overcloud-deploy/dcn2/cli-enable-ssh-admin/hosts.yaml
then the overcloud redeploy is successful.

Version-Release number of selected component (if applicable):
tripleo-ansible-3.3.1-0.20220307002209.130185a.el8ost.noarch
openstack-tripleo-common-15.4.1-0.20220307005139.3db8093.el8ost.noarch
puppet-tripleo-14.2.3-0.20220307022326.30377d8.el8ost.noarch
ansible-role-tripleo-modify-image-1.3.1-0.20220215231337.30d23d5.el8ost.noarch
openstack-tripleo-heat-templates-14.3.1-0.20220307004703.09c1ce2.el8ost.noarch
ansible-tripleo-ipsec-11.0.1-0.20210910002917.b5559c8.el8ost.noarch
openstack-tripleo-common-containers-15.4.1-0.20220307005139.3db8093.el8ost.noarch
python3-tripleo-common-15.4.1-0.20220307005139.3db8093.el8ost.noarch
python3-tripleoclient-16.4.1-0.20220307002209.423daff.el8ost.noarch
openstack-tripleo-validations-14.2.2-0.20220302002301.aa52abe.el8ost.noarch
ansible-tripleo-ipa-0.2.3-0.20220301182436.6b0ed82.el8ost.noarch

How reproducible:
Always

Comment 1 Brendan Shephard 2022-03-27 22:51:18 UTC
Hey Marian,

Did you happen to capture what the permissions were before you did the chmod?

Is this deployment accessible at all?

Comment 2 Brendan Shephard 2022-03-29 09:32:32 UTC
Hey Marian,

I believe this is the problem:
https://review.opendev.org/c/openstack/python-tripleoclient/+/835500

Comment 3 Brendan Shephard 2022-03-29 09:52:45 UTC

*** This bug has been marked as a duplicate of bug 2069313 ***

Comment 4 Marian Krcmarik 2022-03-29 11:57:42 UTC
(In reply to Brendan Shephard from comment #2)
> Hey Marian,
> 
> I believe this is the problem:
> https://review.opendev.org/c/openstack/python-tripleoclient/+/835500

Hi,
Sorry about late reply, Yeah I think that's the problem, I probably try the fix, thanks.


Note You need to log in before you can comment on or make changes to this bug.