Bug 1815178 - Permission denied upon stack redeployment with same name
Summary: Permission denied upon stack redeployment with same name
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Adriano Petrich
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On:
Blocks: 2144851
TreeView+ depends on / blocked
 
Reported: 2020-03-19 17:41 UTC by Andreas Karis
Modified: 2023-09-07 22:29 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2144851 (view as bug list)
Environment:
Last Closed: 2023-01-19 02:07:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-20358 0 None None None 2022-11-22 15:03:08 UTC
Red Hat Knowledge Base (Solution) 4995001 0 None None None 2020-04-17 09:07:30 UTC

Description Andreas Karis 2020-03-19 17:41:23 UTC
Description of problem:

Creating a bug report on behalf of my customer:

------------------------------------------------------------------------------

What problem/issue/behavior are you having trouble with?  What do you expect to see?
Initially, the overcloud deployment failed due to incorrect parameters in rhsm.yaml file. Later I removed the existing stack, fixed the template and tried to redeploy. But then it failed during the deployment due to permission issue on "/var/lib/mistral/stack1/.git/index"[1]. I tried updating the owner:group to mistral. But, no luck. To workaround the issue, I used different overcloud stack name. 

[1]
(undercloud) [stack@786895-director01 scripts]$ bash deploy.sh
[...]
ssh admin enablement workflow - COMPLETE.
Removing TripleO short term key from 172.23.208.99
Warning: Permanently added '172.23.208.99' (ECDSA) to the list of known hosts.
Removing TripleO short term key from 172.23.208.66
Warning: Permanently added '172.23.208.66' (ECDSA) to the list of known hosts.
Removing TripleO short term key from 172.23.208.65
Warning: Permanently added '172.23.208.65' (ECDSA) to the list of known hosts.
Removing TripleO short term key from 172.23.208.94
Warning: Permanently added '172.23.208.94' (ECDSA) to the list of known hosts.
Removing TripleO short term key from 172.23.208.57
Warning: Permanently added '172.23.208.57' (ECDSA) to the list of known hosts.
Removing short term keys locally
Enabling ssh admin - COMPLETE.
Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception [action_ex_id=92ab67d4-b2c0-41c2-91f7-7e86980f8903, msg='[Errno 13] Permission denied: '/var/lib/mistral/stack1/.git/index'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/stack1', 'container_config': 'stack1-config'}']
Overcloud configuration failed.

----------


2020-03-02 14:01:24.598 7 DEBUG swiftclient [req-d0dc741f-80f2-49a2-a7eb-533b9a17424c 1d1d32783fff4ffc81448d5b83e62b25 08a5df2269874d74959f0c03d44ce0b6 - default default] R
EQ: curl -i https://172.23.208.52:13808/v1/AUTH_08a5df2269874d74959f0c03d44ce0b6/stack1-config/.git/index -X GET -H "X-Auth-Token: gAAAAABeXVE7_5TS..." http_log /usr/lib/pyth
on3.6/site-packages/swiftclient/client.py:182
2020-03-02 14:01:24.599 7 DEBUG swiftclient [req-d0dc741f-80f2-49a2-a7eb-533b9a17424c 1d1d32783fff4ffc81448d5b83e62b25 08a5df2269874d74959f0c03d44ce0b6 - default default] R
ESP STATUS: 200 OK http_log /usr/lib/python3.6/site-packages/swiftclient/client.py:183
2020-03-02 14:01:24.599 7 DEBUG swiftclient [req-d0dc741f-80f2-49a2-a7eb-533b9a17424c 1d1d32783fff4ffc81448d5b83e62b25 08a5df2269874d74959f0c03d44ce0b6 - default default] R
ESP HEADERS: {'Content-Type': 'application/octet-stream', 'Etag': '4511e34a0203130e98c7b1c7329c34c7', 'Last-Modified': 'Mon, 02 Mar 2020 19:01:19 GMT', 'X-Timestamp': '1583
175678.52963', 'Accept-Ranges': 'bytes', 'Content-Length': '24408', 'X-Trans-Id': 'txdf08625dfd0840f895758-005e5d5804', 'X-Openstack-Request-Id': 'txdf08625dfd0840f895758-0
05e5d5804', 'Date': 'Mon, 02 Mar 2020 19:01:24 GMT'} http_log /usr/lib/python3.6/site-packages/swiftclient/client.py:184
2020-03-02 14:01:24.600 7 WARNING mistral.executors.default_executor [req-d0dc741f-80f2-49a2-a7eb-533b9a17424c 1d1d32783fff4ffc81448d5b83e62b25 08a5df2269874d74959f0c03d44c
e0b6 - default default] The action raised an exception [action_ex_id=92ab67d4-b2c0-41c2-91f7-7e86980f8903, msg='[Errno 13] Permission denied: '/var/lib/mistral/stack1/.git/in
dex'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/stack1', 'container_config': 'stack1-
config'}']: PermissionError: [Errno 13] Permission denied: '/var/lib/mistral/stack1/.git/index'
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor Traceback (most recent call last):
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 133, in _do_run_act
ion
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/tripleo_common/actions/config.py", line 124, in run
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor     self.work_dir)
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/tripleo_common/utils/swift.py", line 98, in download_container
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor     with open(path, 'wb') as f:
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor PermissionError: [Errno 13] Permission denied: '/var/lib/mistral/stack1/.git/index'
2020-03-02 14:01:24.600 7 ERROR mistral.executors.default_executor
==

[root@786895-director01 mistral]# ll /var/lib/mistral/stack1/.git
total 40
-rw-r--r--.   1 42430 42430    92 Mar  2 14:01 config
-rw-r--r--.   1 42430 42430    73 Mar  2 14:01 description
-rw-r--r--.   1 42430 42430    23 Mar  2 14:01 HEAD
drwxr-xr-x.   2 42430 42430   210 Mar  2 12:29 hooks
-rw-r--r--.   1 root  root  24408 Mar  2 13:33 index
drwxr-xr-x.   2 42430 42430    21 Mar  2 12:29 info
drwxr-xr-x.   3 42430 42430    30 Mar  2 12:29 logs
drwxr-xr-x. 115 42430 42430  4096 Mar  2 12:30 objects
drwxr-xr-x.   3 42430 42430    19 Mar  2 12:30 refs


Where are you experiencing the behavior? What environment?
osp16

Comment 5 Alex Schultz 2020-04-16 17:06:45 UTC
I just tried to reproduce this and was unable to.  I'm going to look further into the logs provided but I don't know how the permissions changed at the moment. In the future if a user hits this, they can simply remove the /var/lib/mistral/<stackname> file and rerun the deployment.

In my testing I did:

1) openstack overcloud deploy <with args>
2) openstack overcloud delete overcloud
3) openstack overcloud deploy <with args>

The redeploy did not fail.

Comment 9 Alex Schultz 2020-04-21 13:12:11 UTC
We're unable to determine how the permissions changed to hit this.  If someone hits this again, please feel free to reopen with additional details on how to reproduce the issue.


Note You need to log in before you can comment on or make changes to this bug.