Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1826981

Summary:	Wrong permissions on /var/lib/neutron/ cause metadata-proxy to fail
Product:	Red Hat OpenStack	Reporter:	ldenny
Component:	openstack-neutron	Assignee:	Dan Radez <dradez>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	amuller, chrisw, dprince, dradez, jschluet, m.andre, scohen
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-09-28 05:00:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description ldenny 2020-04-22 22:43:13 UTC

Description of problem:
Curling the metadata end point fails
~~~
$ curl 169.254.169.254/openstack/latest/meta_data.json
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
~~~

Looking at the original permissions:
~~~
[root@ulposp001 /root]# ls -al /var/lib/neutron
total 64K
drwxrwxr-x+  7 42435 42435 4.0K Apr  1 11:47 ./
drwxr-xr-x. 94 root  root  4.0K Mar 31 15:07 ../
drwxrwxr-x+ 30 42435 42435 4.0K Mar 31 02:44 dhcp/
-rwxrwxr-x+  1 42435 42435 1.1K Mar 28 19:49 dhcp_haproxy_wrapper*
-rwxrwxr-x+  1 42435 42435 1.2K Mar 28 19:48 dibbler_wrapper*
-rwxrwxr-x+  1 42435 42435 1.0K Mar 28 19:49 dnsmasq_wrapper*
drwxrwxr-x+  3 42435 42435   18 Jan 27  2017 external/
drwxrwxr-x+ 14 42435 42435 4.0K Mar 26 07:01 ha_confs/
srwxrwxr-x+  1 42435 42435    0 Mar 31 02:43 keepalived-state-change=
-rwxrwxr-x+  1 42435 42435 1.1K Mar 28 19:48 keepalived_state_change_wrapper*
-rwxrwxr-x+  1 42435 42435 1.1K Mar 28 19:48 keepalived_wrapper*
-rwxrwxr-x+  1 42435 42435 1.1K Mar 28 19:48 l3_haproxy_wrapper*
drwxrwxr-x+  2 42435 42435  16K Dec  9 22:09 lock/
srw-rwxr--+  1 42435 42435    0 Apr  1 11:47 metadata_proxy=
drwxrwxr-x+  2 42435 42435 4.0K Mar 31 02:44 ns-metadata-proxy/
~~~

To fix we need to run the following found here https://review.opendev.org/gitweb?p=openstack%2Ftripleo-heat-templates.git;a=commitdiff;h=818ad752f8b048217a0d5b76ea2c5f86714597f4 from BZ 1563443#c5

~~~
setfacl -d -R -m u:42435:rwx /var/lib/neutron
setfacl -R -m u:42435:rw /var/lib/neutron
find /var/lib/neutron -type d -exec setfacl -m u:42435:rwx '{}' \;
setfacl -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -m u:42435:rwx /var/lib/neutron

setfacl -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -m u:42435:rwx /var/lib/neutron/keepalived-state-change

setfacl -d -R -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -d -R -m u:42435:rwx /var/lib/neutron/keepalived-state-change
setfacl -d -R -m u:42435:rwx /var/lib/neutron
~~~
Which only works until the container is restarted and the permissions revert.


How reproducible:
Every time the container is restarted

Comment 3 Dan Prince 2020-07-06 14:02:42 UTC

I'm setting the component to openstack-neutron as this is a permissions issue related to deployment there.

Comment 4 Dan Radez 2020-07-07 14:57:08 UTC

Has this been resolved?

I'm looking at build openstack-tripleo-heat-templates-8.4.1-42 from February that seems to include the patch that has been linked.
I couldn't find the exact build where this patch was introduced.

If it's not been resolved could you provide the version of openstack-tripleo-heat-templates that you have installed?

Comment 5 ldenny 2020-07-10 01:07:26 UTC

Hi Dan,

Version installed is `openstack-tripleo-heat-templates-8.4.1-16.el7ost.noarch`

We can fix the issue by running the following from https://review.opendev.org/gitweb?p=openstack%2Ftripleo-heat-templates.git;a=commitdiff;h=818ad752f8b048217a0d5b76ea2c5f86714597f4

~~~
setfacl -d -R -m u:42435:rwx /var/lib/neutron
setfacl -R -m u:42435:rw /var/lib/neutron
find /var/lib/neutron -type d -exec setfacl -m u:42435:rwx '{}' \;
setfacl -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -m u:42435:rwx /var/lib/neutron

setfacl -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -m u:42435:rwx /var/lib/neutron/keepalived-state-change

setfacl -d -R -m u:42435:rwx /var/lib/neutron/metadata_proxy
setfacl -d -R -m u:42435:rwx /var/lib/neutron/keepalived-state-change
setfacl -d -R -m u:42435:rwx /var/lib/neutron
~~~

But this only works until the container is restarted and the permissions revert.

Comment 6 Dan Radez 2020-07-10 13:05:13 UTC

Thanks ldenny,
That package was built in Nov 2019.
otoh that patch seems to be included in the package version you have installed.
I'm wondering if this is related to the container build and not THT?

Is there a way you could update to the latest containers in OSP 13?
There have been atleast 2 Z releases GAed I can see since that package was built.

Comment 8 ldenny 2020-07-27 22:36:04 UTC

Hi Dan, 

We can't update the customer to the latest containers sadly.

We are going to try pulling a fresh copy of the current openstack-neutron-metadata-agent:13.0-106 container and launch it with paunch to see if that helps.

Comment 9 Dan Radez 2020-09-24 13:00:11 UTC

Any updates here? What if we close this out and you can reopen it if you need more assistance

Comment 10 ldenny 2020-09-28 05:00:15 UTC

Hi Dan, 

Let's do that, I will close and reopen if needed.

Cheers.