Bug 1566869

Summary: Default Configuration for enabling object storage metrics has a typo which causes the openstack-swift-proxy service fail to start when using swift as a backend for gnocchi
Product: Red Hat OpenStack Reporter: Punit Kundal <pkundal>
Component: puppet-tripleoAssignee: RHOS Maint <rhos-maint>
Status: CLOSED CURRENTRELEASE QA Contact: nlevinki <nlevinki>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: aschultz, jjoyce, jschluet, mabaakou, sacpatil, slinaber, tvignaud
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.2.0-23.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-24 13:51:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Punit Kundal 2018-04-13 06:27:48 UTC
Description of problem:

When using swift as a backend for gnocchi, there is a default configuration which is placed by puppet inside /etc/swift/proxy-server.conf, this configuration has a typo for the project_name parameter.


Below is the snippet of the configuration put by puppet.

+++
[filter:ceilometer]
paste.filter_factory = ceilometermiddleware.swift:filter_factory

url = rabbit://guest:tXxPT6rQMWJdpEhpwQk7B9rsx.125.60:5672,guest:tXxPT6rQMWJdpEhpwQk7B9rsx.125.56:5672,guest:tXxPT6rQMWJdpEhpwQk7B9rsx.125.53:5672//


ignore_projects = ["service"]


auth_uri = http://172.168.125.51:5000/v2.0


auth_url = http://172.168.125.51:5000


auth_type = password


project_name = services << project_name is set to services however the correct name should be 'service'


project_domain_name = Default


user_domain_name = Default


username = swift


password = jf7AF62ZCwau4hMJAX8r4YwQm
+++


Now by default ceilometer is present in the [pipeline:main] section:

+++
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb versioned_writes proxy-logging proxy-server
+++

Once you enable ceilometer in the [pipeline:main] section and then restart the openstack-swift-proxy service it fails with the below error:

+++
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb versioned_writes ceilometer proxy-logging proxy-server


[root@overcloud-controller-0 ~]# systemctl restart openstack-swift-proxy 
[root@overcloud-controller-0 ~]# systemctl status openstack-swift-proxy -l
● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-04-13 11:47:57 IST; 8s ago
  Process: 655493 ExecStart=/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf (code=exited, status=1/FAILURE)
 Main PID: 655493 (code=exited, status=1/FAILURE)

Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 705, in post
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: return self.request(url, 'POST', **kwargs)
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: return wrapped(*args, **kwargs)
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 594, in request
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: raise exceptions.from_response(resp, method, url)
Apr 13 11:47:56 overcloud-controller-0 swift-proxy-server[655493]: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-aeed5fbd-97bd-4d6b-b21c-a728ddbbca92)
Apr 13 11:47:57 overcloud-controller-0 systemd[1]: openstack-swift-proxy.service: main process exited, code=exited, status=1/FAILURE
Apr 13 11:47:57 overcloud-controller-0 systemd[1]: Unit openstack-swift-proxy.service entered failed state.
Apr 13 11:47:57 overcloud-controller-0 systemd[1]: openstack-swift-proxy.service failed.
+++

The simple fix is to correct the project_name parameter in the [filter:ceilometer] section and set it to 'service'

after doing that , restarting the openstack-swift-proxy service fixes the issue:

+++
[root@overcloud-controller-0 ~]# systemctl restart openstack-swift-proxy
[root@overcloud-controller-0 ~]# systemctl status openstack-swift-proxy -l
● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-04-13 11:49:31 IST; 1s ago
 Main PID: 660921 (swift-proxy-ser)
   CGroup: /system.slice/openstack-swift-proxy.service
           └─660921 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb copy dlo versioned_writes ceilometer proxy-logging proxy-server".
Apr 13 11:49:32 overcloud-controller-0 swift-proxy-server[660921]: No handlers could be found for logger "ceilometermiddleware.swift"
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Starting Keystone auth_token middleware
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Using /var/cache/swift as cache directory for signing certificate
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Adding required filter copy to pipeline at position 11
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Adding required filter dlo to pipeline at position 12
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Adding required filter gatekeeper to pipeline at position 1
Apr 13 11:49:32 overcloud-controller-0 proxy-server[660921]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb copy dlo versioned_writes ceilometer proxy-logging proxy-server".
Apr 13 11:49:33 overcloud-controller-0 proxy-server[660921]: Starting Keystone auth_token middleware
Apr 13 11:49:33 overcloud-controller-0 proxy-server[660921]: Using /var/cache/swift as cache directory for signing certificate
+++


The rest of the configuration bits appear to be in place:

+++
[stack@undercloud10 ~]$ openstack role list 
+----------------------------------+-----------------+
| ID                               | Name            |
+----------------------------------+-----------------+
| 3a8122dc41cd4a14af4eda0bbb7c016c | heat_stack_user |
| 534f597bb9ac4e149ead83e2a653b44c | swiftoperator   |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_        |
| a03b5df741ad4afebb14829a9ec5ddc8 | admin           |
| dd10e59988b9460c88c69eec2888dc29 | ResellerAdmin   |  << the ResellerAdmin is created 
+----------------------------------+-----------------+

[stack@undercloud10 ~]$ openstack role assignment list --user ceilometer --project service --names 
+---------------+------------+---------+
| Role          | User       | Project |
+---------------+------------+---------+
| admin         | ceilometer | service |
| ResellerAdmin | ceilometer | service |  << ceilometer user has the ResellerAdmin role in the service project
+---------------+------------+---------+

[filter:keystone]
use = egg:swift#keystoneauth
operator_roles = admin, swiftoperator, ResellerAdmin  << the operator_roles parameter also appears to be set fine.

and [filter:ceilometer] section has the correct configuration for rabbitmq in place

+++


Version-Release number of selected component (if applicable):

+++
[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-tripleo
puppet-tripleo-5.6.4-3.el7ost.noarch

[stack@undercloud10 ~]$ rpm -qa | grep puppet-tripleo
puppet-tripleo-5.6.4-3.el7ost.noarch
+++

How reproducible:
Always with the default configuration in place. Just enable ceilometer in the main pipeline and restart the openstack-swift-proxy service

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Mehdi ABAAKOUK 2018-04-24 13:51:40 UTC
This have been fixed a while ago, according the changelog, it was openstack-tripleo-heat-templates-5.2.0-23.el7ost.

Please update your installation to last zrelease of OSP10. And feel free to reopen if this still occurs with the last release. (openstack-tripleo-heat-templates must be at least: openstack-tripleo-heat-templates-5.3.8-1.el7ost)