Bug 2130849

Summary:	[RHOSP 16.1] swiftclient forbids ceilometer from polling swift objects
Product:	Red Hat OpenStack	Reporter:	Yadnesh Kulkarni <ykulkarn>
Component:	openstack-tripleo-heat-templates	Assignee:	Yadnesh Kulkarni <ykulkarn>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	high	Docs Contact:	Joanne O'Flynn <joflynn>
Priority:	high
Version:	16.1 (Train)	CC:	jelynch, lmadsen, mburns
Target Milestone:	z9	Keywords:	Triaged, ZStream
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.3.2-1.20221004053221.29a02c1.el8ost	Doc Type:	Bug Fix
Doc Text:	Before this update, a Telemetry service (ceilometer) user had insufficient privileges to poll objects from the Object Storage service (swift). The Object Storage service client did not allow the Telemetry service user to fetch object details. With this update, the Telemetry service user is associated with the ResellerAdmin role. + Execute the following command to workaround this issue manually: + ---- $ openstack role add --user ceilometer --project service ResellerAdmin ---- + The associated Telemetry service user can poll Object Storage service object metrics successfully.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-12-07 20:29:53 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2126786
Bug Blocks:

Description Yadnesh Kulkarni 2022-09-29 07:39:18 UTC

This bug was initially created as a copy of Bug #2126786

This bug was initially created as a copy of Bug #2058518


Description of problem:

The following polling.yaml file
---
sources:
    - name: some_pollsters
      interval: 30
      meters:
        - cpu
        - disk.*
        - ip.*
        - image.*
        - memory
        - memory.*
        - network.*
        - perf.*
        - port
        - port.*
        - switch
        - switch.*
        - storage.*
        - volume.*

generated by following the documentation 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/service_telemetry_framework_1.3/assembly-completing-the-stf-configuration_assembly#creating-the-base-configuration-for-stf_assembly-completing-the-stf-configuration



causes the following errors in ceilometer/central.log

2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden:
 swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 113, in get_samples
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:10.929 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.403 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.415 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.451 15 INFO ceilometer.polling.manager [-] Polling pollster image.size in the context of some_pollsters
2022-02-17 17:49:40.092 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.containers in the context of some_pollsters
2022-02-17 17:49:40.416 15 INFO swiftclient [-] REQ: curl -i http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 -I -H "X-Auth-Token: gAAAAABiDoqy4aVP..."
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP STATUS: 403 Forbidden
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP HEADERS: {'Content-Length': '12', 'X-Trans-Id': 'tx0000000000000000018dc-00620e8ab4-806ad4-default', 'X-Openstack-Request-Id': 'tx0000000000000000018dc-00620e8ab
4-806ad4-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Thu, 17 Feb 2022 17:49:40 GMT'}
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects.containers: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403
 Forbidden: swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 149, in get_samples
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:40.442 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.size in the context of some_pollsters

Comment 11 Leonid Natapov 2022-11-06 04:34:28 UTC

OSP16z9 No 403 Forbidden errors in ceilometer logs

---
sources:
    - name: some_pollsters
      interval: 30
      meters:
        - cpu
        - disk.*
        - ip.*
        - image.*
        - memory
        - memory.*
        - network.*
        - perf.*
        - port
        - port.*
        - switch
        - switch.*
        - storage.*
        - volume.*

Comment 15 Jenny-Anne Lynch 2022-11-17 10:22:12 UTC

Hi Yadnesh,

I edited the Doc Text field for 16.1.9 Release Notes. Can you review for technical accuracy and update if needed? 

Thanks,

Jenny-Anne

Comment 20 errata-xmlrpc 2022-12-07 20:29:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack 16.1.9 (openstack-tripleo-heat-templates) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8796