Bug 2126786

Summary: [RHOSP 16.2] swiftclient forbids ceilometer from polling swift objects
Product: Red Hat OpenStack Reporter: Yadnesh Kulkarni <ykulkarn>
Component: openstack-tripleo-heat-templatesAssignee: Yadnesh Kulkarni <ykulkarn>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact: Joanne O'Flynn <joflynn>
Priority: high    
Version: 16.2 (Train)CC: jamsmith, mburns, mmagr
Target Milestone: z4Keywords: Triaged, ZStream
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.6.1-2.20221010235131.e0d438c.el8ost Doc Type: Bug Fix
Doc Text:
This update fixes a bug that prevented the Telemetry service (ceilometer) from polling the Object Storage service (swift) for metrics. The Telemetry service now polls the Object Storage service correctly.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-07 19:24:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2058518, 2162456    
Bug Blocks: 2130849    

Description Yadnesh Kulkarni 2022-09-14 12:08:09 UTC 
This bug was initially created as a copy of Bug #2058518

I am copying this bug because: 



Description of problem:

The following polling.yaml file
---
sources:
    - name: some_pollsters
      interval: 30
      meters:
        - cpu
        - disk.*
        - ip.*
        - image.*
        - memory
        - memory.*
        - network.*
        - perf.*
        - port
        - port.*
        - switch
        - switch.*
        - storage.*
        - volume.*

generated by following the documentation 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/service_telemetry_framework_1.3/assembly-completing-the-stf-configuration_assembly#creating-the-base-configuration-for-stf_assembly-completing-the-stf-configuration



causes the following errors in ceilometer/central.log

2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden:
 swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 113, in get_samples
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:10.929 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.403 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.415 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.451 15 INFO ceilometer.polling.manager [-] Polling pollster image.size in the context of some_pollsters
2022-02-17 17:49:40.092 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.containers in the context of some_pollsters
2022-02-17 17:49:40.416 15 INFO swiftclient [-] REQ: curl -i http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 -I -H "X-Auth-Token: gAAAAABiDoqy4aVP..."
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP STATUS: 403 Forbidden
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP HEADERS: {'Content-Length': '12', 'X-Trans-Id': 'tx0000000000000000018dc-00620e8ab4-806ad4-default', 'X-Openstack-Request-Id': 'tx0000000000000000018dc-00620e8ab
4-806ad4-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Thu, 17 Feb 2022 17:49:40 GMT'}
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects.containers: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403
 Forbidden: swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 149, in get_samples
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:40.442 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.size in the context of some_pollsters
Comment 7 Leonid Natapov 2022-10-31 15:22:42 UTC 
No 403 Forbidden errors in the central.log file
storage pollster enabled in polling.yaml

---
sources:
    - name: some_pollsters
      interval: 30
      meters:
        - cpu
        - disk.*
        - ip.*
        - image.*
        - memory
        - memory.*
        - network.*
        - perf.*
        - port
        - port.*
        - switch
        - switch.*
        - storage.*
        - volume.*
Comment 13 errata-xmlrpc 2022-12-07 19:24:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794