Bug 2162456

Summary: [RHOSP 17.1] swiftclient forbids ceilometer from polling swift objects
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Yadnesh Kulkarni <ykulkarn>
Component: RGWAssignee: Marcus Watts <mwatts>
Status: CLOSED WONTFIX QA Contact: Madhavi Kasturi <mkasturi>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0CC: apevec, augol, ceph-eng-bugs, cephqe-warriors, cschwede, fpantano, gfidente, lmadsen, lnatapov, mbenjamin, mburns, mgeary, mmagr, mrunge, mwatts, vimartin, ykulkarn
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 7.0Flags: fpantano: needinfo? (mwatts)
mbenjamin: needinfo? (mwatts)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2058518 Environment:
Last Closed: 2023-08-16 14:59:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 205851    
Bug Blocks: 2160010, 2126786    

Description Yadnesh Kulkarni 2023-01-19 16:09:04 UTC 
+++ This bug was initially created as a clone of Bug #2058518 +++

Description of problem:

The following polling.yaml file
---
sources:
    - name: some_pollsters
      interval: 30
      meters:
        - cpu
        - disk.*
        - ip.*
        - image.*
        - memory
        - memory.*
        - network.*
        - perf.*
        - port
        - port.*
        - switch
        - switch.*
        - storage.*
        - volume.*

generated by following the documentation 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/service_telemetry_framework_1.3/assembly-completing-the-stf-configuration_assembly#creating-the-base-configuration-for-stf_assembly-completing-the-stf-configuration



causes the following errors in ceilometer/central.log

2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden:
 swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 113, in get_samples
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:10.917 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:10.929 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.403 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.415 15 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found.
2022-02-17 17:49:38.451 15 INFO ceilometer.polling.manager [-] Polling pollster image.size in the context of some_pollsters
2022-02-17 17:49:40.092 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.containers in the context of some_pollsters
2022-02-17 17:49:40.416 15 INFO swiftclient [-] REQ: curl -i http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 -I -H "X-Auth-Token: gAAAAABiDoqy4aVP..."
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP STATUS: 403 Forbidden
2022-02-17 17:49:40.416 15 INFO swiftclient [-] RESP HEADERS: {'Content-Length': '12', 'X-Trans-Id': 'tx0000000000000000018dc-00620e8ab4-806ad4-default', 'X-Openstack-Request-Id': 'tx0000000000000000018dc-00620e8ab
4-806ad4-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Thu, 17 Feb 2022 17:49:40 GMT'}
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager [-] Continue after error from storage.objects.containers: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403
 Forbidden: swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager Traceback (most recent call last):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/polling/manager.py", line 187, in poll_and_notify
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     for sample in samples:
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 149, in get_samples
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     cache, tenants):
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 80, in _iter_accounts
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     ksclient, tenants))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 99, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise e
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     keystone_client.get_auth_token(ksclient)))
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 871, in head_account
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager     raise ClientException.from_response(resp, 'Account HEAD failed', body)
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager swiftclient.exceptions.ClientException: Account HEAD failed: http://10.239.8.138:8080/swift/v1/AUTH_bbeb4893f55840848ad7a598a0e6dcf8 403 Forbidden
2022-02-17 17:49:40.417 15 ERROR ceilometer.polling.manager 
2022-02-17 17:49:40.442 15 INFO ceilometer.polling.manager [-] Polling pollster storage.objects.size in the context of some_pollsters