Description of problem: After solving an issue caused by a strange character ( é ) in the project description (BZ#1434413) the environment is throwing now the following errors: 2017-03-22 14:13:57.197 659050 INFO swiftclient [-] REQ: curl -i http://10.0.122.9:8080/swift/v1/AUTH_1203b91b9a144e378ceed2f830846c11?format=json -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: 2b7c43572a2b4224..." 2017-03-22 14:13:57.197 659050 INFO swiftclient [-] RESP STATUS: 404 Not Found 2017-03-22 14:13:57.197 659050 INFO swiftclient [-] RESP HEADERS: {u'Date': u'Wed, 22 Mar 2017 14:13:57 GMT', u'Content-Length': u'172', u'Content-Type': u'application/json; charset=utf-8', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx0000000000000000005f6-0058d286a5-1f5da-default'} 2017-03-22 14:13:57.197 659050 INFO swiftclient [-] RESP BODY: {"Code":"NoSuchBucket","BucketName":"AUTH_1203b91b9a144e378ceed2f830846c11","RequestId":"tx0000000000000000005f6-0058d286a5-1f5da-default","HostId":"1f5da-default-default"} 2017-03-22 14:13:57.197 659050 WARNING ceilometer.agent.manager [-] Continue after error from storage.containers.objects.size: Account GET failed: http://10.0.122.9:8080/swift/v1/AUTH_1203b91b9a144e378ceed2f830846c11?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_1203b91b9a144e378c 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager Traceback (most recent call last): 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 198, in poll_and_notify 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager for sample in samples: 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 190, in get_samples 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager cache, tenants): 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 84, in _iter_accounts 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager ksclient, tenants)) 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 96, in _get_account_info 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager keystone_client.get_auth_token(ksclient))) 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 753, in get_account 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager raise ClientException.from_response(resp, 'Account GET failed', body) 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://10.0.122.9:8080/swift/v1/AUTH_1203b91b9a144e378ceed2f830846c11?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_1203b91b9a144e378c 2017-03-22 14:13:57.197 659050 ERROR ceilometer.agent.manager Swift seems to be working correctly ( create containers, upload and delete objects ) The swift backed is a radowsgw with the following users created: These are the users in radosgw: # radosgw-admin metadata list user [ "4beb6858b167400bbf8f8624a3fb54b0", "ddf6f9ee53644bedb1f44f76a725a168", "c00768fb02824596a629a360a9f40657", "dac15e87a582447e8d6efdb85478ddb5", "6f2bf9866bb24851ad32fd4f44fa0bbe", "4ecaa56b6d9e4b7e8b277ec2b847bdb0" ] And the details: for uid in $(radosgw-admin metadata list user | cut -d '"' -f2); do radosgw-admin user info --uid=$uid; done { "user_id": "4beb6858b167400bbf8f8624a3fb54b0", "display_name": "service", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } { "user_id": "ddf6f9ee53644bedb1f44f76a725a168", "display_name": "Cloud-Supervision", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } { "user_id": "c00768fb02824596a629a360a9f40657", "display_name": "admin", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } { "user_id": "dac15e87a582447e8d6efdb85478ddb5", "display_name": "Projet-cloud-admin", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } { "user_id": "6f2bf9866bb24851ad32fd4f44fa0bbe", "display_name": "DSN-val", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } { "user_id": "4ecaa56b6d9e4b7e8b277ec2b847bdb0", "display_name": "DSN-val", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "temp_url_keys": [] } Endpoints seems to be configured properly: $ cat pcs_status ip-10.0.122.9 (ocf::heartbeat:IPaddr2): Started clcldhr2 listen ceph_rgw bind 10.0.122.9:13808 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.122.9:8080 transparent server clcldhr1.storage.probtp 10.0.125.11:8080 check fall 5 inter 2000 rise 2 server clcldhr2.storage.probtp 10.0.125.12:8080 check fall 5 inter 2000 rise 2 server clcldhr3.storage.probtp 10.0.125.13:8080 check fall 5 inter 2000 rise 2 2017-03-24 15:54:51.310 900359 ERROR ceilometer.agent.manager ClientException: Account GET failed: http://10.0.122.9:8080/swift/v1/AUTH_4041321e93ee48949e1ac536bd722a13?format=json 404 Not Found [first 60 chars of response] {"Code":"NoSuchBucket","BucketName":"AUTH_4041321e93ee48949e 2017-03-24 15:54:51.310 900359 ERROR ceilometer.agent.manager [client.radosgw.gateway] user = apache rgw_frontends = civetweb port=10.0.125.11:8080 log_file = /var/log/ceph/radosgw.log host = clcldhr1 keyring = /etc/ceph/ceph.client.radosgw.gateway.keyring rgw_keystone_token_cache_size = 500 rgw_keystone_url = http://10.0.122.9:35357 rgw_s3_auth_use_keystone = True rgw_keystone_admin_token = RPwTNHvNMABmEZ6hCfcDaD27r rgw_keystone_accepted_roles = admin,_member_,Member openstack endpoint list +----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+ | 1c3aae191ff44124a726c216c353685a | regionOne | ceilometer | metering | True | internal | http://10.0.122.9:8777 | | 2026e3030d5243af8ecf73cee648165c | regionOne | keystone | identity | True | admin | http://10.0.122.9:35357/v2.0 | | 2a299f35e5ef45bd8b3682489ffbcd5b | regionOne | keystonev3 | identityv3 | True | public | https://openstack-ctn.cloud.probtp.com:13000/v3 | | alarming | True | public | https://openstack-ctn.cloud.probtp.com:13042 | | 4d6fc8002dfc457296cafa52a6f4de09 | regionOne | ceilometer | metering | True | admin | http://10.0.122.9:8777 | | 5186761008444a6bbef34dc610df5f48 | regionOne | keystone | identity | True | internal | http://10.0.122.9:5000/v2.0 | | 5b520db9ba0242f3b7673ed9b027b8e7 | regionOne | keystonev3 | identityv3 | True | internal | http://10.0.122.9:5000/v3 | | 6c39ba033bc04339a4be87ab1f0007be | regionOne | cinderv3 | volumev3 | True | admin | http://10.0.122.9:8776/v3/%(tenant_id)s | | 6e1c21a239724b2dbca55cb9b0caac1c | regionOne | keystonev3 | identityv3 | True | admin | http://10.0.122.9:35357/v3 | | | ccd936b879ee436ea5676ce7383bd15d | regionOne | keystone | identity | True | public | https://openstack-ctn.cloud.probtp.com:13000/v2.0 | | ce15edaf720a4166acc7d30e7c8ffffe | regionOne | swift | object-store | True | internal | http://10.0.122.9:8080/swift/v1 | | | ef24740024ee4e43874d95f6c523b76f | regionOne | swift | object-store | True | admin | http://10.0.122.9:8080/swift/v1 | | +----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+ Version-Release number of selected component (if applicable): RHOS 10 with radosgw How reproducible: always Steps to Reproduce: 1. Create an environment where swift uses rados gw 2. Create a project with strange characters on the description, for example: é 3. remove the strange character from the description Actual results: Traces provided in the description Expected results: no traces
When you delete a project, Ceilometer is not cleaned like all other Openstack components. In this particular case, Ceilometer maintains a cache keystone resources and we should ignore this kind of error when the polled application (here radosgw) doesn't have any resource associated to the keystone resource. If that not the case this is a bug. So, for the fix we are still waiting for next upstream release. I just push it here: https://review.openstack.org/463921. I will build a new package as soon as the upstream release is done.
No errors
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1586