Description of problem: Ceilometer cannot connect to Swift object store. Version-Release number of selected component (if applicable): OSP 7.3 beta How reproducible: 100% Steps to Reproduce: 1. Deploy with IPv6 network isolation 2. 3. Actual results: central.log:2016-02-04 22:16:00.873 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden central.log:2016-02-04 22:16:01.486 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.containers.objects: Account GET failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88?format=json 403 Forbidden [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc Expected results: Ceilometer should be able to connect to Swift proxy. Additional info: I suspect that this is actually related to BZ 1305071, where Swift proxy cannot connect to memcached. This would cause token caching to fail, and might prevent services from authenticating to Swift proxy.
After fixing the errors with swift-proxy and memcached, the Ceilometer authentication errors are still happening, every 10 minutes. This looks like a legitimate problem with authentication in Ceilometer. Note that the environment where I have been reproducing this issue is an HA deployment with IPv6 network isolation. 2016-02-06 07:16:02.056 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects.containers: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base Traceback (most recent call last): 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/agent/base.py", line 177, in poll_and_publish 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base resources=polling_resources 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 146, in get_samples 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base cache, tenants): 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 82, in _iter_accounts 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base ksclient, tenants)) 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base ksclient.auth_token)) 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 541, in head_account 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base http_response_content=body) 2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base ClientException: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
(In reply to Dan Sneddon from comment #2) > After fixing the errors with swift-proxy and memcached, the Ceilometer > authentication errors are still happening, every 10 minutes. This looks like > a legitimate problem with authentication in Ceilometer. I take it back. After fixing the memcached.py in swift common, I also had to restart the ceilometer-central service. After doing that, the errors disappeared and Ceilometer appears to be working.