Bug 1305074

Summary:	Ceilometer Cannot Connect to Swift When Using IPv6
Product:	Red Hat OpenStack	Reporter:	Dan Sneddon <dsneddon>
Component:	openstack-ceilometer	Assignee:	Eoghan Glynn <eglynn>
Status:	CLOSED NOTABUG	QA Contact:	Yurii Prokulevych <yprokule>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.0 (Kilo)	CC:	jruzicka, kbasil, skinjo, yeylon
Target Milestone:	y3
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1305129 (view as bug list)		Environment:
Last Closed:	2016-02-06 12:54:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1305129

Description Dan Sneddon 2016-02-05 14:28:05 UTC

Description of problem:
Ceilometer cannot connect to Swift object store.

Version-Release number of selected component (if applicable):
OSP 7.3 beta

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 network isolation
2.
3.

Actual results:
central.log:2016-02-04 22:16:00.873 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
central.log:2016-02-04 22:16:01.486 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.containers.objects: Account GET failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88?format=json 403 Forbidden  [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc


Expected results:
Ceilometer should be able to connect to Swift proxy. 

Additional info:
I suspect that this is actually related to BZ 1305071, where Swift proxy cannot connect to memcached. This would cause token caching to fail, and might prevent services from authenticating to Swift proxy.

Comment 2 Dan Sneddon 2016-02-06 12:25:05 UTC

After fixing the errors with swift-proxy and memcached, the Ceilometer authentication errors are still happening, every 10 minutes. This looks like a legitimate problem with authentication in Ceilometer.

Note that the environment where I have been reproducing this issue is an HA deployment with IPv6 network isolation.



2016-02-06 07:16:02.056 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects.containers: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base Traceback (most recent call last):
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/agent/base.py", line 177, in poll_and_publish
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     resources=polling_resources
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 146, in get_samples
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     cache, tenants):
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 82, in _iter_accounts
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     ksclient, tenants))
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     ksclient.auth_token))
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 541, in head_account
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     http_response_content=body)
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base ClientException: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden

Comment 3 Dan Sneddon 2016-02-06 12:54:39 UTC

(In reply to Dan Sneddon from comment #2)
> After fixing the errors with swift-proxy and memcached, the Ceilometer
> authentication errors are still happening, every 10 minutes. This looks like
> a legitimate problem with authentication in Ceilometer.

I take it back. After fixing the memcached.py in swift common, I also had to restart the ceilometer-central service. After doing that, the errors disappeared and Ceilometer appears to be working.