Bug 1305074 - Ceilometer Cannot Connect to Swift When Using IPv6
Ceilometer Cannot Connect to Swift When Using IPv6
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
urgent Severity urgent
: y3
: 7.0 (Kilo)
Assigned To: Eoghan Glynn
Yurii Prokulevych
:
Depends On:
Blocks: 1305129
  Show dependency treegraph
 
Reported: 2016-02-05 09:28 EST by Dan Sneddon
Modified: 2016-02-06 07:54 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1305129 (view as bug list)
Environment:
Last Closed: 2016-02-06 07:54:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1305071 None None None 2016-02-05 09:28 EST

  None (edit)
Description Dan Sneddon 2016-02-05 09:28:05 EST
Description of problem:
Ceilometer cannot connect to Swift object store.

Version-Release number of selected component (if applicable):
OSP 7.3 beta

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 network isolation
2.
3.

Actual results:
central.log:2016-02-04 22:16:00.873 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
central.log:2016-02-04 22:16:01.486 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.containers.objects: Account GET failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88?format=json 403 Forbidden  [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc


Expected results:
Ceilometer should be able to connect to Swift proxy. 

Additional info:
I suspect that this is actually related to BZ 1305071, where Swift proxy cannot connect to memcached. This would cause token caching to fail, and might prevent services from authenticating to Swift proxy.
Comment 2 Dan Sneddon 2016-02-06 07:25:05 EST
After fixing the errors with swift-proxy and memcached, the Ceilometer authentication errors are still happening, every 10 minutes. This looks like a legitimate problem with authentication in Ceilometer.

Note that the environment where I have been reproducing this issue is an HA deployment with IPv6 network isolation.



2016-02-06 07:16:02.056 30347 WARNING ceilometer.agent.base [-] Continue after error from storage.objects.containers: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base Traceback (most recent call last):
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/agent/base.py", line 177, in poll_and_publish
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     resources=polling_resources
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 146, in get_samples
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     cache, tenants):
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 82, in _iter_accounts
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     ksclient, tenants))
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/objectstore/swift.py", line 94, in _get_account_info
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     ksclient.auth_token))
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 541, in head_account
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base     http_response_content=body)
2016-02-06 07:16:02.056 30347 TRACE ceilometer.agent.base ClientException: Account HEAD failed: http://[2001:db8:fd00:1000::10]:8080/v1/AUTH_02f0bf561ef842bebc26f0c12d04cd88 403 Forbidden
Comment 3 Dan Sneddon 2016-02-06 07:54:39 EST
(In reply to Dan Sneddon from comment #2)
> After fixing the errors with swift-proxy and memcached, the Ceilometer
> authentication errors are still happening, every 10 minutes. This looks like
> a legitimate problem with authentication in Ceilometer.

I take it back. After fixing the memcached.py in swift common, I also had to restart the ceilometer-central service. After doing that, the errors disappeared and Ceilometer appears to be working.

Note You need to log in before you can comment on or make changes to this bug.