Bug 1062248

Created attachment 860184 [details]
ceilometer_service.patch

Created attachment 860185 [details]
alarm_service.patch

Created attachment 860187 [details]
glance.patch

Backported two of the required fixes from master to stable/havana upstream:

  https://review.openstack.org/72332
  https://review.openstack.org/72333

Hi Alfredo,

Just wondering whether you also tested with the nova interaction over https?

I ask because the insecure flags is not propogated to the novaclient either:

  https://github.com/openstack/ceilometer/blob/master/ceilometer/nova_client.py#L57

Specifically, is the insecure option only required when the cacert is flawed in some way, e.

(In reply to Eoghan Glynn from comment #6)
> Hi Alfredo,
> 
> Just wondering whether you also tested with the nova interaction over https?
> 

Yes, I'm using https in nova as well. Which ceilometer service connects directly to nova API?. I can set debug level in logs and take a look into it.

> I ask because the insecure flags is not propogated to the novaclient either:
> 
>  
> https://github.com/openstack/ceilometer/blob/master/ceilometer/nova_client.
> py#L57
> 
> Specifically, is the insecure option only required when the cacert is flawed
> in some way, e.

Anyway, note that I'm passing os_cacert option so it should work fine even if insecure option is not passed.

(In reply to Alfredo Moralejo from comment #7)
> Yes, I'm using https in nova as well. Which ceilometer service connects
> directly to nova API?. I can set debug level in logs and take a look into it.

The ceilometer-compute-agent interacts with the nova-api service to discover the instances running on the local host.
 
> Anyway, note that I'm passing os_cacert option so it should work fine even
> if insecure option is not passed.

So in that case, I'm confused why propagating the insecure flag was required in the glance case?

Further patch for glance interaction proposed to master upstream:

  https://review.openstack.org/72703

(In reply to Eoghan Glynn from comment #8)
> (In reply to Alfredo Moralejo from comment #7)
> > Yes, I'm using https in nova as well. Which ceilometer service connects
> > directly to nova API?. I can set debug level in logs and take a look into it.
> 
> The ceilometer-compute-agent interacts with the nova-api service to discover
> the instances running on the local host.
>  
> > Anyway, note that I'm passing os_cacert option so it should work fine even
> > if insecure option is not passed.
> 
> So in that case, I'm confused why propagating the insecure flag was required
> in the glance case?

The original code doesn't propagate neither os_cacert or insecure so it didn't work at all. In my case, propagating only os_cacert would be ok. However, the insecure option is documented in config file and exist in other openstack services so I think it must be propataged to glance client as well. Additionally, we've seen issues validating certificates in the past and allowing the insecure connections can be a last resort option in some cases. Anyway, if you think we shouln't propatage it, it's ok for me but then it should be removed from the commented config file as well.

glanceclient-related fix landed upstream:

  https://github.com/openstack/ceilometer/commit/b000317f


and backported to stable/havana upstream:

   https://review.openstack.org/73351

Fixes all now landed on stable/havana upstream and proposed as internal backports:

  https://code.engineering.redhat.com/gerrit/20852
  https://code.engineering.redhat.com/gerrit/20853
  https://code.engineering.redhat.com/gerrit/20854

Backports all landed internally on rhos/rhos-4.0-rhel-6-patches, commits:


  44411da526e65449f5bd77b571b9085b5c36610a
  f4193353a77da018ef6c6f1481a94499f06213cd
  ef625fb1ab6829eef8e35a3b366a01025ba57293

Patch update pushed:

  http://pkgs.devel.redhat.com/cgit/rpms/openstack-ceilometer/commit/?h=rhos-4.0-rhel-6&id=3f02295e

Verified for insecure (i.e. non-cert-verifying) access over HTTPS to the glance API service from the ceilometer central agent, as follows:

# create new self-signed cert and key file
$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365

# remove passphrase from key
$ openssl rsa -in key.pem -out new.pem

# in /etc/glance/glance-api.conf set the following:
cert_file = /etc/openstack/certs/cert.pem
key_file = /etc/openstack/certs/new.pem
registry_client_protocol = https
registry_client_key_file = /etc/openstack/certs/new.pem
registry_client_cert_file = /etc/openstack/certs/cert.pem
registry_client_insecure = True

# in /etc/glance/glance-registry.conf set the following:
cert_file = /etc/openstack/certs/cert.pem
key_file = /etc/openstack/certs/new.pem

# create image endpoint in service catalog with https endpoint

IMAGE_SERVICE=$(keystone service-list | awk '/glance/ {print $2}')
IMAGE_ENDPOINT=$(keystone endpoint-list | awk '/9292/ {print $2}')
IMAGE_IP=<glance host ip>
keystone endpoint-delete $IMAGE_ENDPOINT
keystone endpoint-create  --region RegionOne  --service-id=$IMAGE_SERVICE --publicurl=https://$IMAGE_IP:9292 --internalurl=https://$IMAGE_IP:9292 --adminurl=https://$IMAGE_IP:9292

# restart central agent and glance services

$ for svc in glance-api glance-registry ceilometer-central ; do sudo service openstack-$svc restart ; done

# create a new image

$ glance --insecure image-create --disk-format ami --container-format ami --name test --file /etc/hosts
$ IMAGE_ID=$(glance --insecure image-show test2 | awk '/ id / {print $4}')

# ensure samples for image appear with 600s (i.e. the polling interval for the central agent)

$ sleep 600 ; ceilometer sample-list -m image -q resource=$IMAGE_ID

Failed to verify for insecure (i.e. non-cert-verifying) access over SSL to keystone from the ceilometer api and alarm evaluator, as follows:

# create keystone certs

$ keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone
$ chown keystone:keystone /etc/keystone/ssl/certs/*

# in /etc/keystone/keystone.conf, set the following:

[ssl]
enable = True
certfile = /etc/keystone/ssl/certs/keystone.pem
keyfile = /etc/keystone/ssl/private/keystonekey.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
ca_key = /etc/keystone/ssl/certs/cakey.pem

# re-create identity service endpoint with https URLs:

$ KEYSTONE_SERVICE=$(keystone service-list | awk '/keystone/ {print $2}')
$ KS_IP=keystone_host_ip
$ keystone endpoint-create --region RegionOne --service-id=$KEYSTONE_SERVIVE --publicurl=https://$KS_IP:5000/v2.0 --internalurl=https://$KS_IP:5000/v2.0 --adminurl=https://$KS_IP:35357/v2.0

# delete old keystone endpoint

$ keystone --insecure endpoint-delete OLD_ENDPOINT_UUID

# restart keystone

$ sudo service openstack-keystone restart

# modify ceilometer.conf config as follows:

[DEFAULT]
os_auth_url=http://$KS_IP:35357/v2.0

[service_credentials]
insecure=True

[keystone_authtoken]
auth_protocol=https
auth_uri=https://$KS_IP:5000/
insecure=True

# restart ceilometer alarm-evaluator and api services
$ for svc in api alarm-evaluator ; do sudo service openstack-ceilometer-$svc restart ; done

The alarm evaluator fails to retrieve a token from keystone:

2014-03-13 20:51:34.333 32509 TRACE ceilometer.alarm.service AuthorizationFailure: Authorization Failed: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP Unable to establish connection to https://172.16.12.49:35357/v2.0/tokens)

This is because the insecure option is not being propagated due to:

  https://launchpad.net/bugs/1292130

fix proposed upstream here:

  https://review.openstack.org/80353

which once landed and backported would allow this BZ to be verified.

The alternate approach is to run in more secure mode, by specifying the cacert to keystoneclient, but unfortunately that does not work as expected due to an apparent keystoneclient bug:

  https://bugs.launchpad.net/python-keystoneclient/+bug/1292202

Fix landed on master upstream:

  https://github.com/openstack/ceilometer/commit/eb14d7e8

backported to stable/havana upstream:

  https://review.openstack.org/80613

and backported internally:

  https://code.engineering.redhat.com/gerrit/#/c/21275

Backport landed internally on rhos/rhos-4.0-rhel-6-patches, commit:

  f0db43ea78ea5aadc7ac8599afd0cfa38fe2ff52

One last fix upstream:

  https://review.openstack.org/80642

backported internally:

  https://code.engineering.redhat.com/gerrit/#/c/21281

and committed to rhos-4.0-rhel-6-patches as:

  012b0ac7b6dab28b940b86d1fd63c14b0dbb1244

and pushed to rhos-4.0-rhel-6:

  http://pkgs.devel.redhat.com/cgit/rpms/openstack-ceilometer/commit/?h=rhos-4.0-rhel-6&id=b2d7f40d

Repeated OtherQA process on openstack-ceilometer-2013.2.2-4.el6ost:

# Download rebuilt RPMs from:

https://brewweb.devel.redhat.com/buildinfo?buildID=344014

# Install RPMs:

$ sudo yum install -y openstack-ceilometer-*-2013.2.2-4.el6ost.noarch.rpm python-ceilometer-2013.2.2-4.el6ost.noarch.rpm

# Verify only modified source file were the fixes for the initial OtherQA failure:

$ cd /usr/lib/python2.6/site-packages/ceilometer
$ for f in $(find . -name "*.py"); do diff $f /usr/lib/python2.6/site-packages/ceilometer.cp/$f; if test $? -eq 1 ; then echo $f ; fi ; done

58d57
<                 insecure=auth_config.insecure,
./alarm/evaluator/__init__.py
89d89
<                 insecure=auth_config.insecure,
./alarm/service.py

# Return to the keystone setup described in comment19 above

# Restart ceilometer alarm-evaluator and api services

for svc in api alarm-evaluator ; do sudo service openstack-ceilometer-$svc restart ; done

# Observe in /var/log/ceilometer/alarm-evaluator.log a token being acquired
# sucessfully from keystone and the evaluator determining it has no alarms
# yet to evaluate:

2014-03-14 17:17:29.145 28078 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 0 alarms

# Create an alarm on a non-existent reource

$ OS_AUTH_URL=$(echo $OS_AUTH_URL | sed 's/http:/https:/')
$ ceilometer --insecure alarm-threshold-create --name cpu_high \
     --description 'instance running hot' \
     --meter-name cpu_util  --threshold 70.0 \
     --comparison-operator gt  --statistic avg \
     --period 600 --evaluation-periods 3 --alarm-action 'log://' \
     --query resource_id=NONEXISTENT_INSTANCE --state ok

# Ensure alarm transitions to insufficient_data within the evaluation interval (60s by default)

$ sleep 60 ; ceilometer --insecure alarm-list 
+--------------------------------------+----------+-------------------+---------+------------+---------------------------------+
| Alarm ID                             | Name     | State             | Enabled | Continuous | Alarm condition                 |
+--------------------------------------+----------+-------------------+---------+------------+---------------------------------+
| 440e6358-c41a-4a16-8f59-3c79c9938cc1 | cpu_high | insufficient data | True    | False      | cpu_util > 70.0 during 3 x 600s |
+--------------------------------------+----------+-------------------+---------+------------+---------------------------------+

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0334.html