Bug 1496485

These are the puppet logs from the moment heat-api-cloudwatch is re-configured as an httpd service during the upgrade:
http://pastebin.test.redhat.com/519887

Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Apache/Apache::Vhost[default]/Concat[15-default.conf]/File[/etc/httpd/conf.d/15-default.conf]/ensure: removed",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[/var/www/cgi-bin/heat]/ensure: created",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[heat_api_cloudwatch_wsgi]/ensure: defined content as '{md5}2eb19266988f424046d53acfbcf01c2c'",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/Apache::Vhost[heat_api_cloudwatch_wsgi]/Concat[10-heat_api_cloudwatch_wsgi.conf]/File[/etc/httpd/conf.d/10-heat_api_cloudwatch_wsgi.conf]/ensure: defined content as '{md5}bd90943ed4380f5332bf94387bd4fe06'",
Sep 26 13:53:23 localhost os-collect-config: "Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running'",

Also, it might be useful noticing that there is not heat template for the heat-api-cloudwatch service under /tripleo-heat-templates/docker/services: 
https://github.com/openstack/tripleo-heat-templates/tree/master/docker/services

Hi Jose, the behaviour you're describing in comment #0 sounds right and it is what is currently specified by the tripleo-heat-templates - that is resource registry pointing to the puppet/services/heat-api-cloudwatch.yaml https://github.com/openstack/tripleo-heat-templates/blob/e1a9638732290c247e5dac10392bc8702b531981/overcloud-resource-registry-puppet.j2.yaml#L136 and HeatApiCloudwatch is included in the default controller role services

As you're pointing out in comment #1 there is no /docker/services/heat-api-cloudwatch.yaml i.e. the heat-api-cloudwatch.yaml service is not containerized.

I see that this service was converted to run under httpd with httpd with https://review.openstack.org/#/c/440977/4/puppet/services/heat-api-cloudwatch.yaml@78 on 10 March. That change is not in Ocata so indeed as per your comment #0 you have systemd openstack-heat-api-cloudwatch.service loaded active running before the upgrade, then on Pike you have that service stopped and disabled here https://github.com/openstack/tripleo-heat-templates/blob/e1a9638732290c247e5dac10392bc8702b531981/puppet/services/heat-api-cloudwatch.yaml#L138-L141. Instead heat-api-cloudwatch is being served by httpd.

So to be clear, this BZ is for "why don't we have containerized heat-api-cloudwatch-api lets do it" right? If so we should reach out to deployment/containers team and see what they think/feasible/how much work etc.

Holding on marking triaged incase we go to secondary not primary in tracking.

we discussed this on upgrades scrum yesterday - Jose is going to find out more about why this service was not containerized as per comment #2. 

If it was 'just forgotten' then we can use this BZ to track the effort. 

If there is some legitimate reason we can either close this bz or track whatever the longer term effort is to overcome the problem, if that is possible. 

We can mark triaged once the get the answer.

As discussed in today's scrum meeting there is not a clear idea on how to proceed with the service upon upgrade. The service seems to be deprecated: "This feature will be deprecated or removed during the Havana cycle as we move to using Ceilometer as a metric/alarm service instead [1]", however it is deployed by default in the roles data. 

Also, if it is deprecated, what should be the right way to proceed with it during upgrade? Should the configuration be migrated to the corresponding service (Ceilometer) and stop heat-api-cloudwatch? Or, should it be stopped? (currently the service goes from running as a systemd service to an apache service when upgrading from Ocata to Pike). Or, is the current behavior the right one?

Can someone from the CloudApps or Telemetry DFG's solve our doubts?


[1] https://wiki.openstack.org/wiki/Heat/Using-CloudWatch

Cleaning up the needinfos a bit.

It looks like aschultz has to remove the services from being deployed,but there should be a corresponding upgrade task to clean up as well.

The patch to remove the services from the roles is https://review.openstack.org/#/c/508964/. This does not cleanup the existing running services which would need to be done via the upgrade processes.  For that I'd have to defer to the Upgrades & CloudApp DFGs on those bits.

 thanks Alex going to post something to remove cloudwatch api by default (but allow operator to keep it if wanted)  we can track both of those things here so added the other launchpad to trackers too

Moving this to POST... there are two reviews in trackers... the one at https://review.openstack.org/#/c/508964/ removes the service from the roles files on master. We didn't backport that one to Pike and I don't think we need to/should - adding needinfo Alex what do you think? If you agree we can remove it from trackers so its clearer for the release delivery folks what need to go into the package build

Yes I think that makes sense. For pike we want to clean it up so we need to leave it in the roles so it gets properly disabled.

After upgrade:

[root@controller-0 heat-admin]# docker ps | grep heat
171049f0028a        rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-heat-api-docker:20171103.1                  "kolla_start"            37 minutes ago      Up 37 minutes (healthy)                       heat_api_cron
54d6481ca142        rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-heat-api-cfn-docker:20171103.1              "kolla_start"            37 minutes ago      Up 37 minutes (healthy)                       heat_api_cfn
529c66916005        rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-heat-engine-docker:20171103.1               "kolla_start"            37 minutes ago      Up 37 minutes (healthy)                       heat_engine
344195defd4f        rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-heat-api-docker:20171103.1                  "kolla_start"            37 minutes ago      Up 37 minutes (healthy)                       heat_api
[root@controller-0 heat-admin]# sudo httpd -t -D DUMP_VHOSTS | grep heat
[root@controller-0 heat-admin]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462