Bug 1457421
Summary: | heat_api service is dead on overcloud after deployment | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
Component: | openstack-containers | Assignee: | Martin André <m.andre> |
Status: | CLOSED ERRATA | QA Contact: | Omri Hochman <ohochman> |
Severity: | high | Docs Contact: | Andrew Burden <aburden> |
Priority: | high | ||
Version: | 12.0 (Pike) | CC: | ahrechan, jcoufal, jschluet, lruzicka, m.andre, rhallise, sasha, scorcora |
Target Milestone: | Upstream M2 | Keywords: | Triaged |
Target Release: | 12.0 (Pike) | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | openstack-heat-api-docker-12.0-20170619.1 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-12-13 19:14:33 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Artem Hrechanychenko
2017-05-31 16:44:32 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release. [heat-admin@overcloud-controller-0 ~]$ sudo docker logs heat_api [heat-admin@overcloud-controller-0 ~]$ reproduced. This is likely still an issue. Could I get interactive access to a machine that demonstrates this issue? Please attempt to reproduce with latest and provide environment for Dev to debug on. Had a look at this more specifically this afternoon w/ sasha. It appears that Heat isn't configured correctly within the container. We are missing the /etc/httpd/conf.d/10-heat_api_wsgi.conf file which is what would tell httpd to start Heat API. This could be related to how docker-puppet.py is generating the config files? Or perhaps something else. Looking further... Okay. I found a descrepancy in the /var/lib/docker-puppet/docker-puppet.json. It appears that in this deployment to config generate tool (docker-puppet.py) is configured to generate all of the config files using the 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker:2017-06-15.2 container. For the Heat API and Heat CFN containers we need them to be configured using their respective containers... otherwise puppet fails to run due to missing dependencies in these containers. So... a bit of a guess but can you check and see if we are perhaps setting the following heat parameters incorrectly? DockerHeatConfigImage These should be set to 192.168.24.1:8787/rhosp12/openstack-heat-api-docker and not 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker as they are now. The issue is now resolved - was about having the right entry in the included docker-osp12.yaml: DockerHeatConfigImage: openstack-heat-api-docker:2017-06-19.1 ()[root@controller-0 /]# curl http://172.17.1.16:8004 {"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://172.17.1.16:8004/v1/", "rel": "self"}]}]}curl (http://172.17.1.16:8004/): response: 300, time: 0.002, size: 118 ()[root@controller-0 /]# exit ()[root@controller-0 /]# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 181572 7076 ? Ss 21:47 0:00 /usr/sbin/httpd -DFOREGROUND heat 17 0.6 0.8 577068 70004 ? Sl 21:47 0:01 heat_api_wsgi -DFOREGROUND apache 18 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 19 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 20 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 21 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 22 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 23 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 24 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 25 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND root 38 1.0 0.0 13376 1892 ? Ss 21:50 0:00 bash root 60 0.0 0.0 49052 1808 ? R+ 21:50 0:00 ps aux ()[root@controller-0 /]# (undercloud) [stack@undercloud-0 ~]$ . overcloudrc (overcloud) [stack@undercloud-0 ~]$ heat stack-list WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +----+------------+--------------+---------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +----+------------+--------------+---------------+--------------+ +----+------------+--------------+---------------+--------------+ re(In reply to Omri Hochman from comment #5) > Please attempt to reproduce with latest and provide environment for Dev to > debug on. bug reproduced 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn-docker 2017-07-02.2 (In reply to Artem Hrechanychenko from comment #9) > re(In reply to Omri Hochman from comment #5) > > Please attempt to reproduce with latest and provide environment for Dev to > > debug on. > > bug reproduced > 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn-docker > 2017-07-02.2 I used next config - UC_SSL+OC_SSL(1ctrl(controller.memory=16384, controller.cpu=8+1comp) docker images from 2017-07-02.2 Hi Artem, I believe the issue is caused by a bogus docker-osp12.yaml file you are using for testing. Make sure there is only one entry for DockerHeatConfigImage and that it points to the heat-api image. There were some duplicate heat parameters that were used for different purpose and we fixed that upstream with https://review.openstack.org/#/c/476970/. This commit brings a whole lot of new heat parameters and that needs to be added to the docker-osp12.yaml file you supply when deploying the overcloud. We need to bring the commit downstream and work with jschlueter to update his script that generates docker-osp12.yaml to fix the bug once for all. [stack@undercloud-0 virt]$ cat docker-osp12.yaml |grep Heat DockerHeatApiCfnImage: openstack-heat-api-cfn-docker:2017-07-02.2 DockerHeatApiImage: openstack-heat-api-docker:2017-07-02.2 DockerHeatConfigImage: openstack-heat-api-cfn-docker:2017-07-02.2 DockerHeatConfigImage: openstack-heat-api-docker:2017-07-02.2 DockerHeatEngineImage: openstack-heat-engine-docker:2017-07-02.2 So after looking again at the code there is a bit more involved than just setting the DockerHeatConfigImage to point to heat-api image. Try as a workaround: sed -i 's|DockerHeatConfigImage|DockerHeatApiCfnConfigImage|' docker/services/heat-api-cfn.yaml sed -i 's|DockerHeatConfigImage: openstack-heat-api-cfn-docker|DockerHeatApiCfnConfigImage: openstack-heat-api-cfn-docker|' docker-osp12.yaml w/a is valid Artem, this should now be included in the latest puddle, can you check again? (In reply to Martin André from comment #16) > Artem, this should now be included in the latest puddle, can you check again? Yep, I confirm that in the last puddle I didn't got that issue! (In reply to Artem Hrechanychenko from comment #17) > (In reply to Martin André from comment #16) > > Artem, this should now be included in the latest puddle, can you check again? > > Yep, I confirm that in the last puddle I didn't got that issue! Verified - > Unable to reproduce with: OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457 |