Description of problem: heat_api service inside heat_api docker container is dead on overcloud after deployment (undercloud) [stack@undercloud-0 ~]$ telnet 192.168.24.6 8004 Trying 192.168.24.6... Connected to 192.168.24.6. Escape character is '^]'. GET / <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL / was not found on this server.</p> </body></html> Connection closed by foreign host. ()[root@overcloud-controller-0 /]# ps -aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 180924 6960 ? Ss 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 17 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 18 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 19 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 20 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 21 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 22 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 23 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND apache 24 0.0 0.0 183152 7196 ? S 14:14 0:00 /usr/sbin/httpd -DFOREGROUND root 55 0.7 0.0 13364 1840 ? Ss 16:29 0:00 /bin/bash root 78 0.0 0.0 49040 1840 ? R+ 16:29 0:00 ps -aux ()[root@overcloud-controller-0 /]# Version-Release number of selected component (if applicable): OSP12 How reproducible: Steps to Reproduce: Steps to Reproduce: 1.http://etherpad.corp.redhat.com/testing-osp12-containers, use rhel7.4 for creating vm infrastructure via infrared - --image-url http://download-node-02.eng.bos.redhat.com/brewroot/packages/rhel-guest-image/7.4/135/images/rhel-guest-image-7.4-135.x86_64.qcow2 2.Before deployment of overcloud Apply workarounds for: 1) https://bugzilla.redhat.com/show_bug.cgi?id=1448482 2) https://bugzilla.redhat.com/show_bug.cgi?id=1450370 3) https://bugzilla.redhat.com/show_bug.cgi?id=1452082 4) https://bugzilla.redhat.com/show_bug.cgi?id=1455348 3.Deploy overcloud source /home/stack/stackrc && openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates --libvirt-type kvm -e /home/stack/nodes_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-osp12.yaml --log-file overcloud_deployment_0.log Actual results: (undercloud) [stack@undercloud-0 ~]$ telnet 192.168.24.6 8004 Trying 192.168.24.6... Connected to 192.168.24.6. Escape character is '^]'. GET / <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL / was not found on this server.</p> </body></html> Connection closed by foreign host. Expected results: Additional info:
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
[heat-admin@overcloud-controller-0 ~]$ sudo docker logs heat_api [heat-admin@overcloud-controller-0 ~]$
reproduced.
This is likely still an issue. Could I get interactive access to a machine that demonstrates this issue?
Please attempt to reproduce with latest and provide environment for Dev to debug on.
Had a look at this more specifically this afternoon w/ sasha. It appears that Heat isn't configured correctly within the container. We are missing the /etc/httpd/conf.d/10-heat_api_wsgi.conf file which is what would tell httpd to start Heat API. This could be related to how docker-puppet.py is generating the config files? Or perhaps something else. Looking further...
Okay. I found a descrepancy in the /var/lib/docker-puppet/docker-puppet.json. It appears that in this deployment to config generate tool (docker-puppet.py) is configured to generate all of the config files using the 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker:2017-06-15.2 container. For the Heat API and Heat CFN containers we need them to be configured using their respective containers... otherwise puppet fails to run due to missing dependencies in these containers. So... a bit of a guess but can you check and see if we are perhaps setting the following heat parameters incorrectly? DockerHeatConfigImage These should be set to 192.168.24.1:8787/rhosp12/openstack-heat-api-docker and not 192.168.24.1:8787/rhosp12/openstack-heat-engine-docker as they are now.
The issue is now resolved - was about having the right entry in the included docker-osp12.yaml: DockerHeatConfigImage: openstack-heat-api-docker:2017-06-19.1 ()[root@controller-0 /]# curl http://172.17.1.16:8004 {"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://172.17.1.16:8004/v1/", "rel": "self"}]}]}curl (http://172.17.1.16:8004/): response: 300, time: 0.002, size: 118 ()[root@controller-0 /]# exit ()[root@controller-0 /]# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 181572 7076 ? Ss 21:47 0:00 /usr/sbin/httpd -DFOREGROUND heat 17 0.6 0.8 577068 70004 ? Sl 21:47 0:01 heat_api_wsgi -DFOREGROUND apache 18 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 19 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 20 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 21 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 22 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 23 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 24 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 25 0.0 0.0 183724 7516 ? S 21:47 0:00 /usr/sbin/httpd -DFOREGROUND root 38 1.0 0.0 13376 1892 ? Ss 21:50 0:00 bash root 60 0.0 0.0 49052 1808 ? R+ 21:50 0:00 ps aux ()[root@controller-0 /]# (undercloud) [stack@undercloud-0 ~]$ . overcloudrc (overcloud) [stack@undercloud-0 ~]$ heat stack-list WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +----+------------+--------------+---------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +----+------------+--------------+---------------+--------------+ +----+------------+--------------+---------------+--------------+
re(In reply to Omri Hochman from comment #5) > Please attempt to reproduce with latest and provide environment for Dev to > debug on. bug reproduced 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn-docker 2017-07-02.2
(In reply to Artem Hrechanychenko from comment #9) > re(In reply to Omri Hochman from comment #5) > > Please attempt to reproduce with latest and provide environment for Dev to > > debug on. > > bug reproduced > 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn-docker > 2017-07-02.2
I used next config - UC_SSL+OC_SSL(1ctrl(controller.memory=16384, controller.cpu=8+1comp) docker images from 2017-07-02.2
Hi Artem, I believe the issue is caused by a bogus docker-osp12.yaml file you are using for testing. Make sure there is only one entry for DockerHeatConfigImage and that it points to the heat-api image. There were some duplicate heat parameters that were used for different purpose and we fixed that upstream with https://review.openstack.org/#/c/476970/. This commit brings a whole lot of new heat parameters and that needs to be added to the docker-osp12.yaml file you supply when deploying the overcloud. We need to bring the commit downstream and work with jschlueter to update his script that generates docker-osp12.yaml to fix the bug once for all.
[stack@undercloud-0 virt]$ cat docker-osp12.yaml |grep Heat DockerHeatApiCfnImage: openstack-heat-api-cfn-docker:2017-07-02.2 DockerHeatApiImage: openstack-heat-api-docker:2017-07-02.2 DockerHeatConfigImage: openstack-heat-api-cfn-docker:2017-07-02.2 DockerHeatConfigImage: openstack-heat-api-docker:2017-07-02.2 DockerHeatEngineImage: openstack-heat-engine-docker:2017-07-02.2
So after looking again at the code there is a bit more involved than just setting the DockerHeatConfigImage to point to heat-api image. Try as a workaround: sed -i 's|DockerHeatConfigImage|DockerHeatApiCfnConfigImage|' docker/services/heat-api-cfn.yaml sed -i 's|DockerHeatConfigImage: openstack-heat-api-cfn-docker|DockerHeatApiCfnConfigImage: openstack-heat-api-cfn-docker|' docker-osp12.yaml
w/a is valid
Artem, this should now be included in the latest puddle, can you check again?
(In reply to Martin André from comment #16) > Artem, this should now be included in the latest puddle, can you check again? Yep, I confirm that in the last puddle I didn't got that issue!
(In reply to Artem Hrechanychenko from comment #17) > (In reply to Martin André from comment #16) > > Artem, this should now be included in the latest puddle, can you check again? > > Yep, I confirm that in the last puddle I didn't got that issue! Verified - > Unable to reproduce with: OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457