Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1458861

Summary: rhosp-director: docker error: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: dbecker, emacchi, jcoufal, m.andre, mburns, morazi, ohochman, rhel-osp-director-maint, sasha, scorcora, tvignaud
Target Milestone: betaKeywords: Reopened, Triaged, ZStream
Target Release: 12.0 (Pike)   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-30 21:24:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2017-06-05 16:46:12 UTC
Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn


Environment:
instack-undercloud-7.0.0-0.20170521005009.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170521033302.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch

Steps to reproduce:
Try to deploy overcloud:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/templates/docker-osp12.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
--log-file overcloud_deployment_0.log


Result:
The deployment fails:

(undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1:
  resource_type: OS::Heat::StructuredDeploymentGroup
  physical_resource_id: 51afdef9-48f1-4c36-b5b5-3b84b084b1f3
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: d375179b-e0d9-4c16-b271-ccae5afb91a9
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    ...
    + '[' -d /root/ ']'
    + cp -a /root/ /var/lib/config-data/neutron/root/
    + '[' -d /var/lib/ironic/tftpboot/ ']'
    + '[' -d /var/lib/ironic/httpboot/ ']'
    + '[' -d /var/www/ ']'
    
    2017-06-05 16:06:24,084 INFO: Removing container: docker-puppet-neutron
    2017-06-05 16:06:24,511 DEBUG: docker-puppet-neutron
    
    2017-06-05 16:06:24,512 ERROR: ERROR configuring ceilometer
    (truncated, view all with --long)
  deploy_stderr: |



Note:
(undercloud) [stack@undercloud-0 ~]$ echo -e `heat deployment-show d375179b-e0d9-4c16-b271-ccae5afb91a9`|grep  Error
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
Error: /Stage[main]/Heat::Wsgi::Apache_api/Heat::Wsgi::Apache[api]/Openstacklib::Wsgi::Apache[heat_api_wsgi]/File[heat_api_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api
2017-06-05 16:05:22,444 DEBUG: Error: image rhosp12/centos-binary-ceilometer-central:latest not found
/usr/bin/docker-current: Error: image rhosp12/centos-binary-ceilometer-central:latest not found.

Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn




Note: This BZ if for the second error:
Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn



Checking the controller:
[root@overcloud-controller-0 ~]# journalctl -u docker -perr|grep Error
Jun 05 16:05:15 overcloud-controller-0.localdomain dockerd-current[19039]: Error: /Stage[main]/Heat::Wsgi::Apache_api/Heat::Wsgi::Apache[api]/Openstacklib::Wsgi::Apache[heat_api_wsgi]/File[heat_api_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api
Jun 05 16:06:06 overcloud-controller-0.localdomain dockerd-current[19039]: Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn

Comment 2 Alexander Chuzhoy 2017-06-19 17:41:51 UTC
Re-opening, reproduced.

Comment 3 Alexander Chuzhoy 2017-06-19 17:43:18 UTC
Environment:
openstack-heat-api-cfn-9.0.0-0.20170612043222.a289f55.el7ost.noarch
puppet-heat-11.2.0-0.20170609105517.8d632e1.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
python-heatclient-1.9.0-0.20170418194456.b80166d.el7ost.noarch
python-heat-agent-1.1.1-0.20170602045137.887f06a.el7ost.noarch
openstack-heat-common-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-heat-api-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.0.1-0.20170609013145.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170611233360.el7ost.noarch

The issue reproduced:
Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn"


[root@overcloud-controller-0 ~]# journalctl -u docker -perr|grep Error
Jun 19 17:26:44 overcloud-controller-0 dockerd-current[18784]: Error: /Stage[main]/Heat::Wsgi::Apache_api/Heat::Wsgi::Apache[api]/Openstacklib::Wsgi::Apache[heat_api_wsgi]/File[heat_api_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api
Jun 19 17:26:56 overcloud-controller-0 dockerd-current[18784]: Error: /Stage[main]/Heat::Wsgi::Apache_api_cfn/Heat::Wsgi::Apache[api_cfn]/Openstacklib::Wsgi::Apache[heat_api_cfn_wsgi]/File[heat_api_cfn_wsgi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/bin/heat-wsgi-api-cfn

Comment 4 Alexander Chuzhoy 2017-06-19 19:20:25 UTC
The issue is intermittent and even on the same setup doesn't always reproduce.

Comment 5 Omri Hochman 2017-07-07 15:02:17 UTC
Sasha - according to Martin the latest changes should solve this issue , can you please retest ?

Comment 6 Martin André 2017-07-07 15:04:47 UTC
I believe this is a duplicate of bug 1457421, and the workaround posted there would work for you.

Comment 7 Alexander Chuzhoy 2017-07-28 16:02:56 UTC
Verified.
Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch


The reported issue doesn't reproduce.
Able to deploy overcloud.

sudo journalctl -u docker -perr|grep Error on controller returns an empty list.

Comment 17 errata-xmlrpc 2018-01-30 21:24:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0253