Bug 1451491

Summary: rhosp-director: OC deployment fails: ERROR configuring redis
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-containersAssignee: Jon Schlueter <jschluet>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: urgent Docs Contact: Andrew Burden <aburden>
Priority: urgent    
Version: 12.0 (Pike)CC: ahrechan, dbecker, jcoufal, jschluet, lruzicka, m.andre, mburns, morazi, ohochman, rhel-osp-director-maint
Target Milestone: Upstream M2Keywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-redis-docker-12.0-20170518.1 Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 19:14:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport from the undercloud node none

Description Alexander Chuzhoy 2017-05-16 19:10:04 UTC 
rhosp-director: OC deployment fals: ERROR configuring redis

Environment:
instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170512193554.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch

Steps to reproduce:
1. Deploy overcloud
2. Apply workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1451063
3. Try to deploy overcloud with:
openstack overcloud deploy --templates --libvirt-type kvm -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /home/stack/templates/docker-osp12.yaml -e /home/stack/templates/nodes_data.yaml 

Result:
(undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 7891fe45-3a93-491f-ba60-037594a2154c
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    ...
    + '[' -d /var/lib/ironic/tftpboot/ ']'
    + '[' -d /var/lib/ironic/httpboot/ ']'
    + '[' -d /var/www/ ']'
    + mkdir -p /var/lib/config-data/gnocchi/var/www
    + cp -a /var/www/cgi-bin /var/www/html /var/lib/config-data/gnocchi/var/www/
    
    2017-05-16 18:26:40,924 INFO: Removing container: docker-puppet-gnocchi
    2017-05-16 18:26:41,338 DEBUG: docker-puppet-gnocchi
    
    2017-05-16 18:26:41,340 ERROR: ERROR configuring redis
    (truncated, view all with --long)
  deploy_stderr: |

(undercloud) [stack@undercloud-0 ~]$



(undercloud) [stack@undercloud-0 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                      | physical_resource_id                                                                                                                                                                 | resource_type                                                                                                          | resource_status | updated_time         | stack_name                                                                                                                                               |
+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| AllNodesDeploySteps                                | 9adb29cb-42e8-4de8-83be-7a9c9e58c9b9                                                                                                                                                 | OS::TripleO::PostDeploySteps                                                                                           | CREATE_FAILED   | 2017-05-16T18:10:33Z | overcloud                                                                                                                                                |
| ControllerGenerateConfigDeployment                 | 7a6abaf3-26a1-4a19-837c-20d43d6bb7fe                                                                                                                                                 | OS::Heat::SoftwareDeploymentGroup                                                                                      | CREATE_FAILED   | 2017-05-16T18:21:41Z | overcloud-AllNodesDeploySteps-xeu6lt7bw5qj                                                                                                               |
| 0                                                  | 7891fe45-3a93-491f-ba60-037594a2154c                                                                                                                                                 | OS::Heat::SoftwareDeployment                                                                                           | CREATE_FAILED   | 2017-05-16T18:24:55Z | overcloud-AllNodesDeploySteps-xeu6lt7bw5qj-ControllerGenerateConfigDeployment-t4tq23rd7jmf                                                               |
+----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+




(undercloud) [stack@undercloud-0 ~]$ echo -e `heat deployment-show 7891fe45-3a93-491f-ba60-037594a2154c`|tail
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
+ '[' -d /var/lib/ironic/httpboot/ ']'
+ '[' -d /var/www/ ']'
+ mkdir -p /var/lib/config-data/gnocchi/var/www
+ cp -a /var/www/cgi-bin /var/www/html /var/lib/config-data/gnocchi/var/www/

2017-05-16 18:26:40,924 INFO: Removing container: docker-puppet-gnocchi
2017-05-16 18:26:41,338 DEBUG: docker-puppet-gnocchi

2017-05-16 18:26:41,340 ERROR: ERROR configuring redis
", "deploy_stderr": "", "deploy_status_code": 1 }, "creation_time": "2017-05-16T18:24:56Z", "updated_time": "2017-05-16T18:26:43Z", "input_values": { "NET_HOST": "true" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "7891fe45-3a93-491f-ba60-037594a2154c" }
(undercloud) [stack@undercloud-0 ~]$
Comment 1 Alexander Chuzhoy 2017-05-16 19:11:35 UTC 
Created attachment 1279408 [details]
sosreport from the undercloud node
Comment 2 Martin André 2017-05-17 07:31:41 UTC 
I had to use 'openstack stack failures list --long overcloud' to see the cause of the failure.
The logs show:

2017-05-16 18:25:31,700 DEBUG: Error: image rhosp12/centos-binary-redis:latest not found

Which is not expected as a downstream image name, the image should be openstack-redis-docker:2017-05-02.2:

(undercloud) [stack@undercloud-0 ~]$ docker images | grep redis
(undercloud) [stack@undercloud-0 ~]$ docker pull docker-registry.engineering.redhat.com/rhosp12/centos-binary-redis:latest
Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/centos-binary-redis ... 
Pulling repository docker-registry.engineering.redhat.com/rhosp12/centos-binary-redis
Error: image rhosp12/centos-binary-redis:latest not found
(undercloud) [stack@undercloud-0 ~]$ docker pull docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker:2017-05-02.2
Trying to pull repository docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker ... 
2017-05-02.2: Pulling from docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker
5a865e48f2fd: Already exists 
a778b52f148e: Already exists 
0329156b2484: Already exists 
bfcd1d242b9b: Pull complete 
Digest: sha256:53bb4b15e9f092b4d7517d997934762ea3ed85f89589415eead2475e79c68bf6
(undercloud) [stack@undercloud-0 ~]$ docker images | grep redis
docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker                       2017-05-02.2        840800221d42        2 weeks ago         299.3 MB
Comment 3 Martin André 2017-05-17 07:38:21 UTC 
You need to add redis image to your overcloud_containers.yaml file used to populate the local docker registry on the undercloud, and also to docker-osp12.yaml to point containerized redis service to this image.
Comment 4 Artem Hrechanychenko 2017-05-17 08:53:58 UTC 
Reproduced:
http://pastebin.test.redhat.com/484943
Now I'm trying to apply workaround with openstack-redis-docker image
Comment 5 Artem Hrechanychenko 2017-05-17 11:26:45 UTC 
Yeah, after adding redis image - deployment failed on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1450370
Comment 6 Jon Schlueter 2017-05-19 16:17:21 UTC 
look like issue might be resolved with current build as well as adding
 DockerRedisImage: openstack-redis-docker:<tag>

to the docker-osp12.yaml
Comment 9 Alexander Chuzhoy 2017-06-19 19:44:14 UTC 
Verified:
openstack-redis-docker                       2017-06-15.2 
The reported issue doesn't reproduce.
Comment 14 errata-xmlrpc 2017-12-13 19:14:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3457