Bug 1567966 - Redis container restarts unable to sudo and hangs deployment
Summary: Redis container restarts unable to sudo and hangs deployment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Jon Schlueter
QA Contact: Omri Hochman
Andrew Burden
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-16 13:47 UTC by Jon Schlueter
Modified: 2018-06-28 08:01 UTC (History)
11 users (show)

Fixed In Version: openstack-redis-base-container-13.0-1 openstack-redis-container-13.0-19
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of: 1566684
Environment:
Last Closed: 2018-06-28 08:00:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2085 0 None None None 2018-06-28 08:01:57 UTC

Description Jon Schlueter 2018-04-16 13:47:40 UTC
+++ This bug was initially created as a clone of Bug #1566684 +++

- Attempted overcloud deployment [0] with the latest released OSP12 (April 2018)
- The deployment fails on step3 [1]
- Manually running the failed playbook on the controller node [2] results in a wait for the paunch task to complete [3]
- The controller node's redis container was in a state of restarting [4]
- /var/log/containers/redis/ is empty
- `journalctl CONTAINER_NAME=redis` [5] shows that a process in the container is unable to sudo

[0]
time openstack overcloud deploy \
   --templates /usr/share/openstack-tripleo-heat-templates \
   -r /home/stack/templates/roles_data.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
   -e /home/stack/templates/overcloud_images.yaml \
   -e /home/stack/templates/global-config-simple.yaml \
   -e /home/stack/templates/ceph-config.yaml > /tmp/overcloud.logs 2>&1

[1] 
2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0]: SIGNAL_IN_PROGRESS  Signal: deployment 9452a52b-1661-4dd9-9a1e-a332d51aecec failed (2)
2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2018-04-12 17:23:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED  Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-04-12 17:23:35Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-04-12 17:23:36Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-04-12 17:23:36Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack overcloud CREATE_FAILED



Issue is redis-base was missed for OSP 12 and 13

Comment 4 Leonid Natapov 2018-06-06 13:40:30 UTC
(undercloud) [stack@undercloud-0 ~]$ sudo docker run -ti --rm -u root rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp13/openstack-redis:2018-06-04.2 groups redis
redis : redis kolla
(undercloud) [stack@undercloud-0 ~]$ sudo docker run -ti --rm -u root docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-06-04.2 /bin/bash
Unable to find image 'docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-06-04.2' locally
Trying to pull repository docker-registry.engineering.redhat.com/rhosp13/openstack-redis ...
2018-06-04.2: Pulling from docker-registry.engineering.redhat.com/rhosp13/openstack-redis
Digest: sha256:248752b9b19661722911d8abf929e7fbde3aadefa7b4b1c844f3011056f04f33
Status: Downloaded newer image for docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-06-04.2
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]# /usr/local/bin/kolla_extend_start
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]# ls -al /var/log/redis
total 0
drwxr-x---. 2 redis redis  6 May 24  2017 .
drwxr-xr-x. 1 root  root  19 Jun  4 17:16 ..
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]#
()[root@99e7e4e1ce0c /]# exit
exit
(undercloud) [stack@undercloud-0 ~]$ sudo docker run -ti --rm -u root docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-06-04.2 /bin/bash
()[root@9d13cb5dbd59 /]#
()[root@9d13cb5dbd59 /]#
()[root@9d13cb5dbd59 /]# /usr/local/bin/kolla_extend_start
()[root@9d13cb5dbd59 /]# ls -al /var/log/redis/
total 0
drwxr-x---. 2 redis redis  6 May 24  2017 .
drwxr-xr-x. 1 root  root  19 Jun  4 17:16 ..
()[root@9d13cb5dbd59 /]#

Comment 7 errata-xmlrpc 2018-06-28 08:00:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2085


Note You need to log in before you can comment on or make changes to this bug.