Bug 1566684
| Summary: | Redis container restarts unable to sudo and hangs deployment | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | John Fulton <johfulto> | ||||
| Component: | openstack-containers | Assignee: | Jon Schlueter <jschluet> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Omri Hochman <ohochman> | ||||
| Severity: | high | Docs Contact: | Andrew Burden <aburden> | ||||
| Priority: | high | ||||||
| Version: | 12.0 (Pike) | CC: | jjoyce, jschluet, karan, m.andre, samccann, sasha, tvignaud | ||||
| Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||
| Target Release: | 12.0 (Pike) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-redis-container-12.0-20180420.1 | Doc Type: | Bug Fix | ||||
| Doc Text: |
A refactor introducing redis-base in kolla was missed during the OSP 12 (pike) development cycle. The openstack-redist container image was missing a couple of key pieces - user add and extended_start script.
The fix was to inject the steps from kolla redis-base image into openstack-redis image. The resulting image should have all of the necessary user, extended_start.sh, and permissions setup to work correctly.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1567966 (view as bug list) | Environment: | |||||
| Last Closed: | 2018-05-09 00:04:24 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
[root@controller-0 containers]# docker inspect 0dc0c9363d5a
[
{
"Id": "sha256:0dc0c9363d5ae0fc6d0b18d133f4f91fc135d696281d8dff0e4ffda92bb45dba",
"RepoTags": [
"192.168.120.1:8787/rhosp12/openstack-redis:latest"
],
"RepoDigests": [
"192.168.120.1:8787/rhosp12/openstack-redis@sha256:bcd18b6e1ee817fd658b946f0ab8963ced86b86e15260c7739015f09ba495c0b"
],
"Parent": "",
"Comment": "",
"Created": "2018-03-19T17:06:28.898113Z",
"Container": "",
"ContainerConfig": {
"Hostname": "9817222a9fd1",
"Domainname": "",
"User": "redis",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci",
"KOLLA_BASE_DISTRO=rhel",
"KOLLA_INSTALL_TYPE=binary",
"KOLLA_INSTALL_METATYPE=rhos",
"PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"USER [redis]"
],
"ArgsEscaped": true,
"Image": "sha256:f930816280360a79f8d5dc932650256242798c8638a30c9f14c0868fe23ecd9a",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": [],
"Labels": {
"Kolla-SHA": "5.0.0-39-g6f1b947b",
"architecture": "x86_64",
"authoritative-source-url": "registry.access.redhat.com",
"build-date": "2018-03-19T17:03:07.687625",
"com.redhat.build-host": "osbs-cpt-002.ocp.osbs.upshift.eng.rdu2.redhat.com",
"com.redhat.component": "openstack-redis-docker",
"description": "Red Hat OpenStack Platform 12.0 redis",
"distribution-scope": "public",
"io.k8s.description": "Red Hat OpenStack Platform 12.0 redis",
"io.k8s.display-name": "Red Hat OpenStack Platform 12.0 redis",
"io.openshift.tags": "rhosp osp openstack osp-12.0",
"kolla_version": "stable/pike",
"name": "rhosp12/openstack-redis",
"release": "20180319.1",
"summary": "Red Hat OpenStack Platform 12.0 redis",
"tripleo-common_version": "7.6.3-23-g4891cfe",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-redis/images/12.0-20180319.1",
"vcs-ref": "8b08e7ab1db2c2418f98f34dde384103157ac3e4",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "12.0",
"version-release": "12.0-20180319.1"
}
},
"DockerVersion": "1.12.6",
"Author": "",
"Config": {
"Hostname": "9817222a9fd1",
"Domainname": "",
"User": "redis",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci",
"KOLLA_BASE_DISTRO=rhel",
"KOLLA_INSTALL_TYPE=binary",
"KOLLA_INSTALL_METATYPE=rhos",
"PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "
],
"Cmd": [
"kolla_start"
"ArgsEscaped": true,
"Image": "c336c7eb6c865112442313fc8c8b639164c316277b5f3836c8d13585ca48dc15",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": [],
"Labels": {
"Kolla-SHA": "5.0.0-39-g6f1b947b",
"architecture": "x86_64",
"authoritative-source-url": "registry.access.redhat.com",
"build-date": "2018-03-19T17:03:07.687625",
"com.redhat.build-host": "osbs-cpt-002.ocp.osbs.upshift.eng.rdu2.redhat.com",
"com.redhat.component": "openstack-redis-docker",
"description": "Red Hat OpenStack Platform 12.0 redis",
"distribution-scope": "public",
"io.k8s.description": "Red Hat OpenStack Platform 12.0 redis",
"io.k8s.display-name": "Red Hat OpenStack Platform 12.0 redis",
"io.openshift.tags": "rhosp osp openstack osp-12.0",
"kolla_version": "stable/pike",
"name": "rhosp12/openstack-redis",
"release": "20180319.1",
"summary": "Red Hat OpenStack Platform 12.0 redis",
"tripleo-common_version": "7.6.3-23-g4891cfe",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-redis/images/12.0-20180319.1",
"vcs-ref": "8b08e7ab1db2c2418f98f34dde384103157ac3e4",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "12.0",
"version-release": "12.0-20180319.1"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 826171678,
"VirtualSize": 826171678,
"GraphDriver": {
"Name": "overlay2",
"Data": {
"LowerDir": "/var/lib/docker/overlay2/88d748800b600c09cbe81e0ff0adecf84f734e786c319f22f38257ce022d60de/diff:/var/lib/docker/overlay2/3c2546a28abd99d6d1ce84823af11
be5a96b408f12980f48a90bae60df3af5b2/diff:/var/lib/docker/overlay2/bfce71988fe5cc34d568a4ffb15c896f9270724691bd8908aa8e40b96966b348/diff",
"MergedDir": "/var/lib/docker/overlay2/0ef6e3b1bd484c9314363872bee614e4fbe34e622eb251bff69f3139d9d846b8/merged",
"UpperDir": "/var/lib/docker/overlay2/0ef6e3b1bd484c9314363872bee614e4fbe34e622eb251bff69f3139d9d846b8/diff",
"WorkDir": "/var/lib/docker/overlay2/0ef6e3b1bd484c9314363872bee614e4fbe34e622eb251bff69f3139d9d846b8/work"
}
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:e9fb3906049428130d8fc22e715dc6665306ebbf483290dd139be5d7457d9749",
"sha256:1b0bb3f6ad7e8dbdc1d19cf782dc06227de1d95a5d075efb592196a509e6e3a9",
"sha256:0581e0997f56d1567ab534c1a775641a6f4c9e569141180931c0b847ba7c6740",
"sha256:e0b3e1fdeb38e607eeac999a4c5f4ad0060631807ddac6adefb7d708d551a059"
]
}
}
]
[root@controller-0 containers]#
John also tried with a new redis container version 12.0-20180405.1 , but still the same issue redis container in a state of restarting
"container_name": "redis",
"description": "Red Hat OpenStack Platform 12.0 redis",
"distribution-scope": "public",
"io.k8s.description": "Red Hat OpenStack Platform 12.0 redis",
"io.k8s.display-name": "Red Hat OpenStack Platform 12.0 redis",
"io.openshift.tags": "rhosp osp openstack osp-12.0",
"kolla_version": "stable/pike",
"managed_by": "paunch",
"name": "rhosp12/openstack-redis",
"release": "20180405.1",
"summary": "Red Hat OpenStack Platform 12.0 redis",
"tripleo-common_version": "7.6.3-23-g4891cfe",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-redis/images/12.0-20180405.1",
"vcs-ref": "c9ccbc3600ae45942ed2f1c41023a3116333b181",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "12.0",
"version-release": "12.0-20180405.1"
I have tried cleaning up the failed stack, deleted all the docker images from undercloud. Re-pulled the latest (12.0-20180405.1) docker images and retried OSPd, but stil it fails at Redis container restart issue. [root@controller-0 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS d41a8952a5fe 192.168.120.1:8787/rhosp12/openstack-mariadb:12.0-20180405.1 "kolla_start" 33 minutes ago Up 33 minutes 9ea07279ff08 192.168.120.1:8787/rhosp12/openstack-mariadb:12.0-20180405.1 "kolla_start" 33 minutes ago Up 33 minutes 42c3656378e8 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 43 minutes ago Up 43 minutes 01e12e27c68c 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 44 minutes ago Up 44 minutes d18665b6873b 192.168.120.1:8787/rhosp12/openstack-rabbitmq:12.0-20180405.1 "kolla_start" 49 minutes ago Up 49 minutes (healthy) 39cde3ff48b1 192.168.120.1:8787/rhosp12/openstack-memcached:12.0-20180405.1 "/bin/bash -c 'source" 49 minutes ago Up 49 minutes 226e5fbad0c2 192.168.120.1:8787/rhosp12/openstack-redis:12.0-20180405.1 "kolla_start" 49 minutes ago Restarting (1) 21 minutes ago 91c5a84e36ac 192.168.120.1:8787/rhosp12/openstack-haproxy:12.0-20180405.1 "kolla_start" 49 minutes ago Up 49 minutes [root@controller-0 ~]# [root@controller-0 ~]# docker logs redis
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
I retried this with TAG : 12.0-20180124.1
Still redis container does not starts
[root@controller-0 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c2e73f97e449 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 7 minutes ago Up 7 minutes ceph-mgr-controller-0
eec1544c7dc9 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 9 minutes ago Up 9 minutes ceph-mon-controller-0
dab00dcfba57 192.168.120.1:8787/rhosp12/openstack-rabbitmq:12.0-20180124.1 "kolla_start" 13 minutes ago Up 13 minutes (healthy) rabbitmq
bb75eda6c979 192.168.120.1:8787/rhosp12/openstack-memcached:12.0-20180124.1 "/bin/bash -c 'source" 13 minutes ago Up 13 minutes memcached
c2cbcf3f65a2 192.168.120.1:8787/rhosp12/openstack-redis:12.0-20180124.1 "kolla_start" 13 minutes ago Restarting (1) 6 minutes ago redis
0ba613abbc12 192.168.120.1:8787/rhosp12/openstack-haproxy:12.0-20180124.1 "kolla_start" 13 minutes ago Up 13 minutes haproxy
[root@controller-0 ~]#
sudo: no tty present and no askpass program specified
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
sudo: no tty present and no askpass program specified
User is already set to root in the following:
[root@refarch-r220-04 ~]# grep user /usr/share/openstack-tripleo-heat-templates/docker/services/database/redis.yaml
user: root
user: root
[root@refarch-r220-04 ~]#
[root@refarch-r220-04 ~]# rpm -qa | grep tripleo-heat-templates
openstack-tripleo-heat-templates-7.0.3-22.el7ost.noarch
[root@refarch-r220-04 ~]#
WORKAROUND: Update /usr/share/openstack-tripleo-heat-templates/docker/services/database/redis.yaml to add "user: root" under "restart: always" [1] and redeploy. [1] https://github.com/openstack/tripleo-heat-templates/blob/8d7a9fc3d9a33fcdd070ef43f653c458d30e439c/docker/services/database/redis.yaml#L101 +1 to the workaround that John discovered. Redis container is healthy and clean. I'm seeing this issue too. It appears the redis user on the images does not belong to the kolla group. It's normally added to the group via the configure_user macro [1] and [2]: [heat-admin@openstack-controller-0 ~]$ sudo docker run -ti --rm -u root 192.168.24.1:8787/rhosp13/openstack-redis:13.0-18 groups redis redis : redis [heat-admin@openstack-controller-0 ~]$ sudo docker run -ti --rm -u root 192.168.24.1:8787/rhosp13/openstack-nova-api:13.0-16 groups nova nova : nova kolla qemu A quick look at the downstream scripts to build kolla images tells me the redis image depends on openstack-base, while it should really depend on openstack-redis-base. [1] https://github.com/openstack/kolla/blob/stable/queens/docker/redis/redis-base/Dockerfile.j2#L8 [2] https://github.com/openstack/kolla/blob/stable/queens/docker/macros.j2#L34-L48 Created attachment 1424340 [details]
Proposed dist-git patch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1338 |
- Attempted overcloud deployment [0] with the latest released OSP12 (April 2018) - The deployment fails on step3 [1] - Manually running the failed playbook on the controller node [2] results in a wait for the paunch task to complete [3] - The controller node's redis container was in a state of restarting [4] - /var/log/containers/redis/ is empty - `journalctl CONTAINER_NAME=redis` [5] shows that a process in the container is unable to sudo [0] time openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ -r /home/stack/templates/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/templates/overcloud_images.yaml \ -e /home/stack/templates/global-config-simple.yaml \ -e /home/stack/templates/ceph-config.yaml > /tmp/overcloud.logs 2>&1 [1] 2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0]: SIGNAL_IN_PROGRESS Signal: deployment 9452a52b-1661-4dd9-9a1e-a332d51aecec failed (2) 2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2018-04-12 17:23:34Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2018-04-12 17:23:35Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2018-04-12 17:23:35Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2018-04-12 17:23:36Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2018-04-12 17:23:36Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 Stack overcloud CREATE_FAILED overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 9452a52b-1661-4dd9-9a1e-a332d51aecec status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | ... "Debug: Processing report from controller-0.localdomain with processor Puppet::Reports::Store" ], "changed": false, "failed_when_result": true } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/8ded7ba4-417e-4546-ad30-56ace9dee7c9_plaHeat Stack create failed. Heat Stack create failed. ybook.retry PLAY RECAP ********************************************************************* localhost : ok=4 changed=1 unreachable=0 failed=1 (truncated, view all with --long) deploy_stderr: | [2] cd /var/lib/heat-config/heat-config-ansible ansible-playbook -vvv 8ded7ba4-417e-4546-ad30-56ace9dee7c9_playbook.yaml -e @8ded7ba4-417e-4546-ad30-56ace9dee7c9_variables.json [3] ################################################## # Per step starting of the containers using paunch ################################################## - name: Check if /var/lib/hashed-tripleo-config/docker-container-startup-config-step_{{step}}.json exists stat: path: /var/lib/tripleo-config/hashed-docker-container-startup-config-step_{{step}}.json register: docker_config_json # Note docker-puppet.py generates the hashed-*.json file, which is a copy of # the *step_n.json with a hash of the generated external config added # This acts as a salt to enable restarting the container if config changes - name: Start containers for step {{step}} command: >- paunch --debug apply --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_{{step}}.json --config-id tripleo_step{{step}} --managed-by tripleo-{{role_name}} when: docker_config_json.stat.exists changed_when: false check_mode: no register: outputs failed_when: false no_log: true - debug: var=(outputs.stderr|default('')).split('\n')|union(outputs.stdout_lines|default([])) when: outputs.rc is defined failed_when: outputs.rc != 0 [4] [root@controller-0 containers]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2c0cf16d092f 192.168.120.1:8787/rhosp12/openstack-mariadb:latest "kolla_start" About an hour ago Up About an hour mysql 72aa455cbf67 192.168.120.1:8787/rhosp12/openstack-mariadb:latest "kolla_start" About an hour ago Up About an hour clustercheck 129b89e97e0d 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 2 hours ago Up 2 hours ceph-mgr-controller-0 92b235c77c6b 192.168.120.1:8787/rhceph/rhceph-3-rhel7:latest "/entrypoint.sh" 2 hours ago Up 2 hours ceph-mon-controller-0 fd9470ea8822 192.168.120.1:8787/rhosp12/openstack-rabbitmq:latest "kolla_start" 2 hours ago Up 2 hours (healthy) rabbitmq 4c92a60ea7ef 192.168.120.1:8787/rhosp12/openstack-memcached:latest "/bin/bash -c 'source" 2 hours ago Up 2 hours memcached 88732d26cf83 192.168.120.1:8787/rhosp12/openstack-redis:latest "kolla_start" 2 hours ago Restarting (1) 21 minutes ago redis c74861d9d0a8 192.168.120.1:8787/rhosp12/openstack-haproxy:latest "kolla_start" 2 hours ago Up 2 hours haproxy [root@controller-0 containers]# [5] [root@controller-0 containers]# journalctl CONTAINER_NAME=redis -- Logs begin at Thu 2018-04-12 16:38:56 UTC, end at Thu 2018-04-12 19:18:10 UTC. -- Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: We trust you have received the usual lecture from the local System Apr 12 17:01:02 controller-0 dockerd-current[19670]: Administrator. It usually boils down to these three things: Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: #1) Respect the privacy of others. Apr 12 17:01:02 controller-0 dockerd-current[19670]: #2) Think before you type. Apr 12 17:01:02 controller-0 dockerd-current[19670]: #3) With great power comes great responsibility. Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: sudo: no tty present and no askpass program specified Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: We trust you have received the usual lecture from the local System Apr 12 17:01:02 controller-0 dockerd-current[19670]: Administrator. It usually boils down to these three things: Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: #1) Respect the privacy of others. Apr 12 17:01:02 controller-0 dockerd-current[19670]: #2) Think before you type. Apr 12 17:01:02 controller-0 dockerd-current[19670]: #3) With great power comes great responsibility. Apr 12 17:01:02 controller-0 dockerd-current[19670]: Apr 12 17:01:02 controller-0 dockerd-current[19670]: sudo: no tty present and no askpass program specified Apr 12 17:01:03 controller-0 dockerd-current[19670]: ... Apr 12 18:50:20 controller-0 dockerd-current[19670]: sudo: no tty present and no askpass program specified [root@controller-0 containers]#