Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1446379

Summary: openstack-containers: basic overcloud deployment of 1 controller 1 compute with containers fails over " /rootfs/etc/sysconfig/memcached not a directory"
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: openstack-containersAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Jaroslav Henner <jhenner>
Severity: urgent Docs Contact: Andrew Burden <aburden>
Priority: high    
Version: 12.0 (Pike)CC: jstransk, lruzicka, m.andre, sclewis
Target Milestone: Upstream M2Keywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 19:14:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Omri Hochman 2017-04-27 19:12:48 UTC
openstack-containers: basic overcloud deployment of 1 controller 1 compute with containers fails over " /rootfs/etc/sysconfig/memcached not a directory"  



Environment:
--------------
downstream overcloud docker images from 2017-04-13.1-1



Steps : 
-------
Deploy overcloud 1 controller 1 compute with containers according to http://etherpad.corp.redhat.com/testing-osp12-containers 





Results :  Deployment fails with the following .. 
--------------

2017-04-27 01:09:41Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment]: CREATE_IN_PROGRESS  state changed 
2017-04-27 01:09:42Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment]: CREATE_IN_PROGRESS  Stack CREATE started 
2017-04-27 01:09:42Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment.0]: CREATE_IN_PROGRESS  state changed 
2017-04-27 01:11:16Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment.0]: SIGNAL_IN_PROGRESS  Signal: deployment e0831ded-2f90-4ccf-b3ac-4
28828d2f5a6 succeeded 
2017-04-27 01:11:16Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment.0]: CREATE_COMPLETE  state changed 
2017-04-27 01:11:16Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment]: CREATE_COMPLETE  Stack CREATE completed successfully 
2017-04-27 01:11:17Z [overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment]: CREATE_COMPLETE  state changed 
2017-04-27 01:11:17Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1]: CREATE_IN_PROGRESS  state changed 
2017-04-27 01:11:17Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1]: CREATE_IN_PROGRESS  Stack CREATE started 
2017-04-27 01:11:17Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1.0]: CREATE_IN_PROGRESS  state changed 
2017-04-27 01:11:59Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1.0]: SIGNAL_IN_PROGRESS  Signal: deployment a3b64c74-4199-4d2e-8b6e
-7b490bb1d1dc failed (125) 
2017-04-27 01:12:00Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1.0]: CREATE_FAILED  Error: resources[0]: Deployment to server faile
d: deploy_status_code : Deployment exited with non-zero status code: 125 
2017-04-27 01:12:00Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Depl
oyment to server failed: deploy_status_code : Deployment exited with non-zero status code: 125 
2017-04-27 01:12:00Z [overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1]: CREATE_FAILED  Error: resources.ControllerContainersDeployment_S
tep1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 125 
2017-04-27 01:12:00Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerContainersDeployment_Step1.resource
s[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 125 
2017-04-27 01:12:01Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerContainersDeployment_Step1.re
sources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 125 
2017-04-27 01:12:01Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerContainersDeployment_Step
1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 125 

Stack overcloud CREATE_FAILED  

overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step1.0: 
 resource_type: OS::Heat::StructuredDeployment 
 physical_resource_id: a3b64c74-4199-4d2e-8b6e-7b490bb1d1dc 
 status: CREATE_FAILED 
 status_reason: | 
   Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 125 
 deploy_stdout: | 
   a8cf41852b384413fd6e4d7fe11d0d550118abaffb55670388e94a3f5c6f0433 
    
   3a215ede8508357fc87222607ce80e44d49174fa856c2c466e303c713fb1ee2f 
    
   585b90686fb21c6bf7af095aa5ff0663e9d1cb75d477c0424d9c2faf8160997f 
 deploy_stderr: | 
   /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process 
caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/config-data/memcached/etc/sysconfig/memcached\\\\\\\
" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/90f3945ede18618f21d078b55ceff04c702bc73a296e870014ef78452023818f/rootfs\\\\\\\" at \\\\\\\"/var/lib/do
cker/devicemapper/mnt/90f3945ede18618f21d078b55ceff04c702bc73a296e870014ef78452023818f/rootfs/etc/sysconfig/memcached\\\\\\\" caused \\\\\\\"not a director
y\\\\\\\"\\\"\"\n". 
Heat Stack create failed. 
Heat Stack create failed. 
(undercloud) [root@undercloud-0 tmp]

Comment 1 Omri Hochman 2017-04-27 19:14:17 UTC
adding the comment from Jirka   -  

"
I looked into the machine, there's more errors earlier in the deployment 
(in puppet) which didn't fail the deployment, but they should. E.g.

dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: Digest: 
sha2565d410080d7328cd3e0f5e7a7cdb0b302337b4e019233bb513e0fb9b7fe36d00
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: /usr/bin/docker-current: Error response from 
daemon: linux spec user: unable to find user root: no matching entries 
in passwd file.
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: Failed running docker-puppet.py for mysql
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: Removing container: docker-puppet-mysql
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: docker-puppet-mt/docker-puppet-memcached.sh: 
Permission denied
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: Failed running docker-puppet.py for memcached
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: Removing container: docker-puppet-memcached
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: docker-puppet-memcached
dub 27 01:11:14 overcloud-controller-0.localdomain 
os-collect-config[3301]: ---------

The tripleo-heat-templates are quite outdated, e.g. we're missing this 
commit

https://github.com/openstack/tripleo-heat-templates/commit/af03e4d34bffcabd26e72eb85137b91f348ee2cb

which would fail the deployment on the earlier errors. I'd suggest we 
update to recent t-h-t before spending additional time debugging the errors.

Jirka"

Comment 2 Martin André 2017-05-22 16:37:55 UTC
We've started to see the same issue in upstream CI.

Comment 3 Omri Hochman 2017-07-11 14:59:43 UTC
that was the fix : https://review.openstack.org/#/c/467572/1

Comment 4 Omri Hochman 2017-07-11 15:00:08 UTC
unable to reproduce with latest osp12 puddles.

Comment 9 errata-xmlrpc 2017-12-13 19:14:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3457