Bug 1455472

Summary: Got "chroot: cannot change root directory to /rootfs" when using docker cmd on nodes
Product: OpenShift Container Platform Reporter: Yan Du <yadu>
Component: ReleaseAssignee: Scott Dodson <sdodson>
Status: CLOSED CURRENTRELEASE QA Contact: Gaoyun Pei <gpei>
Severity: urgent Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, dma, gpei, haowang, jokerman, mfiedler, mifiedle, mkargaki, mmccomas, pruan, sdodson, smunilla, vlaad, wmeng, xiazhao
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: aos-scalability-36
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-24 13:01:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yan Du 2017-05-25 09:27:22 UTC 
Description of problem:

SSH into node and run docker cmd after ocp-3.6 fresh installation, docker path is /usr/local/bin/docker and it coluld not work well:

[root@host-8-174-84 ~]# docker images
chroot: cannot change root directory to /rootfs: No such file or directory

[root@host-8-174-84 ~]# docker ps
chroot: cannot change root directory to /rootfs: No such file or directory

[root@host-8-174-84 ~]# which docker
/usr/local/bin/docker

[root@host-8-175-84 ~]# rpm -qf /usr/local/bin/docker
atomic-openshift-node-3.6.84-1.git.0.ba2ffa1.el7.x86_64

docker works well when using /usr/bin/docker 
[root@host-8-174-84 ~]# /usr/bin/docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE


Version-Release number of selected component (if applicable):
openshift v3.6.84
kubernetes v1.6.1+5115d708d7
etcd 3.1.0
docker-client-1.12.6-16.el7.x86_64
docker-1.12.6-16.el7.x86_64
docker-common-1.12.6-16.el7.x86_64
docker-rhel-push-plugin-1.12.6-16.el7.x86_64
atomic-openshift-docker-excluder-3.6.84-1.git.0.ba2ffa1.el7.noarch


Steps to Reproduce:
1. Setup OCP env
2. SSH into node and run docker cmd


Actual results:
Same as description


Expected results:
docker cmd could work well and the docker path should be /usr/bin/docker
Comment 1 Scott Dodson 2017-05-25 12:58:08 UTC 
Broken by this PR, assigning to release team.

https://github.com/openshift/origin/pull/13642
Comment 2 Scott Dodson 2017-05-30 15:47:15 UTC 
https://github.com/openshift/origin/pull/14350 has some discussion on the topic and proposed solution
Comment 3 Mike Fiedler 2017-06-01 15:02:27 UTC 
Changing sev to urgent - openshift-ansible installs are currently broken.
Comment 4 Scott Dodson 2017-06-01 15:21:17 UTC 
(In reply to Mike Fiedler from comment #3)
> Changing sev to urgent - openshift-ansible installs are currently broken.

When you say installs are broken, how are they broken, is there an error?

If the install is failing because of this I'll pick this up and fix it. If it's not actually preventing successful installation I'm going to leave this with the release team as they initiated the breaking change.
Comment 7 Scott Dodson 2017-06-01 17:50:57 UTC 
https://github.com/openshift/origin/pull/14441
Comment 9 Gaoyun Pei 2017-06-05 06:36:58 UTC 
Still reproducible on atomic-openshift-node-3.6.94-1.git.0.ba4aad2.el7.x86_64
Comment 10 Mike Fiedler 2017-06-05 12:14:11 UTC 
Are there any workarounds for this?
Comment 11 Michail Kargakis 2017-06-05 12:20:59 UTC 
The fix will be included in the next OCP build which should land today.

As a workaround, use the full path to /usr/bin/docker.
Comment 12 Scott Dodson 2017-06-05 12:29:32 UTC 
Or rm /usr/local/bin/docker
Comment 13 Michail Kargakis 2017-06-07 15:34:31 UTC 
Should be fixed in v3.6.99-1
Comment 14 Peter Ruan 2017-06-07 23:53:55 UTC 
verified with the latest puddle 

oc v3.6.99
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift-120.lab.sjc.redhat.com:8443
openshift v3.6.99
kubernetes v1.6.1+5115d708d7