Bug 1463574

Summary: Node system container failed to start due to "failed to run Kubelet: failed to create kubelet: mkdir /var/lib/dockershim: read-only file system"
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: InstallerAssignee: Giuseppe Scrivano <gscrivan>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, ghuang, jokerman, mmccomas, smilner
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 21:58:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1482239    
Bug Blocks:    

Comment 1 Giuseppe Scrivano 2017-06-21 14:32:30 UTC
could you please verify if these steps fix the issue for you?

1) create the directory "/var/lib/dockershim" on the host
2) adding this snippet to /var/lib/containers/atomic/atomic-openshift-node.0/config.json (under the mounts section):

        {
            "type": "bind",
            "source": "/var/lib/dockershim",
            "destination": "/var/lib/dockershim",
            "options": [
                "bind",
                "slave",
                "rw",
                "mode=777"
            ]
        },

3) systemctl restart atomic-openshift-node

If you can confirm that, I'll prepare a patch to add the missing bind mount and ensure the directory exists on the host.

Comment 2 Gan Huang 2017-06-22 10:24:25 UTC
Still failed:

[10598]: I0622 06:23:50.028743   10609 iptables.go:562] couldn't get iptables-restore version; assuming it doesn't support --wait
[10598]: I0622 06:23:50.029518   10609 node.go:293] openshift-sdn network plugin ready
[10598]: F0622 06:23:50.029549   10609 node.go:325] error: SDN node startup failed: open /etc/cni/net.d/80-openshift-sdn.conf: read-only file system

Comment 3 Giuseppe Scrivano 2017-06-22 14:04:54 UTC
I've created a PR here:

https://github.com/openshift/origin/pull/14828

This fix is also needed to get openvswitch working as a system container:

https://github.com/openshift/openshift-ansible/pull/4540

Comment 4 Gan Huang 2017-09-30 02:56:31 UTC
Verified with 

openshift v3.7.0-0.127.0
kubernetes v1.7.0+80709908fd
etcd 3.2.1


registry.ops.openshift.com/openshift3/ose                v3.7.0   fe1f4837f73b   2017-09-29 22:27   1.06 GB        ostree

Comment 7 errata-xmlrpc 2017-11-28 21:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188