Bug 1463574 - Node system container failed to start due to "failed to run Kubelet: failed to create kubelet: mkdir /var/lib/dockershim: read-only file system"
Summary: Node system container failed to start due to "failed to run Kubelet: failed t...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.7.0
Assignee: Giuseppe Scrivano
QA Contact: Gan Huang
URL:
Whiteboard:
Depends On: 1482239
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-21 09:10 UTC by Gan Huang
Modified: 2017-11-28 21:58 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-11-28 21:58:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Comment 1 Giuseppe Scrivano 2017-06-21 14:32:30 UTC
could you please verify if these steps fix the issue for you?

1) create the directory "/var/lib/dockershim" on the host
2) adding this snippet to /var/lib/containers/atomic/atomic-openshift-node.0/config.json (under the mounts section):

        {
            "type": "bind",
            "source": "/var/lib/dockershim",
            "destination": "/var/lib/dockershim",
            "options": [
                "bind",
                "slave",
                "rw",
                "mode=777"
            ]
        },

3) systemctl restart atomic-openshift-node

If you can confirm that, I'll prepare a patch to add the missing bind mount and ensure the directory exists on the host.

Comment 2 Gan Huang 2017-06-22 10:24:25 UTC
Still failed:

[10598]: I0622 06:23:50.028743   10609 iptables.go:562] couldn't get iptables-restore version; assuming it doesn't support --wait
[10598]: I0622 06:23:50.029518   10609 node.go:293] openshift-sdn network plugin ready
[10598]: F0622 06:23:50.029549   10609 node.go:325] error: SDN node startup failed: open /etc/cni/net.d/80-openshift-sdn.conf: read-only file system

Comment 3 Giuseppe Scrivano 2017-06-22 14:04:54 UTC
I've created a PR here:

https://github.com/openshift/origin/pull/14828

This fix is also needed to get openvswitch working as a system container:

https://github.com/openshift/openshift-ansible/pull/4540

Comment 4 Gan Huang 2017-09-30 02:56:31 UTC
Verified with 

openshift v3.7.0-0.127.0
kubernetes v1.7.0+80709908fd
etcd 3.2.1


registry.ops.openshift.com/openshift3/ose                v3.7.0   fe1f4837f73b   2017-09-29 22:27   1.06 GB        ostree

Comment 7 errata-xmlrpc 2017-11-28 21:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.