Bug 1463574 - Node system container failed to start due to "failed to run Kubelet: failed to create kubelet: mkdir /var/lib/dockershim: read-only file system"
Node system container failed to start due to "failed to run Kubelet: failed t...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.7.0
Assigned To: Giuseppe Scrivano
Gan Huang
:
Depends On: 1482239
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 05:10 EDT by Gan Huang
Modified: 2017-11-28 16:58 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-28 16:58:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 1 Giuseppe Scrivano 2017-06-21 10:32:30 EDT
could you please verify if these steps fix the issue for you?

1) create the directory "/var/lib/dockershim" on the host
2) adding this snippet to /var/lib/containers/atomic/atomic-openshift-node.0/config.json (under the mounts section):

        {
            "type": "bind",
            "source": "/var/lib/dockershim",
            "destination": "/var/lib/dockershim",
            "options": [
                "bind",
                "slave",
                "rw",
                "mode=777"
            ]
        },

3) systemctl restart atomic-openshift-node

If you can confirm that, I'll prepare a patch to add the missing bind mount and ensure the directory exists on the host.
Comment 2 Gan Huang 2017-06-22 06:24:25 EDT
Still failed:

[10598]: I0622 06:23:50.028743   10609 iptables.go:562] couldn't get iptables-restore version; assuming it doesn't support --wait
[10598]: I0622 06:23:50.029518   10609 node.go:293] openshift-sdn network plugin ready
[10598]: F0622 06:23:50.029549   10609 node.go:325] error: SDN node startup failed: open /etc/cni/net.d/80-openshift-sdn.conf: read-only file system
Comment 3 Giuseppe Scrivano 2017-06-22 10:04:54 EDT
I've created a PR here:

https://github.com/openshift/origin/pull/14828

This fix is also needed to get openvswitch working as a system container:

https://github.com/openshift/openshift-ansible/pull/4540
Comment 4 Gan Huang 2017-09-29 22:56:31 EDT
Verified with 

openshift v3.7.0-0.127.0
kubernetes v1.7.0+80709908fd
etcd 3.2.1


registry.ops.openshift.com/openshift3/ose                v3.7.0   fe1f4837f73b   2017-09-29 22:27   1.06 GB        ostree
Comment 7 errata-xmlrpc 2017-11-28 16:58:09 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.