Description of problem: iscsi volume attached failed in containerized kubelet Version-Release number of selected component (if applicable): oc v3.10.12 openshift v3.10.12 kubernetes v1.10.0+b81c8f8 How reproducible: Always Steps to Reproduce: 1. Create a PVC with gluster-block provisioner gluster-block? 2. Create a Pod using the PVC 3. Check Pod status Actual results: Pod stucks in "ContainerCreating" status and reports: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned cinderpd1 to qe-piqin-node-1 Normal SuccessfulAttachVolume 2m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982" Warning FailedMount 23s (x7 over 57s) kubelet, qe-piqin-node-1 MountVolume.WaitForAttach failed for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 10.73.131.121:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/10.73.131.121,3260 err 30 iscsiadm: Could not open /var/lib/iscsi/send_targets/10.73.131.121,3260: Read-only file system iscsiadm: Could not add new discovery record. , err exit status 6 Warning FailedMount 9s kubelet, qe-piqin-node-1 Unable to mount volumes for pod "cinderpd1_test(1616ee93-7f6f-11e8-9ea0-fa163e2d3982)": timeout expired waiting for volumes to attach or mount for pod "test" Expected results: Pod should run successfully. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
I think we need to add /var/lib/iscsi to the list of system container mounts as it is the directory in which iscsiadm creates discovery records & node records. https://github.com/openshift/origin/blob/b8d718c99325dd3564ff25470994ff18eec8ca06/images/node/system-container/config.json.template . I can confirm 1) adding { "type": "bind", "source": "/var/lib/iscsi", "destination": "/var/lib/iscsi", "options": [ "rbind", "rslave", "rw" ] }, to /var/lib/containers/atomic/atomic-openshift-node.0/config.json 2) `systemctl restart atomic-openshift-node` 3) `runc exec atomic-openshift-node touch /var/lib/iscsi/asdf` succeeds but I'm unable to test if this means iscsi attach would complete successfully. Qin if you don't mind, could you try these steps which should allow the kubelet to write to /var/lib/iscsi and then retry attaching an iscsi volume.
This is not a regression and was broken in earlier releases. In addition containerized kubelet is not supported in 3.10 (outside of Atomic).
I have opened an origin PR at https://github.com/openshift/origin/pull/20221 and confirmed for myself that mounting /var/lib/iscsi allows Attach (and subsequent mount and container running) to succeed.
Verified in OCP: oc v3.11.0-0.22.0 openshift v3.11.0-0.22.0 kubernetes v1.11.0+d4cacc0 # uname -a Linux ip-172-18-6-200.ec2.internal 3.10.0-862.6.3.el7.x86_64 #1 SMP Fri Jun 15 17:57:37 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release Red Hat Enterprise Linux Atomic Host release 7.5
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652