Description of problem:
iscsi volume attached failed in containerized kubelet
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Create a PVC with gluster-block provisioner gluster-block?
2. Create a Pod using the PVC
3. Check Pod status
Pod stucks in "ContainerCreating" status and reports:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned cinderpd1 to qe-piqin-node-1
Normal SuccessfulAttachVolume 2m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982"
Warning FailedMount 23s (x7 over 57s) kubelet, qe-piqin-node-1 MountVolume.WaitForAttach failed for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.73.131.121:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/10.73.131.121,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/10.73.131.121,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6
Warning FailedMount 9s kubelet, qe-piqin-node-1 Unable to mount volumes for pod "cinderpd1_test(1616ee93-7f6f-11e8-9ea0-fa163e2d3982)": timeout expired waiting for volumes to attach or mount for pod "test"
Pod should run successfully.
Node Log (of failed PODs):
StorageClass Dump (if StorageClass used by PV/PVC):
I think we need to add /var/lib/iscsi to the list of system container mounts as it is the directory in which iscsiadm creates discovery records & node records. https://github.com/openshift/origin/blob/b8d718c99325dd3564ff25470994ff18eec8ca06/images/node/system-container/config.json.template . I can confirm
2) `systemctl restart atomic-openshift-node`
3) `runc exec atomic-openshift-node touch /var/lib/iscsi/asdf`
but I'm unable to test if this means iscsi attach would complete successfully.
Qin if you don't mind, could you try these steps which should allow the kubelet to write to /var/lib/iscsi and then retry attaching an iscsi volume.
This is not a regression and was broken in earlier releases.
In addition containerized kubelet is not supported in 3.10 (outside of Atomic).
I have opened an origin PR at https://github.com/openshift/origin/pull/20221 and confirmed for myself that mounting /var/lib/iscsi allows Attach (and subsequent mount and container running) to succeed.
Verified in OCP:
# uname -a
Linux ip-172-18-6-200.ec2.internal 3.10.0-862.6.3.el7.x86_64 #1 SMP Fri Jun 15 17:57:37 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Atomic Host release 7.5
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.