Bug 1598271 - iscsi volume attached failed in containerized kubelet
Summary: iscsi volume attached failed in containerized kubelet
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.0
Assignee: Matthew Wong
QA Contact: Qin Ping
Depends On:
TreeView+ depends on / blocked
Reported: 2018-07-05 02:05 UTC by Qin Ping
Modified: 2018-10-11 07:21 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: kubelet system containers can't write to /var/lib/iscsi Consequence: iscsi volumes can't be attached Fix: mount the host /var/lib/iscsi into the kubelet system container Result: iscsi volumes can be attached
Clone Of:
Last Closed: 2018-10-11 07:21:36 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:21:55 UTC

Description Qin Ping 2018-07-05 02:05:07 UTC
Description of problem:
iscsi volume attached failed in containerized kubelet

Version-Release number of selected component (if applicable):
oc v3.10.12
openshift v3.10.12
kubernetes v1.10.0+b81c8f8

How reproducible:

Steps to Reproduce:
1. Create a PVC with gluster-block provisioner gluster-block?
2. Create a Pod using the PVC
3. Check Pod status

Actual results:
Pod stucks in "ContainerCreating" status and reports:
  Type     Reason                  Age                From                      Message
  ----     ------                  ----               ----                      -------
  Normal   Scheduled               2m                 default-scheduler         Successfully assigned cinderpd1 to qe-piqin-node-1
  Normal   SuccessfulAttachVolume  2m                 attachdetach-controller   AttachVolume.Attach succeeded for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982"
  Warning  FailedMount             23s (x7 over 57s)  kubelet, qe-piqin-node-1  MountVolume.WaitForAttach failed for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6
  Warning  FailedMount  9s  kubelet, qe-piqin-node-1  Unable to mount volumes for pod "cinderpd1_test(1616ee93-7f6f-11e8-9ea0-fa163e2d3982)": timeout expired waiting for volumes to attach or mount for pod "test"

Expected results:
Pod should run successfully.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Matthew Wong 2018-07-05 14:26:57 UTC
I think we need to add /var/lib/iscsi to the list of system container mounts as it is the directory in which iscsiadm creates discovery records & node records. https://github.com/openshift/origin/blob/b8d718c99325dd3564ff25470994ff18eec8ca06/images/node/system-container/config.json.template . I can confirm 
1) adding
            "type": "bind",
            "source": "/var/lib/iscsi",
            "destination": "/var/lib/iscsi",
            "options": [
to /var/lib/containers/atomic/atomic-openshift-node.0/config.json
2) `systemctl restart atomic-openshift-node`
3) `runc exec atomic-openshift-node touch /var/lib/iscsi/asdf`
but I'm unable to test if this means iscsi attach would complete successfully.

Qin if you don't mind, could you try these steps which should allow the kubelet to write to /var/lib/iscsi and then retry attaching an iscsi volume.

Comment 2 Ben Bennett 2018-07-05 15:34:55 UTC
This is not a regression and was broken in earlier releases.

In addition containerized kubelet is not supported in 3.10 (outside of Atomic).

Comment 3 Matthew Wong 2018-07-05 19:10:03 UTC
I have opened an origin PR at https://github.com/openshift/origin/pull/20221 and confirmed for myself that mounting /var/lib/iscsi allows Attach (and subsequent mount and container running) to succeed.

Comment 5 Qin Ping 2018-08-27 06:00:09 UTC
Verified in OCP:
oc v3.11.0-0.22.0
openshift v3.11.0-0.22.0
kubernetes v1.11.0+d4cacc0

# uname -a
Linux ip-172-18-6-200.ec2.internal 3.10.0-862.6.3.el7.x86_64 #1 SMP Fri Jun 15 17:57:37 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

Comment 9 errata-xmlrpc 2018-10-11 07:21:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.