Bug 1598271

Summary:	iscsi volume attached failed in containerized kubelet
Product:	OpenShift Container Platform	Reporter:	Qin Ping <piqin>
Component:	Storage	Assignee:	Matthew Wong <mawong>
Status:	CLOSED ERRATA	QA Contact:	Qin Ping <piqin>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.10.0	CC:	aos-bugs, aos-storage-staff, bbennett, mawong, vwalek
Target Milestone:	---
Target Release:	3.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: kubelet system containers can't write to /var/lib/iscsi Consequence: iscsi volumes can't be attached Fix: mount the host /var/lib/iscsi into the kubelet system container Result: iscsi volumes can be attached	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-10-11 07:21:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Qin Ping 2018-07-05 02:05:07 UTC

Description of problem:
iscsi volume attached failed in containerized kubelet

Version-Release number of selected component (if applicable):
oc v3.10.12
openshift v3.10.12
kubernetes v1.10.0+b81c8f8


How reproducible:
Always

Steps to Reproduce:
1. Create a PVC with gluster-block provisioner gluster-block?
2. Create a Pod using the PVC
3. Check Pod status

Actual results:
Pod stucks in "ContainerCreating" status and reports:
Events:
  Type     Reason                  Age                From                      Message
  ----     ------                  ----               ----                      -------
  Normal   Scheduled               2m                 default-scheduler         Successfully assigned cinderpd1 to qe-piqin-node-1
  Normal   SuccessfulAttachVolume  2m                 attachdetach-controller   AttachVolume.Attach succeeded for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982"
  Warning  FailedMount             23s (x7 over 57s)  kubelet, qe-piqin-node-1  MountVolume.WaitForAttach failed for volume "pvc-a401b46a-7f6e-11e8-9ea0-fa163e2d3982" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.73.131.121:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/10.73.131.121,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/10.73.131.121,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6
  Warning  FailedMount  9s  kubelet, qe-piqin-node-1  Unable to mount volumes for pod "cinderpd1_test(1616ee93-7f6f-11e8-9ea0-fa163e2d3982)": timeout expired waiting for volumes to attach or mount for pod "test"

Expected results:
Pod should run successfully.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Matthew Wong 2018-07-05 14:26:57 UTC

I think we need to add /var/lib/iscsi to the list of system container mounts as it is the directory in which iscsiadm creates discovery records & node records. https://github.com/openshift/origin/blob/b8d718c99325dd3564ff25470994ff18eec8ca06/images/node/system-container/config.json.template . I can confirm 
1) adding
       {
            "type": "bind",
            "source": "/var/lib/iscsi",
            "destination": "/var/lib/iscsi",
            "options": [
                "rbind",
                "rslave",
                "rw"
            ]
        },
to /var/lib/containers/atomic/atomic-openshift-node.0/config.json
2) `systemctl restart atomic-openshift-node`
3) `runc exec atomic-openshift-node touch /var/lib/iscsi/asdf`
succeeds
but I'm unable to test if this means iscsi attach would complete successfully.

Qin if you don't mind, could you try these steps which should allow the kubelet to write to /var/lib/iscsi and then retry attaching an iscsi volume.

Comment 2 Ben Bennett 2018-07-05 15:34:55 UTC

This is not a regression and was broken in earlier releases.

In addition containerized kubelet is not supported in 3.10 (outside of Atomic).

Comment 3 Matthew Wong 2018-07-05 19:10:03 UTC

I have opened an origin PR at https://github.com/openshift/origin/pull/20221 and confirmed for myself that mounting /var/lib/iscsi allows Attach (and subsequent mount and container running) to succeed.

Comment 5 Qin Ping 2018-08-27 06:00:09 UTC

Verified in OCP:
oc v3.11.0-0.22.0
openshift v3.11.0-0.22.0
kubernetes v1.11.0+d4cacc0

# uname -a
Linux ip-172-18-6-200.ec2.internal 3.10.0-862.6.3.el7.x86_64 #1 SMP Fri Jun 15 17:57:37 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

Comment 9 errata-xmlrpc 2018-10-11 07:21:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652