Bug 1686336

Summary: [3.10] Failed to mount iscsi on atomic host
Product: OpenShift Container Platform Reporter: Liang Xia <lxia>
Component: StorageAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: Liang Xia <lxia>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: aos-bugs, aos-storage-staff, jsafrane, sdodson
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The node system container did not properly mount /var/lib/iscsi rw, now it does avoiding problems mounting iscsi volumes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-11 09:30:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1686266    

Description Liang Xia 2019-03-07 09:47:37 UTC
Description of problem:
Set up OCP with atomic host, trying to use iscsi in the pod.
Pod failed with below error:
  Warning  FailedMount             10s (x6 over 26s)  kubelet, qe-lxia-310node-2  MountVolume.WaitForAttach failed for volume "pv-iscsi-zojca" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 172.30.109.235:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.30.109.235,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/172.30.109.235,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6


Version-Release number of selected component (if applicable):
oc v3.10.119
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

# cat /etc/os-release 
NAME="Red Hat Enterprise Linux Atomic Host"
VERSION="7.5.3"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Atomic Host"
VARIANT_ID=atomic.host
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux Atomic Host 7.5.3"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:atomic-host"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION="7.5"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
OSTREE_VERSION=7.5.3


How reproducible:
Always

Steps to Reproduce:
1. Setup OCP with RHEL atomic host.
2. Try to create a pod with iscsi.
3. Check the pod.

Actual results:
# oc get pod -n zojca 
NAME          READY     STATUS              RESTARTS   AGE
iscsi-zojca   0/1       ContainerCreating   0          3m

# oc describe iscsi-zojca pod -n zojca 
error: the server doesn't have a resource type "iscsi-zojca"
[root@qe-lxia-310master-etcd-1 ~]# oc describe pod iscsi-zojca  -n zojca 
Name:         iscsi-zojca
Namespace:    zojca
Node:         qe-lxia-310node-2/10.240.0.181
Start Time:   Thu, 07 Mar 2019 09:40:36 +0000
Labels:       name=iscsi
Annotations:  openshift.io/scc=privileged
Status:       Pending
IP:           
Containers:
  iscsi:
    Container ID:   
    Image:          jhou/hello-openshift
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/iscsi from iscsi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gfbf8 (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  iscsi:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-iscsi-zojca
    ReadOnly:   false
  default-token-gfbf8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gfbf8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     <none>
Events:
  Type     Reason                  Age              From                        Message
  ----     ------                  ----             ----                        -------
  Normal   Scheduled               4m               default-scheduler           Successfully assigned iscsi-zojca to qe-lxia-310node-2
  Normal   SuccessfulAttachVolume  4m               attachdetach-controller     AttachVolume.Attach succeeded for volume "pv-iscsi-zojca"
  Warning  FailedMount             2m               kubelet, qe-lxia-310node-2  Unable to mount volumes for pod "iscsi-zojca_zojca(0f99db82-40bd-11e9-ba5a-42010af000b2)": timeout expired waiting for volumes to attach or mount for pod "zojca"/"iscsi-zojca". list of unmounted volumes=[iscsi]. list of unattached volumes=[iscsi default-token-gfbf8]
  Warning  FailedMount             1m (x9 over 4m)  kubelet, qe-lxia-310node-2  MountVolume.WaitForAttach failed for volume "pv-iscsi-zojca" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 172.30.109.235:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.30.109.235,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/172.30.109.235,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6


Expected results:
Pod is up and running.

Comment 1 Liang Xia 2019-03-07 09:49:05 UTC
The issue exist on atomic host + system container, but does not exist on atomic host + docker container.

Comment 2 Liang Xia 2019-03-07 09:52:16 UTC
Related bug, https://bugzilla.redhat.com/show_bug.cgi?id=1598271

Comment 3 Jan Safranek 2019-03-11 12:53:05 UTC
Filled https://github.com/openshift/origin/pull/22289

In the meantime, can you please check that it really helps? Follow https://bugzilla.redhat.com/show_bug.cgi?id=1598271#c1 and report any issues. Jenkins has failed to give me 3.10 on Atomic Host.

Comment 4 Liang Xia 2019-03-12 01:42:49 UTC
PR https://github.com/openshift/origin/pull/22289 merged.

Comment 5 Liang Xia 2019-03-13 01:04:31 UTC
Build is ready,

Changelog 	
* Mon Mar 11 2019 AOS Automation Release Team <aos-team-art@redhat.com> 3.10.125-1
- Bindmount /var/lib/iscsi rw for iscsi attach (mawong@redhat.com)

Comment 6 Wenqi He 2019-03-22 06:43:03 UTC
Tested on below version:
openshift v3.10.127
kubernetes v1.10.0+b81c8f8

# uname -a
Linux ip-172-18-6-162.ec2.internal 3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10 16:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

ISCSI works well.

# oc get pods -n pxdtd
NAME          READY     STATUS    RESTARTS   AGE
iscsi-pxdtd   1/1       Running   0          16m

# oc exec -it iscsi-pxdtd -n pxdtd bash
bash-4.2$ ls /mnt/iscsi/
hello  iscsi_testfile  lost+found
bash-4.2$ touch /mnt/iscsi/wehe
bash-4.2$ ls /mnt/iscsi/
hello  iscsi_testfile  lost+found  wehe
bash-4.2$ exit

Comment 10 errata-xmlrpc 2019-06-11 09:30:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0786