Description of problem: Set up OCP with atomic host, trying to use iscsi in the pod. Pod failed with below error: Warning FailedMount 10s (x6 over 26s) kubelet, qe-lxia-310node-2 MountVolume.WaitForAttach failed for volume "pv-iscsi-zojca" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 172.30.109.235:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.30.109.235,3260 err 30 iscsiadm: Could not open /var/lib/iscsi/send_targets/172.30.109.235,3260: Read-only file system iscsiadm: Could not add new discovery record. , err exit status 6 Version-Release number of selected component (if applicable): oc v3.10.119 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO # cat /etc/redhat-release Red Hat Enterprise Linux Atomic Host release 7.5 # cat /etc/os-release NAME="Red Hat Enterprise Linux Atomic Host" VERSION="7.5.3" ID="rhel" ID_LIKE="fedora" VARIANT="Atomic Host" VARIANT_ID=atomic.host VERSION_ID="7.5" PRETTY_NAME="Red Hat Enterprise Linux Atomic Host 7.5.3" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:atomic-host" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION="7.5" REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.5" OSTREE_VERSION=7.5.3 How reproducible: Always Steps to Reproduce: 1. Setup OCP with RHEL atomic host. 2. Try to create a pod with iscsi. 3. Check the pod. Actual results: # oc get pod -n zojca NAME READY STATUS RESTARTS AGE iscsi-zojca 0/1 ContainerCreating 0 3m # oc describe iscsi-zojca pod -n zojca error: the server doesn't have a resource type "iscsi-zojca" [root@qe-lxia-310master-etcd-1 ~]# oc describe pod iscsi-zojca -n zojca Name: iscsi-zojca Namespace: zojca Node: qe-lxia-310node-2/10.240.0.181 Start Time: Thu, 07 Mar 2019 09:40:36 +0000 Labels: name=iscsi Annotations: openshift.io/scc=privileged Status: Pending IP: Containers: iscsi: Container ID: Image: jhou/hello-openshift Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /mnt/iscsi from iscsi (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-gfbf8 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: iscsi: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc-iscsi-zojca ReadOnly: false default-token-gfbf8: Type: Secret (a volume populated by a Secret) SecretName: default-token-gfbf8 Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/compute=true Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m default-scheduler Successfully assigned iscsi-zojca to qe-lxia-310node-2 Normal SuccessfulAttachVolume 4m attachdetach-controller AttachVolume.Attach succeeded for volume "pv-iscsi-zojca" Warning FailedMount 2m kubelet, qe-lxia-310node-2 Unable to mount volumes for pod "iscsi-zojca_zojca(0f99db82-40bd-11e9-ba5a-42010af000b2)": timeout expired waiting for volumes to attach or mount for pod "zojca"/"iscsi-zojca". list of unmounted volumes=[iscsi]. list of unattached volumes=[iscsi default-token-gfbf8] Warning FailedMount 1m (x9 over 4m) kubelet, qe-lxia-310node-2 MountVolume.WaitForAttach failed for volume "pv-iscsi-zojca" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 172.30.109.235:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.30.109.235,3260 err 30 iscsiadm: Could not open /var/lib/iscsi/send_targets/172.30.109.235,3260: Read-only file system iscsiadm: Could not add new discovery record. , err exit status 6 Expected results: Pod is up and running.
The issue exist on atomic host + system container, but does not exist on atomic host + docker container.
Related bug, https://bugzilla.redhat.com/show_bug.cgi?id=1598271
Filled https://github.com/openshift/origin/pull/22289 In the meantime, can you please check that it really helps? Follow https://bugzilla.redhat.com/show_bug.cgi?id=1598271#c1 and report any issues. Jenkins has failed to give me 3.10 on Atomic Host.
PR https://github.com/openshift/origin/pull/22289 merged.
Build is ready, Changelog * Mon Mar 11 2019 AOS Automation Release Team <aos-team-art> 3.10.125-1 - Bindmount /var/lib/iscsi rw for iscsi attach (mawong)
Tested on below version: openshift v3.10.127 kubernetes v1.10.0+b81c8f8 # uname -a Linux ip-172-18-6-162.ec2.internal 3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10 16:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release Red Hat Enterprise Linux Atomic Host release 7.5 ISCSI works well. # oc get pods -n pxdtd NAME READY STATUS RESTARTS AGE iscsi-pxdtd 1/1 Running 0 16m # oc exec -it iscsi-pxdtd -n pxdtd bash bash-4.2$ ls /mnt/iscsi/ hello iscsi_testfile lost+found bash-4.2$ touch /mnt/iscsi/wehe bash-4.2$ ls /mnt/iscsi/ hello iscsi_testfile lost+found wehe bash-4.2$ exit
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0786