Hide Forgot
Description of problem: Set up OCP with atomic host, trying to use iscsi in the pod. Pod failed with below error: Warning FailedMount 5s (x7 over 37s) kubelet, ip-172-18-19-181.ec2.internal MountVolume.WaitForAttach failed for volume "pv-iscsi-j4ekr" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 172.31.223.41:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.31.223.41,3260 err 30 iscsiadm: Could not open /var/lib/iscsi/send_targets/172.31.223.41,3260: Read-only file system iscsiadm: Could not add new discovery record. , err exit status 6 Version-Release number of selected component (if applicable): oc v3.9.71 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO # cat /etc/redhat-release Red Hat Enterprise Linux Atomic Host release 7.5 # cat /etc/os-release NAME="Red Hat Enterprise Linux Atomic Host" VERSION="7.1" ID="rhel" ID_LIKE="fedora" VERSION_ID="7.1" PRETTY_NAME="Employee SKU" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.1:GA:atomic-host" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.1 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION=7.1 How reproducible: Always Steps to Reproduce: 1. Setup OCP with RHEL atomic host. 2. Try to create a pod with iscsi. 3. Check the pod. Actual results: $ oc get pods NAME READY STATUS RESTARTS AGE iscsi-j4ekr 0/1 ContainerCreating 0 37s $ oc describe pod iscsi-j4ekr Name: iscsi-j4ekr Namespace: j4ekr Node: ip-172-18-19-181.ec2.internal/172.18.19.181 Start Time: Thu, 07 Mar 2019 14:05:15 +0800 Labels: name=iscsi Annotations: openshift.io/scc: privileged Status: Pending IP: Containers: iscsi: Container ID: Image: jhou/hello-openshift Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /mnt/iscsi from iscsi (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-k6jjc (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: iscsi: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc-iscsi-j4ekr ReadOnly: false default-token-k6jjc: Type: Secret (a volume populated by a Secret) SecretName: default-token-k6jjc Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/compute=true Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 45s default-scheduler Successfully assigned iscsi-j4ekr to ip-172-18-19-181.ec2.internal Normal SuccessfulMountVolume 44s kubelet, ip-172-18-19-181.ec2.internal MountVolume.SetUp succeeded for volume "default-token-k6jjc" Warning FailedMount 5s (x7 over 37s) kubelet, ip-172-18-19-181.ec2.internal MountVolume.WaitForAttach failed for volume "pv-iscsi-j4ekr" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 172.31.223.41:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.31.223.41,3260 err 30 iscsiadm: Could not open /var/lib/iscsi/send_targets/172.31.223.41,3260: Read-only file system iscsiadm: Could not add new discovery record. , err exit status 6 Expected results: Pod is up and running.
Related bug, https://bugzilla.redhat.com/show_bug.cgi?id=1598271
The issue exist on atomic host + system container, but does not exist on atomic host + docker container.
I am not sure if it's installer or container runtime fault, please reassign as necessary. "/" is read-only in atomic-openshift-node container if it runs as system container (/usr/bin/runc --systemd-cgroup run 'atomic-openshift-node'). $ runc exec atomic-openshift-node sh (inside the container)$ mount /dev/mapper/rhel-root on / type xfs (ro,relatime,seclabel,attr2,inode64,noquota) (inside the container)$ touch /var/lib/iscsi/foo touch: cannot touch '/var/lib/iscsi/foo': Read-only file system iscsiadm requires at least /var/lib/iscsi to be writeable. When running atomic-openshift-node as docker container (=/usr/bin/docker run --name atomic-openshift-node ...), "/" is writeable there.
This is just https://github.com/openshift/origin/pull/22289 needing to be back ported to 3.9, right?
> This is just https://github.com/openshift/origin/pull/22289 needing to be back ported to 3.9, right? Yes, that's it.
https://github.com/openshift/ose/pull/1516
Verified the issue has been fixed. # oc version oc v3.9.77 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-10-24.ec2.internal:8443 openshift v3.9.77 kubernetes v1.9.1+a0ce1bc657 # oc describe pod iscsi-1-ultza -n ultza Name: iscsi-1-ultza Namespace: ultza Node: ip-172-18-4-22.ec2.internal/172.18.4.22 Start Time: Wed, 17 Apr 2019 03:42:38 -0400 Labels: <none> Annotations: openshift.io/scc=privileged Status: Running IP: 10.129.0.15 Containers: iscsi: Container ID: docker://f8b590498bd92e7a8ed8b158b06ab352061388ed110de12e6bb68289f1b98f57 Image: aosqe/hello-openshift Image ID: docker-pullable://docker.io/aosqe/hello-openshift@sha256:a2d509d3d5164f54a2406287405b2d114f952dca877cc465129f78afa858b31a Port: <none> State: Running Started: Wed, 17 Apr 2019 03:51:16 -0400 Ready: True Restart Count: 0 Environment: <none> Mounts: /mnt/iscsi from iscsi (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-nqdzl (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: iscsi: Type: ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod) TargetPortal: 172.30.235.169:3260 IQN: iqn.2016-04.test.com:storage.target00 Lun: 0 ISCSIInterface default FSType: ext4 ReadOnly: true Portals: [172.30.235.169:3260 172.30.87.244:3260] DiscoveryCHAPAuth: false SessionCHAPAuth: false SecretRef: <nil> InitiatorName: <none> default-token-nqdzl: Type: Secret (a volume populated by a Secret) SecretName: default-token-nqdzl Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/compute=true Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m default-scheduler Successfully assigned iscsi-1-ultza to ip-172-18-4-22.ec2.internal Normal SuccessfulMountVolume 9m kubelet, ip-172-18-4-22.ec2.internal MountVolume.SetUp succeeded for volume "default-token-nqdzl" Normal SuccessfulMountVolume 30s kubelet, ip-172-18-4-22.ec2.internal MountVolume.SetUp succeeded for volume "iscsi" Normal Pulling 27s kubelet, ip-172-18-4-22.ec2.internal pulling image "aosqe/hello-openshift" Normal Pulled 25s kubelet, ip-172-18-4-22.ec2.internal Successfully pulled image "aosqe/hello-openshift" Normal Created 25s kubelet, ip-172-18-4-22.ec2.internal Created container Normal Started 25s kubelet, ip-172-18-4-22.ec2.internal Started container
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0788