Bug 1686266 - [3.9] Failed to mount iscsi on atomic host
Summary: [3.9] Failed to mount iscsi on atomic host
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.9.z
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1686336
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-07 06:35 UTC by Liang Xia
Modified: 2019-06-06 06:56 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The node system container did not properly mount /var/lib/iscsi rw, now it does avoiding problems mounting iscsi volumes.
Clone Of:
Environment:
Last Closed: 2019-06-06 06:56:05 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0788 None None None 2019-06-06 06:56:14 UTC

Description Liang Xia 2019-03-07 06:35:40 UTC
Description of problem:
Set up OCP with atomic host, trying to use iscsi in the pod.
Pod failed with below error:
  Warning  FailedMount            5s (x7 over 37s)  kubelet, ip-172-18-19-181.ec2.internal  MountVolume.WaitForAttach failed for volume "pv-iscsi-j4ekr" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 172.31.223.41:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.31.223.41,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/172.31.223.41,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6


Version-Release number of selected component (if applicable):
oc v3.9.71
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

# cat /etc/os-release 
NAME="Red Hat Enterprise Linux Atomic Host"
VERSION="7.1"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.1"
PRETTY_NAME="Employee SKU"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.1:GA:atomic-host"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION=7.1


How reproducible:
Always

Steps to Reproduce:
1. Setup OCP with RHEL atomic host.
2. Try to create a pod with iscsi.
3. Check the pod.

Actual results:
$ oc get pods
NAME          READY   STATUS              RESTARTS   AGE
iscsi-j4ekr   0/1     ContainerCreating   0          37s

$ oc describe pod iscsi-j4ekr 
Name:         iscsi-j4ekr
Namespace:    j4ekr
Node:         ip-172-18-19-181.ec2.internal/172.18.19.181
Start Time:   Thu, 07 Mar 2019 14:05:15 +0800
Labels:       name=iscsi
Annotations:  openshift.io/scc: privileged
Status:       Pending
IP:           
Containers:
  iscsi:
    Container ID:   
    Image:          jhou/hello-openshift
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/iscsi from iscsi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-k6jjc (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  iscsi:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-iscsi-j4ekr
    ReadOnly:   false
  default-token-k6jjc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-k6jjc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     <none>
Events:
  Type     Reason                 Age               From                                    Message
  ----     ------                 ----              ----                                    -------
  Normal   Scheduled              45s               default-scheduler                       Successfully assigned iscsi-j4ekr to ip-172-18-19-181.ec2.internal
  Normal   SuccessfulMountVolume  44s               kubelet, ip-172-18-19-181.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-k6jjc"
  Warning  FailedMount            5s (x7 over 37s)  kubelet, ip-172-18-19-181.ec2.internal  MountVolume.WaitForAttach failed for volume "pv-iscsi-j4ekr" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 172.31.223.41:3260 output: iscsiadm: Could not make dir /var/lib/iscsi/send_targets/172.31.223.41,3260 err 30
iscsiadm: Could not open /var/lib/iscsi/send_targets/172.31.223.41,3260: Read-only file system
iscsiadm: Could not add new discovery record.
, err exit status 6


Expected results:
Pod is up and running.

Comment 1 Liang Xia 2019-03-07 06:54:44 UTC
Related bug, https://bugzilla.redhat.com/show_bug.cgi?id=1598271

Comment 2 Liang Xia 2019-03-07 09:51:55 UTC
The issue exist on atomic host + system container, but does not exist on atomic host + docker container.

Comment 3 Jan Safranek 2019-03-29 13:01:01 UTC
I am not sure if it's installer or container runtime fault, please reassign as necessary.

"/" is read-only in atomic-openshift-node container if it runs as system container (/usr/bin/runc --systemd-cgroup run 'atomic-openshift-node').

    $ runc exec atomic-openshift-node sh
    (inside the container)$ mount
    /dev/mapper/rhel-root on / type xfs (ro,relatime,seclabel,attr2,inode64,noquota)

    (inside the container)$ touch /var/lib/iscsi/foo
    touch: cannot touch '/var/lib/iscsi/foo': Read-only file system

iscsiadm requires at least /var/lib/iscsi to be writeable.

When running atomic-openshift-node as docker container (=/usr/bin/docker run --name atomic-openshift-node ...), "/" is writeable there.

Comment 4 Scott Dodson 2019-03-29 13:22:24 UTC
This is just https://github.com/openshift/origin/pull/22289 needing to be back ported to 3.9, right?

Comment 5 Jan Safranek 2019-04-01 07:52:14 UTC
> This is just https://github.com/openshift/origin/pull/22289 needing to be back ported to 3.9, right?

Yes, that's it.

Comment 6 Scott Dodson 2019-04-01 12:27:39 UTC
https://github.com/openshift/ose/pull/1516

Comment 8 Liang Xia 2019-04-17 07:59:13 UTC
Verified the issue has been fixed.

# oc version
oc v3.9.77
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-10-24.ec2.internal:8443
openshift v3.9.77
kubernetes v1.9.1+a0ce1bc657


# oc describe pod iscsi-1-ultza -n ultza
Name:         iscsi-1-ultza
Namespace:    ultza
Node:         ip-172-18-4-22.ec2.internal/172.18.4.22
Start Time:   Wed, 17 Apr 2019 03:42:38 -0400
Labels:       <none>
Annotations:  openshift.io/scc=privileged
Status:       Running
IP:           10.129.0.15
Containers:
  iscsi:
    Container ID:   docker://f8b590498bd92e7a8ed8b158b06ab352061388ed110de12e6bb68289f1b98f57
    Image:          aosqe/hello-openshift
    Image ID:       docker-pullable://docker.io/aosqe/hello-openshift@sha256:a2d509d3d5164f54a2406287405b2d114f952dca877cc465129f78afa858b31a
    Port:           <none>
    State:          Running
      Started:      Wed, 17 Apr 2019 03:51:16 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/iscsi from iscsi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nqdzl (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  iscsi:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       172.30.235.169:3260
    IQN:                iqn.2016-04.test.com:storage.target00
    Lun:                0
    ISCSIInterface      default
    FSType:             ext4
    ReadOnly:           true
    Portals:            [172.30.235.169:3260 172.30.87.244:3260]
    DiscoveryCHAPAuth:  false
    SessionCHAPAuth:    false
    SecretRef:          <nil>
    InitiatorName:      <none>
  default-token-nqdzl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nqdzl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     <none>
Events:
  Type     Reason                 Age               From                                  Message
  ----     ------                 ----              ----                                  -------
  Normal   Scheduled              9m                default-scheduler                     Successfully assigned iscsi-1-ultza to ip-172-18-4-22.ec2.internal
  Normal   SuccessfulMountVolume  9m                kubelet, ip-172-18-4-22.ec2.internal  MountVolume.SetUp succeeded for volume "default-token-nqdzl"
  Normal   SuccessfulMountVolume  30s               kubelet, ip-172-18-4-22.ec2.internal  MountVolume.SetUp succeeded for volume "iscsi"
  Normal   Pulling                27s               kubelet, ip-172-18-4-22.ec2.internal  pulling image "aosqe/hello-openshift"
  Normal   Pulled                 25s               kubelet, ip-172-18-4-22.ec2.internal  Successfully pulled image "aosqe/hello-openshift"
  Normal   Created                25s               kubelet, ip-172-18-4-22.ec2.internal  Created container
  Normal   Started                25s               kubelet, ip-172-18-4-22.ec2.internal  Started container

Comment 10 errata-xmlrpc 2019-06-06 06:56:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0788


Note You need to log in before you can comment on or make changes to this bug.