Bug 1304526 - iSCSI storage does not work after following the documentation
iSCSI storage does not work after following the documentation
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: hchen
Jianwei Hou
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-03 17:40 EST by Eric Jones
Modified: 2016-05-12 12:27 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 12:27:44 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
atomic-openshift-node logs (23.79 KB, text/plain)
2016-02-09 10:52 EST, Eric Jones
no flags Details
atomic-openshift-node logs after changing pv to use IP (17.13 KB, text/plain)
2016-02-09 19:13 EST, Eric Jones
no flags Details
atomic-openshift-node logs after changing pv to use private IP (1.44 MB, text/x-vhdl)
2016-02-09 19:17 EST, Eric Jones
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:1064 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.2 security, bug fix, and enhancement update 2016-05-12 16:19:17 EDT

  None (edit)
Description Eric Jones 2016-02-03 17:40:15 EST
Describe the issue: 
After following the documentation, and confirming that the iSCSI storage can be attached to all of the nodes in the cluster, I am unable to use the PVC for storage for my previously working registry. A Customer is experiencing the same issue.

Version-Release number of selected component (if applicable):
OpenShift 3.1

How reproducible:
100%

Steps to Reproduce:
1. Follow the documentation [0] to setup a pv/pvc using iSCSI
2. Run the following command to attach the successful pvc to the registry `oc volume dc/docker-registry --add  --name=iscsi-thing -t pvc --claim-name=registry-pvc --overwrite`

[0] https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/installation-and-configuration/#using-iscsi

Actual results:
Container fails to create and volume cannot attach

Expected results:
The docker-registry container creates properly after attaching the iSCSI volume

Additional info:
Comment 2 hchen 2016-02-04 09:03:25 EST
Do you have copy of syslog and kubelet (embedded in openshift) logs?
Comment 3 Eric Jones 2016-02-09 10:52 EST
Created attachment 1122441 [details]
atomic-openshift-node logs

Here are the atomic-openshift-node logs from the node where my docker-registry (the pod I am attempting to add my iscsi pvc to) runs, which should include the kublet logs. What other logs are you looking for?
Comment 5 hchen 2016-02-09 13:19:59 EST
Below is the errors related to iSCSI. Can you provide the Pod and PV/PVC you were using? If I can access your setup, that'll be great.


Feb 09 10:47:32 node.test.com atomic-openshift-node[34948]: I0209 10:47:32.717088   34948 helpers.go:96] Unable to get network stats from pid 112926: couldn't read network stats: failure opening /proc/112926/net/dev: open /proc/112926/net/dev: no such file or directory
Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441198   34948 disk_manager.go:49] failed to attach disk
Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441282   34948 iscsi.go:173] iscsi: failed to setup
Comment 7 Eric Jones 2016-02-09 18:07:29 EST
The pod I am adding it to is my docker-registry that is currently setup and running using a PV/PVC based on nfs.

└──> cat iscsipv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  iscsi:
    targetPortal: registry.vault.com
    iqn: iqn.2016-02.com.vault:registry
    lun: 1
    fsType: 'ext4'
    readOnly: false

└──> cat iscsipvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
Comment 8 hchen 2016-02-09 18:47:01 EST
iSCSI targetPortal has to be IP. Your PV uses hostname. Can you try again using IP?
Comment 9 Eric Jones 2016-02-09 19:13 EST
Created attachment 1122563 [details]
atomic-openshift-node logs after changing pv to use IP

└──> cat iscsipv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  iscsi:
    targetPortal: 10.3.11.159
    iqn: iqn.2016-02.com.vault:registry
    lun: 1
    fsType: 'ext4'
    readOnly: false


I changed it to the public IP for my env(its on OS1 openstack) I am testing now using the private IP
Comment 10 Eric Jones 2016-02-09 19:17 EST
Created attachment 1122568 [details]
atomic-openshift-node logs after changing pv to use private IP

└──> cat iscsipv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  iscsi:
    targetPortal: 172.17.29.16
    iqn: iqn.2016-02.com.vault:registry
    lun: 1
    fsType: 'ext4'
    readOnly: false

this is the private IP address
Comment 12 hchen 2016-02-09 19:41:58 EST
Logs from private IP shows the problem is a bad mount. Is your iscsi lun formatted as ext4?

Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: E0209 19:16:07.559423   34948 iscsi_util.go:118] iscsi: failed to mount iscsi volume /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 [ext4] to /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1, error Mount failed: exit status 32
Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Mounting arguments: /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1 ext4 [defaults]
Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Output: mount: wrong fs type, bad option, bad superblock on /dev/sda,
Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: missing codepage or helper program, or other error
Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: In some cases useful info is found in syslog - try
Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: dmesg | tail or so.
Comment 13 Eric Jones 2016-02-10 15:16:33 EST
Potentially? I might have formatted it as xfs instead.
Let me reformat it and try again.
Comment 14 Eric Jones 2016-02-10 15:45:50 EST
After reformatting it per [0] 

parted --script /dev/sdb "mklabel msdos"
parted --script /dev/sdb "mkpart primary 0% 100%"
mkfs.ext4 /dev/sdb1

And I was able to use it as a viable pvc for my docker-registry. Thank you for the help with that. I might've eventually caught the format, but I was unaware that the targetPortal _had_ to be an IP address.

[0] https://access.redhat.com/solutions/2149241
Comment 16 hchen 2016-02-12 11:44:32 EST
target Portal is IP:Port, as defined by SNIA.
Comment 17 Liang Xia 2016-02-15 22:09:02 EST
Based on comment #14, iSCSI storage can be used.

Move bug to verified.
Comment 19 errata-xmlrpc 2016-05-12 12:27:44 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064

Note You need to log in before you can comment on or make changes to this bug.