Describe the issue: After following the documentation, and confirming that the iSCSI storage can be attached to all of the nodes in the cluster, I am unable to use the PVC for storage for my previously working registry. A Customer is experiencing the same issue. Version-Release number of selected component (if applicable): OpenShift 3.1 How reproducible: 100% Steps to Reproduce: 1. Follow the documentation [0] to setup a pv/pvc using iSCSI 2. Run the following command to attach the successful pvc to the registry `oc volume dc/docker-registry --add --name=iscsi-thing -t pvc --claim-name=registry-pvc --overwrite` [0] https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/installation-and-configuration/#using-iscsi Actual results: Container fails to create and volume cannot attach Expected results: The docker-registry container creates properly after attaching the iSCSI volume Additional info:
Do you have copy of syslog and kubelet (embedded in openshift) logs?
Created attachment 1122441 [details] atomic-openshift-node logs Here are the atomic-openshift-node logs from the node where my docker-registry (the pod I am attempting to add my iscsi pvc to) runs, which should include the kublet logs. What other logs are you looking for?
Below is the errors related to iSCSI. Can you provide the Pod and PV/PVC you were using? If I can access your setup, that'll be great. Feb 09 10:47:32 node.test.com atomic-openshift-node[34948]: I0209 10:47:32.717088 34948 helpers.go:96] Unable to get network stats from pid 112926: couldn't read network stats: failure opening /proc/112926/net/dev: open /proc/112926/net/dev: no such file or directory Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441198 34948 disk_manager.go:49] failed to attach disk Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441282 34948 iscsi.go:173] iscsi: failed to setup
The pod I am adding it to is my docker-registry that is currently setup and running using a PV/PVC based on nfs. └──> cat iscsipv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: iscsi-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce iscsi: targetPortal: registry.vault.com iqn: iqn.2016-02.com.vault:registry lun: 1 fsType: 'ext4' readOnly: false └──> cat iscsipvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: registry-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi
iSCSI targetPortal has to be IP. Your PV uses hostname. Can you try again using IP?
Created attachment 1122563 [details] atomic-openshift-node logs after changing pv to use IP └──> cat iscsipv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: iscsi-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce iscsi: targetPortal: 10.3.11.159 iqn: iqn.2016-02.com.vault:registry lun: 1 fsType: 'ext4' readOnly: false I changed it to the public IP for my env(its on OS1 openstack) I am testing now using the private IP
Created attachment 1122568 [details] atomic-openshift-node logs after changing pv to use private IP └──> cat iscsipv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: iscsi-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce iscsi: targetPortal: 172.17.29.16 iqn: iqn.2016-02.com.vault:registry lun: 1 fsType: 'ext4' readOnly: false this is the private IP address
Logs from private IP shows the problem is a bad mount. Is your iscsi lun formatted as ext4? Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: E0209 19:16:07.559423 34948 iscsi_util.go:118] iscsi: failed to mount iscsi volume /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 [ext4] to /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1, error Mount failed: exit status 32 Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Mounting arguments: /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1 ext4 [defaults] Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Output: mount: wrong fs type, bad option, bad superblock on /dev/sda, Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: missing codepage or helper program, or other error Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: In some cases useful info is found in syslog - try Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: dmesg | tail or so.
Potentially? I might have formatted it as xfs instead. Let me reformat it and try again.
After reformatting it per [0] parted --script /dev/sdb "mklabel msdos" parted --script /dev/sdb "mkpart primary 0% 100%" mkfs.ext4 /dev/sdb1 And I was able to use it as a viable pvc for my docker-registry. Thank you for the help with that. I might've eventually caught the format, but I was unaware that the targetPortal _had_ to be an IP address. [0] https://access.redhat.com/solutions/2149241
target Portal is IP:Port, as defined by SNIA.
Based on comment #14, iSCSI storage can be used. Move bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:1064