Bug 1304526
| Summary: | iSCSI storage does not work after following the documentation | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Eric Jones <erjones> | ||||||||
| Component: | Storage | Assignee: | hchen | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Jianwei Hou <jhou> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 3.1.0 | CC: | aos-bugs, bchilds, erjones, hchen, lxia, misalunk, tdawson | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2016-05-12 16:27:44 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Eric Jones
2016-02-03 22:40:15 UTC
Do you have copy of syslog and kubelet (embedded in openshift) logs? Created attachment 1122441 [details]
atomic-openshift-node logs
Here are the atomic-openshift-node logs from the node where my docker-registry (the pod I am attempting to add my iscsi pvc to) runs, which should include the kublet logs. What other logs are you looking for?
Below is the errors related to iSCSI. Can you provide the Pod and PV/PVC you were using? If I can access your setup, that'll be great. Feb 09 10:47:32 node.test.com atomic-openshift-node[34948]: I0209 10:47:32.717088 34948 helpers.go:96] Unable to get network stats from pid 112926: couldn't read network stats: failure opening /proc/112926/net/dev: open /proc/112926/net/dev: no such file or directory Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441198 34948 disk_manager.go:49] failed to attach disk Feb 09 10:47:34 node.test.com atomic-openshift-node[34948]: E0209 10:47:34.441282 34948 iscsi.go:173] iscsi: failed to setup The pod I am adding it to is my docker-registry that is currently setup and running using a PV/PVC based on nfs.
└──> cat iscsipv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: iscsi-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
iscsi:
targetPortal: registry.vault.com
iqn: iqn.2016-02.com.vault:registry
lun: 1
fsType: 'ext4'
readOnly: false
└──> cat iscsipvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
iSCSI targetPortal has to be IP. Your PV uses hostname. Can you try again using IP? Created attachment 1122563 [details]
atomic-openshift-node logs after changing pv to use IP
└──> cat iscsipv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: iscsi-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
iscsi:
targetPortal: 10.3.11.159
iqn: iqn.2016-02.com.vault:registry
lun: 1
fsType: 'ext4'
readOnly: false
I changed it to the public IP for my env(its on OS1 openstack) I am testing now using the private IP
Created attachment 1122568 [details]
atomic-openshift-node logs after changing pv to use private IP
└──> cat iscsipv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: iscsi-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
iscsi:
targetPortal: 172.17.29.16
iqn: iqn.2016-02.com.vault:registry
lun: 1
fsType: 'ext4'
readOnly: false
this is the private IP address
Logs from private IP shows the problem is a bad mount. Is your iscsi lun formatted as ext4? Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: E0209 19:16:07.559423 34948 iscsi_util.go:118] iscsi: failed to mount iscsi volume /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 [ext4] to /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1, error Mount failed: exit status 32 Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Mounting arguments: /dev/disk/by-path/ip-172.17.29.16:3260-iscsi-iqn.2016-02.com.vault:registry-lun-1 /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/172.17.29.16:3260-iqn.2016-02.com.vault:registry-lun-1 ext4 [defaults] Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: Output: mount: wrong fs type, bad option, bad superblock on /dev/sda, Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: missing codepage or helper program, or other error Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: In some cases useful info is found in syslog - try Feb 09 19:16:07 node.test.com atomic-openshift-node[34948]: dmesg | tail or so. Potentially? I might have formatted it as xfs instead. Let me reformat it and try again. After reformatting it per [0] parted --script /dev/sdb "mklabel msdos" parted --script /dev/sdb "mkpart primary 0% 100%" mkfs.ext4 /dev/sdb1 And I was able to use it as a viable pvc for my docker-registry. Thank you for the help with that. I might've eventually caught the format, but I was unaware that the targetPortal _had_ to be an IP address. [0] https://access.redhat.com/solutions/2149241 target Portal is IP:Port, as defined by SNIA. Based on comment #14, iSCSI storage can be used. Move bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:1064 |