Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1669965

Summary:	[3.9]FC volume in used by a node is not updated or appears twice in VolumeInUse
Product:	OpenShift Container Platform	Reporter:	Franck Grosjean <fgrosjea>
Component:	Storage	Assignee:	Jan Safranek <jsafrane>
Status:	CLOSED ERRATA	QA Contact:	Liang Xia <lxia>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, aos-storage-staff, fbertina, fgrosjea, jsafrane, lxia, sponnaga, tsmetana
Target Milestone:	---
Target Release:	3.9.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1679117 (view as bug list)		Environment:
Last Closed:	2019-04-09 14:20:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1679117

Description Franck Grosjean 2019-01-28 08:56:54 UTC

Description of problem:

Since errata tracked in this BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1612783
Information like VolumeInUse on node are not update properly (FC volume appears twice)
Kubelet tries to umount the same volume twice and failed because volume is already umount.
Pods using FC are not rescheduled because FC volume is still mount from a kubelet point of view

Version-Release number of selected component (if applicable):

3.9 with errata RHBA-2018:3748
https://access.redhat.com/errata/RHBA-2018:3748
https://bugzilla.redhat.com/show_bug.cgi?id=1612783

How reproducible:
3.9 with errata RHBA-2018:3748
PV/PVC with fiberchannel lun with multiple path
Drain the node

Actual results:
FC volume apprears twice in VolumeInUse
Kubelet tries to umount twice

Expected results:
Information on node about FC volume should be accurate

Comment 15 Liang Xia 2019-03-22 05:19:40 UTC

Checked on below version,
# oc version
oc v3.9.74
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://xxx-qe-cluster-xxx:8443
openshift v3.9.74
kubernetes v1.9.1+a0ce1bc657


Scenarios tested:
1. Create pod with fc, then drain node.
2. Create pod with fc, then scale down.
3. Create pod with fc, scale down, then scale up.

Then check the logs on the node,
[root@node1 lxia]# journalctl -u atomic-openshift-node | grep -i 'failed to detach'
[root@node1 lxia]# journalctl -u atomic-openshift-node | grep -i 'ConstructVolumeSpec'
Mar 22 01:07:23 xxx-qe-cluster-node1-xxx atomic-openshift-node[18940]: I0322 01:07:23.828142   18940 fc.go:265] ConstructVolumeSpec: TargetWWNs: [50050768030539b6 50050768030539b7], Lun: 0, WWIDs: []
Mar 22 01:10:23 xxx-qe-cluster-node1-xxx atomic-openshift-node[18940]: I0322 01:10:23.841256   18940 fc.go:265] ConstructVolumeSpec: TargetWWNs: [50050768030539b6 50050768030539b7], Lun: 0, WWIDs: []

[root@node2 ~]# journalctl -u atomic-openshift-node | grep -i 'failed to detach'
[root@node2 ~]# journalctl -u atomic-openshift-node | grep -i 'ConstructVolumeSpec'
Mar 22 00:55:23 xxx-qe-cluster-node2-xxx atomic-openshift-node[5467]: I0322 00:55:23.013415    5467 fc.go:265] ConstructVolumeSpec: TargetWWNs: [50050768030539b6 50050768030539b7], Lun: 0, WWIDs: []
Mar 22 00:58:23 xxx-qe-cluster-node2-xxx atomic-openshift-node[5467]: I0322 00:58:23.020057    5467 fc.go:265] ConstructVolumeSpec: TargetWWNs: [50050768030539b6 50050768030539b7], Lun: 0, WWIDs: []
Mar 22 01:01:23 xxx-qe-cluster-node2-xxx atomic-openshift-node[5467]: I0322 01:01:23.034351    5467 fc.go:265] ConstructVolumeSpec: TargetWWNs: [50050768030539b6 50050768030539b7], Lun: 0, WWIDs: []

Comment 17 errata-xmlrpc 2019-04-09 14:20:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0619