Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1937947

Summary:	local storage operator tries to use Ceph RBD volumes - definitely not local storage!
Product:	OpenShift Container Platform	Reporter:	Ben England <bengland>
Component:	Storage	Assignee:	Rohan CJ <rojoseph>
Storage sub component:	Local Storage Operator	QA Contact:	Qin Ping <piqin>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	hekumar
Version:	4.6.z
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-18 09:05:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ben England 2021-03-11 19:01:16 UTC

Description of problem:

I was trying to use the GUI to add local storage devices to create a new LSO storage class. This kept failing. I realized that the devices were already in the LSO storage class. I then went into /mnt/local-storage to see what was there and I noticed that Ceph RBD devices were also in there. I think I tried out the "discovery" feature and this must have accidentally included RBD devices, which is really bad. This could lead to circular dependencies that could break the system. You want to have some filters that exclude devices like this somehow.

Version-Release number of selected component (if applicable):

Openshift 4.6.16
OCS 4.6.3
I0311 18:39:19.523045 1 diskmaker.go:29] local-storage-diskmaker Version: v4.6.0-202103010126.p0-0-g884588f-dirty
Red Hat Enterprise Linux CoreOS release 4.6
Linux f21-h01-000-r640.rdu2.scalelab.redhat.com 4.18.0-193.41.1.el8_2.x86_64 #1 SMP Wed Jan 13 11:33:33 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:

not sure yet

Steps to Reproduce:
1. install OCP 4.6
2. install LSO
3. create a storage class with GUI using LocalVolumeSet and devices of type SSD
4. do a "local Volume Discovery"

Actual results:

RBD devices were included in the "localblock" LVS apparently.

Expected results:

RBD devices should not be included.

Master Log:

how do I get that?

Node Log (of failed PODs):

no pods failed in any obvious sense.

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

[kni@f20-h25-000-r640 ~]$ oc debug node/f21-h01-000-r640.rdu2.scalelab.redhat.com
Creating debug namespace/openshift-debug-node-rnrvc ...
Starting pod/f21-h01-000-r640rdu2scalelabredhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.216.13
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# cd /mnt/local-storage/localblock/
nvme-Dell_Express_Flash_PM1725b_3.2TB_SFF_S5CXNA0N500313 rbd3
rbd1 rbd4p3
rbd10 rbd5
rbd11 rbd6p3
rbd12p3 rbd7
rbd13p3 rbd8p3
rbd14 rbd9p3
rbd15 scsi-362cea7f061673900272194032055ac16
rbd16p3 scsi-362cea7f061673900272194062606449b
rbd2p3

Comment 1 Ben England 2021-03-11 19:49:42 UTC

must-gather data from the cluster is here, forgot to attach URL

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/tmp/ms-etcd/must-gather.local.8808778692104155017.tgz

Comment 2 Ben England 2021-03-11 19:52:49 UTC

opened up the bz to public so Arvin could see it.

Comment 3 Hemant Kumar 2021-03-12 16:26:05 UTC

Can you post output of following command from the node where this happened?



lsblk --pairs -b -o NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,KNAME,SERIAL,PARTLABEL

Comment 4 Hemant Kumar 2021-03-16 19:24:09 UTC

Moving this to Rohan. But we still need that lsblk output, otherwise bug could be closed without inactivity. I do not think must-gather logs will have anything useful that is related to lsblk.

Comment 5 Rohan CJ 2021-03-18 09:05:05 UTC

This is almost certainly a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1937947

The backport has been merged. To cleanup, use remove the symlinks to rbd devices in /mnt/local and remove the pvs with .spec.local.path=/mnt/local/rbd*

Backport BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1937947

*** This bug has been marked as a duplicate of bug 1931766 ***

Comment 6 Rohan CJ 2021-03-18 09:07:10 UTC

Sorry, pasted this bugs url instead of the links, but marked dupllicate to the right bug.

Comment 7 Red Hat Bugzilla 2023-09-15 01:03:15 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days