Bug 2303177

This bug was initially created as a copy of Bug #2297265

I am copying this bug because: 



Description of problem (please be detailed as possible and provide log
snippests):


after upgrade from 4.15.15 to 4.15.18 image registry pods are stuck at "container creating". 

Error in event log:

(combined from similar events): MountVolume.MountDevice failed for volume "pvc-c795a19b-07e7-4594-b998-6f41e537f65d" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 172.30.176.65:3300,172.30.143.191:3300,172.30.124.83:3300:/volumes/csi/csi-vol-393bd202-3750-4d53-bae1-89916bfed53a/3f03bc85-670b-4fd9-9a71-6e24f2d11f5b /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/3c19db5ad93b22a0a40c7b2207f20aa1122367773bd729ef7050476308d75f49/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-1623855236,mds_namespace=ocs-storagecluster-cephfilesystem,ms_mode=prefer-crc,read_from_replica=localize,crush_location=host:srv-ocdev03-demo-smartis-si|zone:,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-07-05T08:16:22.770+0000 7f9af3ce8ec0 -1 failed for service _ceph-mon._tcp mount error 22 = Invalid argument


NAME                                               READY   STATUS              RESTARTS   AGE     IP             NODE                          NOMINATED NODE   READINESS GATES
image-pruner-28670400-ds9vj                        0/1     Error               0          2d6h    10.130.0.247   srv-ocdev03.demo.smartis.si   <none>           <none>
image-pruner-28671840-jjcmj                        0/1     Error               0          30h     10.130.0.106   srv-ocdev03.demo.smartis.si   <none>           <none>
image-pruner-28673280-4x2kn                        0/1     Error               0          6h19m   10.130.1.191   srv-ocdev03.demo.smartis.si   <none>           <none>
image-registry-7bb46cc7f8-c958t                    0/1     ContainerCreating   0          2d23h   <none>         srv-ocdev03.demo.smartis.si   <none>           <none>
image-registry-7bb46cc7f8-qwpzn                    0/1     ContainerCreating   0          2d23h   <none>         srv-ocdev01.demo.smartis.si   <none>           <none>




Version of all relevant components (if applicable):

ODF 4.15.15 

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

image-registry pods are unable to start



Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
N/A

Can this issue reproducible?
N/A

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
upgrade from 4.15.15 to 4.15.18  


Actual results:
image registry pods are stuck at "container creating". 


Expected results:
image registry pods should be running


Additional info:
possibly related BZ
There was an OpenShift Data Foundation bug[1] on this that was fixed in ODF 4.15.0 [2]
looks like this should have been fixed in 4.15.0 and they are on v4.15.3
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2265514
[2] https://access.redhat.com/errata/RHSA-2024:1383