Description of problem:
Tried to upgrade a cluster from 3.3 latest build to 4.x latest using rolling_update. Playbook failed waiting for clean PGs.

Upon observing OSD logs, it appeared that OSD services were saying that Nothing to read on input when trying to open encypted device.


Version-Release number of selected component (if applicable):
ceph-4.1-rhel-8-containers-candidate-35792-20200328071423
ceph version 14.2.8-27.el8cp
ceph-ansible-4.0.17-1.el7.noarch

How reproducible:
Tried Once

Steps to Reproduce:
(What we tried)
1. Configured RHCS cluster using 2.5 live builds
2. Upgraded to 3.3 latest builds (ceph-3.3-rhel-7-containers-candidate-31439-20200312223408)
3. Added some nodes to the cluster
4. Upgraded cluster to 4.1 latest available build.

Actual results:
ceph-osd-run.sh[279395]: 2020-03-30 16:29:13  /opt/ceph-container/bin/entrypoint.sh: Opening encrypted device 03e49abb-3d9a-4541-9512-6a0f13249cc0
ceph-osd-run.sh[279395]: Nothing to read on input.


Expected results:
OSD services must be up and running 

Additional info:
1) Upon initial observations, have selected the component as container , if it is ceph-ansible, please change as required
2) As this is blocking basic usecase - upgrade of encrypted OSDs, have set severity to urgent, please feel free to reset if its not appropriate.
3) Adding the keyword 'regression' as upgrade of encrypted OSDs from 3.x to 4.x has worked before in 4.x.