Bug 1818917

Summary: OSDs not starting after upgrade saying Nothing to read on input trying to open encrypted device
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasishta <vashastr>
Component: ContainerAssignee: Dimitri Savineau <dsavinea>
Status: CLOSED NOTABUG QA Contact: Vasishta <vashastr>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1CC: bniver, ceph-eng-bugs, gabrioux
Target Milestone: rcKeywords: Regression
Target Release: 4.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-30 19:11:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1793564, 1806033    

Description Vasishta 2020-03-30 17:19:12 UTC 
Description of problem:
Tried to upgrade a cluster from 3.3 latest build to 4.x latest using rolling_update. Playbook failed waiting for clean PGs.

Upon observing OSD logs, it appeared that OSD services were saying that Nothing to read on input when trying to open encypted device.


Version-Release number of selected component (if applicable):
ceph-4.1-rhel-8-containers-candidate-35792-20200328071423
ceph version 14.2.8-27.el8cp
ceph-ansible-4.0.17-1.el7.noarch

How reproducible:
Tried Once

Steps to Reproduce:
(What we tried)
1. Configured RHCS cluster using 2.5 live builds
2. Upgraded to 3.3 latest builds (ceph-3.3-rhel-7-containers-candidate-31439-20200312223408)
3. Added some nodes to the cluster
4. Upgraded cluster to 4.1 latest available build.

Actual results:
ceph-osd-run.sh[279395]: 2020-03-30 16:29:13  /opt/ceph-container/bin/entrypoint.sh: Opening encrypted device 03e49abb-3d9a-4541-9512-6a0f13249cc0
ceph-osd-run.sh[279395]: Nothing to read on input.


Expected results:
OSD services must be up and running 

Additional info:
1) Upon initial observations, have selected the component as container , if it is ceph-ansible, please change as required
2) As this is blocking basic usecase - upgrade of encrypted OSDs, have set severity to urgent, please feel free to reset if its not appropriate.
3) Adding the keyword 'regression' as upgrade of encrypted OSDs from 3.x to 4.x has worked before in 4.x.