Bug 1885666

Summary: OSD failed to come up when scaled from 0 to 1
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Filip Balák <fbalak>
Component: rookAssignee: Travis Nielsen <tnielsen>
Status: CLOSED DUPLICATE QA Contact: Elad <ebenahar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.6CC: madam, ocs-bugs, pbalogh, shan, tnielsen
Target Milestone: ---Keywords: Automation
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-08 08:27:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Filip Balák 2020-10-06 16:11:41 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
When OSD deployment was scaled to 0 and after 11 minutes it was scaled back to 1, the OSD failed to come up and cluster remained in HEALTH_WARN 1 osds down; 1 host (1 osds) down; 1 rack (1 osds) down; Degraded data redundancy: 704/2112 objects degraded (33.333%), 88 pgs degraded, 176 pgs undersized

Version of all relevant components (if applicable):
ocs-operator.v4.6.0-111.ci
4.6.0-0.nightly-2020-10-03-051134


Can this issue reproducible?
I am not sure. This seems like a rare issue.


Steps to Reproduce:
1. Scale one of OSDs to 0.
$ oc -n openshift-storage scale --replicas=0 deployment/rook-ceph-osd-2
2. Wait 11 minutes.
3. Scale the OSD to 1.
$ oc -n openshift-storage scale --replicas=1 deployment/rook-ceph-osd-2
4. Check ceph health.
$ oc -n openshift-storage exec [rook-ceph-tools pod] -- ceph health

or run automated test [1]


Actual results:
OSD is down, Ceph is in HEALTH_WARN state.

Expected results:
OSD is up, Ceph is in HEALTH_OK state.


Additional info:
[1] https://github.com/red-hat-storage/ocs-ci/blob/master/tests/manage/monitoring/prometheus/test_deployment_status.py#L86
Comment 3 Travis Nielsen 2020-10-06 17:01:11 UTC 
@Filip I can't connect to the must-gather logs. Can you check the link again? We need to look at the osd logs to see if there was some error after the pod startup. 

Scaling up the osd pod has been many times in the past, so in any case I don't see this as a 4.6 blocker.
Comment 4 Petr Balogh 2020-10-07 07:15:16 UTC 
@Travis I've just checked and can access all the logs. Were you connected to VPN that time? If not, you have to be.
Can you please try again and let us know?

Thanks
Comment 5 Travis Nielsen 2020-10-07 20:10:14 UTC 
It's working now, not sure what happened yesterday even when I was on the VPN.

Are you only seeing this in an encrypted cluster?

In the OSD log [1] I see this message:

2020-10-06T00:39:20.814360264Z /dev/mapper/ocs-deviceset-0-data-0-g8l9q-block-dmcrypt is active.
2020-10-06T00:39:20.814360264Z Underlying device for crypt device ocs-deviceset-0-data-0-g8l9q-block-dmcrypt disappeared.
2020-10-06T00:39:20.814360264Z   type:    n/a
2020-10-06T00:39:20.814360264Z   cipher:  aes-xts-plain64
2020-10-06T00:39:20.814360264Z   keysize: 512 bits
2020-10-06T00:39:20.814360264Z   key location: dm-crypt
2020-10-06T00:39:20.814360264Z   device:  (null)
2020-10-06T00:39:20.814360264Z   sector size:  512
2020-10-06T00:39:20.814360264Z   offset:  32768 sectors
2020-10-06T00:39:20.814360264Z   size:    1073709056 sectors
2020-10-06T00:39:20.814360264Z   mode:    read/write
2020-10-06T00:39:20.814360264Z   flags:   discards 
2020-10-06T00:39:20.814360264Z Command successful.

In the pod description [2] I see that the expand-encrypted-bluefs init container failed:

  - containerID: cri-o://3457d203610cb027e1900399ee8e1c8fa02c2b7d399aec65b09027d5affd7789
    image: quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9
    imageID: quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9
    lastState:
      terminated:
        containerID: cri-o://3457d203610cb027e1900399ee8e1c8fa02c2b7d399aec65b09027d5affd7789
        exitCode: 1
        finishedAt: "2020-10-06T00:55:25Z"
        reason: Error
        startedAt: "2020-10-06T00:55:25Z"
    name: expand-encrypted-bluefs
    ready: false
    restartCount: 8
    state:
      waiting:
        message: back-off 5m0s restarting failed container=expand-encrypted-bluefs pod=rook-ceph-osd-2-9c546484f-jqpjn_openshift-storage(64ce7909-b2a9-434a-982c-4ebc5586fef2)
        reason: CrashLoopBackOff


@Seb Can you take a look? 

[1] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vu1cs33-t4an/jnk-vu1cs33-t4an_20201005T222711/logs/failed_testcase_ocs_logs_1601940776/test_ceph_osd_stopped_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5fe3b72ce75ba0f0bc15dcd51ee1a166b86b0557df8275f3f5bdf62c92630865/namespaces/openshift-storage/pods/rook-ceph-osd-2-9c546484f-jqpjn/encrypted-block-status/encrypted-block-status/logs/current.log
[2] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vu1cs33-t4an/jnk-vu1cs33-t4an_20201005T222711/logs/failed_testcase_ocs_logs_1601940776/test_ceph_osd_stopped_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5fe3b72ce75ba0f0bc15dcd51ee1a166b86b0557df8275f3f5bdf62c92630865/namespaces/openshift-storage/pods/rook-ceph-osd-2-9c546484f-jqpjn/rook-ceph-osd-2-9c546484f-jqpjn.yaml
Comment 6 Sébastien Han 2020-10-08 08:27:53 UTC 
This is a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1885175

*** This bug has been marked as a duplicate of bug 1885175 ***