Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): When OSD deployment was scaled to 0 and after 11 minutes it was scaled back to 1, the OSD failed to come up and cluster remained in HEALTH_WARN 1 osds down; 1 host (1 osds) down; 1 rack (1 osds) down; Degraded data redundancy: 704/2112 objects degraded (33.333%), 88 pgs degraded, 176 pgs undersized Version of all relevant components (if applicable): ocs-operator.v4.6.0-111.ci 4.6.0-0.nightly-2020-10-03-051134 Can this issue reproducible? I am not sure. This seems like a rare issue. Steps to Reproduce: 1. Scale one of OSDs to 0. $ oc -n openshift-storage scale --replicas=0 deployment/rook-ceph-osd-2 2. Wait 11 minutes. 3. Scale the OSD to 1. $ oc -n openshift-storage scale --replicas=1 deployment/rook-ceph-osd-2 4. Check ceph health. $ oc -n openshift-storage exec [rook-ceph-tools pod] -- ceph health or run automated test [1] Actual results: OSD is down, Ceph is in HEALTH_WARN state. Expected results: OSD is up, Ceph is in HEALTH_OK state. Additional info: [1] https://github.com/red-hat-storage/ocs-ci/blob/master/tests/manage/monitoring/prometheus/test_deployment_status.py#L86
@Filip I can't connect to the must-gather logs. Can you check the link again? We need to look at the osd logs to see if there was some error after the pod startup. Scaling up the osd pod has been many times in the past, so in any case I don't see this as a 4.6 blocker.
@Travis I've just checked and can access all the logs. Were you connected to VPN that time? If not, you have to be. Can you please try again and let us know? Thanks
It's working now, not sure what happened yesterday even when I was on the VPN. Are you only seeing this in an encrypted cluster? In the OSD log [1] I see this message: 2020-10-06T00:39:20.814360264Z /dev/mapper/ocs-deviceset-0-data-0-g8l9q-block-dmcrypt is active. 2020-10-06T00:39:20.814360264Z Underlying device for crypt device ocs-deviceset-0-data-0-g8l9q-block-dmcrypt disappeared. 2020-10-06T00:39:20.814360264Z type: n/a 2020-10-06T00:39:20.814360264Z cipher: aes-xts-plain64 2020-10-06T00:39:20.814360264Z keysize: 512 bits 2020-10-06T00:39:20.814360264Z key location: dm-crypt 2020-10-06T00:39:20.814360264Z device: (null) 2020-10-06T00:39:20.814360264Z sector size: 512 2020-10-06T00:39:20.814360264Z offset: 32768 sectors 2020-10-06T00:39:20.814360264Z size: 1073709056 sectors 2020-10-06T00:39:20.814360264Z mode: read/write 2020-10-06T00:39:20.814360264Z flags: discards 2020-10-06T00:39:20.814360264Z Command successful. In the pod description [2] I see that the expand-encrypted-bluefs init container failed: - containerID: cri-o://3457d203610cb027e1900399ee8e1c8fa02c2b7d399aec65b09027d5affd7789 image: quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9 imageID: quay.io/rhceph-dev/rhceph@sha256:22ea8ee38cd8283f636c2eeb640eb4a1bb744efb18abee114517926f4a03bff9 lastState: terminated: containerID: cri-o://3457d203610cb027e1900399ee8e1c8fa02c2b7d399aec65b09027d5affd7789 exitCode: 1 finishedAt: "2020-10-06T00:55:25Z" reason: Error startedAt: "2020-10-06T00:55:25Z" name: expand-encrypted-bluefs ready: false restartCount: 8 state: waiting: message: back-off 5m0s restarting failed container=expand-encrypted-bluefs pod=rook-ceph-osd-2-9c546484f-jqpjn_openshift-storage(64ce7909-b2a9-434a-982c-4ebc5586fef2) reason: CrashLoopBackOff @Seb Can you take a look? [1] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vu1cs33-t4an/jnk-vu1cs33-t4an_20201005T222711/logs/failed_testcase_ocs_logs_1601940776/test_ceph_osd_stopped_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5fe3b72ce75ba0f0bc15dcd51ee1a166b86b0557df8275f3f5bdf62c92630865/namespaces/openshift-storage/pods/rook-ceph-osd-2-9c546484f-jqpjn/encrypted-block-status/encrypted-block-status/logs/current.log [2] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vu1cs33-t4an/jnk-vu1cs33-t4an_20201005T222711/logs/failed_testcase_ocs_logs_1601940776/test_ceph_osd_stopped_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-5fe3b72ce75ba0f0bc15dcd51ee1a166b86b0557df8275f3f5bdf62c92630865/namespaces/openshift-storage/pods/rook-ceph-osd-2-9c546484f-jqpjn/rook-ceph-osd-2-9c546484f-jqpjn.yaml
This is a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1885175 *** This bug has been marked as a duplicate of bug 1885175 ***