Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem (please be detailed as possible and provide log snippests): ---------------------------------------------------------------------- On an OCS 4.4.1-rc2 cluster on Vmware(converged Mode), the ceph cluster in OCS side is in HEALTH_ERR state. Raised this concern in the chatroom but was asked to enable debug_osd to 30 and collect logs. But it was seen that changing the log level from ceph side didnt propagate back to the OCS side. Still raising the issue to track this failure and in case the collected logs are helpful. sh-4.4# ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.7 is active+clean+inconsistent, acting [3,1,2] Following are the only activities known to have been perofrmed on the cluster: 1. Deployment 2. Tier1 execution (the ceph was in HEALTH_OK state after tier1 completion) 3. Some noobaa related buckets were created Snip from OSD.3 logs: 2020-06-30 12:21:26.222886 I | 2020-06-30 12:21:26.221 7f9438664700 -1 log_channel(cluster) log [ERR] : 1.7 scrub : stat mismatch, got 2/2 objects, 0/0 clones, 2/2 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 116/0 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. 2020-06-30 12:21:26.222944 I | 2020-06-30 12:21:26.221 7f9438664700 -1 log_channel(cluster) log [ERR] : 1.7 scrub 1 errors ~ Version of all relevant components (if applicable): ---------------------------------------------------------------------- ========CSV ====== NAME DISPLAY VERSION REPLACES PHASE lib-bucket-provisioner.v1.0.0 lib-bucket-provisioner 1.0.0 Succeeded ocs-operator.v4.4.1-465.ci OpenShift Container Storage 4.4.1-465.ci Succeeded =========OCP========== OCP = 4.4.0-0.nightly-2020-06-29-071755 RHCS = 4.1 (ceph version 14.2.8-59.el8cp (53387608e81e6aa2487c952a604db06faa5b2cd0) nautilus (stable) Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? ---------------------------------------------------------------------- I was able to run IO but the cluster state was in the same state for hours. Is there any workaround available to the best of your knowledge? ---------------------------------------------------------------------- The cluster was in Health_OK state in the morning. Not sure if @Josh performed any repair. Will add a needinfo for update on this. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? ---------------------------------------------------------------------- 3 Can this issue reproducible? ---------------------------------------------------------------------- Not sure Can this issue reproduce from the UI? ---------------------------------------------------------------------- No If this is a regression, please provide more details to justify this: ---------------------------------------------------------------------- No Steps to Reproduce: (Not sure when and why the cluster reached this state, we do not have the exact Steps to reproduce) ---------------------------------------------------------------------- 1. Create an OCP 4.4-nightly and OCS 4.4.1-rc2 cluster 2. Run tier1 on the cluster and upload an object to NooBaa 3. It was found, that @2020-06-30 12:21:26.222944 (much after tier1 completion), 1 PG went to inconsistent state and ceph health changed to ERR Actual results: ---------------------------------------------------------------------- Expected results: ---------------------------------------------------------------------- Additional info: Ceph status @Jun 30 17:26:11 UTC, >5 hrs after the PG issue was reported in OSD.3 logs =====ceph status ==== Tue Jun 30 17:26:11 UTC 2020 cluster: id: ca6c02dd-1942-4f11-bd38-1a03979fc85e health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum a,b,c (age 26h) mgr: a(active, since 30h) mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay osd: 6 osds: 6 up (since 26h), 6 in (since 26h) rgw: 1 daemon active (ocs.storagecluster.cephobjectstore.a) task status: scrub status: mds.ocs-storagecluster-cephfilesystem-a: idle mds.ocs-storagecluster-cephfilesystem-b: idle mds.ocs-storagecluster-cephfilesystem-c: idle mds.ocs-storagecluster-cephfilesystem-d: idle data: pools: 11 pools, 400 pgs objects: 13.49k objects, 47 GiB usage: 104 GiB used, 2.9 TiB / 3.0 TiB avail pgs: 399 active+clean 1 active+clean+inconsistent io: client: 852 B/s rd, 170 KiB/s wr, 2 op/s rd, 3 op/s wr ----------------------------------------------------------------------
that way or another - not 4.5 material
This is not a blocker - as discussed in chat, the next step is to enable debug logging in OCS CI so that if this happens again, we'll have a chance of understanding the root cause. Unfortunately with issues like this, we can't diagnose the root cause without debug logs. A stat mismatch like this is generally not very impactful - running 'ceph pg repair' will fix it, and the inconsistency does not impact client I/O, so changing severity to medium.
Neha, any chance we could repro this with debug logs else we might have to close this.
(In reply to Mudit Agarwal from comment #6) > Neha, any chance we could repro this with debug logs else we might have to > close this. @mudit we have not seen this issue in recent times though we have had tier1 and other noobaa related activities in the builds. It seemed to be a one-off issue where a PG had gotten inconsistent, but I do not have the RCA to confirm what really made the PG inconsistent(to reproduce it correctly ) :)
Thanks Neha. Closing it because of lack of repro.