Bug 1840084
| Summary: | The operator should delete the unused PVCs for mons | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Petr Balogh <pbalogh> |
| Component: | rook | Assignee: | Travis Nielsen <tnielsen> |
| Status: | CLOSED ERRATA | QA Contact: | Aviad Polak <apolak> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.5 | CC: | assingh, ebenahar, madam, nberry, ocs-bugs, prsurve, sostapov, tnielsen |
| Target Milestone: | --- | Keywords: | Automation |
| Target Release: | OCS 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-15 10:17:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Petr Balogh
2020-05-26 10:50:50 UTC
Do you see it in other setups? Post deployment? After some tests? After negative tests? As this is not critical and basically do not break anything we do not have any test for this AFAIK. I noticed it in some of our deployment and remembered this was not first time hence opened the BZ. But as we don't do this check for: len(mon pods) = len(mon PVCs) now we will never have an issue with this if it will not be decided to have that check in place and then we will fail the deployment if we will hit such issue again. I think that Neha know more details about this, can you please answer more info if you've seen this in other cases as well? When I opened this BZ I thought we will get some input from Travis maybe if this can be handled on operator level? Moving to rook, as creation of mon-PVCs is rook's responsibility. (In reply to Petr Balogh from comment #3) > As this is not critical and basically do not break anything As this is not critical, why is it proposed for 4.4.z? Moving to 4.5 for now. @Travis can you confirm what is the intended behavior in rook? Rook should be deleting the PVC that is no longer needed for the mon that was failed over. This is not currently implemented, definitely need to fix it. Upstream fix: https://github.com/rook/rook/pull/5698 @Neha The code path for starting mons the first time when there is a long delay and it creates mons a,d,e is much different from the case where quorum is first established with a,b,c and then failover occurs later. This has long been a bug in Rook where b,c are skipped if the first reconcile takes too long. How about if we consider this issue verified for 4.5 and we open a new BZ for the remaining work? Fixing that scenario will require more than just deleting the unused PVCs. Since it is a bigger change I'd suggest opening it for 4.6. From my point of view I am OK to move it to 4.6 as this is nothing urgent I think. But would like to hear also others opinion like was asked from Neha. @Neha Anytime the operator finds that a mon is unhealthy for more than the timeout (10 min), it will attempt to start a new mon and complete the failover. 1. Deleting a mon deployment: The operator might failover to a new mon, or it might first try to re-create the same mon deployment that is missing. Starting in 4.6 you should see the same mon deployment re-created again, but in 4.5 you likely will see the failover scenario. 2. Node replacement: The mon will be force deleted by the operator if the node is not responding, so the mon likely will already move to another node before the failover will be triggered after 10 minutes. Again, it just depends on timing. 3. The most reliable way to test the failover scenario is likely to set the replicas=0 on the mon deployment. As long as the operator doesn't restart, it won't notice that the deployment doesn't match the desired state of replica=1. *** Bug 1859960 has been marked as a duplicate of this bug. *** with builds: ocs: ocs-operator.v4.5.0-54.ci ocp: 4.5.0-0.nightly-2020-08-15-052753 on vmware, tested with delete deployment, delete pod and set replica=0, looks good moving to verifed Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754 |