Description of problem (please be detailed as possible and provide log snippests): On a 4.14 OCP cluster with >1 nodes, after adding/updating a MCP that's relevant to one node, the MCP update cycle (where the node is cordoned, drained, rebooted etc..) is stuck indefinitely by the ODF PDBs on the openshift-storage namespace. Version of all relevant components (if applicable): OCP 4.14 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? there is a workaround so no. Is there any workaround available to the best of your knowledge? working workaround - remove all the PDBs: oc delete -n openshift-storage pdb --all Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 100% Can this issue reproduce from the UI? yes If this is a regression, please provide more details to justify this: not being able to update MCP is a regression Steps to Reproduce: 1. Follow changes on the MCP on the DPDK-readiness D/S documenation: https://docs.openshift.com/container-platform/4.13/virt/virtual_machines/vm_networking/virt-attaching-vm-to-sriov-network.html#virt-configuring-cluster-dpdk_virt-attaching-vm-to-sriov-network 2. monitor the node's status: `oc get nodes` and notice that the node is not rebooted and stuck during drain. 3. review logs of `machine-config-controller` pods on the `openshift-machine-config-operator` namespace, and see that the ODF pods are not being able to delete, due to their PDB. Actual results: `oc get mcp` - shows that the node is stuck in update Expected results: node should reboot and move to be ready Additional info:
Hi Ram Can you provide the following info: - ODF must gather - What was the ceph health before the MCP operation was performed? Thanks
Hey Santosh can you provide a set of commands for me to follow in order to get the ODF must gather and ceph health?
I currently do not have an available cluster, I'll update when I get the information you need.
>Hey Santosh >can you provide a set of commands for me to follow in order to get the ODF must gather and ceph health?
(In reply to Ram Lavi from comment #5) > >Hey Santosh > >can you provide a set of commands for me to follow in order to get the ODF must gather and ceph health? Hi. Shared the steps on google chat.
Hi Ram. did you get a chance to generate the ODF must gather?
Unfortunately not yet. My cluster is SNO (so there is no drain). I've asked a few cluster admins to tell me when they cause a drain so that we could fetch this info for you.