Bug 1463393
| Summary: | Upgrade process may result in data deletion if running older versions of openshift. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Bradley Childs <bchilds> |
| Component: | Cluster Version Operator | Assignee: | Scott Dodson <sdodson> |
| Status: | CLOSED WONTFIX | QA Contact: | Anping Li <anli> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.2.1 | CC: | aos-bugs, bchilds, eparis, hekumar, jokerman, lfitzger, mmccomas |
| Target Milestone: | --- | ||
| Target Release: | 3.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-15 12:35:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Bradley Childs
2017-06-20 18:49:49 UTC
The proposed solution is that as part of the node upgrade we update the package or container and restart if the version falls into the problematic ranges then perform normal upgrade procedures. 3.2.0 < $current_version < 3.2.1.31 3.3.0 < $current_version < 3.3.1.28 3.4.0 < $current_version < 3.4.1.27 3.5.0 < $current_version < 3.5.5.18 $current_version < 3.5.5.18 AND $target_version >= 3.6 This means, if the version is in the problematic ranges then the node process will be restarted twice during an upgrade. Once for this special task, and then once as part of the normal drain and upgrade process. Isn't it the new node, which starts, finds things in /var/lib/origin-volumes (or whatever) which actually was doing the bad rm? If so, isn't this just fixed by updating to the latest? It doesn't matter where you come from, it only matters where you go to. Also, if that is the case, we could more easily work around things by stopping the node, running umount -l on everything we still have mounted, then starting the new node. Thus the new node couldn't see the thing and delete them. (It also could never properly unmount them or re-use them, but that's a different problem and one I'd rather we had instead of known data deletion...) After some pre-defined internal stopping the node will automatically trigger migration of pods to another node. The migration may trigger code that cleans up orphaned pod directory and thereby triggering the bug.. minor correction - stopping the atomic-openshift-node process while will trigger migration of pods to another node, that will not trigger orphan pod directory deletion code (because node process is not running!). sorry, I had a brain f**t for a moment. Anyways though - stopping node will not stop containers running on it. so, we would be performing unmount on volumes that are being used. The playbooks run `drain` before they stop the node. So there will be no running pods. Yes but drain cause pod eviction and that triggers CleanOrphanPodDir routine (for all evicted pods). Evicted pods are treated differently than deleted pods and somehow we are calling CleanOrphanPodDir for all evicted pods. I have only verified this through experimentation, didn't dig through code yet. So if somehow NFS volume didn't unmount properly during drain then CleanOrphanPodDir routine *will* delete data for versions without fix. We have 2 bugs here: a. Volumes not getting unmounted on pod eviction. b. Data getting deleted on CleanOrphanPodDir call. So, it matters which version we are upgrading from as well. The decision is that we'll do the forced upgrade and then follow with normal upgrade steps afterwards. I think this means we'll need to re-factor the playbook to move the drain operation into openshift_node_upgrade role but I think we should do that anyway. I'll get someone working on this tomorrow. *** Bug 1443623 has been marked as a duplicate of this bug. *** I just realized the proposal would not account for blue green upgrades where the service is never upgraded but instead new hosts are provisioned running the new version and then the old hosts are removed from the cluster in this manner (per the docs, we don't actually automate this). $ oadm manage-node --selector=color=blue --evacuate $ oc delete node --selector=color=blue I assume this would trigger the problem? You mean old hosts are removed from cluster without ever upgrading? In that case - yeah those old hosts must have their version updated (at least with a z-stream version) before draining the old host. This was handled via a release notes entry that advises users to perform a manual update prior to upgrading because it was determined that amending the playbooks was prohibitively complex. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |