We need to create CI job that is able to verify rollback of cluster state. The proposed flow is as follows. ETCD team to augment CI job with relevant steps below. Installer - Provision 4.x cluster MCO - Edit machine config to write file to /etc/ whose content is “A” Etcd - Take etcd backup; capture static pod revision levels ??? - Update release payload MCO - Edit machine config to write file to /etc/ whose content is “B” MCO Wait for roll-out to complete Master - Shutdown kube-apiserver via manual manipulation of static pods on each master Etcd - Restore etcd backup Master - Restart kube-apiserver via manual manipulation of static pods on each master MCO - Wait for cluster to reconcile X period Master - All operators report old payload version MCO - Ensure on each worker /etc/ content is restored to “A” via MCO See https://docs.google.com/document/d/1J-Dy_LoIDPqJ9HqI5iGKbzkzNLNy93b3hmHC-YyNfdI/edit?ts=5cad0560#
We'll track this via Jira instead.