Description of problem: The task [Upgrade all storage] executes oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm) , which fails to migrate some of the network policy objects in the OCP v3.7.9 environment. Version-Release number of the following components: rpm -q openshift-ansible : openshift-ansible-3.7.42-1.git.2.9ee4e71.el7.noarch rpm -q ansible : ansible-2.4.2.0-2.el7.noarch ansible --version : ansible 2.4.2.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Aug 2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] Steps to Reproduce: 1. Cluster v3.7.9 with network plugin as openshift-ovs-networkpolicy 2. Create network policy object cat np.yaml apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: generation: 1 name: allow-all spec: ingress: - {} podSelector: {} <EOF> 3. Run the command oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm Actual results: [root@vm251-93 ~]# oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm E0423 16:04:43.881409 error: -n policy networkpolicies/allow-all: NetworkPolicy.networking.k8s.io "allow-all" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden. summary: total=1061 errors=1 ignored=0 unchanged=1010 migrated=49 info: to rerun only failing resources, add --include=networkpolicies error: 1 resources failed to migrate Expected results: This is the result from OCP v3.7.23 with the same networkpolicy object. [root@vm253-114 ~]# oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm summary: total=1597 errors=0 ignored=0 unchanged=1581 migrated=16 Additional info: It seems v3.7.9 is giving the error to the NP objects but the same is not observed in the later version as tested on v3.7.23 and v3.9.14
> [root@vm251-93 ~]# oc adm --config=/etc/origin/master/admin.kubeconfig migrate storage --include=* --confirm > E0423 16:04:43.881409 error: -n policy networkpolicies/allow-all: NetworkPolicy.networking.k8s.io "allow-all" is invalid: spec: Forbidden: updates to networkpolicy spec are forbidden. So I'm guessing it's trying to copy an object from extensions/v1beta1 NetworkPolicy to networking/v1 NetworkPolicy, except that because they're both treated as the same thing internally, it effectively ends up trying to modify the existing object, which isn't allowed. The fix might be to just remove NetworkPolicy from the list of things that need to be migrated?
Mo: Do you have any idea what we need to do here?
The fix is to correctly handle the no-op update in validation. The migrate command is correct as-is.
> It seems v3.7.9 is giving the error to the NP objects but the same is not > observed in the later version as tested on v3.7.23 and v3.9.14 Uh, wait, if it's fixed in 3.7.23 then what else can we do? We can't retroactively change 3.7.9
If you are on 3.X.Y1 and it got fixed in a later 3.X.Y2 release, then the "solution" is to upgrade your binaries to that 3.X.Y2 release before upgrading to 3.Z. If that is not possible, then you simply run the migrate command before upgrading and let it fail on that resource (it will still process everything else). Then skip the pre-upgrade migration task during the upgrade.
Mo: Thanks for clarifying... is there anything we can do besides document this in the support knowledgebase?
(In reply to Ben Bennett from comment #8) > Mo: Thanks for clarifying... is there anything we can do besides document > this in the support knowledgebase? I suggest adding something like [1] for 3.7 with the specifics of this issue. [1] https://docs.openshift.org/3.6/install_config/upgrading/upgrading_known_issues.html
Documented in https://github.com/openshift/openshift-docs/pull/8986
Docs opened https://github.com/openshift/openshift-docs/pull/9054
This was changed and went in as https://github.com/openshift/openshift-docs/pull/9054
The document is added. Verify the bug
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2009