Bug 1563929
| Summary: | Upgrade from 3.7 to 3.9 fail at the Task [Upgrade all storage] | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sudarshan Chaudhari <suchaudh> |
| Component: | Master | Assignee: | Maciej Szulik <maszulik> |
| Status: | CLOSED WONTFIX | QA Contact: | Wang Haoran <haowang> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, byount, farandac, jliggitt, jokerman, mfojtik, mjahangi, mmccomas, mukeshk2, suchaudh |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-05-03 14:44:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the 3.7 to 3.9 upgrade is multipart, 3.7 to 3.8 to 3.9, can you check which version of openshift is currently running when this happens and/or provide the complete log which would show where in the upgrade phase this fails. Assigning to master team to evaluate the nature of the failure. I've created https://github.com/openshift/openshift-docs/pull/8767 which adds this issue to known issues. The fix is to manually edit the failed services removing the duplicate name+port pairs and re-run the storage. Once the PR merges to docs, I'm going to close this issue as a won't fix. Commits pushed to master at https://github.com/openshift/openshift-docs https://github.com/openshift/openshift-docs/commit/be4ad40e866c425d1b36067017943cc219240578 Bug 1563929 - add a 3.9 upgrade known issues section 3.9 introduces tighter validation for Service objects. During storage upgrade this might require administrators to invoke manual updates to the erroneous objects. https://github.com/openshift/openshift-docs/commit/1e5ceb5bb14a9ef36ec86c795546bfcdb225c0f5 Merge pull request #8781 from soltysh/bug1563929 Bug 1563929 - add a 3.9 upgrade known issues section |
Description of problem: Version-Release number of the following components: :>rpm -qa | grep atomic atomic-openshift-clients-3.9.14-1.git.0.4efa2ca.el7.x86_64 atomic-openshift-master-3.9.14-1.git.0.4efa2ca.el7.x86_64 atomic-openshift-utils-3.9.14-1.git.3.c62bc34.el7.noarch atomic-openshift-3.9.14-1.git.0.4efa2ca.el7.x86_64 atomic-registries-1.20.1-9.git436cf5d.el7.x86_64 atomic-openshift-excluder-3.9.14-1.git.0.4efa2ca.el7.noarch atomic-openshift-sdn-ovs-3.9.14-1.git.0.4efa2ca.el7.x86_64 atomic-openshift-docker-excluder-3.9.14-1.git.0.4efa2ca.el7.noarch atomic-openshift-node-3.9.14-1.git.0.4efa2ca.el7.x86_64 Run the Automated In-Place Upgrade Playbook. # ansible-playbook -i </path/to/inventory/file> /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml -vvv Failing at with the error: TASK [Upgrade all storage] **************************************************************************************************************************************************************************** fatal: [mas-3-01.example.com]: FAILED! => {"changed": true, "cmd": ["oc", "adm", "--config=/etc/origin/master/admin.kubeconfig", "migrate", "storage", "--include=*", "--confirm"], "delta": "0:07:06.574833", "end": "2018-04-03 11:22:07.834827", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-04-03 11:15:01.259994", "stderr": "", "stderr_lines": [], "stdout": "E0403 11:22:04.100394 error: -n oneid-rtp1 services/consul-ingress: Service \"consul-ingress\" is invalid: spec.ports[5]: Duplicate value: api.ServicePort{Name:\"\", Protocol:\"TCP\", Port:8500, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}\nsummary: total=37768 errors=1 ignored=0 unchanged=37767 migrated=0\ninfo: to rerun only failing resources, add --include=services\nerror: 1 resources failed to migrate", "stdout_lines": ["E0403 11:22:04.100394 error: -n oneid-rtp1 services/consul-ingress: Service \"consul-ingress\" is invalid: spec.ports[5]: Duplicate value: api.ServicePort{Name:\"\", Protocol:\"TCP\", Port:8500, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}", "summary: total=37768 errors=1 ignored=0 unchanged=37767 migrated=0", "info: to rerun only failing resources, add --include=services", "error: 1 resources failed to migrate"]} to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.retry When manually ran the command: :>oc adm migrate storage --include=* --confirm E0403 12:07:10.228771 error: -n oneid-rtp1 services/consul-ingress: Service "consul-ingress" is invalid: spec.ports[5]: Duplicate value: api.ServicePort{Name:"", Protocol:"TCP", Port:8500, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0} summary: total=37769 errors=1 ignored=0 unchanged=37768 migrated=0 info: to rerun only failing resources, add --include=services error: 1 resources failed to migrate Expected results: Then upgrade should be successfull