Bug 1623642
| Summary: | Upgrade Fails with "Operation cannot be fulfilled on imagestreams...: the object has been modified" | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jack Ottofaro <jack.ottofaro> |
| Component: | Cluster Version Operator | Assignee: | Scott Dodson <sdodson> |
| Status: | CLOSED ERRATA | QA Contact: | liujia <jiajliu> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, bparees, cscribne, dominik.mierzejewski, jokerman, mmccomas, nnosenzo, scuppett, sdodson |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: In certain environments the image stream content changes rapidly enough to prevent replacement with new content due to the object being modified.
Consequence: The upgrade would fail.
Fix: The image stream update now retries 3 times.
Result: The upgrade should be successful.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-12-13 19:27:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
More accurate description is error occurs sometimes during upgrades from 3.7.z to 3.7.57, 3.7.61 or 3.9.40. Created attachment 1479564 [details]
Ansible log for upgrade of 3.7.61 to 3.9.40
Created attachment 1479565 [details]
Ansible log for upgrade of 3.7.44 to 3.7.57
Ben, Should we just add retries around this or are we doing something wrong here with regard to replacing image streams during upgrade? I don't think there's anything wrong w/ doing the replace. I'm surprised people are managing to hit conflicts here (i don't know what would be changing the imagestream, those imagestreams don't have scheduled import turned on so they should not be changing frequently), but wrapping a retry around it seems reasonable to me. https://github.com/openshift/openshift-ansible/pull/10079 adds 3 retries to all image stream import / replace operations, there will be a 5 second delay between each attempt Can not reproduce on openshift-ansible-3.9.40-1.git.0.188c954.el7.noarch when upgrade ocp v3.7.75 to v3.9.40. QE did not hit this issue in pre-release test, and seems it not 100% happen. According to dev's comments about this issue, we just enhance playbook to add more retry for import images tasks. So QE can only checked pr merged and upgrade works well on latest version. Version: openshift-ansible-3.9.57-1.git.0.9ece605.el7.noarch Upgrade succeed with TASK [openshift_examples : Import xPaas image streams] passed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3748 |
Description of problem: The following error occurs sometimes during upgrades from 3.7.57 to 3.7.61 and 3.9.40: 2018-08-27 17:39:32,744 p=25341 u=ab41553 | TASK [openshift_examples : Import xPaas image streams] ************************************************************************************************************** 2018-08-27 17:39:32,744 p=25341 u=ab41553 | task path: /usr/share/ansible/openshift-ansible/roles/openshift_examples/tasks/main.yml:130 2018-08-27 17:39:32,812 p=25341 u=ab41553 | Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py 2018-08-27 17:39:34,270 p=25341 u=ab41553 | fatal: [sd-c986-41cf.nam.nsroot.net]: FAILED! => { "changed": false, "cmd": [ "oc", "replace", "--config=/etc/origin/master/admin.kubeconfig", "-n", "openshift", "-f", "/usr/share/openshift/examples/xpaas-streams/" ], "delta": "0:00:01.137636", "end": "2018-08-27 17:39:34.234194", "failed": true, "failed_when_result": true, "invocation": { "module_args": { "_raw_params": "oc replace --config=/etc/origin/master/admin.kubeconfig -n openshift -f /usr/share/openshift/examples/xpaas-streams/", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": true } }, "msg": "non-zero return code", "rc": 1, "start": "2018-08-27 17:39:33.096558", "stderr": "Error from server (Conflict): Operation cannot be fulfilled on imagestreams \"jboss-eap64-openshift\": the object has been modified; please apply your changes to the latest version and try again", "stderr_lines": [ "Error from server (Conflict): Operation cannot be fulfilled on imagestreams \"jboss-eap64-openshift\": the object has been modified; please apply your changes to the latest version and try again" ], "stdout": "imagestream \"jboss-amq-62\" replaced\nimagestream \"jboss-amq-63\" replaced\nimagestream \"jboss-datagrid65-openshift\" replaced\nimagestream \"jboss-datagrid65-client-openshift\" replaced\nimagestream \"jboss-datagrid71-openshift\" replaced\nimagestream \"jboss-datagrid71-client-openshift\" replaced\nimagestream \"jboss-datavirt63-openshift\" replaced\nimagestream \"jboss-datavirt63-driver-openshift\" replaced\nimagestream \"jboss-decisionserver62-openshift\" replaced\nimagestream \"jboss-decisionserver63-openshift\" replaced\nimagestream \"jboss-decisionserver64-openshift\" replaced\nimagestream \"eap-cd-openshift\" replaced\nimagestream \"jboss-eap64-openshift\" replaced\nimagestream \"jboss-eap70-openshift\" replaced\nimagestream \"jboss-eap71-openshift\" replaced\nimagestream \"fis-java-openshift\" replaced\nimagestream \"fis-karaf-openshift\" replaced\nimagestream \"jboss-webserver30-tomcat7-openshift\" replaced\nimagestream \"jboss-webserver30-tomcat8-openshift\" replaced\nimagestream \"jboss-webserver31-tomcat7-openshift\" replaced\nimagestream \"jboss-webserver31-tomcat8-openshift\" replaced\nimagestream \"jboss-eap70-openshift\" replaced\nimagestream \"jboss-eap71-openshift\" replaced\nimagestream \"jboss-decisionserver62-openshift\" replaced\nimagestream \"jboss-decisionserver63-openshift\" replaced\nimagestream \"jboss-decisionserver64-openshift\" replaced\nimagestream \"jboss-processserver63-openshift\" replaced\nimagestream \"jboss-processserver64-openshift\" replaced\nimagestream \"jboss-datagrid65-openshift\" replaced\nimagestream \"jboss-datagrid71-openshift\" replaced\nimagestream \"jboss-datagrid65-client-openshift\" replaced\nimagestream \"jboss-datagrid71-client-openshift\" replaced\nimagestream \"jboss-datavirt63-openshift\" replaced\nimagestream \"jboss-datavirt63-driver-openshift\" replaced\nimagestream \"jboss-amq-62\" replaced\nimagestream \"jboss-amq-63\" replaced\nimagestream \"redhat-sso70-openshift\" replaced\nimagestream \"redhat-sso71-openshift\" replaced\nimagestream \"redhat-openjdk18-openshift\" replaced\nimagestream \"jboss-webserver30-tomcat7-openshift\" replaced\nimagestream \"jboss-webserver30-tomcat8-openshift\" replaced\nimagestream \"jboss-webserver31-tomcat7-openshift\" replaced\nimagestream \"jboss-webserver31-tomcat8-openshift\" replaced\nimagestream \"redhat-openjdk18-openshift\" replaced\nimagestream \"java\" replaced\nimagestream \"jboss-processserver63-openshift\" replaced\nimagestream \"jboss-processserver64-openshift\" replaced\nimagestream \"redhat-sso70-openshift\" replaced\nimagestream \"redhat-sso71-openshift\" replaced\nimagestream \"redhat-sso72-openshift\" replaced", "stdout_lines": [ "imagestream \"jboss-amq-62\" replaced", "imagestream \"jboss-amq-63\" replaced", "imagestream \"jboss-datagrid65-openshift\" replaced", "imagestream \"jboss-datagrid65-client-openshift\" replaced", "imagestream \"jboss-datagrid71-openshift\" replaced", "imagestream \"jboss-datagrid71-client-openshift\" replaced", "imagestream \"jboss-datavirt63-openshift\" replaced", "imagestream \"jboss-datavirt63-driver-openshift\" replaced", "imagestream \"jboss-decisionserver62-openshift\" replaced", "imagestream \"jboss-decisionserver63-openshift\" replaced", "imagestream \"jboss-decisionserver64-openshift\" replaced", "imagestream \"eap-cd-openshift\" replaced", "imagestream \"jboss-eap64-openshift\" replaced", "imagestream \"jboss-eap70-openshift\" replaced", "imagestream \"jboss-eap71-openshift\" replaced", "imagestream \"fis-java-openshift\" replaced", "imagestream \"fis-karaf-openshift\" replaced", "imagestream \"jboss-webserver30-tomcat7-openshift\" replaced", "imagestream \"jboss-webserver30-tomcat8-openshift\" replaced", "imagestream \"jboss-webserver31-tomcat7-openshift\" replaced", "imagestream \"jboss-webserver31-tomcat8-openshift\" replaced", "imagestream \"jboss-eap70-openshift\" replaced", "imagestream \"jboss-eap71-openshift\" replaced", "imagestream \"jboss-decisionserver62-openshift\" replaced", "imagestream \"jboss-decisionserver63-openshift\" replaced", "imagestream \"jboss-decisionserver64-openshift\" replaced", "imagestream \"jboss-processserver63-openshift\" replaced", "imagestream \"jboss-processserver64-openshift\" replaced", "imagestream \"jboss-datagrid65-openshift\" replaced", "imagestream \"jboss-datagrid71-openshift\" replaced", "imagestream \"jboss-datagrid65-client-openshift\" replaced", "imagestream \"jboss-datagrid71-client-openshift\" replaced", "imagestream \"jboss-datavirt63-openshift\" replaced", "imagestream \"jboss-datavirt63-driver-openshift\" replaced", "imagestream \"jboss-amq-62\" replaced", "imagestream \"jboss-amq-63\" replaced", "imagestream \"redhat-sso70-openshift\" replaced", "imagestream \"redhat-sso71-openshift\" replaced", "imagestream \"redhat-openjdk18-openshift\" replaced", "imagestream \"jboss-webserver30-tomcat7-openshift\" replaced", "imagestream \"jboss-webserver30-tomcat8-openshift\" replaced", "imagestream \"jboss-webserver31-tomcat7-openshift\" replaced", "imagestream \"jboss-webserver31-tomcat8-openshift\" replaced", "imagestream \"redhat-openjdk18-openshift\" replaced", "imagestream \"java\" replaced", "imagestream \"jboss-processserver63-openshift\" replaced", "imagestream \"jboss-processserver64-openshift\" replaced", "imagestream \"redhat-sso70-openshift\" replaced", "imagestream \"redhat-sso71-openshift\" replaced", "imagestream \"redhat-sso72-openshift\" replaced" ] } In one of their other upgrades the imagestream called out was jboss-amq-63. Upgrade succeeds after re-running the upgrade playbook. This similar bug - https://bugzilla.redhat.com/show_bug.cgi?id=1500931 (Image streams may fail to import during an upgrade blocking the install) - is supposed to have been fixed in 3.7.42 (https://access.redhat.com/errata/RHBA-2018:0636).