Created attachment 1603955 [details] controller runupdate log Description of problem: Minor update of OSP 15 deployed overcloud fails in run_controller_update with the error overcloud-controller-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'docker_image' is un defined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yaml Version-Release number of selected component (if applicable): RHOS_TRUNK-15.0-RHEL-8-20190813.n.0 ansible.noarch 2.8.3-1.el8ae @rhosp-rhel-8.0-ansible ansible-pacemaker.noarch 1.0.4-0.20190418190349.0e4d7c0.el8ost @rhelosp-15.0 ansible-role-atos-hsm.noarch 0.1.1-0.20190422121159.1518dbd.el8ost @rhelosp-15.0 ansible-role-chrony.noarch 1.0.1-0.20190708210510.71b73d6.el8ost @rhelosp-15.0 ansible-role-container-registry.noarch 1.1.1-0.20190715160413.884a8f6.el8ost @rhelosp-15.0 ansible-role-redhat-subscription.noarch 1.0.4-0.20190701140422.fceb51c.el8ost @rhelosp-15.0 ansible-role-thales-hsm.noarch 0.2.1-0.20190812140505.cc299d7.el8ost @rhelosp-15.0 ansible-role-tripleo-modify-image.noarch 1.1.1-0.20190711170427.eabaed0.el8ost @rhelosp-15.0 ansible-tripleo-ipsec.noarch 9.1.1-0.20190513190404.ffe104c.el8ost @rhelosp-15.0 ceph-ansible.noarch 4.0.0-0.1.rc13.el8cp @rhelosp-ceph-4.0-tools-pending openstack-tripleo-common.noarch 10.8.1-0.20190808130448.b942458.el8ost @rhelosp-15.0 openstack-tripleo-common-containers.noarch 10.8.1-0.20190808130448.b942458.el8ost @rhelosp-15.0 openstack-tripleo-heat-templates.noarch 10.6.1-0.20190812140519.2a684c0.el8ost @rhelosp-15.0 openstack-tripleo-image-elements.noarch 10.4.1-0.20190705161217.2c8a6a5.el8ost @rhelosp-15.0 openstack-tripleo-puppet-elements.noarch 10.3.2-0.20190710165331.c89fe3c.el8ost @rhelosp-15.0 openstack-tripleo-validations.noarch 10.5.1-0.20190812110443.80a9f7b.el8ost @rhelosp-15.0 podman.x86_64 1.0.3-1.git9d78c0c.module+el8.0.0.z+3717+fdd07b7c @rhosp-rhel-8.0-appstream puppet-tripleo.noarch 10.5.1-0.20190812120435.ed6c6b0.el8ost @rhelosp-15.0 python3-heat-agent-ansible.noarch 1.8.1-0.20190523210450.1e15344.el8ost @rhelosp-15.0 python3-tripleo-common.noarch 10.8.1-0.20190808130448.b942458.el8ost @rhelosp-15.0 python3-tripleoclient.noarch 11.5.1-0.20190808130445.f83a1ed.el8ost @rhelosp-15.0 python3-tripleoclient-heat-installer.noarch 11.5.1-0.20190808130445.f83a1ed.el8ost @rhelosp-15.0 sshpass.x86_64 1.06-3.el8ae @rhosp-rhel-8.0-ansible How reproducible: consistent Steps to Reproduce: 1.run minor update with ironic in overcloud - 2. 3. Actual results: overcloud update fails Expected results: overcloud updated successfully Additional info: see : https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-15_director-8.0-virthost-3cont_2comp_3ceph_2ironrole_2ironic-ipv4-geneve-composable-minor_update-RHELOSP-48861/5/artifact/.sh/ir-tripleo-overcloud-update.log/*view*/
Hi, the exact error is: 2019-08-15 00:10:08 | TASK [Pull latest ovn-dbs images] ********************************************** 2019-08-15 00:10:08 | Thursday 15 August 2019 00:10:00 +0000 (0:00:00.890) 0:02:43.342 ******* 2019-08-15 00:10:08 | changed: [overcloud-controller-0] => {"changed": true, "cmd": ["podman", "pull", "192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1"], "delta": "0:00:06.836748", "end": "2019-08-15 00:10:07.657482", "rc": 0, "start": "2019-08-15 00:10:00.820734", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1...Getting image source signatures\nCopying blob sha256:e61d8721e62e50814b162c8341bb235d3453b9c95bd26439bf9100fcf88338c7\nCopying blob sha256:c585fd5093c62ee42a56af6c09813ac8384d8145c4285bc62c357b41224b1970\nCopying blob sha256:cac106ef9322f754836534f6172c414a4262077eed63f0a8f65bb8dddd7da2f8\nCopying blob sha256:06da3ba2dc50877087ccaed4e34229dcf86be1a6f4f66337a94920dd6342ead0\nCopying blob sha256:ba2ecfd636bb14cb592501dd2d4e1cc763542ef6eee22c7e791c058d6c45a61a\nCopying blob sha256:9f8e8f301be6ebeb8f11243dc12c7240c135f35b06f7b07495f338cfa25c4ddf\nCopying config sha256:3418ee3ff63d0ce4ffcda6385754ad3c94707535f8386ada3d6fd2f0df2d35a6\nWriting manifest to image destination\nStoring signatures", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1...Getting image source signatures", "Copying blob sha256:e61d8721e62e50814b162c8341bb235d3453b9c95bd26439bf9100fcf88338c7", "Copying blob sha256:c585fd5093c62ee42a56af6c09813ac8384d8145c4285bc62c357b41224b1970", "Copying blob sha256:cac106ef9322f754836534f6172c414a4262077eed63f0a8f65bb8dddd7da2f8", "Copying blob sha256:06da3ba2dc50877087ccaed4e34229dcf86be1a6f4f66337a94920dd6342ead0", "Copying blob sha256:ba2ecfd636bb14cb592501dd2d4e1cc763542ef6eee22c7e791c058d6c45a61a", "Copying blob sha256:9f8e8f301be6ebeb8f11243dc12c7240c135f35b06f7b07495f338cfa25c4ddf", "Copying config sha256:3418ee3ff63d0ce4ffcda6385754ad3c94707535f8386ada3d6fd2f0df2d35a6", "Writing manifest to image destination", "Storing signatures"], "stdout": "3418ee3ff63d0ce4ffcda6385754ad3c94707535f8386ada3d6fd2f0df2d35a6", "stdout_lines": ["3418ee3ff63d0ce4ffcda6385754ad3c94707535f8386ada3d6fd2f0df2d35a6"]} 2019-08-15 00:10:08 | 2019-08-15 00:10:08 | TASK [tripleo-container-tag : Pull {{ container_image }} image] **************** 2019-08-15 00:10:08 | Thursday 15 August 2019 00:10:07 +0000 (0:00:07.230) 0:02:50.572 ******* 2019-08-15 00:10:08 | fatal: [overcloud-controller-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'docker_image' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yaml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Pull {{ container_image }} image\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} This is happening during ovn-dbs images update while retagging the images. I believe this is a duplicate of the situation we found and fix there https://bugzilla.redhat.com/show_bug.cgi?id=1711863 So for now I'm closing this one as a duplication as I believe that the fix attached to the aforementioned bz will solve the issue. If not, we just re-open that bug. Please let us know how your testing goes. Thanks, *** This bug has been marked as a duplicate of bug 1711863 ***