Description of problem: When running the upgrade from OSP15 to 16, the "overcloud upgrade run" operation fails almost at the very beginning with: 2020-01-11 11:24:21 | TASK [tripleo-container-tag : Fail if container image is undefined] ************ 2020-01-11 11:24:21 | Saturday 11 January 2020 11:24:20 +0000 (0:00:00.398) 0:01:17.764 ****** 2020-01-11 11:24:21 | fatal: [controller-0]: FAILED! => {"msg": "The conditional check 'not tripleo_container_image' failed. The error was: error while evaluating conditional (not tripleo_container_image): 'cinder_backup_current_pcmklatest_id' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Fail if container image is undefined\n ^ here\n"} 2020-01-11 11:24:21 | fatal: [controller-1]: FAILED! => {"msg": "The conditional check 'not tripleo_container_image' failed. The error was: error while evaluating conditional (not tripleo_container_image): 'cinder_backup_current_pcmklatest_id' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Fail if container image is undefined\n ^ here\n"} 2020-01-11 11:24:21 | fatal: [controller-2]: FAILED! => {"msg": "The conditional check 'not tripleo_container_image' failed. The error was: error while evaluating conditional (not tripleo_container_image): 'cinder_backup_current_pcmklatest_id' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 20, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Fail if container image is undefined\n ^ here\n"} http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-upgrade-upgrade-15-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-ha/1/undercloud-0.tar.gz?undercloud-0/home/stack/overcloud_upgrade_run_Controller.log The reason is that the existing upgrade tasks found in the workflow belong to the upgrade from OSP14 to 15, which implied an Opearting System upgrade and therefore, they had a different handling of the puppet data. For the OSP15 to 16 workflow the upgrade should go back to the similar workflow that existed before OSP15. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
So, the upgrade is now failing with the error: 2020-01-30 23:13:11 | TASK [tripleo-container-tag : Pull 192.168.24.1:8787/rh-osbs/rhosp15-openstack-cinder-backup:pcmklatest image] *** 2020-01-30 23:13:11 | Thursday 30 January 2020 23:13:10 +0000 (0:00:00.419) 0:01:42.429 ****** 2020-01-30 23:13:11 | skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-30 23:13:11 | skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-30 23:13:11 | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-30 23:13:11 | 2020-01-30 23:13:11 | TASK [tripleo-container-tag : Tag {{ tripleo_container_image_latest }} to latest {{ tripleo_container_image }} image] *** 2020-01-30 23:13:11 | Thursday 30 January 2020 23:13:10 +0000 (0:00:00.457) 0:01:42.886 ****** 2020-01-30 23:13:11 | fatal: [controller-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'cinder_backup_docker_image_latest' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 32, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Tag {{ tripleo_container_image_latest }} to latest {{ tripleo_container_image }} image\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} 2020-01-30 23:13:11 | fatal: [controller-1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'cinder_backup_docker_image_latest' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 32, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Tag {{ tripleo_container_image_latest }} to latest {{ tripleo_container_image }} image\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} 2020-01-30 23:13:11 | fatal: [controller-2]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'cinder_backup_docker_image_latest' is undefined\n\nThe error appears to be in '/usr/share/ansible/roles/tripleo-container-tag/tasks/main.yml': line 32, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Tag {{ tripleo_container_image_latest }} to latest {{ tripleo_container_image }} image\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} 2020-01-30 23:13:11 | 2020-01-30 23:13:11 | NO MORE HOSTS LEFT ************************************************************* 2020-01-30 23:13:11 | 2020-01-30 23:13:11 | PLAY RECAP ********************************************************************* 2020-01-30 23:13:11 | controller-0 : ok=8 changed=4 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 2020-01-30 23:13:11 | controller-1 : ok=6 changed=2 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 2020-01-30 23:13:11 | controller-2 : ok=6 changed=2 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 2020-01-30 23:13:11 | 2020-01-30 23:13:11 | Thursday 30 January 2020 23:13:10 +0000 (0:00:00.292) 0:01:43.178 ****** 2020-01-30 23:13:11 | =============================================================================== 2020-01-30 23:13:11 | 2020-01-30 23:13:11 | Ansible failed, check log at /var/lib/mistral/72438451-13b6-4841-a213-a3ff0757773b/ansible.log. @
Moving target to OSP16.1 as OSP16.0 is EOL. We will use 15 to 16.1 upgrade job to verify this. Patch already merged in 16.1 downstream branch in: commit 3d44b128cf888a9d45d611e86f1d5ee91ca992c7 Merge: 47f6ec208 2367b8aa5 Author: Zuul <zuul.org> Date: Fri May 22 01:44:33 2020 +0000 Merge "Skip operating system upgrade tasks via UpgradeLeappEnabled param." into stable/train Present in openstack-tripleo-heat-templates-11.3.2-0.20200630193615.c20d10d.el8ost.src.rpm : https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1242758
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284