Description of problem: overcloud update converge fails with: ~~~ TASK [octavia-undercloud : get checksum if there\'s an image in glance already] *** task path: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml:42 Monday 26 August 2019 09:12:23 +0200 (0:00:00.071) 0:00:14.578 ********* Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py <localhost> ESTABLISH LOCAL CONNECTION FOR USER: mistral <localhost> EXEC /bin/sh -c \'OS_PROJECT_NAME=service COMPUTE_API_VERSION=1.1 OS_USERNAME=octavia OS_IDENTITY_API_VERSION=3 OS_USER_DOMAIN_NAME=Default OS_CLOUDNAME=overcloud OS_AUTH_URL=https://openstack.ctnpro.cloud.probtp.com:13000/v3 NOVA_VERSION=1.1 OS_IMAGE_API_VERSION=2 OS_PASSWORD=eEGGJqMdMQEwRq8eWCypVbWXz OS_PROJECT_DOMAIN_NAME=Default OS_AUTH_TYPE=password /usr/bin/python && sleep 0\' fatal: [localhost]: FAILED! => { "changed": true, "cmd": "openstack image show -c checksum -f value", "delta": "0:00:01.111391", "end": "2019-08-26 09:12:24.624332", "invocation": { "module_args": { "_raw_params": "openstack image show -c checksum -f value", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": true } }, "msg": "non-zero return code", "rc": 2, "start": "2019-08-26 09:12:23.512941", "stderr": "usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]\ [-c COLUMN] [--max-width <integer>] [--fit-width]\ [--print-empty] [--noindent] [--prefix PREFIX]\ [--human-readable]\ <image>\ openstack image show: error: too few arguments", "stderr_lines": [ "usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]", " [-c COLUMN] [--max-width <integer>] [--fit-width]", " [--print-empty] [--noindent] [--prefix PREFIX]", " [--human-readable]", " <image>", "openstack image show: error: too few arguments" ], "stdout": "", "stdout_lines": [] } ...ignoring ~~~ I looked at the ansible playbook /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml, and this here is the part that creates issues: ~~~ - name: check there's an image in glance already shell: | openstack image list --property owner={{ project_id_result.stdout }} --private --name {{ amphora_image }} -c ID -f value environment: OS_USERNAME: "{{ auth_username }}" OS_PASSWORD: "{{ auth_password }}" OS_PROJECT_NAME: "{{ auth_project_name }}" register: glance_id_result ignore_errors: true - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 - name: get checksum if there's an image in glance already shell: | openstack image show {{ glance_id_result.stdout }} -c checksum -f value environment: OS_USERNAME: "{{ auth_username }}" OS_PASSWORD: "{{ auth_password }}" OS_PROJECT_NAME: "{{ auth_project_name }}" when: image_id is defined register: glance_results ignore_errors: true ~~~ Version-Release number of selected component (if applicable): [root@undercloud-r430 ~]# rpm -qf /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml openstack-tripleo-common-8.6.8-11.el7ost.noarch How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: a) check if there's an octavia image in glance already ... ~~~ TASK [octavia-undercloud : check there\'s an image in glance already] *********** task path: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml:27 Monday 26 August 2019 09:12:20 +0200 (0:00:02.366) 0:00:12.017 ********* Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py <localhost> ESTABLISH LOCAL CONNECTION FOR USER: mistral <localhost> EXEC /bin/sh -c \'OS_PROJECT_NAME=service COMPUTE_API_VERSION=1.1 OS_USERNAME=octavia OS_IDENTITY_API_VERSION=3 OS_USER_DOMAIN_NAME=Default OS_CLOUDNAME=overcloud OS_AUTH_URL=https://openstack.ctnpro.cloud.probtp.com:13000/v3 NOVA_VERSION=1.1 OS_IMAGE_API_VERSION=2 OS_PASSWORD=eEGGJqMdMQEwRq8eWCypVbWXz OS_PROJECT_DOMAIN_NAME=Default OS_AUTH_TYPE=password /usr/bin/python && sleep 0\' changed: [localhost] => { "changed": true, "cmd": "openstack image list --property owner=b8467eb710b2478bb764d4e78c3aaf8a --private --name octavia-amphora-13.0-20190806.1.x86_64 -c ID -f value", "delta": "0:00:02.296875", "end": "2019-08-26 09:12:23.250573", "invocation": { "module_args": { "_raw_params": "openstack image list --property owner=b8467eb710b2478bb764d4e78c3aaf8a --private --name octavia-amphora-13.0-20190806.1.x86_64 -c ID -f value", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": true } }, "rc": 0, "start": "2019-08-26 09:12:20.953698", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [] } ~~~ The above seems to determine that there is an image for octavia. Could you please run the following and provide the output: ~~~ source overcloudrc openstack image list --property owner=b8467eb710b2478bb764d4e78c3aaf8a --private --name octavia-amphora-13.0-20190806.1.x86_64 -c ID -f value source stackrc openstack image list --property owner=b8467eb710b2478bb764d4e78c3aaf8a --private --name octavia-amphora-13.0-20190806.1.x86_64 -c ID -f value ~~~ b) The image id is set to "" in this one here: ~~~ TASK [octavia-undercloud : set image id fact] ********************************** task path: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml:37 Monday 26 August 2019 09:12:23 +0200 (0:00:02.488) 0:00:14.506 ********* ok: [localhost] => { "ansible_facts": { "image_id": "" }, "changed": false } ~~~ c) We are using the empty image ID here and this is why it fails: ~~~ TASK [octavia-undercloud : get checksum if there\'s an image in glance already] *** task path: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml:42 Monday 26 August 2019 09:12:23 +0200 (0:00:00.071) 0:00:14.578 ********* Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py <localhost> ESTABLISH LOCAL CONNECTION FOR USER: mistral <localhost> EXEC /bin/sh -c \'OS_PROJECT_NAME=service COMPUTE_API_VERSION=1.1 OS_USERNAME=octavia OS_IDENTITY_API_VERSION=3 OS_USER_DOMAIN_NAME=Default OS_CLOUDNAME=overcloud OS_AUTH_URL=https://openstack.ctnpro.cloud.probtp.com:13000/v3 NOVA_VERSION=1.1 OS_IMAGE_API_VERSION=2 OS_PASSWORD=eEGGJqMdMQEwRq8eWCypVbWXz OS_PROJECT_DOMAIN_NAME=Default OS_AUTH_TYPE=password /usr/bin/python && sleep 0\' fatal: [localhost]: FAILED! => { "changed": true, "cmd": "openstack image show -c checksum -f value", "delta": "0:00:01.111391", "end": "2019-08-26 09:12:24.624332", "invocation": { "module_args": { "_raw_params": "openstack image show -c checksum -f value", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": true } }, "msg": "non-zero return code", "rc": 2, "start": "2019-08-26 09:12:23.512941", "stderr": "usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]\ [-c COLUMN] [--max-width <integer>] [--fit-width]\ [--print-empty] [--noindent] [--prefix PREFIX]\ [--human-readable]\ <image>\ openstack image show: error: too few arguments", "stderr_lines": [ "usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]", " [-c COLUMN] [--max-width <integer>] [--fit-width]", " [--print-empty] [--noindent] [--prefix PREFIX]", " [--human-readable]", " <image>", "openstack image show: error: too few arguments" ], "stdout": "", "stdout_lines": [] } ...ignoring ~~~
The problem is that glance_id_result.rc is always 0! This needs to be modified: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml From: - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 to - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 and glance_id_result.stdout != "" <------------------------ modify this line to this
I isolated this in my lab: ~~~ [stack@undercloud-r430 ~]$ . overcloudrc (overcloud) [stack@undercloud-r430 ~]$ openstack image list +--------------------------------------+--------+--------+ | ID | Name | Status | +--------------------------------------+--------+--------+ | 90e5ca66-0ccb-4cf2-b8fc-93e11cdc8c79 | cirros | active | | 9e8706dc-23e2-4f12-9241-7eac8456ebf9 | rhel | active | +--------------------------------------+--------+--------+ (overcloud) [stack@undercloud-r430 ~]$ openstack image show rhel +------------------+------------------------------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------------------------------+ | checksum | b7347f03ae4b71fc5294885250d33ee1 | | container_format | bare | | created_at | 2019-08-19T21:08:15Z | | disk_format | qcow2 | | file | /v2/images/9e8706dc-23e2-4f12-9241-7eac8456ebf9/file | | id | 9e8706dc-23e2-4f12-9241-7eac8456ebf9 | | min_disk | 0 | | min_ram | 0 | | name | rhel | | owner | 260bbe0e4ace425eaa524487df2e45f9 | | properties | direct_url='swift+config://ref1/glance/9e8706dc-23e2-4f12-9241-7eac8456ebf9' | | protected | False | | schema | /v2/schemas/image | | size | 625147904 | | status | active | | tags | | | updated_at | 2019-08-26T14:27:18Z | | virtual_size | None | | visibility | private | +------------------+------------------------------------------------------------------------------+ ~~~ My test script in my lab: ~~~ [stack@undercloud-r430 ~]$ [stack@undercloud-r430 ~]$ cat playbook.yml --- - name: TEST hosts: localhost vars: OS_USER_DOMAIN_NAME: "Default" OS_IMAGE_API_VERSION: "2" OS_PROJECT_NAME: "admin" OS_IDENTITY_API_VERSION: "3" OS_PASSWORD: "WZsrxxzHxgNaWV4DgqfADJ3uW" OS_AUTH_TYPE: "password" OS_AUTH_URL: "http://10.0.0.10:5000//v3" OS_USERNAME: "admin" OS_VOLUME_API_VERSION: "3" OS_NO_CACHE: "True" OS_CLOUDNAME: "overcloud" OS_PROJECT_DOMAIN_NAME: "Default" project_id_result: stdout: "260bbe0e4ace425eaa524487df2e45f9" amphora_image: "rhel2" tasks: - name: check there's an image in glance already shell: | openstack image list --property owner={{ project_id_result.stdout }} --private --name {{ amphora_image }} -c ID -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" register: glance_id_result ignore_errors: true - name: akaris debug 1 - glance_id_result debug: var: glance_id_result - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 - name: get checksum if there's an image in glance already shell: | openstack image show {{ glance_id_result.stdout }} -c checksum -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" when: image_id is defined register: glance_results ignore_errors: true ~~~ This fails: ~~~ [stack@undercloud-r430 ~]$ ansible-playbook playbook.yml [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [TEST] ******************************************************************************************************************************************************************************************************************* TASK [Gathering Facts] ******************************************************************************************************************************************************************************************************** ok: [localhost] TASK [check there's an image in glance already] ******************************************************************************************************************************************************************************* changed: [localhost] TASK [akaris debug 1 - glance_id_result] ************************************************************************************************************************************************************************************** ok: [localhost] => { "glance_id_result": { "changed": true, "cmd": "openstack image list --property owner=260bbe0e4ace425eaa524487df2e45f9 --private --name rhel2 -c ID -f value", "delta": "0:00:03.020490", "end": "2019-08-26 10:46:03.664938", "failed": false, "rc": 0, "start": "2019-08-26 10:46:00.644448", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [] } } TASK [set image id fact] ****************************************************************************************************************************************************************************************************** ok: [localhost] TASK [get checksum if there's an image in glance already] ********************************************************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": true, "cmd": "openstack image show -c checksum -f value", "delta": "0:00:01.709394", "end": "2019-08-26 10:46:05.944674", "msg": "non-zero return code", "rc": 2, "start": "2019-08-26 10:46:04.235280", "stderr": "usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]\n [-c COLUMN] [--max-width <integer>] [--fit-width]\n [--print-empty] [--noindent] [--prefix PREFIX]\n [--human-readable]\n <image>\nopenstack image show: error: too few arguments", "stderr_lines": ["usage: openstack image show [-h] [-f {json,shell,table,value,yaml}]", " [-c COLUMN] [--max-width <integer>] [--fit-width]", " [--print-empty] [--noindent] [--prefix PREFIX]", " [--human-readable]", " <image>", "openstack image show: error: too few arguments"], "stdout": "", "stdout_lines": []} ...ignoring PLAY RECAP ******************************************************************************************************************************************************************************************************************** localhost : ok=5 changed=2 unreachable=0 failed=0 ~~~ And with the fix: ~~~ [stack@undercloud-r430 ~]$ cat playbook.yml --- - name: TEST hosts: localhost vars: OS_USER_DOMAIN_NAME: "Default" OS_IMAGE_API_VERSION: "2" OS_PROJECT_NAME: "admin" OS_IDENTITY_API_VERSION: "3" OS_PASSWORD: "WZsrxxzHxgNaWV4DgqfADJ3uW" OS_AUTH_TYPE: "password" OS_AUTH_URL: "http://10.0.0.10:5000//v3" OS_USERNAME: "admin" OS_VOLUME_API_VERSION: "3" OS_NO_CACHE: "True" OS_CLOUDNAME: "overcloud" OS_PROJECT_DOMAIN_NAME: "Default" project_id_result: stdout: "260bbe0e4ace425eaa524487df2e45f9" amphora_image: "rhel2" tasks: - name: check there's an image in glance already shell: | openstack image list --property owner={{ project_id_result.stdout }} --private --name {{ amphora_image }} -c ID -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" register: glance_id_result ignore_errors: true - name: akaris debug 1 - glance_id_result debug: var: glance_id_result - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 and glance_id_result.stdout != "" - name: get checksum if there's an image in glance already shell: | openstack image show {{ glance_id_result.stdout }} -c checksum -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" when: image_id is defined register: glance_results ignore_errors: true ~~~ ~~~ [stack@undercloud-r430 ~]$ cat playbook.yml --- - name: TEST hosts: localhost vars: OS_USER_DOMAIN_NAME: "Default" OS_IMAGE_API_VERSION: "2" OS_PROJECT_NAME: "admin" OS_IDENTITY_API_VERSION: "3" OS_PASSWORD: "WZsrxxzHxgNaWV4DgqfADJ3uW" OS_AUTH_TYPE: "password" OS_AUTH_URL: "http://10.0.0.10:5000//v3" OS_USERNAME: "admin" OS_VOLUME_API_VERSION: "3" OS_NO_CACHE: "True" OS_CLOUDNAME: "overcloud" OS_PROJECT_DOMAIN_NAME: "Default" project_id_result: stdout: "260bbe0e4ace425eaa524487df2e45f9" amphora_image: "rhel2" tasks: - name: check there's an image in glance already shell: | openstack image list --property owner={{ project_id_result.stdout }} --private --name {{ amphora_image }} -c ID -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" register: glance_id_result ignore_errors: true - name: akaris debug 1 - glance_id_result debug: var: glance_id_result - name: set image id fact set_fact: image_id: "{{ glance_id_result.stdout }}" when: glance_id_result.rc == 0 and glance_id_result.stdout != "" - name: get checksum if there's an image in glance already shell: | openstack image show {{ glance_id_result.stdout }} -c checksum -f value environment: OS_USER_DOMAIN_NAME: "{{ OS_USER_DOMAIN_NAME }}" OS_IMAGE_API_VERSION: "{{ OS_IMAGE_API_VERSION }}" OS_PROJECT_NAME: "{{ OS_PROJECT_NAME }}" OS_IDENTITY_API_VERSION: "{{ OS_IDENTITY_API_VERSION }}" OS_PASSWORD: "{{ OS_PASSWORD }}" OS_AUTH_TYPE: "{{ OS_AUTH_TYPE }}" OS_AUTH_URL: "{{ OS_AUTH_URL }}" OS_USERNAME: "{{ OS_USERNAME }}" OS_VOLUME_API_VERSION: "{{ OS_VOLUME_API_VERSION }}" OS_NO_CACHE: "{{ OS_NO_CACHE }}" OS_CLOUDNAME: "{{ OS_CLOUDNAME }}" OS_PROJECT_DOMAIN_NAME: "{{ OS_PROJECT_DOMAIN_NAME }}" when: image_id is defined register: glance_results ignore_errors: true ~~~
We now fail after this: ~~~ TASK [octavia-undercloud : set current_md5 fact from glance if image already exists there] *** task path: /usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml:62 Monday 26 August 2019 17:08:20 +0200 (0:00:00.054) 0:00:15.946 ********* fatal: [localhost]: FAILED! => { "msg": "The conditional check \'glance_results.rc == 0\' failed. The error was: error while evaluating conditional (glance_results.rc == 0): \'dict object\' has no attribute \'rc\'\ \ The error appears to have been in \'/usr/share/openstack-tripleo-common/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml\': line 62, column 5, but may\ be elsewhere in the file depending on the exact syntax problem.\ \ The offending line appears to be:\ \ \ - name: set current_md5 fact from glance if image already exists there\ ^ here\ " } ~~~ Need to change: - name: set current_md5 fact from glance if image already exists there set_fact: current_md5: "{{ glance_results.stdout }}" when: glance_results.rc == 0 to: - name: set current_md5 fact from glance if image already exists there set_fact: current_md5: "{{ glance_results.stdout }}" when: glance_results.rc is defined and glance_results.rc == 0