Description of problem: FFU: running deploy_steps_playbook.yaml on a specific compute node fails with: TASK [check if libvirt is installed] ******************************************* fatal: [192.168.0.24]: FAILED! => {"msg": "The conditional check 'False' failed. The error was: expected string or buffer"} Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Deploy OSP10: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ -e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \ -e ~/openstack_deployment/environments/nodes.yaml \ -e ~/openstack_deployment/environments/network-environment.yaml \ -e ~/openstack_deployment/environments/disk-layout.yaml \ -e ~/openstack_deployment/environments/public_vip.yaml \ -e ~/openstack_deployment/environments/enable-tls.yaml \ -e ~/openstack_deployment/environments/inject-trust-anchor.yaml \ -e ~/openstack_deployment/environments/scheduler_hints_env.yaml \ -e ~/openstack_deployment/environments/ips-from-pool-all.yaml \ -e ~/openstack_deployment/environments/neutron-settings.yaml \ -e ~/openstack_deployment/environments/custom_hiera.yaml \ --log-file overcloud_deployment.log &> overcloud_install.log 2. Upgrade undercloud to OSP11/12/13 3. Update stack outputs: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ -e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \ -e ~/openstack_deployment/environments/nodes.yaml \ -e ~/openstack_deployment/environments/network-environment.yaml \ -e ~/openstack_deployment/environments/disk-layout.yaml \ -e ~/openstack_deployment/environments/public_vip.yaml \ -e ~/openstack_deployment/environments/enable-tls.yaml \ -e ~/openstack_deployment/environments/inject-trust-anchor.yaml \ -e ~/openstack_deployment/environments/scheduler_hints_env.yaml \ -e ~/openstack_deployment/environments/ips-from-pool-all.yaml \ -e ~/openstack_deployment/environments/neutron-settings.yaml \ -e ~/openstack_deployment/environments/custom_hiera.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/fast-forward-upgrade.yaml \ -e /home/stack/ffu_repos.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/config-download-environment.yaml \ -e /home/stack/ceph-ansible-env.yaml \ 4. ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-ureIl_-config/fast_forward_upgrade_playbook.yaml 5. Upgrade controllers: ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-z6Gps8-config/upgrade_steps_playbook.yaml --skip-tags=validation --limit=\!Compute ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-z6Gps8-config/deploy_steps_playbook.yaml --limit=\!Compute 6. Upgrade compute-r00-00 compute node: ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-ureIl_-config/upgrade_steps_playbook.yaml --skip-tags=validation --limit=compute-r00-00 ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-ureIl_-config/deploy_steps_playbook.yaml --limit=compute-r00-00 Actual results: Fails with: TASK [check if libvirt is installed] ******************************************* fatal: [192.168.0.24]: FAILED! => {"msg": "The conditional check 'False' failed. The error was: expected string or buffer"} Expected results: Doesn't fail. Additional info: Attaching full output of the failing playbook.
Created attachment 1419613 [details] logs and playbooks Attaching the playbooks and their outputs.The error can be seen in ffu_deploy_steps_playbook_script_compute-r00-00.log file
Running the playbook with increased verbosity: ansible-playbook --module-path /usr/share/ansible-modules/ -i /home/stack/tripleo-ansible-inventory-static.yaml -b /home/stack/tripleo-ureIl_-config/deploy_steps_playbook.yaml --limit=compute-r00-00 -vvvvvvv TASK [check if libvirt is installed] ********************************************************************************************************************************************************************************************************* task path: /home/stack/tripleo-ureIl_-config/Compute/host_prep_tasks.yaml:88 Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py <192.168.0.24> ESTABLISH SSH CONNECTION FOR USER: heat-admin <192.168.0.24> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <192.168.0.24> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <192.168.0.24> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=heat-admin) <192.168.0.24> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <192.168.0.24> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/stack/.ansible/cp/cf7b81f744) <192.168.0.24> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=heat-admin -o ConnectTimeout=10 -o ControlPath=/home/stack/.ansible/cp/cf7b81f744 192.168.0.24 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"'' <192.168.0.24> (0, '/home/heat-admin\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /home/stack/.ssh/config\r\ndebug1: /home/stack/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21364\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <192.168.0.24> ESTABLISH SSH CONNECTION FOR USER: heat-admin <192.168.0.24> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <192.168.0.24> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <192.168.0.24> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=heat-admin) <192.168.0.24> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <192.168.0.24> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/stack/.ansible/cp/cf7b81f744) <192.168.0.24> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=heat-admin -o ConnectTimeout=10 -o ControlPath=/home/stack/.ansible/cp/cf7b81f744 192.168.0.24 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768 `" && echo ansible-tmp-1523318477.13-97902695833768="` echo /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768 `" ) && sleep 0'"'"'' <192.168.0.24> (0, 'ansible-tmp-1523318477.13-97902695833768=/home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /home/stack/.ssh/config\r\ndebug1: /home/stack/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21364\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <192.168.0.24> PUT /tmp/tmp_bN881 TO /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/command.py <192.168.0.24> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <192.168.0.24> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <192.168.0.24> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=heat-admin) <192.168.0.24> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <192.168.0.24> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/stack/.ansible/cp/cf7b81f744) <192.168.0.24> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=heat-admin -o ConnectTimeout=10 -o ControlPath=/home/stack/.ansible/cp/cf7b81f744 '[192.168.0.24]' <192.168.0.24> (0, 'sftp> put /tmp/tmp_bN881 /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/command.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /home/stack/.ssh/config\r\ndebug1: /home/stack/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21364\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename" revision 1\r\ndebug2: Server supports extension "statvfs" revision 2\r\ndebug2: Server supports extension "fstatvfs" revision 2\r\ndebug2: Server supports extension "hardlink" revision 1\r\ndebug2: Server supports extension "fsync" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/heat-admin size 0\r\ndebug3: Looking up /tmp/tmp_bN881\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/command.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:1279\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 1279 bytes at 65536\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <192.168.0.24> ESTABLISH SSH CONNECTION FOR USER: heat-admin <192.168.0.24> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <192.168.0.24> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <192.168.0.24> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=heat-admin) <192.168.0.24> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <192.168.0.24> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/stack/.ansible/cp/cf7b81f744) <192.168.0.24> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=heat-admin -o ConnectTimeout=10 -o ControlPath=/home/stack/.ansible/cp/cf7b81f744 192.168.0.24 '/bin/sh -c '"'"'chmod u+x /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/ /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/command.py && sleep 0'"'"'' <192.168.0.24> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /home/stack/.ssh/config\r\ndebug1: /home/stack/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21364\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <192.168.0.24> ESTABLISH SSH CONNECTION FOR USER: heat-admin <192.168.0.24> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <192.168.0.24> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <192.168.0.24> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=heat-admin) <192.168.0.24> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <192.168.0.24> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/stack/.ansible/cp/cf7b81f744) <192.168.0.24> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=heat-admin -o ConnectTimeout=10 -o ControlPath=/home/stack/.ansible/cp/cf7b81f744 -tt 192.168.0.24 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gtwvdcxkztsyjqwylkvbvygpzgseraly; /usr/bin/python /home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/command.py; rm -rf "/home/heat-admin/.ansible/tmp/ansible-tmp-1523318477.13-97902695833768/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"'' <192.168.0.24> (0, '\r\n{"changed": true, "end": "2018-04-10 00:01:17.438080", "stdout": "libvirt-daemon-3.9.0-14.el7.x86_64", "cmd": ["/usr/bin/rpm", "-q", "libvirt-daemon"], "rc": 0, "start": "2018-04-10 00:01:17.395856", "stderr": "", "delta": "0:00:00.042224", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "/usr/bin/rpm -q libvirt-daemon", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "warnings": ["Consider using yum, dnf or zypper module rather than running rpm"]}\r\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /home/stack/.ssh/config\r\ndebug1: /home/stack/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21364\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 192.168.0.24 closed.\r\n') fatal: [192.168.0.24]: FAILED! => { "msg": "The conditional check 'False' failed. The error was: expected string or buffer" } Running the failing task adhoc works fine: [stack@undercloud-0 ~]$ ansible -i tripleo-ansible-inventory-static.yaml -m command -a '/usr/bin/rpm -q libvirt-daemon' compute-r00-00 [WARNING]: Skipping unexpected key (hostvars) in group (_meta), only "vars", "children" and "hosts" are valid [WARNING]: Consider using yum, dnf or zypper module rather than running rpm 192.168.0.24 | SUCCESS | rc=0 >> libvirt-daemon-3.9.0-14.el7.x86_64
*** Bug 1565395 has been marked as a duplicate of this bug. ***
Changed assignee from default (me) to UA of the DFG who owns this BZ.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086