Description of problem: After upgrading RHOSP13 to RHOSP16.1, I'm following the documentation below to migrate ml2ovs to ovn. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/networking_with_open_virtual_network/migrating-ml2ovs-to-ovn While running "ovn_migration.sh setup-mtu-t1", I noticed that neutron dhcp agent containers are not restarted because of missing "docker" command in overcloud nodes. The playbook execution shows success because it ignores these failures. ~~~ (overcloud) [stack@undercloud-0 ovn_migration]$ ovn_migration.sh setup-mtu-t1 ansible-playbook 2.9.13 config file = /home/stack/ovn_migration/ansible.cfg configured module search path = ['/home/stack/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.6/site-packages/ansible executable location = /usr/bin/ansible-playbook python version = 3.6.8 (default, Dec 5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] ... TASK [Get the neutron dhcp agent docker id] ************************************************************************************************************************************************************************************************** task path: /home/stack/ovn_migration/playbooks/reduce-dhcp-renewal-time.yml:16 Sunday 13 September 2020 14:50:26 +0000 (0:00:00.836) 0:00:04.606 ****** changed: [controller-0] => {"changed": true, "cmd": "docker ps | grep neutron_dhcp | awk '{print $1}'", "delta": "0:00:00.006763", "end": "2020-09-13 14:50:26.998275", "rc": 0, "start": "2020-09-13 14:50:26.991512", "stderr": "/bin/sh: docker: command not found", "stderr_lines": ["/bin/sh: docker: command not found"], "stdout": "", "stdout_lines": []} ... TASK [Restart neutron dhcp agent] ************************************************************************************************************************************************************************************************************ task path: /home/stack/ovn_migration/playbooks/reduce-dhcp-renewal-time.yml:22 Sunday 13 September 2020 14:50:27 +0000 (0:00:00.715) 0:00:05.322 ****** fatal: [controller-0]: FAILED! => {"changed": false, "cmd": "docker restart", "msg": "[Errno 2] No such file or directory: b'docker': b'docker'", "rc": 2} ...ignoring ... META: ran handlers META: ran handlers PLAY RECAP *********************************************************************************************************************************************************************************************************************************** controller-0 : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1 controller-1 : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1 controller-2 : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1 ~~~ Actually docker command is not available in overcloud nodes after upgrade. The migration playbook should use podman, or install podman-docker if it relies on docker command. ~~~ [heat-admin@controller-0 ~]$ rpm -qa | grep podman | sort podman-1.6.4-16.module+el8.2.0+7659+b700d80e.x86_64 [heat-admin@controller-0 ~]$ which docker /usr/bin/which: no docker in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/heat-admin/.local/bin:/home/heat-admin/bin) ~~~ After installing podman-docker in overcloud nodes, dhcp agents are restarted properly by the playbook. ~~~ (overcloud) [stack@undercloud-0 ovn_migration]$ ovn_migration.sh setup-mtu-t1 ... TASK [Get the neutron dhcp agent docker id] ************************************************************************************************************************************************************************************************** task path: /home/stack/ovn_migration/playbooks/reduce-dhcp-renewal-time.yml:16 Sunday 13 September 2020 14:59:33 +0000 (0:00:01.136) 0:00:01.252 ****** changed: [controller-0] => {"changed": true, "cmd": "docker ps | grep neutron_dhcp | awk '{print $1}'", "delta": "0:00:00.125135", "end": "2020-09-13 14:59:34.368226", "rc": 0, "start": "2020-09-13 14:59:34.243091", "stderr": "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.", "stderr_lines": ["Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg."], "stdout": "efb2283e823b", "stdout_lines": ["efb2283e823b"]} ... TASK [Restart neutron dhcp agent] ************************************************************************************************************************************************************************************************************ task path: /home/stack/ovn_migration/playbooks/reduce-dhcp-renewal-time.yml:22 Sunday 13 September 2020 14:59:34 +0000 (0:00:00.897) 0:00:02.150 ****** changed: [controller-0] => {"changed": true, "cmd": ["docker", "restart", "efb2283e823b"], "delta": "0:00:10.882992", "end": "2020-09-13 14:59:45.909760", "rc": 0, "start": "2020-09-13 14:59:35.026768", "stderr": "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.\ntime=\"2020-09-13T14:59:45Z\" level=error msg=\"Failed to remove paths: map[cpuset:/sys/fs/cgroup/cpuset/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope freezer:/sys/fs/cgroup/freezer/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope hugetlb:/sys/fs/cgroup/hugetlb/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope net_cls:/sys/fs/cgroup/net_cls,net_prio/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope net_prio:/sys/fs/cgroup/net_cls,net_prio/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope perf_event:/sys/fs/cgroup/perf_event/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope]\"", "stderr_lines": ["Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.", "time=\"2020-09-13T14:59:45Z\" level=error msg=\"Failed to remove paths: map[cpuset:/sys/fs/cgroup/cpuset/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope freezer:/sys/fs/cgroup/freezer/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope hugetlb:/sys/fs/cgroup/hugetlb/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope net_cls:/sys/fs/cgroup/net_cls,net_prio/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope net_prio:/sys/fs/cgroup/net_cls,net_prio/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope perf_event:/sys/fs/cgroup/perf_event/machine.slice/libpod-efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563.scope]\""], "stdout": "efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563", "stdout_lines": ["efb2283e823bd7f630f5ebe583ef57428bc7b9b4e5649cd06454168039bb0563"]} ... META: ran handlers META: ran handlers PLAY RECAP *********************************************************************************************************************************************************************************************************************************** controller-0 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 controller-1 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 controller-2 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ~~~ Version-Release number of selected component (if applicable): python3-networking-ovn-migration-tool-7.2.1-0.20200611133439.15f2281.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. Upgrade RHOSP13 to RHOSP16.1 (*1) 2. Follow Migration document (*1) We can reproduce the same situation by uninstalling podman-docker package from deployed overcloud nodes Actual results: neutron dhcp containers are not restarted during "ovn_migration.sh setup-mtu-t1" Expected results: neutron dhcp containers are restarted during "ovn_migration.sh setup-mtu-t1" Additional info:
*** This bug has been marked as a duplicate of bug 1855360 ***