Description of problem: While we are upgrading the existing cluster from 7.9+ OSP13 to 8.2-OSP16.1. Upgrading of two contrail-controllers and openstack worked, but while upgrade contrail-controller-2 which is currently in 7.9 failed to upgrade to 8.2. ======== TASK [Check that the /etc/leapp/files/repomap.csv exists on UC] **************** Friday 24 September 2021 15:10:27 -0400 (0:00:00.075) 0:00:09.548 ****** skipping: [overcloud-controller-2] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Stop all services by stopping all docker containers] ********************* Friday 24 September 2021 15:10:26 -0400 (0:00:01.752) 0:00:10.433 ****** TASK [tripleo-podman : Check if docker is enabled in the system] *************** Friday 24 September 2021 15:10:27 -0400 (0:00:00.101) 0:00:10.534 ****** ok: [overcloud-contrailcontroller-2] => {"changed": false, "failed_when_result": false, "stat": {"atime": 1631949801.3817253, "attr_flags": "", "attributes": [], "block_size": 4096, "blocks": 0, "charset": "binary", "ctime": 1631949801.3817253, "dev": 20, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 7167609, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1631949801.3817253, "nlink": 2, "path": "/var/run/docker.sock", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 40, "uid": 0, "version": null, "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}} TASK [tripleo-podman : Stop all services by stopping all Docker containers] **** Friday 24 September 2021 15:10:27 -0400 (0:00:00.235) 0:00:10.770 ****** fatal: [overcloud-contrailcontroller-2]: FAILED! => {"changed": true, "cmd": ["docker", "ps", "-aq"], "delta": "0:00:00.026731", "end": "2021-09-24 19:10:27.497949", "msg": "non-zero return code", "rc": 1, "start": "2021-09-24 19:10:27.471218", "stderr": "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?", "stderr_lines": ["Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"], "stdout": "", "stdout_lines": []} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* overcloud-contrailcontroller-2 : ok=20 changed=3 unreachable=0 failed=1 skipped=21 rescued=0 ignored=0 Friday 24 September 2021 15:10:27 -0400 (0:00:00.262) 0:00:11.033 ****** =============================================================================== Render all_nodes data as group_vars for overcloud ----------------------- 2.35s Destroy pacemaker cluster ----------------------------------------------- 1.75s Gathering Facts --------------------------------------------------------- 1.17s Stop pacemaker cluster before stopping all docker containers ------------ 0.63s Gather missing facts ---------------------------------------------------- 0.43s zk volume path ---------------------------------------------------------- 0.43s Check if config zk data not moved yet ----------------------------------- 0.43s include_tasks ----------------------------------------------------------- 0.32s cassandra volume path --------------------------------------------------- 0.27s tripleo-podman : Stop all services by stopping all Docker containers ---- 0.26s tripleo-podman : Check if docker is enabled in the system --------------- 0.24s Check if pcs is present ------------------------------------------------- 0.22s Check if config db data not moved yet ----------------------------------- 0.22s Set all_nodes data as group_vars for overcloud -------------------------- 0.15s include_tasks ----------------------------------------------------------- 0.11s Stop all services by stopping all docker containers --------------------- 0.10s include_tasks ----------------------------------------------------------- 0.09s include_vars ------------------------------------------------------------ 0.09s move config database zk data -------------------------------------------- 0.08s Version-Release number of selected component (if applicable): How reproducible: Upgrade procedure using leapp Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
[root@overcloud-contrailcontroller-2 heat-admin]# systemctl restart docker Sep 24 19:32:31 overcloud-contrailcontroller-2 systemd: Starting SystemWide Container Registries... Sep 24 19:32:31 overcloud-contrailcontroller-2 systemd: Starting SystemWide Container Registries... Sep 24 19:32:31 overcloud-contrailcontroller-2 systemd: Starting Docker Storage Setup... Sep 24 19:32:31 overcloud-contrailcontroller-2 systemd: Starting Docker Storage Setup...Sep 24 19:32:32 overcloud-contrailcontroller-2 container-storage-setup: INFO: Volume group backing root filesystem could not be determinedSep 24 19:32:32 overcloud-contrailcontroller-2 container-storage-setup: INFO: Volume group backing root filesystem could not be determined Sep 24 19:32:32 overcloud-contrailcontroller-2 container-storage-setup: INFO: Storage is already configured with o driver. Can't configure it with overlay2 driver. To override, remove /etc/sysconfig/docker-storage and retry. Sep 24 19:32:32 overcloud-contrailcontroller-2 container-storage-setup: INFO: Storage is already configured with o driver. Can't configure it with overlay2 driver. To override, remove /etc/sysconfig/docker-storage and retry. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Started Docker Storage Setup. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Started Docker Storage Setup. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Started SystemWide Container Registries. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Started SystemWide Container Registries. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Starting Docker Application Container Engine... Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Starting Docker Application Container Engine... Sep 24 19:32:32 overcloud-contrailcontroller-2 dockerd-current: can't create unix socket /run/docker.sock: is a directorySep 24 19:32:32 overcloud-contrailcontroller-2 dockerd-current: can't create unix socket /run/docker.sock: is a directorySep 24 19:32:32 overcloud-contrailcontroller-2 systemd: docker.service: main process exited, code=exited, status=1/FAILURE Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: docker.service: main process exited, code=exited, status=1/FAILURE Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Failed to start Docker Application Container Engine. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Unit docker.service entered failed state. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: docker.service failed. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Failed to start Docker Application Container Engine. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: Unit docker.service entered failed state. Sep 24 19:32:32 overcloud-contrailcontroller-2 systemd: docker.service failed. Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
Drop-In: /etc/systemd/system/docker.service.d └─99-unset-mountflags.conf Active: failed (Result: exit-code) since Fri 2021-09-24 19:31:16 UTC; 12s ago Docs: http://docs.docker.com Process: 529768 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE) Main PID: 529768 (code=exited, status=1/FAILURE) Tasks: 195 Memory: 92.0M CGroup: /system.slice/docker.service ├─3720 /usr/bin/docker-containerd-shim-current 0c8a94f24fd8190c077ff8424682aa671dec8074f2f4141043aecd3783ea2252 /var/run/docker/libcontainerd/0c8a94f24fd8190c077ff8424682aa671dec8074f2f4141043aecd3783ea2252 /u... ├─3739 /usr/bin/docker-containerd-shim-current 1a6a52f658117b600d9b56c2d976af5657ab89e0e6d2c979d2c9ea17278aa707 /var/run/docker/libcontainerd/1a6a52f658117b600d9b56c2d976af5657ab89e0e6d2c979d2c9ea17278aa707 /u... ├─3804 /usr/bin/docker-containerd-shim-current 17a3ba2d3b9da20cd1d4f31ed9addd78a7840336f3dda927a6dcccbdb68a9329 /var/run/docker/libcontainerd/17a3ba2d3b9da20cd1d4f31ed9addd78a7840336f3dda927a6dcccbdb68a9329 /u... ├─3833 /usr/bin/docker-containerd-shim-current aab24bb4d493175e243e028b710ddb038883d18f346a0b53c58f576e1718c00c /var/run/docker/libcontainerd/aab24bb4d493175e243e028b710ddb038883d18f346a0b53c58f576e1718c00c /u... ├─3861 /usr/bin/docker-containerd-shim-current 6a6209383dc18948fa84cb19f9fcbd611a49cdcfb81faf6e66e223716045864a /var/run/docker/libcontainerd/6a6209383dc18948fa84cb19f9fcbd611a49cdcfb81faf6e66e223716045864a /u... ├─3918 /usr/bin/docker-containerd-shim-current 4880ef39edfa8417d963c08858d2774b2f174efa07ff5b0af926693000612846 /var/run/docker/libcontainerd/4880ef39edfa8417d963c08858d2774b2f174efa07ff5b0af926693000612846 /u... ├─3946 /usr/bin/docker-containerd-shim-current e744e6aae60201628ab8195f0ae5eb0d3bfc56ef740fc2592657d8180b787607 /var/run/docker/libcontainerd/e744e6aae60201628ab8195f0ae5eb0d3bfc56ef740fc2592657d8180b787607 /u... ├─4147 /usr/bin/docker-containerd-shim-current e7068b5e0cdf974457e3267b21ff2948219966b53b5e2d3c5c9a14acdb5ceb02 /var/run/docker/libcontainerd/e7068b5e0cdf974457e3267b21ff2948219966b53b5e2d3c5c9a14acdb5ceb02 /u... ├─4243 /usr/bin/docker-containerd-shim-current 5e2e688989b10a3ae61de717e7030f134db538039700df69987e5ce285f58108 /var/run/docker/libcontainerd/5e2e688989b10a3ae61de717e7030f134db538039700df69987e5ce285f58108 /u... ├─4969 /usr/bin/docker-containerd-shim-current 2afd76526d5427203d0486e6325ddd3dc2e594b2f6ed5b5836639f8b58430401 /var/run/docker/libcontainerd/2afd76526d5427203d0486e6325ddd3dc2e594b2f6ed5b5836639f8b58430401 /u... ├─5460 /usr/bin/docker-containerd-shim-current e4d195865e33db36938faed976540629d2308defbc44ca262513eaa99bec395a /var/run/docker/libcontainerd/e4d195865e33db36938faed976540629d2308defbc44ca262513eaa99bec395a /u... ├─5708 /usr/bin/docker-containerd-shim-current 7a52d66fc7d7be59835de9af16fa9c15e8b6463adca33e400e027b3084ef4118 /var/run/docker/libcontainerd/7a52d66fc7d7be59835de9af16fa9c15e8b6463adca33e400e027b3084ef4118 /u... ├─8006 /usr/bin/docker-containerd-shim-current 4adc170661c7902c3c8d7231a2f6171923c75cd0877b086076792c94fd2bc269 /var/run/docker/libcontainerd/4adc170661c7902c3c8d7231a2f6171923c75cd0877b086076792c94fd2bc269 /u... ├─8250 /usr/bin/docker-containerd-shim-current 07e1197994511f3af94ca987e6179377b122ece3feaccd4214161f7cb9560d42 /var/run/docker/libcontainerd/07e1197994511f3af94ca987e6179377b122ece3feaccd4214161f7cb9560d42 /u... ├─8325 /usr/bin/docker-containerd-shim-current 8adc4aa4b70dc656579730d411e0ac257b61683c90d53a2bb79d77945c521e4e /var/run/docker/libcontainerd/8adc4aa4b70dc656579730d411e0ac257b61683c90d53a2bb79d77945c521e4e /u... ├─8442 /usr/bin/docker-containerd-shim-current 86785649d84cc7473311ce2f7824476c37cedf9a494d214f8249606616097c7c /var/run/docker/libcontainerd/86785649d84cc7473311ce2f7824476c37cedf9a494d214f8249606616097c7c /u... ├─8591 /usr/bin/docker-containerd-shim-current 48e6e2c49a84936f0d0205b7a861ba21b330ad1e1824142695ff4b44625f467d /var/run/docker/libcontainerd/48e6e2c49a84936f0d0205b7a861ba21b330ad1e1824142695ff4b44625f467d /u... └─8787 /usr/bin/docker-containerd-shim-current 7513fd164f29fc2a67ef0216f5371242e9a4b467fd505e1a80c631572126aa04 /var/run/docker/libcontainerd/7513fd164f29fc2a67ef0216f5371242e9a4b467fd505e1a80c631572126aa04 /u... Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local systemd[1]: Starting Docker Application Container Engine... Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local dockerd-current[529768]: can't create unix socket /run/docker.sock: is a directory Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local systemd[1]: Failed to start Docker Application Container Engine. Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local systemd[1]: Unit docker.service entered failed state. Sep 24 19:31:16 overcloud-contrailcontroller-2.5b6s1.local systemd[1]: docker.service failed. [root@overcloud-contrailcontroller-2 heat-admin]# systemctl stop docker [root@overcloud-contrailcontroller-2 heat-admin]# systemctl start docker Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. [root@overcloud-contrailcontroller-2 heat-admin]# systemctl status docker.service ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─99-unset-mountflags.conf Active: failed (Result: exit-code) since Fri 2021-09-24 19:31:39 UTC; 10s ago Docs: http://docs.docker.com Process: 529843 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE) Main PID: 529843 (code=exited, statu
This turned out to be due to https://bugzilla.redhat.com/show_bug.cgi?id=1966968