Description of problem:

Updating from Z12 to latest is failing on ceph update which fails the overcloud update.
the task is failing inspect ceph mon container.
using ceph-ansible 3.3
http://download.eng.bos.redhat.com/rhel-7/composes/auto/ceph-3.3-rhel-7/latest-RHCEPH-3-RHEL-7/compose/Tools/x86_64/os/Packages/ceph-ansible-3.2.45-1.el7cp.noarch.rpm

Version-Release number of selected component (if applicable):
core_puddle: 2020-06-11.1

How reproducible:
100%

Steps to Reproduce:
1. deploy RHOS13 Z12 with ceph
2. update the overcloud and undercloud to latest

Actual results:
overcloud update failed

Expected results:
pass

Additional info:


inspect ceph mon container] *************************\ntask path: /usr/share/ceph-ansible/roles/ceph-docker-common/tasks/fetch_image.yml:3\nSunday 12 July 2020  13:17:19 -0400 (0:00:00.385)       0:00:50.368 *********** \nok: [192.168.24.20] => {"changed": false, "cmd": ["docker", "inspect", "06e9dee9cd59"], "delta": "0:00:00.037744", "end": "2020-07-12 17:17:19.834438", "rc": 0, "start": "2020-07-12 17:17:19.796694", "stderr": "", "stderr_lines": [], "stdout": "[\\n    {\\n        \\"Id\\": \\"06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f\\",\\n        \\"Created\\": \\"2020-07-12T11:34:11.720829555Z\\",\\n        \\"Path\\": \\"/entrypoint.sh\\",\\n        \\"Args\\": [],\\n        \\"State\\": {\\n            \\"Status\\": \\"running\\",\\n            \\"Running\\": true,\\n            \\"Paused\\": false,\\n            \\"Restarting\\": false,\\n            \\"OOMKilled\\": false,\\n            \\"Dead\\": false,\\n            \\"Pid\\": 43265,\\n            \\"ExitCode\\": 0,\\n            \\"Error\\": \\"\\",\\n            \\"StartedAt\\": \\"2020-07-12T11:34:12.066302937Z\\",\\n            \\"FinishedAt\\": \\"0001-01-01T00:00:00Z\\"\\n        },\\n        \\"Image\\": \\"sha256:4464c3e4b038e3501bb31a61098d3edace33a045bfa22d4fc8b1e2a4d016fd44\\",\\n        \\"ResolvConfPath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/resolv.conf\\",\\n        \\"HostnamePath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/hostname\\",\\n        \\"HostsPath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/hosts\\",\\n        \\"LogPath\\": \\"\\",\\n        \\"Name\\": \\"/ceph-mon-controller-1\\",\\n        \\"RestartCount\\": 0,\\n        \\"Driver\\": \\"overlay2\\",\\n        \\"MountLabel\\": \\"\\",\\n        \\"ProcessLabel\\": \\"\\",\\n        \\"AppArmorProfile\\": \\"\\",\\n        \\"ExecIDs\\": null,\\n        \\"HostConfig\\": {\\n            \\"Binds\\": [\\n                \\"/var/lib/ceph:/var/lib/ceph:z\\",\\n                \\"/etc/ceph:/etc/ceph:z\\",\\n                \\"/var/run/ceph:/var/run/ceph:z\\",\\n                \\"/etc/localtime:/etc/localtime:ro\\"\\n            ],\\n            \\"ContainerIDFile\\": \\"\\",\\n            \\"LogConfig\\": {\\n                \\"Type\\": \\"journald\\",\\n                \\"Config\\": {}\\n            },\\n            \\"NetworkMode\\": \\"host\\",\\n            \\"PortBindings\\": {},\\n            \\"RestartPolicy\\": {\\n                \\"Name\\": \\"no\\",\\n                \\"MaximumRetryCount\\": 0\\n            },\\n            \\"AutoRemove\\": true,\\n            \\"VolumeDriver\\": \\"\\",\\n            \\"VolumesFrom\\": null,\\n            \\"CapAdd\\": null,\\n            \\"CapDrop\\": null,\\n            \\"Dns\\": [],\\n            \\"DnsOptions\\": [],\\n            \\"DnsSearch\\": [],\\n            \\"ExtraHosts\\": null,\\n            \\"GroupAdd\\": null,\\n            \\"IpcMode\\": \\"\\",\\n            \\"Cgroup\\": \\"\\",\\n            \\"Links\\": null,\\n            \\"OomScoreAdj\\": 0,\\n            \\"PidMode\\": \\"\\",\\n            \\"Privileged\\": false,\\n            \\"PublishAllPorts\\": false,\\n            \\"ReadonlyRootfs\\": false,\\n            \\"SecurityOpt\\": null,\\n            \\"UTSMode\\": \\"\\",\\n            \\"UsernsMode\\": \\"\\",\\n            \\"ShmSize\\": 67108864,\\n            \\"Runtime\\": \\"docker-runc\\",\\n            \\"ConsoleSize\\": [\\n                0,\\n                0\\n            ],\\n            \\"Isolation\\": \\"\\",\\n            \\"CpuShares\\": 0,\\n            \\"Memory\\": 33565966336,\\n            \\"NanoCpus\\": 0,\\n            \\"CgroupParent\\": \\"\\",\\n            \\"BlkioWeight\\": 0,\\n            \\"BlkioWeightDevice\\": null,\\n            \\"BlkioDeviceReadBps\\": null,\\n            \\"BlkioDeviceWriteBps\\": null,\\n            \\"BlkioDeviceReadIOps\\": null,\\n            \\"BlkioDeviceWriteIOps\\": null,\\n            \\"CpuPeriod\\": 0,\\n            \\"CpuQuota\\": 100000,\\n            \\"CpuRealtimePeriod\\": 0,\\n            \\"CpuRealtimeRuntime\\": 0,\\n            \\"CpusetCpus\\": \\"\\",\\n            \\"CpusetMems\\": \\"\\",\\n            \\"Devices\\": [],\\n            \\"DiskQuota\\": 0,\\n            \\"KernelMemory\\": 0,\\n            \\"MemoryReservation\\": 0,\\n            \\"MemorySwap\\": 67131932672,\\n            \\"MemorySwappiness\\": -1,\\n            \\"OomKillDisable\\": false,\\n            \\"PidsLimit\\": 0,\\n            \\"Ulimits\\": null,\\n            \\"CpuCount\\": 0,\\n            \\"CpuPercent\\": 0,\\n            \\"IOMaximumIOps\\": 0,\\n            \\"IOMaximumBandwidth\\": 0\\n        },\\n        \\"GraphDriver\\": {\\n            \\"Name\\": \\"overlay2\\",\\n            \\"Data\\": {\\n                \\"LowerDir\\": \\"/var/lib/docker/overlay2/58a2c14d5b6d1205df578622c7138a53a39b2ef496d6f80008de2c9b8f6ecbe1-init/diff:/var/lib/docker/overlay2/ef3aff6e3251eb14f99f98374932981584a0ccf0f4bf7ce271c4fd7f79038335/diff:/var/lib/docker/overlay2/e549b239529262391c0b6a60c38bb0ed1edbf78d1b23c0c8677a88effd05a0b6/diff:/var/lib/docker/overlay2/36dd6962d271d9a9a8d09dfef3ce8f61ac5c3ebf68d8f68f1de6a47c929bf856/diff\\",\\n                \\"MergedDir\\": \\"/var/lib/docker/overlay2/58a2c14d5b6d1205df578622c7138a53a39b2ef496d6f80008de2c9b8f6ecbe1/merged\\",\\n                \\"UpperDir\\": \\"/var/lib/docker/overlay2/58a2c14d5b6d1205df578622c7138a53a39b2ef496d6f80008de2c9b8f6ecbe1/diff\\",\\n                \\"WorkDir\\": \\"/var/lib/docker/overlay2/58a2c14d5b6d1205df578622c7138a53a39b2ef496d6f80008de2c9b8f6ecbe1/work\\"\\n            }\\n        },\\n        \\"Mounts\\": [\\n            {\\n                \\"Type\\": \\"bind\\",\\n                \\"Source\\": \\"/etc/ceph\\",\\n                \\"Destination\\": \\"/etc/ceph\\",\\n                \\"Mode\\": \\"z\\",\\n                \\"RW\\": true,\\n                \\"Propagation\\": \\"rprivate\\"\\n            },\\n            {\\n                \\"Type\\": \\"bind\\",\\n                \\"Source\\": \\"/etc/localtime\\",\\n                \\"Destination\\": \\"/etc/localtime\\",\\n                \\"Mode\\": \\"ro\\",\\n                \\"RW\\": false,\\n                \\"Propagation\\": \\"rprivate\\"\\n            },\\n            {\\n                \\"Type\\": \\"bind\\",\\n                \\"Source\\": \\"/var/lib/ceph\\",\\n                \\"Destination\\": \\"/var/lib/ceph\\",\\n                \\"Mode\\": \\"z\\",\\n                \\"RW\\": true,\\n                \\"Propagation\\": \\"rprivate\\"\\n            },\\n            {\\n                \\"Type\\": \\"bind\\",\\n                \\"Source\\": \\"/var/run/ceph\\",\\n                \\"Destination\\": \\"/var/run/ceph\\",\\n                \\"Mode\\": \\"z\\",\\n                \\"RW\\": true,\\n                \\"Propagation\\": \\"rprivate\\"\\n            }\\n        ],\\n        \\"Config\\": {\\n            \\"Hostname\\": \\"controller-1\\",\\n            \\"Domainname\\": \\"\\",\\n            \\"User\\": \\"\\",\\n            \\"AttachStdin\\": false,\\n            \\"AttachStdout\\": true,\\n            \\"AttachStderr\\": true,\\n            \\"ExposedPorts\\": {\\n                \\"5000/tcp\\": {},\\n                \\"6789/tcp\\": {},\\n                \\"6800/tcp\\": {},\\n                \\"6801/tcp\\": {},\\n                \\"6802/tcp\\": {},\\n                \\"6803/tcp\\": {},\\n                \\"6804/tcp\\": {},\\n                \\"6805/tcp\\": {},\\n                \\"80/tcp\\": {}\\n            },\\n            \\"Tty\\": false,\\n            \\"OpenStdin\\": false,\\n            \\"StdinOnce\\": false,\\n            \\"Env\\": [\\n                \\"IP_VERSION=4\\",\\n                \\"MON_IP=172.17.3.41\\",\\n                \\"CLUSTER=ceph\\",\\n                \\"FSID=dd86df8c-c42e-11ea-8899-52540092e928\\",\\n                \\"CEPH_PUBLIC_NETWORK=172.17.3.0/24\\",\\n                \\"CEPH_DAEMON=MON\\",\\n                \\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\\",\\n                \\"container=oci\\",\\n                \\"CEPH_VERSION=luminous\\",\\n                \\"CEPH_POINT_RELEASE=\\"\\n            ],\\n            \\"Cmd\\": null,\\n            \\"ArgsEscaped\\": true,\\n            \\"Image\\": \\"192.168.24.1:8787/rhceph/rhceph-3-rhel7:3-43\\",\\n            \\"Volumes\\": null,\\n            \\"WorkingDir\\": \\"/\\",\\n            \\"Entrypoint\\": [\\n                \\"/entrypoint.sh\\"\\n            ],\\n            \\"OnBuild\\": null,\\n            \\"Labels\\": {\\n                \\"CEPH_POINT_RELEASE\\": \\"\\",\\n                \\"GIT_BRANCH\\": \\"stable-3.2\\",\\n                \\"GIT_CLEAN\\": \\"True\\",\\n                \\"GIT_COMMIT\\": \\"0329ef42593abbec5d460096db46d915016af4ab\\",\\n                \\"GIT_REPO\\": \\"https://github.com/ceph/ceph-container.git\\",\\n                \\"RELEASE\\": \\"stable-3.2\\",\\n                \\"architecture\\": \\"x86_64\\",\\n                \\"authoritative-source-url\\": \\"registry.access.redhat.com\\",\\n                \\"build-date\\": \\"2020-03-31T06:53:40.672493\\",\\n                \\"com.redhat.build-host\\": \\"cpt-1003.osbs.prod.upshift.rdu2.redhat.com\\",\\n                \\"com.redhat.component\\": \\"rhceph-rhel7-container\\",\\n                \\"com.redhat.license_terms\\": \\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements\\",\\n                \\"description\\": \\"Red Hat Ceph Storage 3\\",\\n                \\"distribution-scope\\": \\"public\\",\\n                \\"install\\": \\"/usr/bin/docker run --rm --privileged -v /:/host -e MON_IP=${MON_IP}  -e CEPH_PUBLIC_NETWORK=${CEPH_PUBLIC_NETWORK} -e CEPH_DAEMON=${CEPH_DAEMON} -e MON_NAME=${MON_NAME} -e OSD_DEVICE=${OSD_DEVICE} -e HOST=/host -e IMAGE=${IMAGE} --entrypoint=/install.sh ${IMAGE}\\",\\n                \\"io.k8s.description\\": \\"Red Hat Ceph Storage 3\\",\\n                \\"io.k8s.display-name\\": \\"Red Hat Ceph Storage 3 on RHEL 7\\",\\n                \\"io.openshift.tags\\": \\"rhceph ceph\\",\\n                \\"maintainer\\": \\"Dimitri Savineau <dsavinea>\\",\\n                \\"name\\": \\"rhceph\\",\\n                \\"release\\": \\"43\\",\\n                \\"run\\": \\"/usr/bin/docker run -d --net=host --pid=host -e MON_NAME=${MON_NAME} -e MON_IP=${MON_IP}  -e CEPH_PUBLIC_NETWORK=${CEPH_PUBLIC_NETWORK} -e CEPH_DAEMON=${CEPH_DAEMON} -v /etc/ceph:/etc/ceph -v /var/lib/ceph:/var/lib/ceph ${IMAGE}\\",\\n                \\"summary\\": \\"Provides the latest Red Hat Ceph Storage 3 on RHEL 7 in a fully featured and supported base image.\\",\\n                \\"url\\": \\"https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/3-43\\",\\n                \\"vcs-ref\\": \\"ada53692c7adc6a56fb45054cac07803c1a97ddc\\",\\n                \\"vcs-type\\": \\"git\\",\\n                \\"vendor\\": \\"Red Hat, Inc.\\",\\n                \\"version\\": \\"3\\"\\n            }\\n        },\\n        \\"NetworkSettings\\": {\\n            \\"Bridge\\": \\"\\",\\n            \\"SandboxID\\": \\"0704cf6bf3f9992cc0c7cffbc0035f2cb32ef869d73e0b3653c609b2e173d316\\",\\n            \\"HairpinMode\\": false,\\n            \\"LinkLocalIPv6Address\\": \\"\\",\\n            \\"LinkLocalIPv6PrefixLen\\": 0,\\n            \\"Ports\\": {},\\n            \\"SandboxKey\\": \\"/var/run/docker/netns/default\\",\\n            \\"SecondaryIPAddresses\\": null,\\n            \\"SecondaryIPv6Addresses\\": null,\\n            \\"EndpointID\\": \\"\\",\\n            \\"Gateway\\": \\"\\",\\n            \\"GlobalIPv6Address\\": \\"\\",\\n            \\"GlobalIPv6PrefixLen\\": 0,\\n            \\"IPAddress\\": \\"\\",\\n            \\"IPPrefixLen\\": 0,\\n            \\"IPv6Gateway\\": \\"\\",\\n            \\"MacAddress\\": \\"\\",\\n            \\"Networks\\": {\\n                \\"host\\": {\\n                    \\"IPAMConfig\\": null,\\n                    \\"Links\\": null,\\n                    \\"Aliases\\": null,\\n                    \\"NetworkID\\": \\"5cb7428dbf5639b94d674702a4c5231e930331b83492f3d812e93a9bbcf101e3\\",\\n                    \\"EndpointID\\": \\"66fe5e74f176c9c3f94607e2d8468a621dfe5146c63d7cd64a485b63c91dde22\\",\\n                    \\"Gateway\\": \\"\\",\\n                    \\"IPAddress\\": \\"\\",\\n                    \\"IPPrefixLen\\": 0,\\n                    \\"IPv6Gateway\\": \\"\\",\\nHeat Stack update failed.
2020-07-12 13:19:47 | Heat Stack update failed.
2020-07-12 13:19:47 |                     \\"GlobalIPv6Address\\": \\"\\",\\n                    \\"GlobalIPv6PrefixLen\\": 0,\\n                    \\"MacAddress\\": \\"\\"\\n                }\\n            }\\n        }\\n    }\\n]", "stdout_lines": ["[", "    {", "        \\"Id\\": \\"06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f\\",", "        \\"Created\\": \\"2020-07-12T11:34:11.720829555Z\\",", "        \\"Path\\": \\"/entrypoint.sh\\",", "        \\"Args\\": [],", "        \\"State\\": {", "            \\"Status\\": \\"running\\",", "            \\"Running\\": true,", "            \\"Paused\\": false,", "            \\"Restarting\\": false,", "            \\"OOMKilled\\": false,", "            \\"Dead\\": false,", "            \\"Pid\\": 43265,", "            \\"ExitCode\\": 0,", "            \\"Error\\": \\"\\",", "            \\"StartedAt\\": \\"2020-07-12T11:34:12.066302937Z\\",", "            \\"FinishedAt\\": \\"0001-01-01T00:00:00Z\\"", "        },", "        \\"Image\\": \\"sha256:4464c3e4b038e3501bb31a61098d3edace33a045bfa22d4fc8b1e2a4d016fd44\\",", "        \\"ResolvConfPath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/resolv.conf\\",", "        \\"HostnamePath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/hostname\\",", "        \\"HostsPath\\": \\"/var/lib/docker/containers/06e9dee9cd59646eaafc0397dfd4e55dab5ab3bc3aae353947ee8ad714c41e3f/hosts\\",", "        \\"LogPath\\": \\"\\",", "        \\"Name\\": \\"/ceph-mon-controller-1\\",", "        \\"RestartCount\\": 0,", "        \\"Driver\\": \\"overlay2\\",", "        \\"MountLabel\\": \\"\\",", "        \\"ProcessLabel\\": \\"\\",", "        \\"AppArmorProfile\\": \\"\\",", "        \\"ExecIDs\\": null,", "        \\"HostConfig\\": {", "            \\"Binds\\": [", "                \\"/var/lib/ceph:/var/lib/ceph:z\\",", "                \\"/etc/ceph:/etc/ceph:z\\",", "                \\"/var/run/ceph:/var/run/ceph:z\\",", "                \\"/etc/localtime:/etc/localtime:ro\\"", "            ],", "            \\"ContainerIDFile\\": \\"\\",", "            \\"LogConfig\\": {", "                \\"Type\\": \\"journald\\",", "                \\"Config\\": {}", "            },", "            \\"NetworkMode\\": \\"host\\",", "            \\"PortBindings\\": {},", "            \\"RestartPolicy\\": {", "                \\"Name\\": \\"no\\",", "                \\"MaximumRetryCount\\": 0", "            },", "            \\"AutoRemove\\": true,", "            \\"VolumeDriver\\": \\"\\",", "            \\"VolumesFrom\\": null,", "            \\"CapAdd\\": null,", "            \\"CapDrop\\": null,", "            \\"Dns\\": [],", "            \\"DnsOptions\\": [],", "            \\"DnsSearch\\": [],", "            \\"ExtraHosts\\": null,", "            \\"GroupAdd\\": null,", "            \\"IpcMode\\": \\"\\",", "            \\"Cgroup\\": \\"\\",", "            \\"Links\\": null,", "            \\"OomScoreAdj\\": 0,", "            \\"PidMode\\": \\"\\",", "            \\"Privileged\\": false,", "            \\"PublishAllPorts\\": false,", "            \\"ReadonlyRootfs\\": false,", "            \\"SecurityOpt\\": null,", "            \\"UTSMode\\": \\"\\",", "            \\"UsernsMode\\": \\"\\",", "            \\"ShmSize\\": 67108864,", "            \\"Runtime\\": \\"docker-runc\\",", "            \\"ConsoleSize\\": [", "                0,", "                0", "            ],", "            ...