Description of problem: Time out starting containers in overcloud update Version-Release number of selected component (if applicable): How reproducible: 3 cont - 2 comp - 2 net - vlan - ipv4 Steps to Reproduce: 1.Update from core_puddle: RHOS_TRUNK-16.0-RHEL-8-20200204.n.1 to RHOS_TRUNK-16.0-RHEL-8-20200305.n.1 2. 3. Actual results: Expected results: Additional info: 020-03-09 15:55:58 | Monday 09 March 2020 15:55:45 +0000 (0:00:00.265) 0:23:43.672 ********** 2020-03-09 15:55:58 | changed: [controller-0] => {"ansible_job_id": "162495403102.467470", "changed": true, "finished": 0, "results_file": "/root/.ansible_async/162495403102.467470", "started": 1} 2020-03-09 15:55:58 | 2020-03-09 15:55:58 | TASK [Wait for containers to start for step 4 using paunch] ******************** 2020-03-09 15:55:58 | Monday 09 March 2020 15:55:46 +0000 (0:00:00.773) 0:23:44.445 ********** 2020-03-09 15:55:58 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1200 retries left). 2020-03-09 15:55:58 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1199 retries left). 2020-03-09 15:55:58 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1198 retries left). 2020-03-09 15:55:58 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1197 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1196 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1195 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1194 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1193 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1192 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1191 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1190 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1189 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1188 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1187 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1186 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1185 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1184 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1183 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1182 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1181 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1180 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1179 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1178 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1177 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1176 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1175 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1174 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1173 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1172 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1171 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1170 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1169 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1168 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1167 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1166 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1165 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1164 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1163 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1162 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1161 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1160 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1159 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1158 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1157 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1156 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1155 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1154 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1153 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1152 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1151 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1150 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1149 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1148 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1147 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1146 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1145 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1144 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1143 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1142 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1141 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1140 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1139 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1138 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1137 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1136 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1135 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1134 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1133 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1132 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1131 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1130 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1129 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1128 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1127 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1126 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1125 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1124 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1123 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1122 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1121 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1120 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1119 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1118 retries left). 2020-03-09 16:00:39 | 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1117 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1116 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1115 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1114 retries left). 2020-03-09 16:00:39 | FAILED - RETRYING: Wait for containers to start for step 4 using paunch (1113 retries left). 2020-03-09 16:00:40 | fatal: [controller-0]: FAILED! => {"ansible_job_id": "162495403102.467470", "attempts": 89, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step4", "rc": 1, "stderr": "$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cinder-api:20200305.1\nb''\nb''\n$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cinder-scheduler:20200305.1\nb''\nb''\n$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cron:20200305.1\nb''\nb''\n$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-glance-api:20200305.1\nb''\nb''\n$ podman image exists undercloud- .... \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:9696\", \"region\": \"regionOne\"}, {\"id\": \"a740a78f518a4b498f8dbd0759a49c7f\", \"interface\": \"public\", \"region_id\": \"regionOne\", \"url\": \"http://10.0.0.123:9696\", \"region\": \"regionOne\"}, {\"id\": \"d390d44208544f468cf590f2e86c7d9b\", \"interface\": \"internal\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:9696\", \"region\": \"regionOne\"}], \"id\": \"8ba0e99095df44819e987b941fa11418\", \"type\": \"network\", \"name\": \"neutron\"}, {\"endpoints\": [{\"id\": \"2f63cba56251453fac9cd720ca8e60e8\", \"interface\": \"public\", \"region_id\": \"regionOne\", \"url\": \"http://10.0.0.123:8778/placement\", \"region\": \"regionOne\"}, {\"id\": \"7ac2948d287d4e0aba25fb3d5da09e4d\", \"interface\": \"admin\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:8778/placement\", \"region\": \"regionOne\"}, {\"id\": \"e67a87fcf76f432ba74cb1df395a455b\", \"interface\": \"internal\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:8778/placement\", \"region\": \"regionOne\"}], \"id\": \"c4e8ab8764c84336a5b038ebbc128a64\", \"type\": \"placement\", \"name\": \"placement\"}, {\"endpoints\": [{\"id\": \"97684e6e02134b4da19d590ec8ec5fcd\", \"interface\": \"admin\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:8774/v2.1\", \"region\": \"regionOne\"}, {\"id\": \"ba4a2cae995a43c5960d8d16b624f49b\", \"interface\": \"public\", \"region_id\": \"regionOne\", \"url\": \"http://10.0.0.123:8774/v2.1\", \"region\": \"regionOne\"}, {\"id\": \"c5572cbb2a134b418a2aa48d9baac153\", \"interface\": \"internal\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:8774/v2.1\", \"region\": \"regionOne\"}], \"id\": \"cf091f91436144f78f2656c417ee0775\", \"type\": \"compute\", \"name\": \"nova\"}, {\"endpoints\": [{\"id\": \"54ea075f537b4af08e9b157b43f70643\", \"interface\": \"public\", \"region_id\": \"regionOne\", \"url\": \"http://10.0.0.123:5000\", \"region\": \"regionOne\"}, {\"id\": \"bb534ba447ad45e7bd2c86ac506971d0\", \"interface\": \"admin\", \"region_id\": \"regionOne\", \"url\": \"http://192.168.24.22:35357\", \"region\": \"regionOne\"}, {\"id\": \"cc86d121bfd34d28a1ed1852f264f147\", \"interface\": \"internal\", \"region_id\": \"regionOne\", \"url\": \"http://172.17.1.139:5000\", \"region\": \"regionOne\"}], \"id\": \"d3c4aa05440e4cd4bd2b176f67f652a8\", \"type\": \"identity\", \"name\": \"keystone\"}]}}", "DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://172.17.1.139:8774 -H \"Accept: application/json\" -H \"User-Agent: python-novaclient\" -H \"X-Auth-Token: {SHA256}9b76dae0f97b6b5257477e0d17eb593a01676ad73739fe0f2a9dca556d2c1e8c\" -H \"X-OpenStack-Nova-API-Version: 2.11\"", "DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.17.1.139:8774", "DEBUG:urllib3.connectionpool:http://172.17.1.139:8774 \"GET / HTTP/1.1\" 200 None", "DEBUG:keystoneauth.session:RESP: [200] Content-Encoding: gzip Content-Type: application/json Date: Mon, 09 Mar 2020 16:00:29 GMT Server: Apache Transfer-Encoding: chunked Vary: Accept-Encoding", "DEBUG:keystoneauth.session:RESP BODY: {\"versions\": [{\"id\": \"v2.0\", \"status\": \"SUPPORTED\", \"version\": \"\", \"min_version\": \"\", \"updated\": \"2011-01-21T11:33:21Z\", \"links\": [{\"rel\": \"self\", \"href\": \"http://172.17.1.139:8774/v2/\"}]}, {\"id\": \"v2.1\", \"status\": \"CURRENT\", \"version\": \"2.79\", \"min_version\": \"2.1\", \"updated\": \"2013-07-23T11:33:21Z\", \"links\": [{\"rel\": \"self\", \"href\": \"http://172.17.1.139:8774/v2.1/\"}]}]}", "INFO:nova_wait_for_api_service:Nova-api service active", "", "ea47a1a478ffb176c73ebcfe159e84b0966b95da7b3655fad002773a7876eaff", "", "95ae90196104e8a615d10092e5ac2de4d56ec2ec07ca3d5125e611f29c3966c9", "", "627b31e5339d8a43180947aaf9b071913cb92262b090b4fcbabd4e81c22178b3"]} 2020-03-09 16:00:40 | 2020-03-09 16:00:40 | NO MORE HOSTS LEFT ************************************************************* 2020-03-09 16:00:40 | 2020-03-09 16:00:40 | PLAY RECAP ********************************************************************* 2020-03-09 16:00:40 | controller-0 : ok=302 changed=142 unreachable=0 failed=1 skipped=405 rescued=0 ignored=2 2020-03-09 16:00:40 | controller-1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 2020-03-09 16:00:40 | controller-2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 2020-03-09 16:00:40 | 2020-03-09 16:00:40 | Monday 09 March 2020 16:00:38 +0000 (0:04:52.968) 0:28:37.413 ********** 2020-03-09 16:00:40 | =============================================================================== 2020-03-09 16:00:40 | 2020-03-09 16:00:40 | Ansible failed, check log at /var/lib/mistral/1c218df9-f32c-4fba-8c85-03ce34e58d54/ansible.log. 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/1c218df9-f32c-4fba-8c85-03ce34e58d54/ansible.log. 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun Traceback (most recent call last): 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun super(Command, self).run(parsed_args) 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun return super(Command, self).run(parsed_args) 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun return_code = self.take_action(parsed_args) or 0 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_update.py", line 171, in take_action 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun priv_key=key) 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1191, in run_update_ansible_action 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun verbosity=verbosity, extra_vars=extra_vars) 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun raise RuntimeError('Update failed with: {}'.format(payload['message'])) 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/1c218df9-f32c-4fba-8c85-03ce34e58d54/ansible.log. 2020-03-09 16:00:40 | 2020-03-09 16:00:40.239 72969 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun [00m 2020-03-09 16:00:40 | 2020-03-09 16:00:40.242 72969 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/lib/mistral/1c218df9-f32c-4fba-8c85-03ce34e58d54/ansible.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/1c218df9-f32c-4fba-8c85-03ce34e58d54/ansible.log.[00m 2020-03-09 16:00:40 | 2020-03-09 16:00:40.242 72969 INFO osc_lib.shell [-] END return value: 1[00m
Moving to verified. The overcloud update in the job reporting the problem passed. Note: the tempest failure in that job can be ignored. Its unrelated: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/neutron/job/DFG-network-neutron-update-16_director-rhel-virthost-3cont_2comp_2net-ipv4-vlan-composable-ml2ovs/74/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2114