Bug 1732355 - [OSP15] instance-ha fails with an ansible timeout error
Summary: [OSP15] instance-ha fails with an ansible timeout error
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-23 08:33 UTC by Cesar Fernandez
Modified: 2019-10-12 07:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-12 07:01:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Cesar Fernandez 2019-07-23 08:33:54 UTC
Description of problem:

"instance-ha" job is failing with an ansible timeout error in "udercloud-0/var/log/containers/mistral/mistral-db-manage.log":

2019-07-22 21:10:39.111 13 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: nova.certs_convert_into_with_meta: AttributeError: 'Client' object has no attribute 'certs'

Additional info:

Jenkins Job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/instance-ha/job/DFG-pidone-instance-ha-15_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-instance-ha-test-suite/9/console

Comment 1 Luca Miccini 2019-07-23 08:54:02 UTC
(In reply to Cesar Fernandez from comment #0)
> Description of problem:
> 
> "instance-ha" job is failing with an ansible timeout error in
> "udercloud-0/var/log/containers/mistral/mistral-db-manage.log":
> 
> 2019-07-22 21:10:39.111 13 ERROR
> mistral.actions.openstack.action_generator.base [-] Failed to create action:
> nova.certs_convert_into_with_meta: AttributeError: 'Client' object has no
> attribute 'certs'
> 
> Additional info:
> 
> Jenkins Job:
> https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/
> view/instance-ha/job/DFG-pidone-instance-ha-15_director-rhel-virthost-
> 3cont_2comp_3ceph-ipv4-geneve-instance-ha-test-suite/9/console

that ERROR message is due to https://bugzilla.redhat.com/show_bug.cgi?id=1709566.

As for the actual failure I am going to have a look :)

Comment 2 Luca Miccini 2019-07-23 09:18:50 UTC
here the step where it hanged/failed:

TASK [configure tmpwatch on the host] ******************************************
Monday 22 July 2019  22:09:01 +0000 (0:00:00.552)       0:12:38.772 *********** 
skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [overcloud-novacomputeiha-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [overcloud-novacomputeiha-1] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [tripleo-container-tag : Pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1 image] ***
Monday 22 July 2019  22:09:02 +0000 (0:00:00.353)       0:12:39.125 *********** 
skipping: [overcloud-novacomputeiha-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [overcloud-novacomputeiha-1] => {"changed": false, "skip_reason": "Conditional result was False"}
changed: [controller-0] => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1", "delta": "0:00:06.547186", "end": "2019-07-22 22:09:08.906103", "rc": 0, "start": "2019-07-22 22:09:02.358917", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1...Getting image source signatures\nCopying blob sha256:3fd1d6bddf52f5739ebe7a52906714c0142e430b2305e8e9e663ee8d19c5d5fd\nCopying blob sha256:3b3d402417deac332eda774ac4b9fbc0e7f4da02558b0b3a0ba54dc283203047\nCopying blob sha256:6bc8d2c3e35d7762fdc45ab9ac14c152b308d183680ac0b5bb85f17bd0962dbf\nCopying blob sha256:0db9c662d67b89727a5a3a6aebbf04c67b1d711e6ac133f34634d10da67822e4\nCopying blob sha256:3dc708edb94b8390cd03c6e55a89cca0483cec8d9be8db5b2d08f1eb95d1012e\nCopying blob sha256:3506f1c05051545bc85d863feb4c990abe6735fb09fea7280854d4d92c9c881c\nCopying config sha256:134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1\nWriting manifest to image destination\nStoring signatures", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1...Getting image source signatures", "Copying blob sha256:3fd1d6bddf52f5739ebe7a52906714c0142e430b2305e8e9e663ee8d19c5d5fd", "Copying blob sha256:3b3d402417deac332eda774ac4b9fbc0e7f4da02558b0b3a0ba54dc283203047", "Copying blob sha256:6bc8d2c3e35d7762fdc45ab9ac14c152b308d183680ac0b5bb85f17bd0962dbf", "Copying blob sha256:0db9c662d67b89727a5a3a6aebbf04c67b1d711e6ac133f34634d10da67822e4", "Copying blob sha256:3dc708edb94b8390cd03c6e55a89cca0483cec8d9be8db5b2d08f1eb95d1012e", "Copying blob sha256:3506f1c05051545bc85d863feb4c990abe6735fb09fea7280854d4d92c9c881c", "Copying config sha256:134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1", "Writing manifest to image destination", "Storing signatures"], "stdout": "134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1", "stdout_lines": ["134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1"]}
changed: [controller-1] => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1", "delta": "0:00:06.514072", "end": "2019-07-22 22:09:08.945439", "rc": 0, "start": "2019-07-22 22:09:02.431367", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1...Getting image source signatures\nCopying blob sha256:3fd1d6bddf52f5739ebe7a52906714c0142e430b2305e8e9e663ee8d19c5d5fd\nCopying blob sha256:3b3d402417deac332eda774ac4b9fbc0e7f4da02558b0b3a0ba54dc283203047\nCopying blob sha256:6bc8d2c3e35d7762fdc45ab9ac14c152b308d183680ac0b5bb85f17bd0962dbf\nCopying blob sha256:0db9c662d67b89727a5a3a6aebbf04c67b1d711e6ac133f34634d10da67822e4\nCopying blob sha256:3dc708edb94b8390cd03c6e55a89cca0483cec8d9be8db5b2d08f1eb95d1012e\nCopying blob sha256:3506f1c05051545bc85d863feb4c990abe6735fb09fea7280854d4d92c9c881c\nCopying config sha256:134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1\nWriting manifest to image destination\nStoring signatures", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190715.1...Getting image source signatures", "Copying blob sha256:3fd1d6bddf52f5739ebe7a52906714c0142e430b2305e8e9e663ee8d19c5d5fd", "Copying blob sha256:3b3d402417deac332eda774ac4b9fbc0e7f4da02558b0b3a0ba54dc283203047", "Copying blob sha256:6bc8d2c3e35d7762fdc45ab9ac14c152b308d183680ac0b5bb85f17bd0962dbf", "Copying blob sha256:0db9c662d67b89727a5a3a6aebbf04c67b1d711e6ac133f34634d10da67822e4", "Copying blob sha256:3dc708edb94b8390cd03c6e55a89cca0483cec8d9be8db5b2d08f1eb95d1012e", "Copying blob sha256:3506f1c05051545bc85d863feb4c990abe6735fb09fea7280854d4d92c9c881c", "Copying config sha256:134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1", "Writing manifest to image destination", "Storing signatures"], "stdout": "134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1", "stdout_lines": ["134cbb7d77dc9efab004099fb8432c85fc44baaf3cc5f47599499faafed85cf1"]}
Overcloud configuration failed.

Ansible timed out at 5259 seconds.


looks like something happened on controller-2, unfortunately infrared did not grab anything from that vm.
Can you try rerunning this job?

Thanks
Luca


Note You need to log in before you can comment on or make changes to this bug.