Bug 1732599

Hello Shaju,

After reviewing your issue and as the fix for the bug #1722147 doesn't work well with DCI's OpenStack agent,
could you try the following:

- add a line with "tempest_track_resources: false" in your /etc/dci-openstack-agent/settings.yml (or the settings file you're using).

This will disable all the tempest cleanup tasks (tempest won't inventory the initial state of the resources & won't clean after its run).
In DCI context this should not be an issue because the environment is new before running tempest and not kept for long after.

Please tell us if it works and if it is acceptable for you.

Hi,

I have made this change and restarted dci-openstack-agent.timer. I will update you after the next job is completed.

Thanks,

Hi,
After trying the above suggested change, still I see RHOSP13 jobs are failing

Something went wrong during the running. Review the log at: https://www.distributed-ci.io/jobs/eba1ca01-14df-4c23-959a-54a4226d048b/jobStates
[{'_ansible_no_log': False,
  'changed': False,
  'msg': u'Something went wrong during the running. Review the log at: https://www.distributed-ci.io/jobs/eba1ca01-14df-4c23-959a-54a4226d048b/jobStates'}]

If you look at the step the error happened[1], you'll see that there's an issue with the deployment because introspection fails:

[1] https://www.distributed-ci.io/jobs/eba1ca01-14df-4c23-959a-54a4226d048b/jobStates#52c427c6-41ab-4972-bb56-76d7b951cccd:file0

With the following meaningful excerpt formatted properly:

    Retrying 4 nodes that failed introspection. Attempt 3 of 3
    Introspection of node 341c63e2-93d9-4d39-b4ad-5d9f0ca69b6c timed out.
    Introspection of node 61367517-5b24-4e68-b004-458507c4a4e4 timed out.
    Introspection of node 3a06408b-7479-453d-ba76-018ebd4d4d9c timed out.
    Introspection of node fa263143-1534-47fd-aaa3-01e1142cafb6 timed out.
    Retry limit reached with 4 nodes still failing introspection

These errors are the same that were occurring from June 20th to July 2nd on your remoteCI.

To be clear, this has nothing to do with the "tempest_track_resources: false" flag which effect happens only when running Tempest.

Looks like deployment of undercloud task skipped after we tried with changes made to reconfiguration,

The condition rconfiguration == 'RHEL7-CDN' didn’t meet.

Logs:
====
 
Aug 02 18:55:03 rhel-cache.englab.juniper.net ansible-playbook[13131]: TASK [Deploy undercloud] *******************************************************
Aug 02 18:55:03 rhel-cache.englab.juniper.net ansible-playbook[13131]: task path: /etc/dci-openstack-agent/hooks/running.yml:8
Aug 02 18:55:03 rhel-cache.englab.juniper.net ansible-playbook[13131]: skipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}
Aug 02 18:55:03 rhel-cache.englab.juniper.net ansible-playbook[13131]: TASK [Deploy overcloud] ********************************************************
Aug 02 18:55:03 rhel-cache.englab.juniper.net ansible-playbook[13131]: task path: /etc/dci-openstack-agent/hooks/running.yml:15
 
 
Code:
===
- name: Deploy undercloud
  shell: "ansible-playbook -i inventory playbooks/prepare_undercloud.yml"
  args:
    chdir: /var/lib/dci-openstack-agent/ansible
  when: rconfiguration == 'RHEL7-CDN'. ==================è this condition didn’t met
 
- name: Deploy undercloud
  shell: "ansible-playbook -i inventory playbooks/prepare_undercloud.yml -e director_extra_yum_repos=ftp://partners.redhat.com/1c5d859a/juniper-4dbbfc30a829359271b73ab5b6ad965a/RHEL-7.6-Snapshot-3/RHEL-7-server.repo"
  args:
    chdir: /var/lib/dci-openstack-agent/ansible
  when: rconfiguration == 'RHEL7-partner-beta'

This works now.. Closing the bug.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days