Bug 1910508

Summary: [OSP 16.2] No such file or directory: '/var/log/validations
Product: Red Hat OpenStack Reporter: Pradipta Kumar Sahoo <psahoo>
Component: validations-commonAssignee: mathieu bultel <mbultel>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: low Docs Contact:
Priority: low    
Version: 16.2 (Train)CC: aschultz, drosenfe, gchamoul, gregraka, jbuchta, jpodivin, mburns, michele
Target Milestone: rcKeywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: validations-common-1.1.2-2.20210316005404.4c29579.el8ost python-validations-libs-1.0.5-2.20210313005027.6c1b8a5.el8ost Doc Type: Bug Fix
Doc Text:
Before this update, validation results were not logged and validation artifacts were not collected because the permissions required to access the requested logging directory were not granted. This update resolves the issue, and validation results are successfully logged and validation artifacts are collected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-15 07:11:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pradipta Kumar Sahoo 2020-12-24 09:09:23 UTC
Description of problem:
During the deployment, the overcloud stack failed with below error.

[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
(<ansible.plugins.callback.validation_json.CallbackModule object at
0x7fb0bae71e80>): [Errno 2] No such file or directory: '/var/log/validations/52
5400f8-462c-4ece-1fd9-000000000009_deploy_steps_playbook_2020-12-23T17:12:41.01
5043Z.json'


Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 16.2.0 Beta (Train)

How reproducible: 100% reproduced in nfv perf lab


Steps to Reproduce:
1. Deployed Undercloud successfully with the following puddle: RHOS-16.2-RHEL-8-20201215.n.1
2. NFV templates are validated in 16.1 and reuse the same template for osp16.2.
3. The overcloud stack has failed with below status:


2020-12-23 17:23:28.609106 | 525400f8-462c-4ece-1fd9-00000000014b |    SKIPPED | Restart services | nfv-controller-0 | item=rsyslog
2020-12-23 17:23:28.609966 | 525400f8-462c-4ece-1fd9-00000000014b |    SKIPPED | Restart services | nfv-controller-0 | item=crond
2020-12-23 17:23:28.610922 | 525400f8-462c-4ece-1fd9-00000000014b |     TIMING | Restart services | nfv-controller-0 | 0:10:47.596016 | 0.06s

PLAY RECAP *********************************************************************
nfv-compute-offload-0      : ok=112  changed=68   unreachable=0    failed=1    skipped=122  rescued=0    ignored=0   
nfv-controller-0           : ok=170  changed=108  unreachable=0    failed=0    skipped=126  rescued=0    ignored=0   
undercloud                 : ok=10   changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

2020-12-23 17:23:28.901873 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.902029 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 343        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.902147 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:10:47.887248 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.902311 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2020-12-23 17:23:28.902459 | 525400f8-462c-4ece-1fd9-000000000972 |    SUMMARY | nfv-compute-offload-0 | tripleo-kernel : Reboot after kernel args update | 241.62s
2020-12-23 17:23:28.902583 | 525400f8-462c-4ece-1fd9-0000000001a9 |    SUMMARY | nfv-compute-offload-0 | Gathering Facts | 93.04s
2020-12-23 17:23:28.902702 | 525400f8-462c-4ece-1fd9-0000000001cd |    SUMMARY | nfv-compute-offload-0 | Gathering Facts | 91.52s
2020-12-23 17:23:28.902817 | 525400f8-462c-4ece-1fd9-00000000004a |    SUMMARY | nfv-compute-offload-0 | tripleo-network-config : Run NetworkConfig script | 76.07s
2020-12-23 17:23:28.902929 | 525400f8-462c-4ece-1fd9-000000000090 |    SUMMARY | nfv-controller-0 | Run puppet on the host to apply IPtables rules | 24.42s
2020-12-23 17:23:28.903057 | 525400f8-462c-4ece-1fd9-00000000004a |    SUMMARY | nfv-controller-0 | tripleo-network-config : Run NetworkConfig script | 13.83s
2020-12-23 17:23:28.903166 | 525400f8-462c-4ece-1fd9-000000000aaf |    SUMMARY | nfv-controller-0 | tripleo-hieradata : Render hieradata from template | 4.97s
2020-12-23 17:23:28.903286 | 525400f8-462c-4ece-1fd9-000000000aaf |    SUMMARY | nfv-compute-offload-0 | tripleo-hieradata : Render hieradata from template | 4.55s
2020-12-23 17:23:28.903401 | 525400f8-462c-4ece-1fd9-000000000d23 |    SUMMARY | nfv-compute-offload-0 | tripleo-kernel : Set extra sysctl options | 3.05s
2020-12-23 17:23:28.903517 | 525400f8-462c-4ece-1fd9-000000000d23 |    SUMMARY | nfv-controller-0 | tripleo-kernel : Set extra sysctl options | 2.95s
2020-12-23 17:23:28.903613 | 525400f8-462c-4ece-1fd9-0000000001a9 |    SUMMARY | undercloud | Gathering Facts | 2.67s
2020-12-23 17:23:28.903740 | 525400f8-462c-4ece-1fd9-0000000001a9 |    SUMMARY | nfv-controller-0 | Gathering Facts | 2.53s
2020-12-23 17:23:28.903842 | 525400f8-462c-4ece-1fd9-0000000001cd |    SUMMARY | nfv-controller-0 | Gathering Facts | 1.93s
2020-12-23 17:23:28.903942 | 525400f8-462c-4ece-1fd9-0000000001bd |    SUMMARY | undercloud | Gathering Facts | 1.89s
2020-12-23 17:23:28.904051 | 525400f8-462c-4ece-1fd9-0000000000ea |    SUMMARY | nfv-controller-0 | Populate service facts (chrony) | 1.89s
2020-12-23 17:23:28.904159 | 525400f8-462c-4ece-1fd9-000000000d1e |    SUMMARY | nfv-compute-offload-0 | tripleo-kernel : Remove dracut-config-generic | 1.78s
2020-12-23 17:23:28.904285 | 525400f8-462c-4ece-1fd9-0000000000f1 |    SUMMARY | nfv-controller-0 | Populate service facts | 1.69s
2020-12-23 17:23:28.904399 | 525400f8-462c-4ece-1fd9-00000000010d |    SUMMARY | nfv-compute-offload-0 | enable virt_sandbox_use_netlink for healtcheck | 1.67s
2020-12-23 17:23:28.904528 | 525400f8-462c-4ece-1fd9-000000000d1e |    SUMMARY | nfv-controller-0 | tripleo-kernel : Remove dracut-config-generic | 1.62s
2020-12-23 17:23:28.904640 | 525400f8-462c-4ece-1fd9-00000000006c |    SUMMARY | nfv-controller-0 | enable virt_sandbox_use_netlink for healthcheck | 1.55s
2020-12-23 17:23:28.904756 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.904874 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.904991 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2020-12-23 17:23:28.905085 |  The following node(s) had failures: nfv-compute-offload-0
2020-12-23 17:23:28.905208 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
(<ansible.plugins.callback.validation_json.CallbackModule object at
0x7fb0bae71e80>): [Errno 2] No such file or directory: '/var/log/validations/52
5400f8-462c-4ece-1fd9-000000000009_deploy_steps_playbook_2020-12-23T17:12:41.01
5043Z.json'
Host 192.168.24.8 not found in /home/stack/.ssh/known_hosts
Overcloud configuration failed.


Actual results: Overcloud stack failed


Additional info: The same issue has reported in upstream, kindly share us the downstream patch if it available: 
https://bugs.launchpad.net/tripleo/+bug/1904781

BR,
Pradipta

Comment 2 Alex Schultz 2020-12-25 21:52:12 UTC
The warning is not a fatal error and does not affect the deployment.  We'll use this bz to track this warning and get it resolved.  I think it might be fixed in packaging in the upstream.  You'll need to look back in the logs to see what failed on nfv-compute-offload-0 and open a separate issue if necessary.

nfv-compute-offload-0      : ok=112  changed=68   unreachable=0    failed=1    skipped=122  rescued=0    ignored=0

Comment 7 David Rosenfeld 2021-06-07 19:59:53 UTC
The No such file or directory error no longer appears during deploy:

more overcloud_install.log | grep validations

In addition the file: 5254007a-5179-1d59-299f-000000000007_deploy_steps_playbook_2021-06-04T15:28:52.923574Z.json is created in the /home/stack directory.

Comment 9 errata-xmlrpc 2021-09-15 07:11:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483