Bug 1895045

Summary:	Validations in FFU 13->16.1: validations before 'openstack undercloud upgrade' failed with:Permission denied: '/var/log/validations'
Product:	Red Hat OpenStack	Reporter:	Jose Luis Franco <jfrancoa>
Component:	validations-common	Assignee:	Jiri Podivin <jpodivin>
Status:	CLOSED ERRATA	QA Contact:	nlevinki <nlevinki>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.1 (Train)	CC:	aschultz, hbrock, jamsmith, jbuchta, jslagle, mburns
Target Milestone:	z6	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-validations-libs-1.0.5-1.20210326153612.4a660ec.el8ost validations-common-1.1.2-1.20210326153615.7f3c565.el8ost	Doc Type:	Bug Fix
Doc Text:	This update fixes a bug that caused failure of validations before `openstack undercloud upgrade` in some cases. Before this upgrade, a lack of permissions needed to access the requested logging directory sometimes resulted in the following failures: Failure to log validation results Failure of the validation run ** Failure of artifacts collection from validation. + This update adds a fallback logging directory. Validation results are logged and artifacts collected.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-26 13:49:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jose Luis Franco 2020-11-05 15:59:45 UTC

Description of problem:
After applying in my local environment the patch https://review.opendev.org/760351 which fixes https://bugzilla.redhat.com/show_bug.cgi?id=1892323  I got the following error when trying to run any validation/group:

(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory inventory.yaml --group pre-upgrade              
Error while creating Ansible artifacts log file.Please check the access rights for {}                                                
Traceback (most recent call last):                                                                                                                                   
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir                             
    os.makedirs(log_dir)                                                                                                                            
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs                                                                            
    makedirs(head, mode, exist_ok)                                                                                                                                   
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs                                                                        
    makedirs(head, mode, exist_ok)                                                                                                                   
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs                                                                                      
    mkdir(name, mode)                                                                                                                 
PermissionError: [Errno 13] Permission denied: '/var/log/validations'                                                                        
Exception occured while running the command                                                                                           
Traceback (most recent call last):                                                                                                                  
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir                                   
    os.makedirs(log_dir)                                                                                                                     
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs                                                                           
    makedirs(head, mode, exist_ok)                                                                                                    
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs                                                                              
    makedirs(head, mode, exist_ok)                                                                                                
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs                                                                           
    mkdir(name, mode)                                                                                                                             
PermissionError: [Errno 13] Permission denied: '/var/log/validations'            
During handling of the above exception, another exception occurred:                                                                   
                                                                                                                                     
Traceback (most recent call last):                                                                                                                                   
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run                                                     
    super(Command, self).run(parsed_args)                                                                                            
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run                                                 
    return super(Command, self).run(parsed_args)                                                                                                    
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 398, in take_action
    self._run_validator_run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 369, in _run_validator_run
    quiet=parsed_args.quiet)
  File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 100, in run_validations
    prefix=os.path.basename(playbook))
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 48, in create_artifacts_dir
    LOG.exception("Error while creating Ansible artifacts log file."
AttributeError: 'NoneType' object has no attribute 'format'
'NoneType' object has no attribute 'format'

(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory inventory.yaml --validation 512e
Error while creating Ansible artifacts log file.Please check the access rights for {}
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir
    os.makedirs(log_dir)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/validations'
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir
    os.makedirs(log_dir)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/validations'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 398, in take_action
    self._run_validator_run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 369, in _run_validator_run
    quiet=parsed_args.quiet)
  File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 100, in run_validations
    prefix=os.path.basename(playbook))
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 48, in create_artifacts_dir
    LOG.exception("Error while creating Ansible artifacts log file."
AttributeError: 'NoneType' object has no attribute 'format'
'NoneType' object has no attribute 'format'

(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator show history
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 460, in take_action
    return actions.show_history(parsed_args.validation)
  File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 164, in show_history
    if validation_id else vlogs.get_all_logfiles(extension))
  File "/usr/lib/python3.6/site-packages/validations_libs/validation_logs.py", line 209, in get_all_logfiles
    return [join(self.logs_path, f) for f in os.listdir(self.logs_path) if
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/validations/'
[Errno 2] No such file or directory: '/var/log/validations/'

When checking the system, the /var/log/validations directory wasn't created yet.



Version-Release number of selected component (if applicable):

(undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa | egrep 'validations|ansible'
openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch
ansible-tripleo-ipsec-9.2.1-0.20200311073016.0c8693c.el8ost.noarch
tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch
python3-heat-agent-ansible-1.10.1-0.20200311091123.96b819c.el8ost.noarch
validations-common-1.1.2-1.20200914180304.7f3c565.el8ost.noarch
ansible-config_template-1.1.2-1.20200818183400.ea07ed0.el8ost.noarch
ceph-ansible-4.0.31-1.el8cp.noarch
ansible-role-thales-hsm-0.2.1-0.20200311074756.2803c6c.el8ost.noarch
ansible-freeipa-0.1.8-3.el8.noarch
python3-ansible-runner-1.3.4-2.el8ar.noarch
ansible-role-chrony-1.0.2-0.20200311064420.03e7fbe.el8ost.noarch
ansible-pacemaker-1.0.4-1.20200907073409.accaf26.el8ost.noarch
ansible-role-tripleo-modify-image-1.2.1-0.20200804085623.1dffa21.el8ost.noarch
ansible-role-container-registry-1.1.1-0.20200311065947.7eca2dd.el8ost.noarch
python3-validations-libs-1.0.5-1.20200914174429.4a660ec.el8ost.noarch
ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.noarch
ansible-role-atos-hsm-0.1.1-0.20200318203421.1269408.el8ost.noarch
ansible-tripleo-ipa-0.2.1-1.20200813093411.3bb3c53.el8ost.noarch
ansible-2.9.15-1.el8ae.noarch
ansible-role-redhat-subscription-1.1.1-0.20200716163409.6d26afa.el8ost.noarch


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jiri Podivin 2021-03-11 08:58:18 UTC

I deployed the 16.1 undecloud and attempted to replicate. 
The directory /var/log/validations was present immediately after deployment and I was unable to replicate.
Requesting more information.

Comment 2 Jiri Podivin 2021-03-26 07:49:19 UTC

Both patches are now backported to 16.2. Backport to 16.1 pending...

Comment 3 Jiri Podivin 2021-03-26 15:19:21 UTC

16.1 trunk patches merged.

Comment 4 Jiri Podivin 2021-03-29 15:30:49 UTC

Fix for the validations-libs in 16.2 https://pkgs.devel.redhat.com/cgit/rpms/python-validations-libs/commit/?h=rhos-16.2-rhel-8-trunk&id=9cf2d6882e2780d683e4a4afd7584ae7f795f4c6

Comment 16 errata-xmlrpc 2021-05-26 13:49:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097