Bug 1876040

Summary: validate-selinux fails because of missing /var/log/validations
Product: Red Hat OpenStack Reporter: Takashi Kajinami <tkajinam>
Component: validations-commonAssignee: Cédric Jeanneret <cjeanner>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: astupnik, cjeanner, emacchi, gchamoul, jjoyce, jschluet, mbultel, slinaber, tvignaud
Target Milestone: z3Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: validations-common-1.1.2-1.20200914180305.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-15 18:36:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Takashi Kajinami 2020-09-05 05:51:04 UTC
Description of problem:

validate-selinux fails in overcloud nodes when running post-upgrade validation
after upgrading deployment from 13 to 16.1 [1].

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#validating-the-post-upgrade-functionality

~~~
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --group post-upgrade
...
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+
| UUID                                 | Validations               | Status | Host Group(s)         | Status by Host                                                             | Unreachable Host(s) | Duration    |
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+
| 525400df-30c2-0be7-b62a-00000000000b | container-status          | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:02.904 |
| 525400df-30c2-6363-38bd-00000000000b | openstack-endpoints       | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:01.985 |
| 525400df-30c2-65ef-60ed-00000000000b | image-serve               | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:02.214 |
| 525400df-30c2-8476-978f-00000000000b | stack-health              | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:02.124 |
| 525400df-30c2-8b44-9836-00000000000b | undercloud-service-status | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:01.781 |
| 525400df-30c2-9240-3462-00000000000b | service-status            | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:00.997 |
| 525400df-30c2-c635-1f3d-00000000000b | validate-selinux          | FAILED | all                   | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:04.763 |
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+

It seems that the failure is caused by missing /var/log/validations in overcloud nodes
~~~
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator show run 525400df-30c2-c635-1f3d-00000000000b
{
    "task": {
        "hosts": {
            "controller-0": {
                "_ansible_no_log": false,
                "action": "copy",
                "changed": false,
                "failed": true,
                "invocation": {
                    "module_args": {
                        "_original_basename": null,
                        "attributes": null,
                        "backup": false,
                        "checksum": null,
                        "content": null,
                        "delimiter": null,
                        "dest": "/var/log/validations/denials-filtered.log",
                        "directory_mode": null,
                        "follow": false,
                        "force": true,
                        "group": null,
                        "local_follow": null,
                        "mode": null,
                        "owner": null,
                        "regexp": null,
                        "remote_src": true,
                        "selevel": null,
                        "serole": null,
                        "setype": null,
                        "seuser": null,
                        "src": "/tmp/denials.log",
                        "unsafe_writes": null,
                        "validate": null
                    }
                },
                "msg": "Destination directory /var/log/validations does not exist"
            }
        },
        "name": "No skip_list",
        "status": "FAILED"
    }
}
...
~~~


Version-Release number of selected component (if applicable):

The following tripleo packages are installed in undercloud nodes
~~~
ansible-role-tripleo-modify-image-1.2.1-0.20200527233426.bc21900.el8ost.noarch
ansible-tripleo-ipa-0.2.1-0.20200611104546.c22fc8d.el8ost.noarch
ansible-tripleo-ipsec-9.2.1-0.20200311073016.0c8693c.el8ost.noarch
openstack-tripleo-common-11.3.3-0.20200611110657.f7715be.el8ost.noarch
openstack-tripleo-common-containers-11.3.3-0.20200611110657.f7715be.el8ost.noarch
openstack-tripleo-heat-templates-11.3.2-0.20200616081539.396affd.el8ost.noarch
openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.noarch
openstack-tripleo-puppet-elements-11.2.2-0.20200527003426.226ce95.el8ost.noarch
openstack-tripleo-validations-11.3.2-0.20200611115253.08f469d.el8ost.noarch
puppet-tripleo-11.5.0-0.20200616033428.8ff1c6a.el8ost.noarch
python3-tripleoclient-12.3.2-0.20200615103427.6f877f6.el8ost.noarch
python3-tripleoclient-heat-installer-12.3.2-0.20200615103427.6f877f6.el8ost.noarch
python3-tripleo-common-11.3.3-0.20200611110657.f7715be.el8ost.noarch
tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8ost.noarch
~~~

How reproducible:
Always

Steps to Reproduce:
1. Follow the upgrade documentation and upgrade osp13 to 16.1
2. Run post-upgrade validation

Actual results:
validate-selinux fails in overcloud nodes

Expected results:
validate-selinux succeeds in overcloud nodes

Additional info:

Comment 1 Takashi Kajinami 2020-09-05 05:55:16 UTC
I confirmed that the validate-selinux validation succeeds after I manually create /var/log/validations directory in overcloud nodes.

~~~
(undercloud) [stack@undercloud-0 ~]$ cat playbook-validations.yaml 
---
- name: Copy leapp data
  hosts: overcloud
  tasks:

  - name: Create validation log directory
    file:
      path: /var/log/validations
      state: directory
      owner: heat-admin
      group: heat-admin
      mode: 0755
    become: yes
(undercloud) [stack@undercloud-0 ~]$ ansible-playbook -i ~/inventory.yaml playbook-validations.yaml
...
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --group post-upgrade
...
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+
| UUID                                 | Validations               | Status | Host Group(s)         | Status by Host                                                             | Unreachable Host(s) | Duration    |
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+
| 525400df-30c2-281f-631e-00000000000b | container-status          | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:02.854 |
| 525400df-30c2-351d-e55f-00000000000b | image-serve               | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:02.126 |
| 525400df-30c2-9e5b-d59e-00000000000b | service-status            | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:01.012 |
| 525400df-30c2-9fd4-0fbf-00000000000b | stack-health              | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:02.159 |
| 525400df-30c2-b11d-ca63-00000000000b | openstack-endpoints       | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:01.915 |
| 525400df-30c2-b526-b5de-00000000000b | undercloud-service-status | PASSED | undercloud            | undercloud                                                                 |                     | 0:00:01.868 |
| 525400df-30c2-be75-2273-00000000000b | validate-selinux          | PASSED | all                   | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud |                     | 0:00:05.173 |
+--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+
~~~

Comment 2 Takashi Kajinami 2020-10-23 03:02:38 UTC
It seems that the same issue exists for pre-upgrade validation.

I didn't detect this during my trial because most of pre-upgrade validation fails because of another bug[1]
 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1873470

Comment 3 Takashi Kajinami 2020-10-27 01:02:00 UTC
@Cédric

I found you submitted the fix for this issue in upstream.
Will you backport that patch to RHOSP16.1 as well ?

I tried to find the bug report associated with that patch but couldn't find it
because of wrong number in the commit message, but please close this bug as
a duplicate if there are always a bug report for the same issue in bugzilla.

Comment 4 Cédric Jeanneret 2020-10-27 06:16:37 UTC
Hello Takashi,

I apparently pointed to the actual review instead of the launchpad bug ID X(.
The LP is https://bugs.launchpad.net/tripleo/+bug/1892356 - I just closed it since everything merged...

Regarding downstream: yep, my intend is to backport it, for z3 - on its way as of now.

Sorry for the confused IDs..

Cheers,

C.

Comment 6 Cédric Jeanneret 2020-10-27 06:33:01 UTC
Me again.

Apparently the tripleo-validations patches came late in, and the validation was moved to validations-common at some point, without the correction :(. So I've re-done the patch, pointing to the right LP and current BZ:
https://review.opendev.org/759815

Sorry for the additional delay... Putting back ON_DEV.

Comment 17 errata-xmlrpc 2020-12-15 18:36:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5413