Description of problem: validate-selinux fails in overcloud nodes when running post-upgrade validation after upgrading deployment from 13 to 16.1 [1]. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index#validating-the-post-upgrade-functionality ~~~ (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --group post-upgrade ... +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ | UUID | Validations | Status | Host Group(s) | Status by Host | Unreachable Host(s) | Duration | +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ | 525400df-30c2-0be7-b62a-00000000000b | container-status | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:02.904 | | 525400df-30c2-6363-38bd-00000000000b | openstack-endpoints | PASSED | undercloud | undercloud | | 0:00:01.985 | | 525400df-30c2-65ef-60ed-00000000000b | image-serve | PASSED | undercloud | undercloud | | 0:00:02.214 | | 525400df-30c2-8476-978f-00000000000b | stack-health | PASSED | undercloud | undercloud | | 0:00:02.124 | | 525400df-30c2-8b44-9836-00000000000b | undercloud-service-status | PASSED | undercloud | undercloud | | 0:00:01.781 | | 525400df-30c2-9240-3462-00000000000b | service-status | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:00.997 | | 525400df-30c2-c635-1f3d-00000000000b | validate-selinux | FAILED | all | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:04.763 | +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ It seems that the failure is caused by missing /var/log/validations in overcloud nodes ~~~ (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator show run 525400df-30c2-c635-1f3d-00000000000b { "task": { "hosts": { "controller-0": { "_ansible_no_log": false, "action": "copy", "changed": false, "failed": true, "invocation": { "module_args": { "_original_basename": null, "attributes": null, "backup": false, "checksum": null, "content": null, "delimiter": null, "dest": "/var/log/validations/denials-filtered.log", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": null, "owner": null, "regexp": null, "remote_src": true, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/tmp/denials.log", "unsafe_writes": null, "validate": null } }, "msg": "Destination directory /var/log/validations does not exist" } }, "name": "No skip_list", "status": "FAILED" } } ... ~~~ Version-Release number of selected component (if applicable): The following tripleo packages are installed in undercloud nodes ~~~ ansible-role-tripleo-modify-image-1.2.1-0.20200527233426.bc21900.el8ost.noarch ansible-tripleo-ipa-0.2.1-0.20200611104546.c22fc8d.el8ost.noarch ansible-tripleo-ipsec-9.2.1-0.20200311073016.0c8693c.el8ost.noarch openstack-tripleo-common-11.3.3-0.20200611110657.f7715be.el8ost.noarch openstack-tripleo-common-containers-11.3.3-0.20200611110657.f7715be.el8ost.noarch openstack-tripleo-heat-templates-11.3.2-0.20200616081539.396affd.el8ost.noarch openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.noarch openstack-tripleo-puppet-elements-11.2.2-0.20200527003426.226ce95.el8ost.noarch openstack-tripleo-validations-11.3.2-0.20200611115253.08f469d.el8ost.noarch puppet-tripleo-11.5.0-0.20200616033428.8ff1c6a.el8ost.noarch python3-tripleoclient-12.3.2-0.20200615103427.6f877f6.el8ost.noarch python3-tripleoclient-heat-installer-12.3.2-0.20200615103427.6f877f6.el8ost.noarch python3-tripleo-common-11.3.3-0.20200611110657.f7715be.el8ost.noarch tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8ost.noarch ~~~ How reproducible: Always Steps to Reproduce: 1. Follow the upgrade documentation and upgrade osp13 to 16.1 2. Run post-upgrade validation Actual results: validate-selinux fails in overcloud nodes Expected results: validate-selinux succeeds in overcloud nodes Additional info:
I confirmed that the validate-selinux validation succeeds after I manually create /var/log/validations directory in overcloud nodes. ~~~ (undercloud) [stack@undercloud-0 ~]$ cat playbook-validations.yaml --- - name: Copy leapp data hosts: overcloud tasks: - name: Create validation log directory file: path: /var/log/validations state: directory owner: heat-admin group: heat-admin mode: 0755 become: yes (undercloud) [stack@undercloud-0 ~]$ ansible-playbook -i ~/inventory.yaml playbook-validations.yaml ... (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --group post-upgrade ... +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ | UUID | Validations | Status | Host Group(s) | Status by Host | Unreachable Host(s) | Duration | +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ | 525400df-30c2-281f-631e-00000000000b | container-status | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:02.854 | | 525400df-30c2-351d-e55f-00000000000b | image-serve | PASSED | undercloud | undercloud | | 0:00:02.126 | | 525400df-30c2-9e5b-d59e-00000000000b | service-status | PASSED | undercloud, overcloud | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:01.012 | | 525400df-30c2-9fd4-0fbf-00000000000b | stack-health | PASSED | undercloud | undercloud | | 0:00:02.159 | | 525400df-30c2-b11d-ca63-00000000000b | openstack-endpoints | PASSED | undercloud | undercloud | | 0:00:01.915 | | 525400df-30c2-b526-b5de-00000000000b | undercloud-service-status | PASSED | undercloud | undercloud | | 0:00:01.868 | | 525400df-30c2-be75-2273-00000000000b | validate-selinux | PASSED | all | compute-0, compute-1, controller-0, controller-1, controller-2, undercloud | | 0:00:05.173 | +--------------------------------------+---------------------------+--------+-----------------------+----------------------------------------------------------------------------+---------------------+-------------+ ~~~
It seems that the same issue exists for pre-upgrade validation. I didn't detect this during my trial because most of pre-upgrade validation fails because of another bug[1] [1] https://bugzilla.redhat.com/show_bug.cgi?id=1873470
@Cédric I found you submitted the fix for this issue in upstream. Will you backport that patch to RHOSP16.1 as well ? I tried to find the bug report associated with that patch but couldn't find it because of wrong number in the commit message, but please close this bug as a duplicate if there are always a bug report for the same issue in bugzilla.
Hello Takashi, I apparently pointed to the actual review instead of the launchpad bug ID X(. The LP is https://bugs.launchpad.net/tripleo/+bug/1892356 - I just closed it since everything merged... Regarding downstream: yep, my intend is to backport it, for z3 - on its way as of now. Sorry for the confused IDs.. Cheers, C.
Me again. Apparently the tripleo-validations patches came late in, and the validation was moved to validations-common at some point, without the correction :(. So I've re-done the patch, pointing to the right LP and current BZ: https://review.opendev.org/759815 Sorry for the additional delay... Putting back ON_DEV.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5413