Bug 1710020

Summary: Installtion of OCP 3.11 fails due to missing validation of SELinux and/or NetworkManager
Product: OpenShift Container Platform Reporter: Jay Cromer <jcromer>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Weihua Meng <wmeng>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: jcallen, jialiu, rteague, vjaypurk, wmeng
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
SELinux check was in the openshift_node role and has been moved to the init role so the check runs earlier in the installation process. The Ansible play which ran the NetworkManager check was not being run in prerequisites due to the way dynamic host groups were being populated. Groups have been adjusted to ensure prerequisite checks will be run any time prerequistes.yml is run. SELinux and NetworkManager checks are now run before cluster deployment beings.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-23 19:56:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1724718    
Bug Blocks:    

Description Jay Cromer 2019-05-14 18:29:35 UTC
Description of problem:
Using openshift-ansible, it is not validating if SELinux is enabled on the nodes, additionally it does not appear to be validating the installation or enablement of NetworkManager either

Version-Release number of the following components:
rpm -q openshift-ansible - openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7.noarch
rpm -q ansible           - ansible-2.6.16-1.el7ae.noarch
ansible --version        - ansible 2.6.16

How reproducible:
Every attempt to run install fails because SELinux and/or NetworkManager are not enabled 

Steps to Reproduce:
1. Run /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml 
2. Run /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml
3.

Actual results:
TASK [openshift_node : fail] **************************************************************************************************************************

fatal: [tpavcpax101s4.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s5.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s6.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s7.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s8.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s9.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s10.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

fatal: [tpavcpax101s13.vici.verizon.com]: FAILED! => {"changed": false, "msg": "SELinux is disabled, This deployment type requires that SELinux is enabled."}

[WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'.         [Errno 13] Permission denied:

u'/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'

Expected results:
Successful installation

Additional info:

Comment 1 Jay Cromer 2019-05-14 18:33:33 UTC
TASK [fail] *******************************************************************************************************************************************

fatal: [tpavcpax101s4.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s5.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s6.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s7.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s8.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s9.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s10.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

fatal: [tpavcpax101s13.vici.verizon.com]: FAILED! => {"changed": false, "msg": "Currently, NetworkManager must be installed and enabled prior to installation."}

[WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'.         [Errno 13] Permission denied:

u'/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'

Comment 2 Russell Teague 2019-05-14 18:54:36 UTC
Are you stating that selinux is enabled and the installer is failing to correctly identify the state?
SELinux is a documented prerequisite, https://docs.openshift.com/container-platform/3.11/install/prerequisites.html#prereq-selinux

Are you stating that NetworkManager is enable and the installer is failing to correctly identigy the state?
NetworkManager is a documented prerequisite, https://docs.openshift.com/container-platform/3.11/install/prerequisites.html#prereq-networkmanager

Comment 3 Jay Cromer 2019-05-14 19:26:39 UTC
No, I am stating SELinux is disabled and the prerequisites playbook is failing to identify the state. Shouldn't the prereq playbook identify this as an issue?

No, I am stating NetworkManager is disabled and the prerequisites playbook is failing to identify the state. Shouldn't the prereq playbook identify this as an issue?

Comment 4 Russell Teague 2019-06-14 12:01:15 UTC
https://github.com/openshift/openshift-ansible/pull/11695
NetworkManager and SELinux checks will now run in prerequisites.yml.

Comment 5 Russell Teague 2019-06-24 15:01:15 UTC
Fixed in openshift-ansible-3.11.120-1 and later

Comment 6 Johnny Liu 2019-06-28 02:28:54 UTC
Seem like this PR introduce some regression:
https://bugzilla.redhat.com/show_bug.cgi?id=1724718

Comment 7 Weihua Meng 2019-06-28 08:28:51 UTC
(In reply to Johnny Liu from comment #6)
> Seem like this PR introduce some regression:
> https://bugzilla.redhat.com/show_bug.cgi?id=1724718

That bug is not introduced by PR for this bug, it has been there silently before PR for this bug.
It is not triggered because the first condition is false, then the other conditions are not tested.

- fail:	
    msg: "SELinux is disabled, This deployment type requires that SELinux is enabled."	
  when:	
    - (ansible_selinux is not defined or ansible_selinux.status != 'enabled')	
    - openshift_deployment_type == 'openshift-enterprise'	
    - not openshift_use_crio | bool

Comment 8 Weihua Meng 2019-06-28 09:05:18 UTC
Fixed.

openshift-ansible-3.11.124

TASK [Verify SELinux is enabled] ***********************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/sanity_checks.yml:45
fatal: [ci-vm-10-0-151-214.hosted.upshift.rdu2.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "The openshift-enterprise deployment type requires SELinux is 'enforcing' with type 'targeted'\nSELinux status: \"enabled\"\nSELinux mode: \"permissive\"\nSELinux type: \"targeted\"\n"
}

	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/prerequisites.retry

Comment 15 errata-xmlrpc 2019-07-23 19:56:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1753