Bug 1658841

Summary:	Options openshift_node_min_tls_version and openshift_node_cipher_suites removed in OpenShift 3.10 and later
Product:	OpenShift Container Platform	Reporter:	Brian J. Beaudoin <bbeaudoi>
Component:	Installer	Assignee:	Scott Dodson <sdodson>
Installer sub component:	openshift-ansible	QA Contact:	Johnny Liu <jialiu>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	bbeaudoi, grodrigu, knewcomer
Version:	3.11.0
Target Milestone:	---
Target Release:	3.11.z
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-12-14 21:56:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brian J. Beaudoin 2018-12-13 00:51:53 UTC

Description of problem:

Version-Release number of the following components:
[bbeaudoin@control ~]$ rpm -q openshift-ansible
openshift-ansible-3.11.43-1.git.0.fa69a02.el7.noarch
[bbeaudoin@control ~]$ ansible --version
ansible 2.6.7
  config file = /home/bbeaudoin/ansible.cfg
  configured module search path = [u'/home/bbeaudoin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
[bbeaudoin@master1 ~]$ oc version
oc v3.11.43
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://cluster1.openshifthappens.com:443
openshift v3.11.43
kubernetes v1.11.0+d4cacc0
[bbeaudoin@master1 ~]$ rpm -q atomic-openshift-node
atomic-openshift-node-3.11.43-1.git.0.647ac05.el7.x86_64


How reproducible:

Steps to Reproduce:

1. Install a new OpenShift 3.11 cluster with minimum tls version and cipher suites specified for masters, nodes, and etcd:

openshift_master_min_tls_version=VersionTLS12
openshift_node_min_tls_version=VersionTLS12

openshift_master_cipher_suites=['TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256','TLS_RSA_WITH_AES_256_CBC_SHA','TLS_RSA_WITH_AES_128_CBC_SHA']
openshift_node_cipher_suites=['TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256','TLS_RSA_WITH_AES_256_CBC_SHA','TLS_RSA_WITH_AES_128_CBC_SHA']
etcd_cipher_suites="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA"

2. Validate the settings were applied to:
   - /etc/origin/master/master-config.yaml
   - /etc/origin/node/node-config.yaml
   - /etc/etcd/etcd.conf

Actual results:

Correct settings were applied to:
   - /etc/origin/master/master-config.yaml
   - /etc/etcd/etcd.conf

The installation failed to set the options on
   - /etc/origin/node/node-config.yaml

Expected results:

OpenShift 3.7 supported these options for master and node configuration. It was expected the configuration options would continue to configure the node service properly in OpenShift 3.11 and later so minimum TLS version was enforced and weak ciphers were disabled through whitelisting of strong ciphers.

This issue also affects OpenShift 3.10.

Additional info:

The template openshift-ansible/roles/openshift_node/templates/node.yaml.v1.j2 contained the following block of code to ensure this was set properly.

~~~
{% if openshift_node_min_tls_version is defined %}
  minTLSVersion: {{ openshift_node_min_tls_version }}
{% endif %}
{% if openshift_node_cipher_suites is defined %}
  cipherSuites:
{% for cipher_suite in openshift_node_cipher_suites %}
  - {{ cipher_suite }}
{% endfor %}
{% endif %}
~~~

openshift-ansible/roles/openshift_node_group/templates/node-config.yaml.j2 is missing the block so nodes are not getting configured properly. This affects the configuration deployed to the nodes during bootstrapping as the following configmaps in the openshift-node project do not have the configuration required:

node-config-all-in-one
node-config-all-in-one-crio
node-config-compute
node-config-compute-crio
node-config-infra
node-config-infra-crio
node-config-master
node-config-master-crio
node-config-master-infra
node-config-master-infra-crio

While it is possible to configure this post-install, it's desirable to have the ability to ensure compliance out-of-box without making changes to configmaps later. There is also a risk changes to the configmaps could be reverted during updates or upgrades.

Comment 2 Scott Dodson 2018-12-13 01:25:24 UTC

I think our advice at this point in time is to apply the desired edits to the node groups. That's the long term solution for providing config for 3.10 and 3.11 nodes. We could add the same tls options to the node group generation template but we'd prefer having one method for defining node config edits.

Would that work for you?

Comment 6 Scott Dodson 2018-12-13 20:44:16 UTC

They can do this at install time today. They just need to define their own openshift_node_groups structure which is defaulted here.

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_facts/defaults/main.yml#L144-L195

for instance, put a file like the following at inventory/group_vars/all.yaml assuming your inventory is at inventory/hosts

openshift_node_groups:
  - name: node-config-master
    labels:
      - 'node-role.kubernetes.io/master=true'
    edits: 
    - key: servingInfo.minTLSVersion
      value: "VersionTLS12"
    - key: servingInfo.cipherSuites
      value: 
      - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
      - "TLS_RSA_WITH_AES_256_CBC_SHA"
      - "TLS_RSA_WITH_AES_128_CBC_SHA"

This is an all or nothing thing, you either get the default node groups or you need to define the entire structure, so probably best to start by copying the default and editing from there.

Docs for this here https://docs.openshift.com/container-platform/3.11/install/configuring_inventory_file.html#configuring-inventory-node-group-definitions

Comment 7 Brian J. Beaudoin 2018-12-13 23:24:12 UTC

Scott, thank you, this method of overriding the node configurations work.

Comment 9 Scott Dodson 2018-12-14 21:56:16 UTC

Thanks for confirming that the custom definition of openshift_node_groups worked. I'm going to close this.