Bug 1559876 - [RFE]/etc/etcd/etcd.conf variables are different between fresh install vs upgrade of OCP 3.7
Summary: [RFE]/etc/etcd/etcd.conf variables are different between fresh install vs upg...
Keywords:
Status: CLOSED DUPLICATE of bug 1529575
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.7.z
Assignee: Scott Dodson
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-23 13:00 UTC by Nick Poyant - npoyant@redhat.com
Modified: 2018-04-16 20:54 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-16 20:54:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Nick Poyant - npoyant@redhat.com 2018-03-23 13:00:58 UTC
Description of problem:
I was testing ETCD by following the below article and found some ETCD configuration differences between fresh install vs in place upgrade of OCP 3.7.
https://access.redhat.com/articles/3093761

At face value I have not seen any negative effects on the cluster but there are variables called out in the article that should exist when they do not. I have posted the detailed config file in a below section.

I'm filing this bug because the variables assigned to the /etc/etcd/etcd.conf should be the same whether i choose to do a new install or an upgrade. The upgrade installer should detect the differences between the changes and migrate to the new variables. IMO

Most of the new variables are commented options but the main concern is the conversion of the certificate file variable options in the security section which are active.


Version-Release number of selected component (if applicable):

# oc version
oc v3.7.23
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://console.ocp-int.fqdn:8443
openshift v3.7.23
kubernetes v1.7.6+a08f5eeb62



How reproducible:
Install new OCP 3.7 cluster
vs
In place upgrade of OCP 3.6 to 3.7

Steps to Reproduce:
Install new OCP 3.7 cluster
vs
In place upgrade of OCP 3.6 to 3.7

Actual results:

# cd /etc/etcd
# sdiff etcd.conf-after-upgrade etcd.conf-fresh-install

ETCD_NAME=npoyant-ocp0002.fqdn		                        ETCD_NAME=npoyant-ocp0002.fqdn
ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380			ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380
ETCD_DATA_DIR=/var/lib/etcd/					ETCD_DATA_DIR=/var/lib/etcd/
#ETCD_SNAPSHOT_COUNTER=10000				      |	#ETCD_WAL_DIR=""
							      >	#ETCD_SNAPSHOT_COUNT=10000
ETCD_HEARTBEAT_INTERVAL=500					ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500					ETCD_ELECTION_TIMEOUT=2500
ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379		ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379
#ETCD_MAX_SNAPSHOTS=5						#ETCD_MAX_SNAPSHOTS=5
#ETCD_MAX_WALS=5						#ETCD_MAX_WALS=5
#ETCD_CORS=							#ETCD_CORS=


#[cluster]							#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380	ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380
ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=h	ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=h
ETCD_INITIAL_CLUSTER_STATE=new					ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1			ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
#ETCD_DISCOVERY=						#ETCD_DISCOVERY=
#ETCD_DISCOVERY_SRV=						#ETCD_DISCOVERY_SRV=
#ETCD_DISCOVERY_FALLBACK=proxy					#ETCD_DISCOVERY_FALLBACK=proxy
#ETCD_DISCOVERY_PROXY=						#ETCD_DISCOVERY_PROXY=
ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379		ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379
							      >	#ETCD_STRICT_RECONFIG_CHECK="false"
							      >	#ETCD_AUTO_COMPACTION_RETENTION="0"
							      >	#ETCD_ENABLE_V2="true"
							      >	ETCD_QUOTA_BACKEND_BYTES=4294967296

#[proxy]							#[proxy]
#ETCD_PROXY=off							#ETCD_PROXY=off
							      >	#ETCD_PROXY_FAILURE_WAIT="5000"
							      >	#ETCD_PROXY_REFRESH_INTERVAL="30000"
							      >	#ETCD_PROXY_DIAL_TIMEOUT="1000"
							      >	#ETCD_PROXY_WRITE_TIMEOUT="5000"
							      >	#ETCD_PROXY_READ_TIMEOUT="0"

#[security]							#[security]
ETCD_CA_FILE=/etc/etcd/ca.crt				      |	ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
							      >	ETCD_CLIENT_CERT_AUTH="true"
ETCD_CERT_FILE=/etc/etcd/server.crt				ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key				ETCD_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CA_FILE=/etc/etcd/ca.crt			      |	#ETCD_AUTO_TLS="false"
							      >	ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
							      >	ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt				ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key				ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
							      >	#ETCD_PEER_AUTO_TLS="false"
							      >
							      >	#[logging]
							      >	ETCD_DEBUG="False"
							      >
							      >	#[profiling]
							      >	#ETCD_ENABLE_PPROF="false"
							      >	#ETCD_METRICS="basic"
							      >	#
							      >	#[auth]
							      >	#ETCD_AUTH_TOKEN="simple"


Expected results:
I would expect that both the new installer vs upgrade playbooks would make the same changes to etcd.conf

Additional info:


Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.7.23-1.git.0.bc406aa.el7.noarch

rpm -q ansible
ansible-2.4.1.0-1.el7.noarch

ansible --version
ansible 2.4.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

How reproducible:
Install new OCP 3.7 cluster
vs
In place upgrade of OCP 3.6 to 3.7

Steps to Reproduce:
Install new OCP 3.7 cluster
vs
In place upgrade of OCP 3.6 to 3.7

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Nick Poyant - npoyant@redhat.com 2018-03-23 13:05:01 UTC
Reposting separately incase diff formatting gets jacked.

###################
Fresh Install etcd.conf


ETCD_NAME=npoyant-ocp0002.rhc-lab.iad.redhat.com
ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380
ETCD_DATA_DIR=/var/lib/etcd/
#ETCD_WAL_DIR=""
#ETCD_SNAPSHOT_COUNT=10000
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500
ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379
#ETCD_MAX_SNAPSHOTS=5
#ETCD_MAX_WALS=5
#ETCD_CORS=


#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380
ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=https://10.15.69.154:2380,npoyant-ocp0003.rhc-lab.iad.redhat.com=https://10.15.69.165:2380,npoyant-ocp0004.rhc-lab.iad.redhat.com=https://10.15.69.160:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
#ETCD_DISCOVERY=
#ETCD_DISCOVERY_SRV=
#ETCD_DISCOVERY_FALLBACK=proxy
#ETCD_DISCOVERY_PROXY=
ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379
#ETCD_STRICT_RECONFIG_CHECK="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#ETCD_ENABLE_V2="true"
ETCD_QUOTA_BACKEND_BYTES=4294967296

#[proxy]
#ETCD_PROXY=off
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"

#[security]
ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt
ETCD_CLIENT_CERT_AUTH="true"
ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key
#ETCD_AUTO_TLS="false"
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
#ETCD_PEER_AUTO_TLS="false"

#[logging]
ETCD_DEBUG="False"

#[profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[auth]
#ETCD_AUTH_TOKEN="simple"


########################################

In place upgrade etcd.conf

ETCD_NAME=npoyant-ocp0002.rhc-lab.iad.redhat.com
ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380
ETCD_DATA_DIR=/var/lib/etcd/
#ETCD_SNAPSHOT_COUNTER=10000
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500
ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379
#ETCD_MAX_SNAPSHOTS=5
#ETCD_MAX_WALS=5
#ETCD_CORS=


#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380
ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=https://10.15.69.154:2380,npoyant-ocp0003.rhc-lab.iad.redhat.com=https://10.15.69.165:2380,npoyant-ocp0004.rhc-lab.iad.redhat.com=https://10.15.69.160:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
#ETCD_DISCOVERY=
#ETCD_DISCOVERY_SRV=
#ETCD_DISCOVERY_FALLBACK=proxy
#ETCD_DISCOVERY_PROXY=
ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379

#[proxy]
#ETCD_PROXY=off

#[security]
ETCD_CA_FILE=/etc/etcd/ca.crt
ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CA_FILE=/etc/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key

Comment 2 Nick Poyant - npoyant@redhat.com 2018-03-23 13:11:46 UTC
After a new upgrade:

[root@npoyant-ocp0002 etcd]# grep storage-backend -A1 /etc/origin/master/master-config.yaml
    storage-backend:
    - etcd3

[root@npoyant-ocp0002 etcd]# source /etc/etcd/etcd.conf
[root@npoyant-ocp0002 etcd]# export ETCDCTL_API=3
[root@npoyant-ocp0002 etcd]# ETCD_ALL_ENDPOINTS=` etcdctl  --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS --write-out=fields   member list | awk '/ClientURL/{printf "%s%s",sep,$3; sep=","}'`
Error:  empty string is passed to --cacert option



This is it was noticed that the variables changed.

Comment 4 Scott Dodson 2018-04-16 20:54:37 UTC
Going to close as a dupe of bug 1529575. We should now be ensuring that both 3.1 and 3.2 compatible flags are set during upgrade.

3.7 backport https://bugzilla.redhat.com/show_bug.cgi?id=1563376
3.6 backport https://bugzilla.redhat.com/show_bug.cgi?id=1563375

We're working to ensure that before etcd 3.3 ships we have 100% assurance that no new flags are necessary and if they are they'll be added during the upgrade process.

*** This bug has been marked as a duplicate of bug 1529575 ***


Note You need to log in before you can comment on or make changes to this bug.