Description of problem: I was testing ETCD by following the below article and found some ETCD configuration differences between fresh install vs in place upgrade of OCP 3.7. https://access.redhat.com/articles/3093761 At face value I have not seen any negative effects on the cluster but there are variables called out in the article that should exist when they do not. I have posted the detailed config file in a below section. I'm filing this bug because the variables assigned to the /etc/etcd/etcd.conf should be the same whether i choose to do a new install or an upgrade. The upgrade installer should detect the differences between the changes and migrate to the new variables. IMO Most of the new variables are commented options but the main concern is the conversion of the certificate file variable options in the security section which are active. Version-Release number of selected component (if applicable): # oc version oc v3.7.23 kubernetes v1.7.6+a08f5eeb62 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://console.ocp-int.fqdn:8443 openshift v3.7.23 kubernetes v1.7.6+a08f5eeb62 How reproducible: Install new OCP 3.7 cluster vs In place upgrade of OCP 3.6 to 3.7 Steps to Reproduce: Install new OCP 3.7 cluster vs In place upgrade of OCP 3.6 to 3.7 Actual results: # cd /etc/etcd # sdiff etcd.conf-after-upgrade etcd.conf-fresh-install ETCD_NAME=npoyant-ocp0002.fqdn ETCD_NAME=npoyant-ocp0002.fqdn ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380 ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380 ETCD_DATA_DIR=/var/lib/etcd/ ETCD_DATA_DIR=/var/lib/etcd/ #ETCD_SNAPSHOT_COUNTER=10000 | #ETCD_WAL_DIR="" > #ETCD_SNAPSHOT_COUNT=10000 ETCD_HEARTBEAT_INTERVAL=500 ETCD_HEARTBEAT_INTERVAL=500 ETCD_ELECTION_TIMEOUT=2500 ETCD_ELECTION_TIMEOUT=2500 ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379 ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379 #ETCD_MAX_SNAPSHOTS=5 #ETCD_MAX_SNAPSHOTS=5 #ETCD_MAX_WALS=5 #ETCD_MAX_WALS=5 #ETCD_CORS= #ETCD_CORS= #[cluster] #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380 ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380 ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=h ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=h ETCD_INITIAL_CLUSTER_STATE=new ETCD_INITIAL_CLUSTER_STATE=new ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1 ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1 #ETCD_DISCOVERY= #ETCD_DISCOVERY= #ETCD_DISCOVERY_SRV= #ETCD_DISCOVERY_SRV= #ETCD_DISCOVERY_FALLBACK=proxy #ETCD_DISCOVERY_FALLBACK=proxy #ETCD_DISCOVERY_PROXY= #ETCD_DISCOVERY_PROXY= ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379 ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379 > #ETCD_STRICT_RECONFIG_CHECK="false" > #ETCD_AUTO_COMPACTION_RETENTION="0" > #ETCD_ENABLE_V2="true" > ETCD_QUOTA_BACKEND_BYTES=4294967296 #[proxy] #[proxy] #ETCD_PROXY=off #ETCD_PROXY=off > #ETCD_PROXY_FAILURE_WAIT="5000" > #ETCD_PROXY_REFRESH_INTERVAL="30000" > #ETCD_PROXY_DIAL_TIMEOUT="1000" > #ETCD_PROXY_WRITE_TIMEOUT="5000" > #ETCD_PROXY_READ_TIMEOUT="0" #[security] #[security] ETCD_CA_FILE=/etc/etcd/ca.crt | ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt > ETCD_CLIENT_CERT_AUTH="true" ETCD_CERT_FILE=/etc/etcd/server.crt ETCD_CERT_FILE=/etc/etcd/server.crt ETCD_KEY_FILE=/etc/etcd/server.key ETCD_KEY_FILE=/etc/etcd/server.key ETCD_PEER_CA_FILE=/etc/etcd/ca.crt | #ETCD_AUTO_TLS="false" > ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt > ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt ETCD_PEER_KEY_FILE=/etc/etcd/peer.key ETCD_PEER_KEY_FILE=/etc/etcd/peer.key > #ETCD_PEER_AUTO_TLS="false" > > #[logging] > ETCD_DEBUG="False" > > #[profiling] > #ETCD_ENABLE_PPROF="false" > #ETCD_METRICS="basic" > # > #[auth] > #ETCD_AUTH_TOKEN="simple" Expected results: I would expect that both the new installer vs upgrade playbooks would make the same changes to etcd.conf Additional info: Description of problem: Version-Release number of the following components: rpm -q openshift-ansible openshift-ansible-3.7.23-1.git.0.bc406aa.el7.noarch rpm -q ansible ansible-2.4.1.0-1.el7.noarch ansible --version ansible 2.4.1.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] How reproducible: Install new OCP 3.7 cluster vs In place upgrade of OCP 3.6 to 3.7 Steps to Reproduce: Install new OCP 3.7 cluster vs In place upgrade of OCP 3.6 to 3.7 Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Reposting separately incase diff formatting gets jacked. ################### Fresh Install etcd.conf ETCD_NAME=npoyant-ocp0002.rhc-lab.iad.redhat.com ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380 ETCD_DATA_DIR=/var/lib/etcd/ #ETCD_WAL_DIR="" #ETCD_SNAPSHOT_COUNT=10000 ETCD_HEARTBEAT_INTERVAL=500 ETCD_ELECTION_TIMEOUT=2500 ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379 #ETCD_MAX_SNAPSHOTS=5 #ETCD_MAX_WALS=5 #ETCD_CORS= #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380 ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=https://10.15.69.154:2380,npoyant-ocp0003.rhc-lab.iad.redhat.com=https://10.15.69.165:2380,npoyant-ocp0004.rhc-lab.iad.redhat.com=https://10.15.69.160:2380 ETCD_INITIAL_CLUSTER_STATE=new ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1 #ETCD_DISCOVERY= #ETCD_DISCOVERY_SRV= #ETCD_DISCOVERY_FALLBACK=proxy #ETCD_DISCOVERY_PROXY= ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379 #ETCD_STRICT_RECONFIG_CHECK="false" #ETCD_AUTO_COMPACTION_RETENTION="0" #ETCD_ENABLE_V2="true" ETCD_QUOTA_BACKEND_BYTES=4294967296 #[proxy] #ETCD_PROXY=off #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" #[security] ETCD_TRUSTED_CA_FILE=/etc/etcd/ca.crt ETCD_CLIENT_CERT_AUTH="true" ETCD_CERT_FILE=/etc/etcd/server.crt ETCD_KEY_FILE=/etc/etcd/server.key #ETCD_AUTO_TLS="false" ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ca.crt ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt ETCD_PEER_KEY_FILE=/etc/etcd/peer.key #ETCD_PEER_AUTO_TLS="false" #[logging] ETCD_DEBUG="False" #[profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" # #[auth] #ETCD_AUTH_TOKEN="simple" ######################################## In place upgrade etcd.conf ETCD_NAME=npoyant-ocp0002.rhc-lab.iad.redhat.com ETCD_LISTEN_PEER_URLS=https://10.15.69.154:2380 ETCD_DATA_DIR=/var/lib/etcd/ #ETCD_SNAPSHOT_COUNTER=10000 ETCD_HEARTBEAT_INTERVAL=500 ETCD_ELECTION_TIMEOUT=2500 ETCD_LISTEN_CLIENT_URLS=https://10.15.69.154:2379 #ETCD_MAX_SNAPSHOTS=5 #ETCD_MAX_WALS=5 #ETCD_CORS= #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.15.69.154:2380 ETCD_INITIAL_CLUSTER=npoyant-ocp0002.rhc-lab.iad.redhat.com=https://10.15.69.154:2380,npoyant-ocp0003.rhc-lab.iad.redhat.com=https://10.15.69.165:2380,npoyant-ocp0004.rhc-lab.iad.redhat.com=https://10.15.69.160:2380 ETCD_INITIAL_CLUSTER_STATE=new ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1 #ETCD_DISCOVERY= #ETCD_DISCOVERY_SRV= #ETCD_DISCOVERY_FALLBACK=proxy #ETCD_DISCOVERY_PROXY= ETCD_ADVERTISE_CLIENT_URLS=https://10.15.69.154:2379 #[proxy] #ETCD_PROXY=off #[security] ETCD_CA_FILE=/etc/etcd/ca.crt ETCD_CERT_FILE=/etc/etcd/server.crt ETCD_KEY_FILE=/etc/etcd/server.key ETCD_PEER_CA_FILE=/etc/etcd/ca.crt ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt ETCD_PEER_KEY_FILE=/etc/etcd/peer.key
After a new upgrade: [root@npoyant-ocp0002 etcd]# grep storage-backend -A1 /etc/origin/master/master-config.yaml storage-backend: - etcd3 [root@npoyant-ocp0002 etcd]# source /etc/etcd/etcd.conf [root@npoyant-ocp0002 etcd]# export ETCDCTL_API=3 [root@npoyant-ocp0002 etcd]# ETCD_ALL_ENDPOINTS=` etcdctl --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS --write-out=fields member list | awk '/ClientURL/{printf "%s%s",sep,$3; sep=","}'` Error: empty string is passed to --cacert option This is it was noticed that the variables changed.
Going to close as a dupe of bug 1529575. We should now be ensuring that both 3.1 and 3.2 compatible flags are set during upgrade. 3.7 backport https://bugzilla.redhat.com/show_bug.cgi?id=1563376 3.6 backport https://bugzilla.redhat.com/show_bug.cgi?id=1563375 We're working to ensure that before etcd 3.3 ships we have 100% assurance that no new flags are necessary and if they are they'll be added during the upgrade process. *** This bug has been marked as a duplicate of bug 1529575 ***