Bug 1810133

Summary: connect to etcd using ips
Product: OpenShift Container Platform Reporter: David Eads <deads>
Component: kube-apiserverAssignee: Luis Sanchez <sanchezl>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: aos-bugs, geliu, mfojtik, sanchezl, sbatsche, xxia, yanyang
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1812071 (view as bug list) Environment:
Last Closed: 2020-07-13 17:18:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1807194, 1812071    

Description David Eads 2020-03-04 15:29:07 UTC
Using IPs instead of dns names reduces dependencies and gives more flexibility to the etcd operator maintain reliability.

Comment 1 Luis Sanchez 2020-03-05 14:07:26 UTC
*** Bug 1809732 has been marked as a duplicate of this bug. ***

Comment 2 Luis Sanchez 2020-03-05 14:11:22 UTC
*** Bug 1807194 has been marked as a duplicate of this bug. ***

Comment 5 Ke Wang 2020-03-12 10:12:48 UTC
Tried to verify with OCP build 
oc get kubeapiserver/cluster -o json | jq .spec.observedConfig.storageConfig
{
  "urls": [
    "https://10.0.142.186:2379",
    "https://10.0.154.171:2379",
    "https://10.0.170.94:2379",
    "https://localhost:2379"
  ]
}

logged into the master node and searching related keywords.,

# grep -nr 'https://etcd' openshift*

Nothing found.

# grep -nr 'host-etcd' openshift*
openshift-apiserver-operator_openshift-apiserver-operator-57854c4dcb-jcpvq_8d362102-0fe0-4496-8b79-d0c353ed6dcd/openshift-apiserver-operator/2.log:53:2020-03-12T02:19:21.935209526+00:00 stderr F I0312 02:19:21.935170       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-apiserver-operator", Name:"openshift-apiserver-operator", UID:"e8915d9b-6bc2-4a38-8dc1-406cf5424204", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ObserveStorageFailed' Required endpoints/host-etcd-2 in the openshift-etcd namespace not found, falling back to default etcd service
...
openshift-sdn_sdn-ssv96_1f74a551-69c5-467f-a28d-5e94de3f2f75/sdn/0.log:25:2020-03-12T02:12:30.507255093+00:00 stderr F I0312 02:12:30.505087    2139 roundrobin.go:235] LoadBalancerRR: Setting endpoints for openshift-etcd/host-etcd:etcd to [192.0.2.2:2379 192.0.2.200:2379 192.0.2.3:2379 192.0.2.4:2379]
openshift-sdn_sdn-ssv96_1f74a551-69c5-467f-a28d-5e94de3f2f75/sdn/0.log:103:2020-03-12T02:13:32.701757672+00:00 stderr F I0312 02:13:32.701493    2139 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-etcd/host-etcd-2:etcd to [10.0.142.186:2379 10.0.154.171:2379 10.0.170.94:2379]

Let's see the above second result, still found ‘host-etcd’ in log file. From PR https://github.com/openshift/cluster-kube-apiserver-operator/pull/787 change, it should not be find. Anyone can confirm if some codes are not completely updated for this?

Comment 6 Ke Wang 2020-03-12 10:47:26 UTC
Please ignore the above comments, here is for 4.5, 

With OCP build 4.5.0-0.nightly-2020-03-11-080149,

$ oc get kubeapiserver/cluster -o json | jq .spec.observedConfig.storageConfig
{
  "urls": [
    "https://192.168.0.14:2379",
    "https://192.168.0.24:2379",
    "https://192.168.0.26:2379",
    "https://localhost:2379"
  ]
}


logged into the master node and searching related keywords.,

# grep -nr 'https://etcd' openshift*

Nothing found.

# grep -nr 'host-etcd' openshift*
openshift-apiserver-operator_openshift-apiserver-operator-7f4784bc46-hkhnz_5fdcbcc6-5228-4abb-9c52-371217e8f0fb/openshift-apiserver-operator/0.log:52:2020-03-11T11:11:57.760368140+00:00 stderr F I0311 11:11:57.760323       1 event.go:281] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-apiserver-operator", Name:"openshift-apiserver-operator", UID:"6cbde1b3-b5d8-4d1f-8ee1-d63d6149468a", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ObserveStorageFallback' endpoints/host-etcd-2 in the openshift-etcd namespace: no etcd endpoint addresses found, falling back to default etcd service
...
openshift-sdn_sdn-24ggm_bc1940f2-625f-44f9-8025-efea60845133/sdn/0.log:42:2020-03-11T11:11:35.839990269+00:00 stderr F I0311 11:11:35.839976    3374 roundrobin.go:235] LoadBalancerRR: Setting endpoints for openshift-etcd/host-etcd:etcd to [192.0.2.2:2379 192.0.2.200:2379 192.0.2.3:2379 192.0.2.4:2379]
openshift-sdn_sdn-24ggm_bc1940f2-625f-44f9-8025-efea60845133/sdn/0.log:117:2020-03-11T11:11:58.275364864+00:00 stderr F I0311 11:11:58.275288    3374 roundrobin.go:267] LoadBalancerRR: Setting endpoints for openshift-etcd/host-etcd-2:etcd to [192.168.0.14:2379 192.168.0.24:2379 192.168.0.26:2379]

Let's see the above second result, still found ‘host-etcd’ in log file. From PR https://github.com/openshift/cluster-kube-apiserver-operator/pull/787 change, it should not be find. Anyone can confirm if some codes are not completely updated for this?

Comment 8 Luis Sanchez 2020-03-16 15:22:48 UTC
host-etcd is still created in 4.4, but unused (as you verified in the list of storage urls). It will be removed completely in 4.5 (https://bugzilla.redhat.com/show_bug.cgi?id=1813012).

Comment 9 Ke Wang 2020-03-17 03:50:17 UTC
Since the Luis explained the problem that found ‘host-etcd’ in log file, it is reasonable at present stage, move the bug verify.

Comment 11 errata-xmlrpc 2020-07-13 17:18:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409