Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1387391

Summary: api-server never initializes when configured with storage-backend=etcd3 - logs flooded with client transport error msgs
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: NodeAssignee: Timothy St. Clair <tstclair>
Status: CLOSED ERRATA QA Contact: Mike Fiedler <mifiedle>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, bleanhar, ccoleman, dma, jeder, jokerman, mifiedle, mmccomas, tdawson, tstclair, vlaad
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: aos-scalability-34
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-18 12:43:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
api server logs
none
etcd3 cluster leader log with --debug=true
none
api server logs at loglevel=8 none

Description Mike Fiedler 2016-10-20 18:39:13 UTC 
Description of problem:

After updating the etcd systems in an HA cluster from 2.3.7 to 3.0.12, migrating the data and configuring the master-api servers to use storage-backend=etcd3, the api-server will not completely start.    The logs are flooded with the following errors:

Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.1.11.211:33308->192.1.11.215:2379: read: connection reset by peer.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.1.11.211:34574->192.1.11.214:2379: read: connection reset by peer.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.1.11.211:48334->192.1.11.216:2379: read: connection reset by peer.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.1.11.211:34570->192.1.11.214:2379: read: connection reset by peer.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
Oct 20 13:41:17 svt-m-1.localdomain openshift[44796]: transport: http2Client.notifyError got notified that the client transport was broken read tcp 192.1.11.211:48374->192.1.11.216:2379: read: connection reset by peer.


Version-Release number of selected component (if applicable): 3.4.0.11


How reproducible: Always


Steps to Reproduce:

Environment:

3.4.0.11 cluster installed with 3 masters and 3 2.3.7 etcd servers and 300 nodes.   Ran the conformance tests in the cluster and then created 1K projects with 4K pods.   Everything successful up to this point.

This issue has also been reproduced in a smaller cluster with 1 etcd, 1 master and 3 nodes.  Cluster size not an issue.

0. Shutdown all OCP masters
1. Shutdown etcd servers and update (yum swap) to install etcd 3.0.12
2. On each etcd:  etcdctl migrate --data-dir=/var/lib/etcd
3. Verify the migration completes with no errors and restart all etcds
4. On each master, edit master-config.yaml and add the following to apiServerArguments:

  apiServerArguments:
    storage-backend:
    - "etcd3"

5. Restart the master services

Actual results:

The master-api services will fail to initialize.   The logs will be overrun with the messages above.  Attached are the master-api journal logs and etcd with --debug=true.   The etcd logs show no errors.


Expected results:

Migrated cluster operates as before the etcd upgrade.

Additional info:

I ran tcpdump/wireshark and every time the master sends something to the etcd system, the etcd system sends back a reset/RST
Comment 1 Mike Fiedler 2016-10-20 18:39:50 UTC 
Created attachment 1212615 [details]
api server logs
Comment 2 Mike Fiedler 2016-10-20 18:40:26 UTC 
Created attachment 1212616 [details]
etcd3 cluster leader log with --debug=true
Comment 3 Mike Fiedler 2016-10-20 18:42:22 UTC 
Removing storage-backend=etcd3 from master-config.yaml restores cluster functionality.
Comment 4 Mike Fiedler 2016-10-20 19:04:24 UTC 
One note.   For a successful migration (etcdctl migrate) you might need a patch to etcdctl.  See https://bugzilla.redhat.com/show_bug.cgi?id=1386963#c3
Comment 5 Mike Fiedler 2016-10-20 19:46:09 UTC 
Created attachment 1212633 [details]
api server logs at loglevel=8
Comment 6 Timothy St. Clair 2016-10-20 19:57:17 UTC 
Client in openshift is v3.1.0-alpha.1, and I'm guessing it's un-vetted for version compat as things have changed.
Comment 7 Clayton Coleman 2016-10-21 15:41:58 UTC 
This is https://github.com/openshift/origin/pull/10980
Comment 10 Troy Dawson 2016-10-27 16:06:18 UTC 
This has been merged into ose and is in OSE v3.4.0.16 or newer.
Comment 12 Mike Fiedler 2016-11-03 11:39:34 UTC 
This specific issue is resolved, although additional issues with running in migrated etcd2->etcd3 clusters are still being investigated.
Comment 15 errata-xmlrpc 2017-01-18 12:43:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066