Bug 1386963

Summary: etcd2 -> etcd3 migration utility fails against OpenShift 3.4 configuration data
Product: Red Hat Enterprise Linux 7 Reporter: Mike Fiedler <mifiedle>
Component: etcd3Assignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: medium    
Version: 7.2CC: anli, aos-bugs, jokerman, mifiedle, mmccomas, sdodson, tstclair, vlaad, xtian, ypu
Target Milestone: rcKeywords: Extras
Target Release: 7.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: etcd3-3.0.14-3.el7 Doc Type: If docs needed, set a value
Doc Text:
This enhancement update fixes migration utility that was failing when migrating from v2 to v3
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-06 17:43:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Fiedler 2016-10-19 23:31:49 UTC
Description of problem:

The migration of OpenShift data from etcd2 -> etcd3 currently fails while running the etcdctl migrate tool.   It errors out immediately whether a custom transformer or the default one is used.

The error (with some extra debugging in the custom transformer used) is:

Starting...
ERROR CODING NODE 0 {Key: , CreatedIndex: 0, ModifiedIndex: 0, TTL: 0}
Reader(0)[write /dev/stdin: bad file descriptor] = 
Err: write /dev/stdin: bad file descriptor
Error:  exit status 1
failed to transform keys

Version-Release number of selected component (if applicable):   3.4.0.11 and etcd 2.3.7 -> etcd3 3.0.12


How reproducible:  always (at least on the OCP 3.4 data sets we have tried)


Steps to Reproduce:
1.   Install OCP 3.4.0.11 with a separate etcd 2.3.7 
2.   Create some projects, pods, etc.   General OCP data
3.   Backup the etcd environment
4.   yum swap etcd etcd3 and verify the OCP env is still operational running etcd3 in etcd2 mode
5.   etcdctl --endpoints $endpoints migrate --data-dir=/var/lib/etcd


Actual results:  Error message (as above)


Expected results:

Successful migration

Comment 3 Timothy St. Clair 2016-10-20 14:19:07 UTC
fix is here: https://github.com/coreos/etcd/pull/6690, we'll need to update our etcdctl binaries.  

Hold off on final update until we've had a bit more time to validate.

Comment 5 Mike Fiedler 2016-10-20 20:47:51 UTC
Changing sev to high since it blocks migration.

Comment 6 Jan Chaloupka 2016-10-21 11:53:54 UTC
Merged upstream, updating rpm.

Comment 8 Joy Pu 2016-11-15 05:45:03 UTC
Hi Mike,

Can you help to provide some data for etcd for testing this? Seems I can not find OCP 3.4 env for testing it. And I try to reproduce it with some key value with '%' such as "test%" and "%", then migrate datedir from etcd-2.3.7-4 to etcd3-3.0.12-3, also borrow a openshift 3.3 env to test it. But it won't trigger the problem.

Comment 12 Joy Pu 2016-11-15 23:21:08 UTC
It's OK. :) Will following the new bug for the migration problem. And thanks for providing the information.

Comment 14 errata-xmlrpc 2016-12-06 17:43:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2890.html