Bug 1541550

Summary:	Enabling At-Rest Secret Encryption & Migrating Etcd V2 To V3 Data Leaves Unencrypted Secrets Behind.
Product:	OpenShift Container Platform	Reporter:	Bernie Hoefer <bhoefer>
Component:	Cluster Version Operator	Assignee:	Scott Dodson <sdodson>
Status:	CLOSED DUPLICATE	QA Contact:	Weihua Meng <wmeng>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	aos-bugs, cscribne, dmoessne, joedward, jokerman, mmccomas, pdwyer, scuppett, sdodson, thunt
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-08-23 20:43:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Bernie Hoefer 2018-02-02 20:47:16 UTC

Description of problem:
The etcd datastore migration playbook was run after upgrading from OCP 3.5 to 3.6.  The datastore was been updated to v3; existing secrets can be seen to be encrypted when using th3 v3 API to query etcd.  However, querying the etcd database using the v2 API shows the old, unencrypted (just base64 encoded) secret.

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2018-02-02 21:18:28 UTC

Ok, understanding this a bit more the process was upgrade from 3.5 to 3.6. Migrate from v2 to v3. Enable etcd encryption at rest per https://docs.openshift.com/container-platform/3.6/admin_guide/encrypting_data.html

Yes, the v2 keys are not removed during the etcd v2 to v3 migration.

They can be removed by running the following on your master:
etcdctl2 rm -r /kubernetes.io
etcdctl2 rm -r /openshift.io

Please note that etcd backups are also taken during the upgrade and migration processes. If you truly want to eliminate all traces of unencrypted data you'll also want to carefully remove /var/lib/etcd/openshift-backup* making sure not to touch /var/lib/etcd/member which is the live database.

Example directory listing

# ls -la /var/lib/etcd
total 4
drwxr-xr-x.  7 etcd etcd  218 Feb  2 14:48 .
drwxr-xr-x. 44 root root 4096 Feb  2 14:51 ..
drwx------.  4 etcd etcd   29 Feb  2 13:42 member
drwx------.  3 root root   20 Feb  2 14:36 openshift-backup-post-3.0-20180202143640
drwx------.  3 root root   20 Feb  2 14:48 openshift-backup-post-3.0-20180202144816
drwx------.  3 root root   20 Feb  2 14:36 openshift-backup-pre-upgrade-20180202143626
drwx------.  3 root root   20 Feb  2 14:48 openshift-backup-pre-upgrade-20180202144759

Comment 10 Scott Dodson 2018-08-23 20:43:51 UTC


*** This bug has been marked as a duplicate of bug 1514487 ***