Bug 1502767

Description of problem:
Following the guide here: https://docs.openshift.com/container-platform/3.6/admin_guide/encrypting_data.html

Customer was attempting to have all secrets in the cluster become encrypted. Steps followed below.

0. Generate encryption key
[root@br1endvosm01 ~]# head -c 32 /dev/urandom | base64
1. Update kubernetesMasterConfig in master-config.yaml on all masters with:
kubernetesMasterConfig:
  admissionConfig:
  apiServerArguments:
# BEGIN Encrypt secrets
    experimental-encryption-provider-config:
    - /usr/local/src/encryption_config.yml
# END Encrypt secrets

2. Add /usr/local/src/encryption_config.yml on all masters
[root@br1endvosm01 ~]# ls -al /usr/local/src/encryption_config.yml
-rw-------. 1 root root 225 Oct  4 10:44 /usr/local/src/encryption_config.yml
[root@br1endvosm01 ~]# cat /usr/local/src/encryption_config.yml 
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: "<REDACTED>"
    - identity: {}
3. Restart api service on all masters:
systemctl restart atomic-openshift-master-api

4. Create new secret
[root@br1endvosm01 ~]# oc create secret generic secret1 -n default --from-literal=mykey=mydata
secret "secret1" created

5. Attempt to confirm secret is encrypted, but it isn't
[root@br1endvosm01 ~]# etcdctl --cert-file='/etc/origin/master/master.etcd-client.crt' --key-file='/etc/origin/master/master.etcd-client.key' --endpoints='https://10.24.249.21:2379' --ca-file='/etc/origin/master/master.etcd-ca.crt' get /kubernetes.io/secrets/default/secret1 
2017-10-04 11:06:28.581200 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"kind":"Secret","apiVersion":"v1","metadata":{"name":"secret1","namespace":"default","uid":"b299b1d4-a91d-11e7-bbf7-005056ab6e19","creationTimestamp":"2017-10-04T16:04:27Z"},"data":{"mykey":"bXlkYXRh"},"type":"Opaque"}



The service is running as root, who owns the encryption_config.yaml file. 


Version-Release number of selected component (if applicable):
oc v3.6.173.0.21
kubernetes v1.6.1+5115d708d7
openshift v3.6.173.0.21
kubernetes v1.6.1+5115d708d7


How reproducible:
Unconfirmed