Bug 1749653

Summary: EC certificates are not working for routes in OCP
Product: OpenShift Container Platform Reporter: Neeraj <nbhatt>
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: alwyn, aos-bugs, freark+1
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-07 15:13:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Neeraj 2019-09-06 07:18:22 UTC
Description of problem:
EC certificates are not working for routes in OCP

Version-Release number of selected component (if applicable):
3.x

How reproducible:
100%

Steps to Reproduce:
~~~
[root@master-0 router-certs]# cat ec.key 
-----BEGIN EC PARAMETERS-----
BggqhkjOPQMBBw==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIO74ZSKtpel3dG1HtgUsHjERaoAt3l61HZ54POHxYCgioAoGCCqGSM49
AwEHoUQDQgAEcRn09VRIeVuzVVfDd7LSRxkIZamqu/4e0s8uj+o44x43XXU2Oweb
/7Y4pZ6y2UWgyuIwrtcwPsTzUpHorb7Uaw==
-----END EC PRIVATE KEY-----
~~~

Here are the steps I have done to redeploy the custom EC certs for router.

~~~
> Create a config:

# cat >>  san.cnf <<EOF
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
organizationName = Organization Name (eg, company)
commonName = Common Name (e.g. server FQDN or YOUR name)
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[ req_ext ]
subjectAltName = @al  325t_names
# Put the SAN values acc to your requirement. Below is the example how to configure it.
[alt_names]
DNS.1 = *.apps.devilamycry.lab.pnq2.cee.redhat.com
DNS.2 = apps.devilamycry.lab.pnq2.cee.redhat.com
EOF

> created CA key and cert
# openssl ecparam -out ec.key -name prime256v1 -genkey 
#openssl req -in ec.csr  -x509 -nodes -days 5000 -key ec.key  -out ec-ca.crt 

>> generated device CSR and CRT
#openssl req -new -key ec.key -out ec.csr -sha256
#openssl req -in ec.csr  -x509 -nodes -days 5000 -key ec.key  -out ec.crt  -extensions v3_req -config  san.cnf

>> Verfiy
# openssl x509  -text -noout -in ec.crt

> Step need for haproxy:
#  cat ec.key ec1.crt ec.crt > router.pem 

>> Create a route:


-  In case of edge:
  #  oc create route edge  console-2 --service=registry-console --ca-cert=ec.crt --cert=router.pem  --key=ec.key --hostname=abcdeas.apps.devilamycry.lab.pnq2.cee.redhat.com

   - In this the route unable to add the key.

~~~~
Requested Host:	abcdeas.apps.devilamycry.lab.pnq2.cee.redhat.com
		  rejected by router router: ExtendedValidationFailed (21 minutes ago)
		    
  - spec.tls.certificate: Invalid value: "redacted certificate data": unrecognized PEM block EC PRIVATE KEY
  - spec.tls.key: Invalid value: "redacted key data": unrecognized PEM block EC PRIVATE KEY
~~~


Router logs:
~~~
E0906 06:49:01.279798       1 extended_validator.go:55] Skipping route default/console-2 due to invalid configuration: 
  - spec.tls.certificate: Invalid value: "redacted certificate data": unrecognized PEM block EC PRIVATE KEY
  - spec.tls.key: Invalid value: "redacted key data": unrecognized PEM block EC PRIVATE KEY
E0906 06:49:01.279868       1 router_controller.go:250] invalid route configuration

~~~~




 In case of reencrypt:

~~~~
  #  oc create route reencrypt console-3 --service=registry-console --ca-cert=ec.crt --cert=ec1.crt --dest-ca-cert=ec.crt --key=pkcs8_key.pem --hostname=asadbcd.apps.devilamycry.lab.pnq2.cee.redhat.com

- Route added but application is not avaibale.

- router logs
~~~
E0906 07:00:25.906850       1 limiter.go:137] error reloading router: exit status 1
[ALERT] 248/070025 (1857) : parsing [/var/lib/haproxy/conf/haproxy.config:116] : 'bind 127.0.0.1:10444' : 'crt-list' : error processing line 1 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load SSL private key from PEM file '/var/lib/haproxy/router/certs/default:console-3.pem'.
[ALERT] 248/070025 (1857) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
~~~
~~~~




Additional info:

I reckon the culprit here is the conversion of key inside router pod, however this is properly working when I redeploy the router certs.


[root@master-0 router-certs]# cat ec.key
-----BEGIN EC PARAMETERS-----
BggqhkjOPQMBBw==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIO74ZSKtpel3dG1HtgUsHjERaoAt3l61HZ54POHxYCgioAoGCCqGSM49
AwEHoUQDQgAEcRn09VRIeVuzVVfDd7LSRxkIZamqu/4e0s8uj+o44x43XXU2Oweb
/7Y4pZ6y2UWgyuIwrtcwPsTzUpHorb7Uaw==
-----END EC PRIVATE KEY-----


INSIDE ROUTER:

[root@master-0 router-certs]# oc rsh router-3-7scn7
sh-4.2$ cat ../router/certs/default\:console-3.pem 
-----BEGIN ECDSA PRIVATE KEY-----
MHcCAQEEIO74ZSKtpel3dG1HtgUsHjERaoAt3l61HZ54POHxYCgioAoGCCqGSM49
AwEHoUQDQgAEcRn09VRIeVuzVVfDd7LSRxkIZamqu/4e0s8uj+o44x43XXU2Oweb
/7Y4pZ6y2UWgyuIwrtcwPsTzUpHorb7Uaw==
-----END ECDSA PRIVATE KEY-----

Comment 1 Alwyn Kik 2019-10-07 15:10:27 UTC
I believe the following PR intends to fix another related issue and also fixes this one: https://github.com/openshift/origin/pull/23918

Comment 2 Dan Mace 2019-10-07 15:13:15 UTC

*** This bug has been marked as a duplicate of bug 1723400 ***

Comment 3 Dan Mace 2019-10-07 15:13:47 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1723400 is the canonical tracker for this issue. Thanks!