Bug 1768751 - certificate renewal script
Summary: certificate renewal script
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 1.4.1
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 1.4.2
Assignee: Roman Mohr
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On: 1768484
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-05 07:53 UTC by Fabian Deutsch
Modified: 2020-03-25 08:48 UTC (History)
12 users (show)

Fixed In Version: https://raw.githubusercontent.com/kubevirt/hyperconverged-cluster-operator/901d257cd1c5ca9ff1e66a80284bedc7c7bb4139/tools/rotate-certs.sh
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1768484
Environment:
Last Closed: 2020-03-25 08:44:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
verified rotation script - githash 901d257 (3.22 KB, text/plain)
2020-03-25 08:48 UTC, Dan Kenigsberg
no flags Details

Description Fabian Deutsch 2019-11-05 07:53:25 UTC
As a CNV 1.4 admin I'd like to easily check if my CNV certificates are nearing their expiry date, and to easily regenerate new certificate.

I'd like to have a script such as cnv_certs.sh that can be used as follows:

$ oc login 
$ ./cnv_certs.sh examine
...
list of existing certs, each with its expiry date

$ ./cnv_certs.sh renew
...
list of new certs each with its expiry date


The script should work against cnv-1.4 and cnv-2 (possibly two different scripts).

Comment 1 Fabian Deutsch 2019-11-05 07:53:57 UTC
CNV 2 bug can be found here bug 1768484

Comment 2 Roman Mohr 2019-12-19 09:15:35 UTC
There is now a script in HCO which roughly does what was requested here:

https://github.com/kubevirt/hyperconverged-cluster-operator/pull/372
https://github.com/kubevirt/hyperconverged-cluster-operator/pull/352

Comment 3 Dan Kenigsberg 2019-12-19 09:24:25 UTC
The documentation on how to test this is here: https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/tools/README.md#rotating-certificates

Comment 4 Nelly Credi 2019-12-30 07:31:38 UTC
Please add 'fixed in version'

Comment 5 Dan Kenigsberg 2019-12-30 11:02:58 UTC
The Fixed in Version field is not very meaningful here, as what we deliver is an upstream script. Nonetheless, I've added a link to it.

Comment 8 Roman Mohr 2020-02-05 10:01:36 UTC
A fix posted: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/424

Comment 9 Roman Mohr 2020-02-05 16:41:23 UTC
PR is merged and can be taken from master. Since it is meant just for the knowledge-base, I don't think that this needs a backport anywhere. Let me know if that is not true.

Comment 14 Marek Libra 2020-02-18 10:52:31 UTC
Yes, the secret and the pod "console-*" which is for the web-ui application. There's no need to remove the operator pod.

Comment 20 Dan Kenigsberg 2020-03-18 09:01:29 UTC
No build is expected, this should be ON_QA now.

Comment 22 Israel Pinto 2020-03-18 20:03:51 UTC
kubevirt-web-ui certificate rotating failed with error:
Error from server (NotFound): kwebuis.kubevirt.io "kubevirt-web-ui" not found
Also web console is not reachable: https://kubevirt-web-ui.cloudapps.example.com/

Moving to assign.

Comment 23 Israel Pinto 2020-03-18 20:28:54 UTC
Update: web console is reachable , problem with sshuttle

Comment 26 Israel Pinto 2020-03-19 21:44:16 UTC
The secret under kubevirt-web-ui not renew, and we have error in the script.
Moving to assign.

Comment 27 Tomas Jelinek 2020-03-23 06:27:07 UTC
https://github.com/kubevirt/hyperconverged-cluster-operator/pull/504 merged,  moving to on_qa

Comment 28 Israel Pinto 2020-03-23 10:22:53 UTC
Verify: 
Steps:
1. Run script
2. Check secret under: cdi/kubevirt/kubevirt-web-ui
3. Check the exist VMs are running
4. Check Web UI is responsive
6. Add new project: Create new VM / connect via console
5. Check that we can view new VM from UI  

Output:

$bash rotate-certs.sh --namespace kubevirt --cdi-namespace cdi
# Rotating kubemacpool certificates ...
No resources found
# Rotating cdi certificates ...
deployment.extensions/cdi-operator scaled
secret "cdi-api-server-cert" deleted
secret "cdi-api-signing-key" deleted
secret "cdi-upload-proxy-ca-key" deleted
secret "cdi-upload-proxy-server-key" deleted
secret "cdi-upload-server-ca-key" deleted
secret "cdi-upload-server-client-ca-key" deleted
secret "cdi-upload-server-client-key" deleted
pod "cdi-apiserver-586b64b5d-kvvl8" deleted
pod "cdi-deployment-85dff44b86-hqshl" deleted
pod "cdi-uploadproxy-5b5659fc76-mv8sh" deleted
NAME      HOST/PORT                               PATH      SERVICES   PORT      TERMINATION          WILDCARD
console   kubevirt-web-ui.cloudapps.example.com             console    https     reencrypt/Redirect   None
pod "cdi-apiserver-586b64b5d-nphjm" deleted
pod "cdi-deployment-85dff44b86-5b9sw" deleted
pod "cdi-uploadproxy-5b5659fc76-9674k" deleted
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
deployment.extensions/cdi-operator scaled
# Rotating kubevirt certificates ...
secret "kubevirt-virt-api-certs" deleted
pod "virt-api-669f897b49-ddjpn" deleted
pod "virt-api-669f897b49-r59g9" deleted
pod "virt-controller-64dc697446-m2zc7" deleted
pod "virt-controller-64dc697446-xj6bj" deleted
pod "virt-handler-h7mvb" deleted
pod "virt-handler-z8vxp" deleted
# Rotating SSP certificates ...
No resources found
# Rotating Web UI certificates ...
Detected former Web UI version: v1.4.1
kwebui.kubevirt.io/kubevirt-web-ui patched
console   1         1         1         1         48m
Waiting for Web UI ...
console   1         1         1         1         48m
Waiting for Web UI ...
kwebui.kubevirt.io/kubevirt-web-ui patched
[cnv-qe-jenkins@cnv-executor-cnv14 tools]$  oc get secret -n cdi | grep Opaque 
cdi-api-server-cert               Opaque                                3         1m
cdi-api-signing-key               Opaque                                2         1m
cdi-upload-proxy-ca-key           Opaque                                2         45s
cdi-upload-proxy-server-key       Opaque                                2         44s
cdi-upload-server-ca-key          Opaque                                2         45s
cdi-upload-server-client-ca-key   Opaque                                2         45s
cdi-upload-server-client-key      Opaque                                3         45s
[cnv-qe-jenkins@cnv-executor-cnv14 tools]$  oc get secret -n kubevirt | grep Opaque 
kubevirt-virt-api-certs                      Opaque                                3         1m
[cnv-qe-jenkins@cnv-executor-cnv14 tools]$  oc get secret -n kubevirt-web-ui | grep Opaque 
console-oauth-config                       Opaque                                1         47s
[cnv-qe-jenkins@cnv-executor-cnv14 tools]$ oc get vmi --all-namespaces 
NAMESPACE   NAME          AGE       PHASE     IP             NODENAME
default     vm-cirros-2   4d        Running   10.130.0.21    cnv-executor-cnv14-node-20c0b0-1.example.com
test-1      vm-cirros-2   2d        Running   10.130.0.76    cnv-executor-cnv14-node-20c0b0-1.example.com
test-2      vm-cirros-2   2d        Running   10.130.0.110   cnv-executor-cnv14-node-20c0b0-1.example.com
test-3      vm-cirros-2   21h       Running   10.130.0.122   cnv-executor-cnv14-node-20c0b0-1.example.com
test-4      vm-cirros-2   4m        Running   10.130.0.134   cnv-executor-cnv14-node-20c0b0-1.example.com

New VM in project test-5:
[cnv-qe-jenkins@cnv-executor-cnv14 tools]$ oc get vmi --all-namespaces 
NAMESPACE   NAME          AGE       PHASE     IP             NODENAME
default     vm-cirros-2   4d        Running   10.130.0.21    cnv-executor-cnv14-node-20c0b0-1.example.com
test-1      vm-cirros-2   2d        Running   10.130.0.76    cnv-executor-cnv14-node-20c0b0-1.example.com
test-2      vm-cirros-2   2d        Running   10.130.0.110   cnv-executor-cnv14-node-20c0b0-1.example.com
test-3      vm-cirros-2   21h       Running   10.130.0.122   cnv-executor-cnv14-node-20c0b0-1.example.com
test-4      vm-cirros-2   10m       Running   10.130.0.134   cnv-executor-cnv14-node-20c0b0-1.example.com
test-5      vm-cirros-2   58s       Running   10.130.0.143   cnv-executor-cnv14-node-20c0b0-1.example.com


ALL PASS

Comment 29 Dan Kenigsberg 2020-03-25 08:43:56 UTC
Thanks for the fix and verification. Closing as there is no errata delivery for this script.

Comment 30 Dan Kenigsberg 2020-03-25 08:48:06 UTC
Created attachment 1673330 [details]
verified rotation script - githash 901d257


Note You need to log in before you can comment on or make changes to this bug.