Created attachment 1324318 [details] ansible log - updating router certificates Description of problem: After run redeploy-router-certificates.yml playbook against an ocp-3.6 cluster, router pod became CrashLoopBackOff status. Version-Release number of the following components: openshift-ansible-3.6.173.0.31-1.git.0.c9aeacc.el7.noarch ansible-2.3.2.0-2.el7.noarch openshift v3.6.173.0.30 How reproducible: Always Steps to Reproduce: 1.Setup an ocp-3.6 env, router pod is running well [root@qe-gpei-36-3-master-1 ~]# oc get pod NAME READY STATUS RESTARTS AGE docker-registry-3-p3x4b 1/1 Running 0 10m registry-console-1-vbfzd 1/1 Running 0 11m router-1-1vh7q 1/1 Running 0 12m 2.Run redeploy-router-certificates.yml playbook ansible-playbook -i host /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-router-certificates.yml -v The detailed log could be found in the attachment Actual results: [root@qe-gpei-36-3-master-1 ~]# oc get pod NAME READY STATUS RESTARTS AGE docker-registry-3-p3x4b 1/1 Running 0 26m registry-console-1-vbfzd 1/1 Running 0 27m router-2-mp2wg 0/1 CrashLoopBackOff 7 12m [root@qe-gpei-36-3-master-1 ~]# oc logs router-2-mp2wg I0911 03:16:32.737459 1 template.go:246] Starting template router (v3.6.173.0.30) I0911 03:16:32.766937 1 metrics.go:43] Router health and metrics port listening at 0.0.0.0:1936 I0911 03:16:33.644177 1 router.go:240] Router is including routes in all namespaces E0911 03:16:33.690427 1 ratelimiter.go:52] error reloading router: exit status 1 [ALERT] 253/031633 (45) : parsing [/var/lib/haproxy/conf/haproxy.config:123] : 'bind 127.0.0.1:10444' : unable to load SSL private key from PEM file '/etc/pki/tls/private/tls.crt'. [ALERT] 253/031633 (45) : parsing [/var/lib/haproxy/conf/haproxy.config:164] : 'bind 127.0.0.1:10443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/tls.crt'. [ALERT] 253/031633 (45) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] 253/031633 (45) : Proxy 'fe_sni': no SSL certificate specified for bind '127.0.0.1:10444' at [/var/lib/haproxy/conf/haproxy.config:123] (use 'crt'). [ALERT] 253/031633 (45) : Proxy 'fe_no_sni': no SSL certificate specified for bind '127.0.0.1:10443' at [/var/lib/haproxy/conf/haproxy.config:164] (use 'crt'). [ALERT] 253/031633 (45) : Fatal errors found in configuration. Expected results: Additional info: [root@qe-gpei-36-3-master-1 ~]# oc rsh router-2-mp2wg cat /etc/pki/tls/private/tls.crt -----BEGIN CERTIFICATE----- MIIDhDCCAmygAwIBAgIIOCo5zN6WPgswDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTUwNTA5Nzg0MDAe Fw0xNzA5MTEwMzA4NDlaFw0xOTA5MTEwMzA4NTBaMB0xGzAZBgNVBAMTEnJvdXRl ci5kZWZhdWx0LnN2YzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJD3 GsBrp0SgiBod5XmPSEvST9BwlAVYeyJjvOMsffelbQ+nqkSY53X/QH2hfVx2nJsR HjBDFgTW8AnHo58eo/P8/webZpRWLZI5+qK7CK8WJNblCcQaRgY+oXspU1wOC9T2 aGRKw6ha0WZKf8MhB5Tz1ZUTgQrTynd3s/QQWxhYYqQ+O7xeJwRF1cX26xyVMEUt Ui6MZNiysWbjzdaAk23jRsUsFgJS8Whnli3shOyCBtdHGdly1BlLEOo4/hURscrd Ixd0iobyV7CSgZmgZ7Bp6hKWtDn1OV94yotgFNN6rxZ7b/IXzisbg0Ej/72qdB+Z gN0hZ0BZckCa6+CWj3MCAwEAAaOBrjCBqzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADA/BgNVHREEODA2ghJyb3V0ZXIu ZGVmYXVsdC5zdmOCIHJvdXRlci5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsMDUG CysGAQQBkggRZAIBBCYTJDE5Y2RjYWZmLTk2OWMtMTFlNy1hMzQ5LTQyMDEwYWYw MDAxMzANBgkqhkiG9w0BAQsFAAOCAQEAASsLz2cYDMXxI+WplkH6XiRVbMqS2NPF ZDiWYiIBJPg44mv9x32WCeqE6kyv4qgS9cXkI/WC2M8z0NHiFcQCn+zHUz03JdvP vnMH0p7apuokkjwReT3QYosXaoLjcfVngv4tW9laORT3u6SmMSWCe+wzlDzHy2Rl Z0awwYyzwD8ikihwkfCEBue1KDk9LlXTFQ7Lsq73p3TJSevwMEaAW5sfFR5HkNhq 9RTQ2cbNsqQby3+pmMtQs8t13x+mDcldhGivvMLoFlZqnCkCEQN9Hmz2jU4vj1yp y+yexaFaMqNg5953oiWuMh4z9RHngLqsCTGjL/HZ9MtA/fLxk3GSmA== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDCjCCAfKgAwIBAgIBATANBgkqhkiG9w0BAQsFADA2MTQwMgYDVQQDDCtvcGVu c2hpZnQtc2VydmljZS1zZXJ2aW5nLXNpZ25lckAxNTA1MDk3ODQwMB4XDTE3MDkx MTAyNDQwMFoXDTIyMDkxMDAyNDQwMVowNjE0MDIGA1UEAwwrb3BlbnNoaWZ0LXNl cnZpY2Utc2VydmluZy1zaWduZXJAMTUwNTA5Nzg0MDCCASIwDQYJKoZIhvcNAQEB BQADggEPADCCAQoCggEBALRiyy6kpm0zgTnPcU3uAiC7zXr9HaPrlNLV7k0R4ZSg fY1jaJbN/KZXsUJhvPf72xBSjC+obvo0OQWSWYnw4IFgOMG1EeYFRWfoWKW4edp0 GWsU4CMVZXve0u6tVSNH7815YZk/9PWaSC9rBwYBO5OKuNXdo1KnWsFlQ3riyY/9 joLzpHqAHyo6FDTFG6wynVq7yBkJN9JLicNggDhH0sJJ2+Tu5dT8HMgroOJHx28H bWZ9gZAhPa6GxO4EbDIQa8jxLSG+VXk9AgTteu3Dr4RrUKcASvy+qtVXeOJE2/YG CpDUskVIziq/5nV0IVIoHj5bQlEyjhv8tA1GjrRl4vECAwEAAaMjMCEwDgYDVR0P AQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACUr 8L3txGwv0xvXpVgPZOU0dJIQYp/P0WkjR+WZzyU6CPiN6Wz+CiU1+MnATbztNU9q mK5XiBNGB2gIXzBKlmXz+Mm23tG7SFB9w1eT1HA1Xj85YGOz+bx+rc/ovSHd0dXm FVasZTy0b6rLTa/tuxcCKwnzloXzJaMLfVHrRWy6vRD20t9ASUwE1cS+fB5hSGg3 YaccgG9HuBxajMl+dVIPfF3FjoLOlvyK//jGdiFInWehXDZfNx3kqJv7h76viApE U4vh04ZB5FUq44xrj/2ywLQqWYyycvhOc59fuUety87kSxk+l/YFWEj9sz+CPpre jpVDGX7ToCyNRq7sSSU= -----END CERTIFICATE----- [root@qe-gpei-36-3-master-1 ~]# oc get secret router-certs -o yaml apiVersion: v1 data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURoRENDQW15Z0F3SUJBZ0lJT0NvNXpONldQZ3N3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRVd05UQTVOemcwTURBZQpGdzB4TnpBNU1URXdNekE0TkRsYUZ3MHhPVEE1TVRFd016QTROVEJhTUIweEd6QVpCZ05WQkFNVEVuSnZkWFJsCmNpNWtaV1poZFd4MExuTjJZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFKRDMKR3NCcnAwU2dpQm9kNVhtUFNFdlNUOUJ3bEFWWWV5Smp2T01zZmZlbGJRK25xa1NZNTNYL1FIMmhmVngybkpzUgpIakJERmdUVzhBbkhvNThlby9QOC93ZWJacFJXTFpJNStxSzdDSzhXSk5ibENjUWFSZ1krb1hzcFUxd09DOVQyCmFHUkt3NmhhMFdaS2Y4TWhCNVR6MVpVVGdRclR5bmQzcy9RUVd4aFlZcVErTzd4ZUp3UkYxY1gyNnh5Vk1FVXQKVWk2TVpOaXlzV2JqemRhQWsyM2pSc1VzRmdKUzhXaG5saTNzaE95Q0J0ZEhHZGx5MUJsTEVPbzQvaFVSc2NyZApJeGQwaW9ieVY3Q1NnWm1nWjdCcDZoS1d0RG4xT1Y5NHlvdGdGTk42cnhaN2IvSVh6aXNiZzBFai83MnFkQitaCmdOMGhaMEJaY2tDYTYrQ1dqM01DQXdFQUFhT0JyakNCcXpBT0JnTlZIUThCQWY4RUJBTUNCYUF3RXdZRFZSMGwKQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVEFRSC9CQUl3QURBL0JnTlZIUkVFT0RBMmdoSnliM1YwWlhJdQpaR1ZtWVhWc2RDNXpkbU9DSUhKdmRYUmxjaTVrWldaaGRXeDBMbk4yWXk1amJIVnpkR1Z5TG14dlkyRnNNRFVHCkN5c0dBUVFCa2dnUlpBSUJCQ1lUSkRFNVkyUmpZV1ptTFRrMk9XTXRNVEZsTnkxaE16UTVMVFF5TURFd1lXWXcKTURBeE16QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFBU3NMejJjWURNWHhJK1dwbGtINlhpUlZiTXFTMk5QRgpaRGlXWWlJQkpQZzQ0bXY5eDMyV0NlcUU2a3l2NHFnUzljWGtJL1dDMk04ejBOSGlGY1FDbit6SFV6MDNKZHZQCnZuTUgwcDdhcHVva2tqd1JlVDNRWW9zWGFvTGpjZlZuZ3Y0dFc5bGFPUlQzdTZTbU1TV0NlK3d6bER6SHkyUmwKWjBhd3dZeXp3RDhpa2lod2tmQ0VCdWUxS0RrOUxsWFRGUTdMc3E3M3AzVEpTZXZ3TUVhQVc1c2ZGUjVIa05ocQo5UlRRMmNiTnNxUWJ5MytwbU10UXM4dDEzeCttRGNsZGhHaXZ2TUxvRmxacW5Da0NFUU45SG16MmpVNHZqMXlwCnkreWV4YUZhTXFOZzU5NTNvaVd1TWg0ejlSSG5nTHFzQ1RHakwvSFo5TXRBL2ZMeGszR1NtQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDakNDQWZLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREEyTVRRd01nWURWUVFEREN0dmNHVnUKYzJocFpuUXRjMlZ5ZG1salpTMXpaWEoyYVc1bkxYTnBaMjVsY2tBeE5UQTFNRGszT0RRd01CNFhEVEUzTURreApNVEF5TkRRd01Gb1hEVEl5TURreE1EQXlORFF3TVZvd05qRTBNRElHQTFVRUF3d3JiM0JsYm5Ob2FXWjBMWE5sCmNuWnBZMlV0YzJWeWRtbHVaeTF6YVdkdVpYSkFNVFV3TlRBNU56ZzBNRENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTFJpeXk2a3BtMHpnVG5QY1UzdUFpQzd6WHI5SGFQcmxOTFY3azBSNFpTZwpmWTFqYUpiTi9LWlhzVUpodlBmNzJ4QlNqQytvYnZvME9RV1NXWW53NElGZ09NRzFFZVlGUldmb1dLVzRlZHAwCkdXc1U0Q01WWlh2ZTB1NnRWU05INzgxNVlaay85UFdhU0M5ckJ3WUJPNU9LdU5YZG8xS25Xc0ZsUTNyaXlZLzkKam9MenBIcUFIeW82RkRURkc2d3luVnE3eUJrSk45SkxpY05nZ0RoSDBzSkoyK1R1NWRUOEhNZ3JvT0pIeDI4SApiV1o5Z1pBaFBhNkd4TzRFYkRJUWE4anhMU0crVlhrOUFnVHRldTNEcjRSclVLY0FTdnkrcXRWWGVPSkUyL1lHCkNwRFVza1ZJemlxLzVuVjBJVklvSGo1YlFsRXlqaHY4dEExR2pyUmw0dkVDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUNVcgo4TDN0eEd3djB4dlhwVmdQWk9VMGRKSVFZcC9QMFdralIrV1p6eVU2Q1BpTjZXeitDaVUxK01uQVRienROVTlxCm1LNVhpQk5HQjJnSVh6QktsbVh6K01tMjN0RzdTRkI5dzFlVDFIQTFYajg1WUdPeitieCtyYy9vdlNIZDBkWG0KRlZhc1pUeTBiNnJMVGEvdHV4Y0NLd256bG9YekphTUxmVkhyUld5NnZSRDIwdDlBU1V3RTFjUytmQjVoU0dnMwpZYWNjZ0c5SHVCeGFqTWwrZFZJUGZGM0Zqb0xPbHZ5Sy8vakdkaUZJbldlaFhEWmZOeDNrcUp2N2g3NnZpQXBFClU0dmgwNFpCNUZVcTQ0eHJqLzJ5d0xRcVdZeXljdmhPYzU5ZnVVZXR5ODdrU3hrK2wvWUZXRWo5c3orQ1BwcmUKanBWREdYN1RvQ3lOUnE3c1NTVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBa1BjYXdHdW5SS0NJR2gzbGVZOUlTOUpQMEhDVUJWaDdJbU84NHl4OTk2VnRENmVxClJKam5kZjlBZmFGOVhIYWNteEVlTUVNV0JOYndDY2Vqbng2ajgvei9CNXRtbEZZdGtqbjZvcnNJcnhZazF1VUoKeEJwR0JqNmhleWxUWEE0TDFQWm9aRXJEcUZyUlprcC93eUVIbFBQVmxST0JDdFBLZDNlejlCQmJHRmhpcEQ0Nwp2RjRuQkVYVnhmYnJISlV3UlMxU0xveGsyTEt4WnVQTjFvQ1RiZU5HeFN3V0FsTHhhR2VXTGV5RTdJSUcxMGNaCjJYTFVHVXNRNmpqK0ZSR3h5dDBqRjNTS2h2SlhzSktCbWFCbnNHbnFFcGEwT2ZVNVgzaktpMkFVMDNxdkZudHYKOGhmT0t4dURRU1AvdmFwMEg1bUEzU0ZuUUZseVFKcnI0SmFQY3dJREFRQUJBb0lCQUVtL3ZMZFp6VDNub1hZUAo4WmQzSW1PWWVneGp5Wm95bW1vemJYdnhQYUNJWTBTTW10M05JQ3Z5a0orWlVZcUJ1VWhXWEJKWGZ0TTFlZWF2Cnp2ajdnSk4zQ1ZPWVpvWmVsVTZIK285aE1aakpYaktKVVArWUVqbkpFU1AwTVRscTEvQWU3NXJlUWszYTdoN3AKRmJoeWpFWG1DcW1OdzU5N2djeG5kSUJjOWNzYklBakhvdVRsV0tLMkVaRXFick5JQ2VIT0NIelRKMm1uMFpXMApTOWJhWDFuMG9YVWRaQzNITW5iWVRxUnBIUnl5bkpTNm16YlV3MUwzcXpCczZhbEx3NUYvUTNLTzJZdWhrd0sxCnlpY1J2T0JuZFk2R2l6NTR1SGZGaWFBMlZFaVdCdkZGS2c2aXVka0lBN2x6SmxoVUR5TDRQSlFTV0NCQUJmMmkKV3RFbDQ3a0NnWUVBd1NQbjBHOWx6bVY3ZWVxbkNuWldMOHh4djV0N0dRT0RmeXBRemJBd2tpWnBHamdsWDBqVwo0Y0FmL2JQTTdZZXpuTWZTNHJVRXdSWUQzcTVsbnU0cjR0VzRGelBwT0Fnb0tqd1BhNmE1Qm1PMERaZjg1QjJPCkFWSXIzYXpkNG1UVUZ0MW1aaytVbVUwejY3eUh5b253ZVNuUXR3cm1OZVdvdk5zWFVoQ1QvZmNDZ1lFQXdDVlkKYVFvSGg1WnNaK0w5Z2FnUFp6a1N0K0JkK2hkNzhEY1QyRWg2b2dzbVJWS2tZRm5Ob2dNSHpSVXg1R3hiSzJkQwp4SnhEcklLczZKbEUzTzF3TGM0SWQrNCswcjI2TWJqWG1lS1F2VUxCdjFrSjJ3R1pTQkNoL3o2L2R5c2VTVnVJCm1hZ3dPMWRuOVYyUlBMa2FHN1VoZDc4SFR6ZlFpSE5tdmhpVVMyVUNnWUJMbmo0MW1sZld2bDdndGVHZXRBUTgKaEtRSWE5eC9SNlBsVnZENkIwQ05DQ0xnVkJrMXZEUDlnUlpGWTI4WmxLYmEvRlZ4MDRQQUFWbVVodDFjbzAxbAphMTIvSCtEd3dpalBtR0pGT0twNmVQRFczYW55dWl1T0V0ai8ycDJvOU9jZzZLQWQ4SXh5bFFwK0FudDc3aFBMCnJIWitJL05NS05XdTAyekhBWnlmc3dLQmdDcElqWm9mNE9SUGorK1VuaURvTU91UVBiUkZKYWtoZmQ2dzRDY2EKWEJKSXpkVTNpSUREZ21NQ3M3YkgwVURwTDI3dnZGdXozNWlZOFNBR2lvYnJvV0lURXY0QmFjWEtocXptZGx1Zwo1aVVnQ2E3b0V5a2t3dXgrN0lkVThhRngxYXBqQUQ2QUtvMEVnUllYenh4RHg2RU5UUjRpNWFMakwzcTNtb1orCnRqT1pBb0dBSFdiZFgvTk80RUpTcmtVUzVLS2taZDFPcjhZWGRKS3BQVUl2TzNOS1h2QnlLdnBBdXFsT0RtZU0KMnhabFgyUHF3NmNsT2NjanpFRmZIV0VnYXFuQ0RTYXNqS2xzWXRNNG1rbkFLRFAxWVl5N3U2ZXpTVG9yRkx0TApuS1FDcDB3SGpmenpQdTdZYnU1THlia3NGQS9DT2hZSkZqOXk5SkNmdUI4VGtVV08wV3c9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== kind: Secret metadata: annotations: service.alpha.openshift.io/expiry: 2019-09-11T03:08:50Z service.alpha.openshift.io/originating-service-name: router service.alpha.openshift.io/originating-service-uid: 19cdcaff-969c-11e7-a349-42010af00013 creationTimestamp: 2017-09-11T03:08:50Z name: router-certs namespace: default resourceVersion: "2380" selfLink: /api/v1/namespaces/default/secrets/router-certs uid: 88ccb212-969e-11e7-a349-42010af00013 type: kubernetes.io/tls [root@qe-gpei-36-3-master-1 ~]# oc get dc router -o yaml .. containers: - env: - name: DEFAULT_CERTIFICATE_DIR value: /etc/pki/tls/private - name: DEFAULT_CERTIFICATE_PATH value: /etc/pki/tls/private/tls.crt .. volumeMounts: - mountPath: /etc/pki/tls/private name: server-certificate readOnly: true .. volumes: - name: server-certificate secret: defaultMode: 420 secretName: router-certs
Commit pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/62cb2a8d573928cb54a7d0ba475d61a6b65e0307 Merge pull request #5449 from abutcher/wildcard-router-cert-redeploy Automatic merge from submit-queue. Bug 1490186: Router pod not running after router certificates redeployment This carries https://github.com/openshift/openshift-ansible/pull/5417. More of the router cert redeploy logic could be moved into the `openshift_hosted` role with a flag. I may pull those over. https://bugzilla.redhat.com/show_bug.cgi?id=1490186
https://github.com/openshift/openshift-ansible/pull/5449
Verify this bug with openshift-ansible-3.7.0-0.143.2.git.0.39404c5.el7.noarch After run redeploy-router-certificates.yml playbook against an ocp-3.7 cluster, router pod was still running well, no error in router pod log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188
Is there a 3.6 fix around?
I have the same issue and support give the solution via this articule: https://access.redhat.com/solutions/2650171 In my case I was trying to replace the self-signed certificate with one of my own, but the root case is the same: tls.crt should include not only the cert + intermediate + ca .. but also the key. I have successfully solve my problem with: $ cat mycert.crt mycert.key > router.pem $ oc delete router-certs $ oc secrets new router-certs tls.crt=router.pem tls.key=mycert.key