Created attachment 1364385 [details] ansible-playbook-vvvv Description of problem: Cluster upgrade from v3.6 to v3.7 worked correctly, but any attempt to install the service catalog fails: FAILED - RETRYING: wait for api server to be ready (1 retries left). fatal: [master1.36c.alxrh.ose]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz"], "delta": "0:00:01.012446", "end": "2017-12-07 15:57:06.057559", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2017-12-07 15:57:05.045113", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused"], "stdout": "", "stdout_lines": []} The log of the apiserver pod shows that the requestheader-client-ca-file doesn't exist: 1207 14:56:01.841961 1 round_trippers.go:445] Cache-Control: no-store I1207 14:56:01.841964 1 round_trippers.go:445] Content-Type: application/json I1207 14:56:01.841966 1 round_trippers.go:445] Content-Length: 1442 I1207 14:56:01.841969 1 round_trippers.go:445] Date: Thu, 07 Dec 2017 14:56:01 GMT I1207 14:56:01.841986 1 request.go:836] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"1c5f97f6-db41-11e7-a05b-52540032d975","resourceVersion":"1357","creationTimestamp":"2017-12-07T11:23:55Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTI2NDU3MDEwHhcNMTcxMjA3MTEyMTQwWhcNMjIxMjA2\nMTEyMTQxWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTI2NDU3MDEw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXOtymj+4fYtC8mnFkzQVW\nvdUiL43OQXqtmoSHGPnm+OrRlJe0OfRHRZc75RfA/t/Y80D7Kzxrd5fViqZ3lpE1\ndzj1DldvgTeonMui30DTn6JaFeKd9VrCnnvQ94GbumDkAInGLQas0Mlz9hmHeVA4\nawgZQH2KU4JaI+tsb/FLgqJB8F7To7HX6+vchNzbrS8rziCUabdt32Z+joCKHA10\nwnN+oRQJz4culn9amVvSRSnWSeX+V613Np2OJSbeGzXch1n/Hri3rbcynhRGlP7T\nJdew+4Tr6bEbuQw3EwG1mdi3uJOfUFqMaryMEHbocPTN5E7qYxeynSC9ScFVAH3T\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQC2thVnKBlRWCi2GPLg9/j5h13DbrzWABT9nLVR0HrHUbbx\nsQpONO0dM64wx5EhcHNbtiwqoJgc99M1m1snC+U+joHpFGImvUdYVxn7ErkVGMq9\nypbUob6NSJ8S1x7Nu+TK6gmu838dCrl2fIc2rH71/EY410LnbxUxYnmRfaaCAUCr\nSet8ld+cHS+UjU9w9vPbexmIGaqrNtjZHyrurvO+oNXFFofcX0gNwDJKiBxUEPmk\nnViYstJI1xDpwOI6mQVCdT7/pgQTv0YJW+vTtCjHigOeEc24kIoZuIUI9A1C/Opj\nzzB531kk7YbIFSvFFyA0zzmRK+YSQuGl2Vlvkgr/\n-----END CERTIFICATE-----\n"}} Error: cluster doesn't provide requestheader-client-ca-file Version-Release number of the following components: openshift-ansible-3.7.9-1.git.7.eedd332.el7.noarch ansible-2.4.1.0-1.el7.noarch ansible 2.4.1.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] How reproducible: Always Steps to Reproduce: 1. Upgrade the cluster to v3.7 2. Launch the playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.yml to install the service catalog Actual results: TASK [openshift_service_catalog : wait for api server to be ready] **************************************************************************************************** FAILED - RETRYING: wait for api server to be ready (120 retries left). FAILED - RETRYING: wait for api server to be ready (119 retries left). FAILED - RETRYING: wait for api server to be ready (118 retries left). [...] FAILED - RETRYING: wait for api server to be ready (1 retries left). fatal: [master1.36c.alxrh.ose]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz"], "delta": "0:00:01.012446", "end": "2017-12-07 15:57:06.05 7559", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2017-12-07 15:57:05.045113", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:- - 0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:0 1 --:--:-- 0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused"], "stdout": "", "stdout_lines": []} to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.retry Expected results: Service catalog is installed without errors Additional info: Comparing the status against a freshly installed 3.7 cluster, I see that the following files are missing under /etc/origin/master: client-ca-bundle.crt front-proxy-ca.crt frontproxy-ca.crt front-proxy-ca.key frontproxy-ca.key frontproxy-ca.serial.txt openshift-aggregator.crt openshift-aggregator.key Also, the extension-apiserver-authentication ConfigMap in the kube-system namespace doesn't include the certificate for requestheader-client-ca-file: In the upgraded cluster it is: apiVersion: v1 data: client-ca-file: | -----BEGIN CERTIFICATE----- MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1MTI2NDU3MDEwHhcNMTcxMjA3MTEyMTQwWhcNMjIxMjA2 MTEyMTQxWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTI2NDU3MDEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXOtymj+4fYtC8mnFkzQVW vdUiL43OQXqtmoSHGPnm+OrRlJe0OfRHRZc75RfA/t/Y80D7Kzxrd5fViqZ3lpE1 dzj1DldvgTeonMui30DTn6JaFeKd9VrCnnvQ94GbumDkAInGLQas0Mlz9hmHeVA4 awgZQH2KU4JaI+tsb/FLgqJB8F7To7HX6+vchNzbrS8rziCUabdt32Z+joCKHA10 wnN+oRQJz4culn9amVvSRSnWSeX+V613Np2OJSbeGzXch1n/Hri3rbcynhRGlP7T Jdew+4Tr6bEbuQw3EwG1mdi3uJOfUFqMaryMEHbocPTN5E7qYxeynSC9ScFVAH3T AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG SIb3DQEBCwUAA4IBAQC2thVnKBlRWCi2GPLg9/j5h13DbrzWABT9nLVR0HrHUbbx sQpONO0dM64wx5EhcHNbtiwqoJgc99M1m1snC+U+joHpFGImvUdYVxn7ErkVGMq9 ypbUob6NSJ8S1x7Nu+TK6gmu838dCrl2fIc2rH71/EY410LnbxUxYnmRfaaCAUCr Set8ld+cHS+UjU9w9vPbexmIGaqrNtjZHyrurvO+oNXFFofcX0gNwDJKiBxUEPmk nViYstJI1xDpwOI6mQVCdT7/pgQTv0YJW+vTtCjHigOeEc24kIoZuIUI9A1C/Opj zzB531kk7YbIFSvFFyA0zzmRK+YSQuGl2Vlvkgr/ -----END CERTIFICATE----- kind: ConfigMap metadata: creationTimestamp: 2017-12-07T11:23:55Z name: extension-apiserver-authentication namespace: kube-system resourceVersion: "1357" selfLink: /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication uid: 1c5f97f6-db41-11e7-a05b-52540032d975 In the fresh installation: apiVersion: v1 data: client-ca-file: | -----BEGIN CERTIFICATE----- MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1MTI2NDYyOTIwHhcNMTcxMjA3MTEzMTMxWhcNMjIxMjA2 MTEzMTMyWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTI2NDYyOTIw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4LUZQjMk6Fr4EpzWGV/n9 iTv+H2cJ1VPN6o+VML4BgQgGzmoGp69ZyAlsZ1vQMU6+omQqUhi+cdluOdCOkOO0 GtsB+o05i+Dh91OzujhYhNoPaFsVI92l9yOx8oKLAQL4P6Yjk7fcy8cu2oq5zycd 9qe/rmGxISLr4LtNmUTUuJpjgGjT/++9RjmVuHkKQE0ZqJdprOGzBAd/oekrA1Nl hOU4BlRjZczGQR9JjT/LXjEaGP8J93osK3aG6sMq66oOVqtss+/lv6MZ4kAg+xX6 QsPH3fY2y3nFBcawfaiAwqiyTRDf3d3pQwLirizVJdiZD+a4awlhuRcld4Tk5+Dl AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG SIb3DQEBCwUAA4IBAQC2HGClXu7euONLb4WhlW8xcdkkhhyQ7U6V57JxDmnpRBxU 5rL5wzEsPaAs2drlhBMqbsa6/lI3GyRnALW0DMQFX1eKGQwC2mbM6kaHIaD0Jk0n 142oIK1FwTklReemckXyraliRJwflyZTnjEjD5b9EnrB3UrGm6EQN1FbmRH+MzQL 2AtOszQ1RoI0D//V2KYn2QYOEudpPtdvRfU8pquwER6uJI/b14fB9XXIVhWoCPWE ebizo63HN+seg8yBGjbT8YOCO/g41LWuSJkLzWppwkynBtCvVs8TZGRn/aSCqmys U1abn5TemLXZWLWjomgHwhQZ8/s0SkhwuR4fyvb0 -----END CERTIFICATE----- requestheader-allowed-names: '["aggregator-front-proxy"]' requestheader-client-ca-file: | -----BEGIN CERTIFICATE----- MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1MTI2NDY1MzcwHhcNMTcxMjA3MTEzNTM2WhcNMjIxMjA2 MTEzNTM3WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTI2NDY1Mzcw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDDgpz8OgWP61SDsuk7O2o9 4g27jREPiS8y1OFCKemodKnkNEZ4jQ97ZYkvIrQgA8nBWXY3w2cc/5Ue25TKY81q ndLwyY3m0oo0irPTLXCwUly5G62jEmjvFHg5cB+Xggdd/i4y/kYDtzLwdv8JA+Cv 1C5ARwYpFjrwMhX1qoCO3B7I4bDcawo1BF9xMj0I9jV0GgcOKsDObnW+Hse8ncxt 0jflQIWeYPDFQRpU+MDIqcBgSyGLhR6zwHtuet0Czk0m7g6FDN2sTF7RtFigmDWk CyFYDpmnTZbdVQZm65YGIjvXwtQgO8H0AscOc9yXUySlCSTFNC1oxw9ANX36fsTD AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG SIb3DQEBCwUAA4IBAQC6DnMNcOBQSRhqyAcsZCumOQdaIPb0F0V6k1jYKgWdEPBn noE6zgArWYEGFsGiSyOSm7R+Pr597jNkMqsoRW5kIBr7zgebvESafEEW+7naZ7IP VLoEvnQF9YOnEQYUum7gf01K3HIq2zo8xmx9meUN95nnxH60sLszvkFYlS89r93H +2t1lcdM72FJjA1unmPKIKUtQMVjm8z28ZjpucdNuQP8y/IvMPmAKiIpx3hFarVB /F8xmqe33WHaToqLgFrnbu/p4tItTAAZJpJKJrSiDoRXtTsENns0HcQ1k0VQcEVs ecdSulg8pa7oCSz2eTE98ElatseN1HONcTGTXvyv -----END CERTIFICATE----- requestheader-extra-headers-prefix: '["X-Remote-Extra-"]' requestheader-group-headers: '["X-Remote-Group"]' requestheader-username-headers: '["X-Remote-User"]' kind: ConfigMap metadata: creationTimestamp: 2017-12-07T11:34:56Z name: extension-apiserver-authentication namespace: kube-system resourceVersion: "11037" selfLink: /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication uid: a64c8fa3-db42-11e7-b6ae-525400fa9407 ansible-playbook output attached
Did you make any progress here? I am also trying to install the ServiceCatalog after Upgrading to v 3.7 . My Installation fails with the same errors that you get.
Posted upstream for review: https://github.com/openshift/openshift-ansible/pull/6736
Looking at c24, I wasn't able to see exactly what the problem was. It looks like the etcd PV bound properly, were the pods in the openshift-ansible-service-broker namespace in an error state? This also should probably live in its own BZ, or we should move discussion to https://bugzilla.redhat.com/show_bug.cgi?id=1520291 if it's a dupe.
*** Bug 1522805 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0636