Description of problem: The ansible upgrader for control plane pods fails due to this API endpoint is not accessible. Version-Release number of the following components: rpm -q openshift-ansible openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7.noarch rpm -q ansible ansible-2.6.16-1.el7ae.noarch ansible --version ansible 2.6.16 How reproducible: Steps to Reproduce: 1. Prepare upgrade from 3.11.69 to 3.11.98 2. Run the upgrade playbook for control plane 3. Wait until playbook fails Actual results: TASK [openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered] ******************************************************************************************************** Thursday 23 May 2019 11:21:31 +0200 (0:00:00.498) 0:06:21.136 ********** FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (30 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (29 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (28 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (27 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (26 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (25 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (24 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (23 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (22 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (21 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (20 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (19 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (18 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (17 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (16 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (15 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (14 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (13 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (12 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (11 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (10 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (9 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (8 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (7 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (6 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (5 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (4 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (3 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (2 retries left). FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (1 retries left). fatal: [dfvvejaspcpm2.dzbank.vrnet]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/metrics.k8s.io/v1beta1"], "delta": "0:00:00.198934", "end": "2019-05-23 11:24:15.228616", "msg": "non-zero return code", "rc": 1, "start": "2019-05-23 11:24:15.029682", "stderr": "Error from server (ServiceUnavailable): the server is currently unable to handle the request", "stderr_lines": ["Error from server (ServiceUnavailable): the server is currently unable to handle the request"], "stdout": "", "stdout_lines": []} Expected results: The API endpoint should be available. oc get apiservices lists the metrics.k8s.io API as available with a creation time just a few minutes ago. However, the result of the actual command which is carried out by ansible is: # oc --config=/etc/origin/master/admin.kubeconfig get --raw /apis/metrics.k8s.io/v1beta1 --loglevel=8 I0523 11:40:53.265798 106651 loader.go:359] Config loaded from file /etc/origin/master/admin.kubeconfig I0523 11:40:53.267029 106651 round_trippers.go:383] GET https://dzecp-cluster.dzbank.vrnet:8443/apis/metrics.k8s.io/v1beta1 I0523 11:40:53.267052 106651 round_trippers.go:390] Request Headers: I0523 11:40:53.267060 106651 round_trippers.go:393] Accept: application/json, */* I0523 11:40:53.267068 106651 round_trippers.go:393] User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0 I0523 11:40:53.294517 106651 round_trippers.go:408] Response Status: 503 Service Unavailable in 27 milliseconds I0523 11:40:53.294549 106651 round_trippers.go:411] Response Headers: I0523 11:40:53.294559 106651 round_trippers.go:414] Cache-Control: no-store I0523 11:40:53.294567 106651 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I0523 11:40:53.294578 106651 round_trippers.go:414] X-Content-Type-Options: nosniff I0523 11:40:53.294586 106651 round_trippers.go:414] Content-Length: 20 I0523 11:40:53.294593 106651 round_trippers.go:414] Date: Thu, 23 May 2019 09:40:53 GMT I0523 11:40:53.294601 106651 round_trippers.go:414] Audit-Id: 0bba89bd-7e50-4778-bee4-6eea94712e04 I0523 11:40:53.294636 106651 request.go:897] Response Body: service unavailable I0523 11:40:53.294705 106651 helpers.go:201] server response object: [{ "metadata": {}, "status": "Failure", "message": "the server is currently unable to handle the request", "reason": "ServiceUnavailable", "details": { "causes": [ { "reason": "UnexpectedServerResponse", "message": "service unavailable" } ] }, "code": 503 }] F0523 11:40:53.294787 106651 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request
Marking as duplicate. Please update https://bugzilla.redhat.com/show_bug.cgi?id=1719362 instead. *** This bug has been marked as a duplicate of bug 1719362 ***