Bug 1713267 - Upgrade from 3.11.69 to 3.11.98 fails due to /apis/metrics.k8s.io/v1beta1 not accessible
Summary: Upgrade from 3.11.69 to 3.11.98 fails due to /apis/metrics.k8s.io/v1beta1 not...
Keywords:
Status: CLOSED DUPLICATE of bug 1719362
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.11.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.11.z
Assignee: Joseph Callen
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-23 09:50 UTC by szustkowski
Modified: 2019-06-17 22:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-17 22:16:04 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description szustkowski 2019-05-23 09:50:06 UTC
Description of problem:

The ansible upgrader for control plane pods fails due to this API endpoint is not accessible. 

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7.noarch

rpm -q ansible
ansible-2.6.16-1.el7ae.noarch

ansible --version
ansible 2.6.16


How reproducible:

Steps to Reproduce:
1. Prepare upgrade from 3.11.69 to 3.11.98
2. Run the upgrade playbook for control plane
3. Wait until playbook fails

Actual results:
TASK [openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered] ********************************************************************************************************
Thursday 23 May 2019  11:21:31 +0200 (0:00:00.498)       0:06:21.136 **********
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (1 retries left).
fatal: [dfvvejaspcpm2.dzbank.vrnet]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/metrics.k8s.io/v1beta1"], "delta": "0:00:00.198934", "end": "2019-05-23 11:24:15.228616", "msg": "non-zero return code", "rc": 1, "start": "2019-05-23 11:24:15.029682", "stderr": "Error from server (ServiceUnavailable): the server is currently unable to handle the request", "stderr_lines": ["Error from server (ServiceUnavailable): the server is currently unable to handle the request"], "stdout": "", "stdout_lines": []}


Expected results:

The API endpoint should be available. oc get apiservices lists the metrics.k8s.io API as available with a creation time just a few minutes ago. 
However, the result of the actual command which is carried out by ansible is:

# oc --config=/etc/origin/master/admin.kubeconfig get --raw /apis/metrics.k8s.io/v1beta1 --loglevel=8
I0523 11:40:53.265798  106651 loader.go:359] Config loaded from file /etc/origin/master/admin.kubeconfig
I0523 11:40:53.267029  106651 round_trippers.go:383] GET  https://dzecp-cluster.dzbank.vrnet:8443/apis/metrics.k8s.io/v1beta1
I0523 11:40:53.267052  106651 round_trippers.go:390] Request Headers:
I0523 11:40:53.267060  106651 round_trippers.go:393]     Accept: application/json, */*
I0523 11:40:53.267068  106651 round_trippers.go:393]     User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0
I0523 11:40:53.294517  106651 round_trippers.go:408] Response Status: 503 Service Unavailable in 27 milliseconds
I0523 11:40:53.294549  106651 round_trippers.go:411] Response Headers:
I0523 11:40:53.294559  106651 round_trippers.go:414]     Cache-Control: no-store
I0523 11:40:53.294567  106651 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
I0523 11:40:53.294578  106651 round_trippers.go:414]     X-Content-Type-Options: nosniff
I0523 11:40:53.294586  106651 round_trippers.go:414]     Content-Length: 20
I0523 11:40:53.294593  106651 round_trippers.go:414]     Date: Thu, 23 May 2019 09:40:53 GMT
I0523 11:40:53.294601  106651 round_trippers.go:414]     Audit-Id: 0bba89bd-7e50-4778-bee4-6eea94712e04
I0523 11:40:53.294636  106651 request.go:897] Response Body: service unavailable
I0523 11:40:53.294705  106651 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request",
  "reason": "ServiceUnavailable",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "service unavailable"
      }
    ]
  },
  "code": 503
}]
F0523 11:40:53.294787  106651 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request

Comment 1 Joseph Callen 2019-06-17 22:16:04 UTC
Marking as duplicate.  Please update https://bugzilla.redhat.com/show_bug.cgi?id=1719362 instead.

*** This bug has been marked as a duplicate of bug 1719362 ***


Note You need to log in before you can comment on or make changes to this bug.