Bug 1620367 - master controllers static pod restart failed for apiservice "v1beta1.servicecatalog.k8s.io" can not handle the request.
Summary: master controllers static pod restart failed for apiservice "v1beta1.servicec...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.0
Assignee: David Eads
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-23 05:27 UTC by Qin Ping
Modified: 2018-10-11 07:25 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-11 07:25:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:25:43 UTC

Description Qin Ping 2018-08-23 05:27:00 UTC
Description of problem:
master controllers static pod restart failed for apiservice "v1beta1.servicecatalog.k8s.io" can not handle the request.

Version-Release number of selected component (if applicable):
oc v3.11.0-0.20.0
openshift v3.11.0-0.20.0
kubernetes v1.11.0+d4cacc0


How reproducible:
Sometimes

Steps to Reproduce:
1. Modify master-config.yaml file
2. Restart apiserver
3. Restart controllers

Actual results:
master controllers static pod ran into "CrashLoopBackOff" status.

Expected results:
master controllers static pad can be run successfully.

Additional info:
controllers static pod logs:
I0822 09:25:16.822146       1 request.go:1099] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0822 09:25:16.822751       1 shared_informer.go:123] caches populated
F0822 09:25:16.825636       1 controller_manager.go:127] Error starting "openshift.io/cluster-quota-reconciliation" (unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request)

Pod status:
# oc get pods --all-namespaces
NAMESPACE                           NAME                                        READY     STATUS             RESTARTS   AGE
default                             docker-registry-2-4bqc5                     1/1       Running            0          7h
default                             registry-console-1-n9m5q                    1/1       Running            1          7h
default                             router-1-rtf6h                              1/1       Running            0          7h
install-test                        mongodb-1-jvsdt                             1/1       Running            0          7h
install-test                        nodejs-mongodb-example-1-build              0/1       Completed          0          7h
install-test                        nodejs-mongodb-example-1-s2dnw              1/1       Running            0          7h
kube-service-catalog                apiserver-9qnpn                             1/1       Running            0          7h
kube-service-catalog                controller-manager-m7t7s                    0/1       CrashLoopBackOff   13         7h
kube-system                         master-api-qe-piqin-master-etcd-1           1/1       Running            2          7h
kube-system                         master-controllers-qe-piqin-master-etcd-1   0/1       CrashLoopBackOff   47         6m
kube-system                         master-etcd-qe-piqin-master-etcd-1          1/1       Running            2          7h
kube-system                         prometheus-0                                6/6       Running            0          7h
openshift-ansible-service-broker    asb-1-s9xbq                                 1/1       Running            19         7h
openshift-console                   console-788c8fc9dd-44sxg                    1/1       Running            1          7h
openshift-node                      sync-29jwx                                  1/1       Running            1          7h
openshift-node                      sync-plwcl                                  1/1       Running            0          7h
openshift-node                      sync-zhh2c                                  1/1       Running            0          7h
openshift-sdn                       ovs-fg2x7                                   1/1       Running            0          7h
openshift-sdn                       ovs-krrwj                                   1/1       Running            1          7h
openshift-sdn                       ovs-pp6b4                                   1/1       Running            0          7h
openshift-sdn                       sdn-4lh45                                   1/1       Running            0          7h
openshift-sdn                       sdn-5n7k2                                   1/1       Running            0          7h
openshift-sdn                       sdn-t5d4j                                   1/1       Running            1          7h
openshift-template-service-broker   apiserver-bk4p6                             1/1       Running            0          7h
openshift-web-console               webconsole-6b8bdf69cf-8ggd4                 1/1       Running            1          7h

Comment 1 Qin Ping 2018-08-23 05:28:59 UTC
Workaround for this:

1) Delete apiservice "v1beta1.servicecatalog.k8s.io"
2) Restart master controllers.

Comment 2 Qin Ping 2018-08-23 06:35:33 UTC
Sorry, OCP version is: v3.11.0-0.19.0.0

Comment 3 Michal Fojtik 2018-08-23 07:54:59 UTC
This should be fixed here: https://github.com/openshift/origin/pull/20693

Comment 4 Qin Ping 2018-08-28 06:52:55 UTC
Verified in OCP:
oc v3.11.0-0.24.0
openshift v3.11.0-0.24.0
kubernetes v1.11.0+d4cacc0


# uname -a
Linux qe-piqin-master-etcd-1 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.5

Comment 6 errata-xmlrpc 2018-10-11 07:25:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.