Description of problem: Failed to install service catalog with oc cluster up Version-Release number of selected component (if applicable): oc v3.6.131 kubernetes v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1.install service catalog with oc cluster up $ oc cluster up --image=brew-pulp.../openshift3/ose --version=v3.6 --service-catalog 2. 3. Actual results: I0704 13:58:20.830677 26716 servicecatalog.go:178] retrying registration after error the server could not find the requested resource (post brokers.servicecatalog.k8s.io) I0704 13:58:20.830702 26716 request.go:991] Request Body: {"apiVersion":"servicecatalog.k8s.io/v1alpha1","kind":"Broker","metadata":{"name":"template-broker"},"spec":{"url":"https://kubernetes.default.svc:443/brokers/template.openshift.io"}} I0704 13:58:20.830726 26716 round_trippers.go:383] POST https://127.0.0.1:8443/apis/servicecatalog.k8s.io/v1alpha1/brokers I0704 13:58:20.830734 26716 round_trippers.go:390] Request Headers: I0704 13:58:20.830740 26716 round_trippers.go:393] Content-Type: application/json I0704 13:58:20.830746 26716 round_trippers.go:393] Accept: application/json I0704 13:58:20.830751 26716 round_trippers.go:393] User-Agent: oc/v1.6.1+5115d708d7 (linux/amd64) kubernetes/314edd5 I0704 13:58:20.831521 26716 round_trippers.go:408] Response Status: 404 Not Found in 0 milliseconds I0704 13:58:20.831533 26716 round_trippers.go:411] Response Headers: I0704 13:58:20.831539 26716 round_trippers.go:414] Date: Tue, 04 Jul 2017 05:58:20 GMT I0704 13:58:20.831544 26716 round_trippers.go:414] Cache-Control: no-store I0704 13:58:20.831550 26716 round_trippers.go:414] Content-Type: application/json I0704 13:58:20.831555 26716 round_trippers.go:414] Content-Length: 109 I0704 13:58:20.831571 26716 request.go:991] Response Body: { "paths": [ "/apis", "/brokers/template.openshift.io/v2", "/controllers", "/version" ] } I0704 13:58:20.831613 26716 request.go:1190] body was not decodable (unable to check for Status): Object 'Kind' is missing in '{ "paths": [ "/apis", "/brokers/template.openshift.io/v2", "/controllers", "/version" ] }' I0704 13:58:20.831623 26716 servicecatalog.go:178] retrying registration after error the server could not find the requested resource (post brokers.servicecatalog.k8s.io) FAIL Error: failed to register broker with service catalog: timed out waiting for the condition Expected results: Display successful info without error Additional info: But openshift cluster is launched, when I log into webconsole, I cannot see installed templates in web catalog, only can see images
I'm not sure if this bug related to https://github.com/openshift/origin/pull/14896 that fix for bz1463630
Not sure if it even matters yet, but the alpha apis are enabled: kubernetesMasterConfig: admissionConfig: pluginConfig: null apiLevels: null apiServerArguments: runtime-config: - apis/settings.k8s.io/v1alpha1=true Registered the SC api aggregation here: I0704 12:48:14.194239 18853 servicecatalog.go:110] setting up the api aggregator I0704 12:48:14.194996 18853 request.go:991] Request Body: {"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1beta1","metadata":{"name":"v1alpha1.servicecatalog.k8s.io","creationTimestamp":null},"spec":{"service":{"namespace":"service-catalog","name":"apiserver"},"group":"servicecatalog.k8s.io","version":"v1alpha1","insecureSkipTLSVerify":true,"caBundle":"","groupPriorityMinimum":200,"versionPriority":20},"status":{}} I0704 12:48:14.202321 18853 request.go:991] Response Body: {"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1beta1","metadata":{"name":"v1alpha1.servicecatalog.k8s.io","selfLink":"/apis/apiregistration.k8s.io/v1beta1/apiservices/v1alpha1.servicecatalog.k8s.io","uid":"924f571b-60d8-11e7-90c1-507b9d27b5d9","resourceVersion":"1288","creationTimestamp":"2017-07-04T16:48:14Z"},"spec":{"service":{"namespace":"service-catalog","name":"apiserver"},"group":"servicecatalog.k8s.io","version":"v1alpha1","insecureSkipTLSVerify":true,"caBundle":"","groupPriorityMinimum":200,"versionPriority":20},"status":{}} The apiserver endpoint is responsive: $ oc project Using project "service-catalog" on server "https://127.0.0.1:8443". $ oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE apiserver 172.30.1.2 <none> 80/TCP,443/TCP 8m $ curl 172.30.1.2 { "paths": [ "/apis", "/apis/servicecatalog.k8s.io", "/apis/servicecatalog.k8s.io/v1alpha1", "/healthz", "/healthz/ping", "/healthz/poststarthook/start-service-catalog-apiserver-informers", "/metrics", "/swaggerapi/", "/version" ] } but POSTs to the service catalog brokers endpoint are rejected: I0704 12:48:15.202908 18853 round_trippers.go:383] POST https://127.0.0.1:8443/apis/servicecatalog.k8s.io/v1alpha1/brokers I0704 12:48:15.204314 18853 request.go:991] Response Body: { "paths": [ "/apis", "/brokers/template.openshift.io/v2", "/controllers", "/version" ] } and the servicecatalog group doesn't show up in the api resource list: $ curl -k https://127.0.0.1:8443 { "paths": [ "/api", "/api/v1", "/apis", "/apis/apiregistration.k8s.io", "/apis/apiregistration.k8s.io/v1beta1", "/apis/apps", "/apis/apps.openshift.io", "/apis/apps.openshift.io/v1", "/apis/apps/v1beta1", "/apis/authentication.k8s.io", "/apis/authentication.k8s.io/v1", "/apis/authentication.k8s.io/v1beta1", "/apis/authorization.k8s.io", "/apis/authorization.k8s.io/v1", "/apis/authorization.k8s.io/v1beta1", "/apis/authorization.openshift.io", "/apis/authorization.openshift.io/v1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/autoscaling/v2alpha1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/build.openshift.io", "/apis/build.openshift.io/v1", "/apis/certificates.k8s.io", "/apis/certificates.k8s.io/v1beta1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/image.openshift.io", "/apis/image.openshift.io/v1", "/apis/network.openshift.io", "/apis/network.openshift.io/v1", "/apis/oauth.openshift.io", "/apis/oauth.openshift.io/v1", "/apis/policy", "/apis/policy/v1beta1", "/apis/project.openshift.io", "/apis/project.openshift.io/v1", "/apis/quota.openshift.io", "/apis/quota.openshift.io/v1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1beta1", "/apis/route.openshift.io", "/apis/route.openshift.io/v1", "/apis/security.openshift.io", "/apis/security.openshift.io/v1", "/apis/settings.k8s.io", "/apis/settings.k8s.io/v1alpha1", "/apis/storage.k8s.io", "/apis/storage.k8s.io/v1", "/apis/storage.k8s.io/v1beta1", "/apis/template.openshift.io", "/apis/template.openshift.io/v1", "/apis/user.openshift.io", "/apis/user.openshift.io/v1", "/brokers/template.openshift.io/v2", "/controllers", "/healthz", "/healthz/autoregister-completion", "/healthz/ping", "/healthz/poststarthook/apiservice-registration-controller", "/healthz/poststarthook/apiservice-status-available-controller", "/healthz/poststarthook/bootstrap-controller", "/healthz/poststarthook/ca-registration", "/healthz/poststarthook/kube-apiserver-autoregistration", "/healthz/poststarthook/start-kube-aggregator-informers", "/healthz/ready", "/metrics", "/oapi", "/oapi/v1", "/swaggerapi", "/version", "/version/openshift" ] } and the aggregator seems to think it can't reach the SC apis: $ oc get apiservices v1alpha1.servicecatalog.k8s.io -o yaml apiVersion: apiregistration.k8s.io/v1beta1 kind: APIService metadata: creationTimestamp: 2017-07-04T16:48:14Z name: v1alpha1.servicecatalog.k8s.io resourceVersion: "1322" selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1alpha1.servicecatalog.k8s.io uid: 924f571b-60d8-11e7-90c1-507b9d27b5d9 spec: caBundle: "" group: servicecatalog.k8s.io groupPriorityMinimum: 200 insecureSkipTLSVerify: true service: name: apiserver namespace: service-catalog version: v1alpha1 versionPriority: 20 status: conditions: - lastTransitionTime: 2017-07-04T16:48:14Z message: 'no response from https://172.30.1.2/apis: context deadline exceeded' reason: FailedDiscoveryCheck status: "False" type: Available but i can: $ curl -k https://172.30.1.2/apis { "kind": "APIGroupList", "groups": [ { "name": "servicecatalog.k8s.io", "versions": [ { "groupVersion": "servicecatalog.k8s.io/v1alpha1", "version": "v1alpha1" } ], "preferredVersion": { "groupVersion": "servicecatalog.k8s.io/v1alpha1", "version": "v1alpha1" }, "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "" } ] } ] } David, thoughts/other things to look at?
Is there a window here where i might be registering the api w/ the aggregator "too soon" (before it's actually available) and so it fails discovery and never gets re-checked? The logic currently waits for the sc api server deployment to have an available endpoint before registering, but nothing guarantees that endpoint is actually serving successfully yet (i.e. no readiness probe). (That said, it should be pretty soon after, if not already available by the time the registration call is made).
Looks similar to what https://github.com/openshift/origin/pull/15022 fixed.
appears to be working again w/ https://github.com/openshift/origin/pull/15022 merged. Thanks David.
oc v3.6.135 kubernetes v1.6.1+5115d708d7