Bug 1467519 - Failed to install service catalog with oc cluster up
Failed to install service catalog with oc cluster up
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Command Line Interface (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Ben Parees
Dongbo Yan
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-04 02:24 EDT by Dongbo Yan
Modified: 2017-08-16 15 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-01 11:23:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dongbo Yan 2017-07-04 02:24:29 EDT
Description of problem:
Failed to install service catalog with oc cluster up

Version-Release number of selected component (if applicable):
oc v3.6.131
kubernetes v1.6.1+5115d708d7

How reproducible:
Always

Steps to Reproduce:
1.install service catalog with oc cluster up
 $ oc cluster up --image=brew-pulp.../openshift3/ose --version=v3.6 --service-catalog
2.
3.

Actual results:
I0704 13:58:20.830677   26716 servicecatalog.go:178] retrying registration after error the server could not find the requested resource (post brokers.servicecatalog.k8s.io)
I0704 13:58:20.830702   26716 request.go:991] Request Body: {"apiVersion":"servicecatalog.k8s.io/v1alpha1","kind":"Broker","metadata":{"name":"template-broker"},"spec":{"url":"https://kubernetes.default.svc:443/brokers/template.openshift.io"}}
I0704 13:58:20.830726   26716 round_trippers.go:383] POST https://127.0.0.1:8443/apis/servicecatalog.k8s.io/v1alpha1/brokers
I0704 13:58:20.830734   26716 round_trippers.go:390] Request Headers:
I0704 13:58:20.830740   26716 round_trippers.go:393]     Content-Type: application/json
I0704 13:58:20.830746   26716 round_trippers.go:393]     Accept: application/json
I0704 13:58:20.830751   26716 round_trippers.go:393]     User-Agent: oc/v1.6.1+5115d708d7 (linux/amd64) kubernetes/314edd5
I0704 13:58:20.831521   26716 round_trippers.go:408] Response Status: 404 Not Found in 0 milliseconds
I0704 13:58:20.831533   26716 round_trippers.go:411] Response Headers:
I0704 13:58:20.831539   26716 round_trippers.go:414]     Date: Tue, 04 Jul 2017 05:58:20 GMT
I0704 13:58:20.831544   26716 round_trippers.go:414]     Cache-Control: no-store
I0704 13:58:20.831550   26716 round_trippers.go:414]     Content-Type: application/json
I0704 13:58:20.831555   26716 round_trippers.go:414]     Content-Length: 109
I0704 13:58:20.831571   26716 request.go:991] Response Body: {
  "paths": [
    "/apis",
    "/brokers/template.openshift.io/v2",
    "/controllers",
    "/version"
  ]
}
I0704 13:58:20.831613   26716 request.go:1190] body was not decodable (unable to check for Status): Object 'Kind' is missing in '{
  "paths": [
    "/apis",
    "/brokers/template.openshift.io/v2",
    "/controllers",
    "/version"
  ]
}'
I0704 13:58:20.831623   26716 servicecatalog.go:178] retrying registration after error the server could not find the requested resource (post brokers.servicecatalog.k8s.io)
FAIL
   Error: failed to register broker with service catalog: timed out waiting for the condition


Expected results:
Display successful info without error

Additional info:
But openshift cluster is launched, when I log into webconsole, I cannot see installed templates in web catalog, only can see images
Comment 1 Dongbo Yan 2017-07-04 02:28:15 EDT
I'm not sure if this bug related to https://github.com/openshift/origin/pull/14896 that fix for bz1463630
Comment 2 Ben Parees 2017-07-04 13:06:28 EDT
Not sure if it even matters yet, but the alpha apis are enabled:

kubernetesMasterConfig:
  admissionConfig:
    pluginConfig: null
  apiLevels: null
  apiServerArguments:
    runtime-config:
    - apis/settings.k8s.io/v1alpha1=true


Registered the SC api aggregation here:
I0704 12:48:14.194239   18853 servicecatalog.go:110] setting up the api aggregator
I0704 12:48:14.194996   18853 request.go:991] Request Body: {"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1beta1","metadata":{"name":"v1alpha1.servicecatalog.k8s.io","creationTimestamp":null},"spec":{"service":{"namespace":"service-catalog","name":"apiserver"},"group":"servicecatalog.k8s.io","version":"v1alpha1","insecureSkipTLSVerify":true,"caBundle":"","groupPriorityMinimum":200,"versionPriority":20},"status":{}}
I0704 12:48:14.202321   18853 request.go:991] Response Body: {"kind":"APIService","apiVersion":"apiregistration.k8s.io/v1beta1","metadata":{"name":"v1alpha1.servicecatalog.k8s.io","selfLink":"/apis/apiregistration.k8s.io/v1beta1/apiservices/v1alpha1.servicecatalog.k8s.io","uid":"924f571b-60d8-11e7-90c1-507b9d27b5d9","resourceVersion":"1288","creationTimestamp":"2017-07-04T16:48:14Z"},"spec":{"service":{"namespace":"service-catalog","name":"apiserver"},"group":"servicecatalog.k8s.io","version":"v1alpha1","insecureSkipTLSVerify":true,"caBundle":"","groupPriorityMinimum":200,"versionPriority":20},"status":{}}



The apiserver endpoint is responsive:

$ oc project
Using project "service-catalog" on server "https://127.0.0.1:8443".


$ oc get svc
NAME                 CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
apiserver            172.30.1.2       <none>        80/TCP,443/TCP   8m

$ curl 172.30.1.2
{
  "paths": [
    "/apis",
    "/apis/servicecatalog.k8s.io",
    "/apis/servicecatalog.k8s.io/v1alpha1",
    "/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/start-service-catalog-apiserver-informers",
    "/metrics",
    "/swaggerapi/",
    "/version"
  ]
}


but POSTs to the service catalog brokers endpoint are rejected:

I0704 12:48:15.202908   18853 round_trippers.go:383] POST https://127.0.0.1:8443/apis/servicecatalog.k8s.io/v1alpha1/brokers
I0704 12:48:15.204314   18853 request.go:991] Response Body: {
  "paths": [
    "/apis",
    "/brokers/template.openshift.io/v2",
    "/controllers",
    "/version"
  ]
}


and the servicecatalog group doesn't show up in the api resource list:

$ curl -k  https://127.0.0.1:8443
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1beta1",
    "/apis/apps",
    "/apis/apps.openshift.io",
    "/apis/apps.openshift.io/v1",
    "/apis/apps/v1beta1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/authorization.openshift.io",
    "/apis/authorization.openshift.io/v1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/autoscaling/v2alpha1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/build.openshift.io",
    "/apis/build.openshift.io/v1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/image.openshift.io",
    "/apis/image.openshift.io/v1",
    "/apis/network.openshift.io",
    "/apis/network.openshift.io/v1",
    "/apis/oauth.openshift.io",
    "/apis/oauth.openshift.io/v1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/project.openshift.io",
    "/apis/project.openshift.io/v1",
    "/apis/quota.openshift.io",
    "/apis/quota.openshift.io/v1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/route.openshift.io",
    "/apis/route.openshift.io/v1",
    "/apis/security.openshift.io",
    "/apis/security.openshift.io/v1",
    "/apis/settings.k8s.io",
    "/apis/settings.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/apis/template.openshift.io",
    "/apis/template.openshift.io/v1",
    "/apis/user.openshift.io",
    "/apis/user.openshift.io/v1",
    "/brokers/template.openshift.io/v2",
    "/controllers",
    "/healthz",
    "/healthz/autoregister-completion",
    "/healthz/ping",
    "/healthz/poststarthook/apiservice-registration-controller",
    "/healthz/poststarthook/apiservice-status-available-controller",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/kube-apiserver-autoregistration",
    "/healthz/poststarthook/start-kube-aggregator-informers",
    "/healthz/ready",
    "/metrics",
    "/oapi",
    "/oapi/v1",
    "/swaggerapi",
    "/version",
    "/version/openshift"
  ]
}



and the aggregator seems to think it can't reach the SC apis:
$ oc get apiservices v1alpha1.servicecatalog.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  creationTimestamp: 2017-07-04T16:48:14Z
  name: v1alpha1.servicecatalog.k8s.io
  resourceVersion: "1322"
  selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1alpha1.servicecatalog.k8s.io
  uid: 924f571b-60d8-11e7-90c1-507b9d27b5d9
spec:
  caBundle: ""
  group: servicecatalog.k8s.io
  groupPriorityMinimum: 200
  insecureSkipTLSVerify: true
  service:
    name: apiserver
    namespace: service-catalog
  version: v1alpha1
  versionPriority: 20
status:
  conditions:
  - lastTransitionTime: 2017-07-04T16:48:14Z
    message: 'no response from https://172.30.1.2/apis: context deadline exceeded'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available


but i can:
$ curl -k https://172.30.1.2/apis
{
  "kind": "APIGroupList",
  "groups": [
    {
      "name": "servicecatalog.k8s.io",
      "versions": [
        {
          "groupVersion": "servicecatalog.k8s.io/v1alpha1",
          "version": "v1alpha1"
        }
      ],
      "preferredVersion": {
        "groupVersion": "servicecatalog.k8s.io/v1alpha1",
        "version": "v1alpha1"
      },
      "serverAddressByClientCIDRs": [
        {
          "clientCIDR": "0.0.0.0/0",
          "serverAddress": ""
        }
      ]
    }
  ]
}

David, thoughts/other things to look at?
Comment 3 Ben Parees 2017-07-04 13:10:29 EDT
Is there a window here where i might be registering the api w/ the aggregator "too soon" (before it's actually available) and so it fails discovery and never gets re-checked?

The logic currently waits for the sc api server deployment to have an available endpoint before registering, but nothing guarantees that endpoint is actually serving successfully yet (i.e. no readiness probe).  (That said, it should be pretty soon after, if not already available by the time the registration call is made).
Comment 4 David Eads 2017-07-05 08:46:23 EDT
Looks similar to what https://github.com/openshift/origin/pull/15022 fixed.
Comment 5 Ben Parees 2017-07-05 10:15:21 EDT
appears to be working again w/  https://github.com/openshift/origin/pull/15022 merged.  Thanks David.
Comment 7 Dongbo Yan 2017-07-06 02:51:04 EDT
oc v3.6.135
kubernetes v1.6.1+5115d708d7

Note You need to log in before you can comment on or make changes to this bug.