Bug 1595573
| Summary: | New 3.9 install asb-1-deploy and asb-etcd-1-deploy stay in error status with update acceptor rejected | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | David Caldwell <dcaldwel> |
| Component: | openshift-controller-manager | Assignee: | Michal Fojtik <mfojtik> |
| Status: | CLOSED NOTABUG | QA Contact: | Wang Haoran <haowang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, cstark, dapark, dcaldwel, maszulik, mfojtik, syangsao |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-24 14:32:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
|
Description
David Caldwell
2018-06-27 07:46:17 UTC
Can we get the following: * oc describe rs/asb-1 * oc describe pod/asb-1-deploy * oc describe pod/asb-etcd-1-deploy * controller logs (ideally with loglevel>3) * oc get dc,rc,pod -o yaml Created attachment 1454994 [details]
oc describe asb-1-deploy
Created attachment 1454995 [details]
oc describe asb-etcd
Created attachment 1454996 [details]
oc get dc,rc,pod -o yaml
$ oc describe rs/asb-1 Error from server (NotFound): replicasets.extensions "asb-1" not found $ oc get rs --all-namespaces NAMESPACE NAME DESIRED CURRENT READY AGE openshift-web-console webconsole-746dbc7568 3 3 3 19h Controller logs to follow. Once the loglevel is set, is there an action/command you would like me to collect logs for? Also: oc logs asb-1-deploy That should tell us what the deployer pod was doing for 10 minutes after it timeouted. $ oc logs asb-1-deploy --> Scaling asb-1 to 1 error: update acceptor rejected asb-1: pods for rc 'openshift-ansible-service-broker/asb-1' took longer than 600 seconds to become available Created attachment 1455069 [details]
complete journald since install filtered for unit openshift-controllers
Created attachment 1455070 [details]
journal log for the time period during which I changed to loglevel=4 only - filtered for unit openshift-controllers
Same issue in 3.9.33. New install (on upshift quicklab) and the asb deployment pods are stuck in error: [quicklab@master-0 ~]$ oc logs asb-1-deploy -n openshift-ansible-service-broker --> Scaling asb-1 to 1 error: update acceptor rejected asb-1: pods for rc 'openshift-ansible-service-broker/asb-1' took longer than 600 seconds to become available [quicklab@master-0 ~]$ oc version oc v3.9.33 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO openshift v3.9.33 kubernetes v1.9.1+a0ce1bc657 [quicklab@master-0 ~]$ oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default docker-registry-1-dn2zt 1/1 Running 1 2h default registry-console-1-szqjp 1/1 Running 1 2h default router-2-t9hm8 1/1 Running 0 1h kube-service-catalog apiserver-g4r9k 1/1 Running 0 1h kube-service-catalog apiserver-g5hxw 1/1 Running 0 1h kube-service-catalog apiserver-q62jm 1/1 Running 0 1h kube-service-catalog controller-manager-6gvfb 1/1 Running 1 2h kube-service-catalog controller-manager-rhn2x 1/1 Running 2 2h kube-service-catalog controller-manager-t2pf2 1/1 Running 6 2h openshift-ansible-service-broker asb-1-deploy 0/1 Error 0 2h openshift-ansible-service-broker asb-etcd-1-deploy 0/1 Error 0 2h openshift-template-service-broker apiserver-bkqg5 1/1 Running 1 2h openshift-web-console webconsole-68b848cb77-72vmj 1/1 Running 1 2h openshift-web-console webconsole-68b848cb77-89ctr 1/1 Running 1 2h openshift-web-console webconsole-68b848cb77-9lggd 1/1 Running 1 2h I also met this issue, and I could resolve it using "oc rollout latest" as workaround. My verification steps is as follows. ~~~ # oc version oc v3.9.43 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO openshift v3.9.43 kubernetes v1.9.1+a0ce1bc657 # oc get pod NAME READY STATUS RESTARTS AGE asb-1-deploy 0/1 Error 0 12d asb-etcd-1-deploy 0/1 Error 0 12d # oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE etcd Bound etcd-pv 1Gi RWO 12d # oc rollout latest dc/asb-etcd deploymentconfig "asb-etcd" rolled out # oc get pod NAME READY STATUS RESTARTS AGE asb-1-deploy 0/1 Error 0 12d asb-etcd-2-deploy 0/1 ContainerCreating 0 2s # oc get pod NAME READY STATUS RESTARTS AGE asb-1-deploy 0/1 Error 0 12d asb-etcd-2-t6wwj 1/1 Running 0 2m ~~~ the solution should be to check the pvc in the project (named etcd). I had the same behaviour and then found that it was pending. Looks like the previous comment solved the issue, as well as original reporter found a working solution. I'm going to close this, feel free to re-open if it's still a problem. |