Bug 1278630
| Summary: | oc rollback says it is doing something, but doesn't appear to do it | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Erik M Jacobs <ejacobs> |
| Component: | openshift-controller-manager | Assignee: | Dan Mace <dmace> |
| Status: | CLOSED ERRATA | QA Contact: | zhou ying <yinzhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.1.0 | CC: | aos-bugs, bleanhar, dmace, dmcphers, pruan, pweil, wsun |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-01-26 19:16:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[alice@ose3-master ~]$ oc describe dc ruby-hello-world
Name: ruby-hello-world
Created: 14 minutes ago
Labels: app=ruby-hello-world
Annotations: openshift.io/generated-by=OpenShiftNewApp
Latest Version: 3
Triggers: Config, Image(ruby-hello-world@latest, auto=false)
Strategy: Rolling
Template:
Selector: app=ruby-hello-world,deploymentconfig=ruby-hello-world
Replicas: 1
Containers:
NAME IMAGE ENV
ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 MYSQL_DATABASE=mydb,MYSQL_PASSWORD=redhat,MYSQL_USER=redhat
Latest Deployment: <none>
Deployment #2:
Created: 10 minutes ago
Status: Complete
Replicas: 1 current / 1 desired
Deployment #1:
Created: 13 minutes ago
Status: Complete
Replicas: 0 current / 0 desired
No events.
I use the application-template-stibuild.json, can't reproduce this issue: [root@zhouy testjson]# oc get pods NAME READY STATUS RESTARTS AGE database-1-mq2up 1/1 Running 0 <invalid> frontend-2-e16tg 1/1 Running 0 <invalid> frontend-2-g7arq 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 5s [root@zhouy testjson]# oc rollback frontend --to-version=1 #3 rolled back to frontend-1 Warning: the following images triggers were disabled: origin-ruby-sample You can re-enable them with: oc deploy frontend --enable-triggers -n zhouyt [root@zhouy testjson]# oc get pods NAME READY STATUS RESTARTS AGE database-1-mq2up 1/1 Running 0 1m frontend-3-deploy 1/1 Running 0 <invalid> frontend-3-posthook 0/1 Error 0 <invalid> frontend-3-prehook 0/1 Completed 0 <invalid> frontend-3-vl53j 1/1 Running 0 <invalid> frontend-3-zqdtd 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 1m [root@zhouy testjson]# oc get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE database-1 ruby-helloworld-database openshift/mysql-55-centos7:latest deployment=database-1,deploymentconfig=database,name=database 1 2m frontend-1 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-1,deploymentconfig=frontend,name=frontend 0 1m frontend-2 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-2,deploymentconfig=frontend,name=frontend 0 <invalid> frontend-3 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-3,deploymentconfig=frontend,name=frontend 2 <invalid> Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.093561 116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2 Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.096765 116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.096805 116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.104967 116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2 Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.113940 116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.114096 116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers So, the "issue" is actually not an issue in anything native to OpenShift, it's actually an issue on quota reporting. It appears that my quota prohibits deploying another RC, but this information (forbidden due to rc limits) doesn't appear to be displayed to the user anywhere. Since the RC is never created, you can't describe it. Describing the DC doesn't propagate the event that the RC creation was prohibited. https://github.com/openshift/origin/pull/5875 introduces an event when the RC couldn't be created, which will resolve the reporting issue. Dan, Is this fixed now? The linked PR is merged Confirmed on ami devenv-rhel7_3076, but the issue still appears.
[root@ip-172-18-11-38 amd64]# openshift version
openshift v1.1-668-g705a7dc
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2
When the quota prohibits deploying, there was no event describe the RC creation was prohibited.
[root@ip-172-18-11-38 amd64]# oc get quota
NAME AGE
quota 3s
[root@ip-172-18-11-38 amd64]# oc get pods
NAME READY STATUS RESTARTS AGE
hooks-1-13ghh 1/1 Running 0 1m
[root@ip-172-18-11-38 amd64]# oc deploy hooks --latest
Started deployment #2
[root@ip-172-18-11-38 amd64]# oc get pods
NAME READY STATUS RESTARTS AGE
hooks-1-13ghh 1/1 Running 0 1m
[root@ip-172-18-11-38 amd64]# oc get pods
NAME READY STATUS RESTARTS AGE
hooks-1-13ghh 1/1 Running 0 1m
[root@ip-172-18-11-38 amd64]# oc describe dc hooks
Name: hooks
Created: 2 minutes ago
Labels: name=mysql
Latest Version: 2
Triggers: Config
Strategy: Recreate
Pre-deployment hook (pod type, failure policy: Ignore):
Container: mysql-55-rhel7
Command: /bin/bash -c /usr/bin/sleep 6
Env: VAR=pre-deployment
Post-deployment hook (pod type, failure policy: Retry):
Container: mysql-55-rhel7
Command: /bin/bash -c /usr/bin/sleep 2
Env: VAR=post-deployment
Template:
Selector: name=mysql
Replicas: 1
Containers:
NAME IMAGE ENV
mysql-55-rhel7 openshift/mysql-55-centos7:latest MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2
Deployment #2 (latest):
Name: hooks-2
Created: 13 seconds ago
Status: New
Replicas: 0 current / 0 desired
Selector: deployment=hooks-2,deploymentconfig=hooks,name=mysql
Labels: name=mysql,openshift.io/deployment-config.name=hooks
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #1:
Created: 2 minutes ago
Status: Complete
Replicas: 1 current / 1 desired
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
2m 2m 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-1" for version 1
13s 13s 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-2" for version 2
[root@ip-172-18-11-38 amd64]# oc get rc
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
hooks-1 mysql-55-rhel7 openshift/mysql-55-centos7:latest deployment=hooks-1,deploymentconfig=hooks,name=mysql 1 2m
hooks-2 mysql-55-rhel7 openshift/mysql-55-centos7:latest deployment=hooks-2,deploymentconfig=hooks,name=mysql 0 34s
[root@ip-172-18-11-38 amd64]# oc describe dc hooks
Name: hooks
Created: 2 minutes ago
Labels: name=mysql
Latest Version: 2
Triggers: Config
Strategy: Recreate
Pre-deployment hook (pod type, failure policy: Ignore):
Container: mysql-55-rhel7
Command: /bin/bash -c /usr/bin/sleep 6
Env: VAR=pre-deployment
Post-deployment hook (pod type, failure policy: Retry):
Container: mysql-55-rhel7
Command: /bin/bash -c /usr/bin/sleep 2
Env: VAR=post-deployment
Template:
Selector: name=mysql
Replicas: 1
Containers:
NAME IMAGE ENV
mysql-55-rhel7 openshift/mysql-55-centos7:latest MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2
Deployment #2 (latest):
Name: hooks-2
Created: 54 seconds ago
Status: New
Replicas: 0 current / 0 desired
Selector: deployment=hooks-2,deploymentconfig=hooks,name=mysql
Labels: name=mysql,openshift.io/deployment-config.name=hooks
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #1:
Created: 2 minutes ago
Status: Complete
Replicas: 1 current / 1 desired
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
2m 2m 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-1" for version 1
54s 54s 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-2" for version 2
The reported issue was a lack of event for the deployment itself when the RC quota had been reached; there's no event currently produced for an inability to create hook pods as described in https://bugzilla.redhat.com/show_bug.cgi?id=1278630#c7. We could consider trying to produce events for the hook pod creation part, but that would need to be a separate RFE. Let's keep this bug scoped to the original report by testing to ensure an event is created when the deployment itself can't be created due to a quota limit. Confirmed on devenv-rhel7_3093
could see the event from original reported:
[root@ip-172-18-4-210 amd64]# oc get events|grep Failed
14m 14m 1 hooks-1 ReplicationController FailedUpdate {deployer } Error updating deployment zhouy/hooks-1 status to Pending
10m 8m 9 hooks-2 ReplicationController FailedCreate {deployer } Error creating deployer pod for zhouy/hooks-2: Pod "hooks-2-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota.
5m 30s 12 hooks-3 ReplicationController FailedCreate {deployer } Error creating deployer pod for zhouy/hooks-3: Pod "hooks-3-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:0070 |
atomic-openshift-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-clients-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-master-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-sdn-ovs-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-utils-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-filter-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-lookup-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-playbooks-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-roles-3.0.7-1.git.76.c73ec7b.el7aos.noarch tuned-profiles-atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 [alice@ose3-master ~]$ oc rollback ruby-hello-world-1 #3 rolled back to ruby-hello-world-1 Warning: the following images triggers were disabled: ruby-hello-world You can re-enable them with: oc deploy ruby-hello-world --enable-triggers -n wiring [alice@ose3-master ~]$ oc get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE database-1 mysql registry.access.redhat.com/openshift3/mysql-55-rhel7:latest deployment=database-1,deploymentconfig=database,name =database 1 4m ruby-hello-world-1 ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c66 5e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 app=ruby-hello-world,deployment=ruby-hello-world-1,deploymentconfig=ruby-hello-world 0 4m ruby-hello-world-2 ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e app=ruby-hello-world,deployment=ruby-hello-world-2,d eploymentconfig=ruby-hello-world 1 1m It appears that the DC was changed, and a new version was created... but no new deployment ever happened: apiVersion: v1 kind: DeploymentConfig metadata: annotations: openshift.io/generated-by: OpenShiftNewApp creationTimestamp: 2015-11-06T02:31:52Z labels: app: ruby-hello-world name: ruby-hello-world namespace: wiring resourceVersion: "4058" selfLink: /oapi/v1/namespaces/wiring/deploymentconfigs/ruby-hello-world uid: 8a29a27f-842e-11e5-af5d-525400b33d1d spec: replicas: 1 selector: app: ruby-hello-world deploymentconfig: ruby-hello-world strategy: resources: {} rollingParams: intervalSeconds: 1 maxSurge: 25% maxUnavailable: 25% timeoutSeconds: 600 updatePeriodSeconds: 1 type: Rolling template: metadata: annotations: openshift.io/generated-by: OpenShiftNewApp creationTimestamp: null labels: app: ruby-hello-world deploymentconfig: ruby-hello-world spec: containers: - env: - name: MYSQL_USER value: redhat - name: MYSQL_PASSWORD value: redhat - name: MYSQL_DATABASE value: mydb image: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 imagePullPolicy: Always name: ruby-hello-world ports: - containerPort: 8080 protocol: TCP resources: {} terminationMessagePath: /dev/termination-log dnsPolicy: ClusterFirst restartPolicy: Always securityContext: {} terminationGracePeriodSeconds: 30 triggers: - type: ConfigChange - imageChangeParams: containerNames: - ruby-hello-world from: kind: ImageStreamTag name: ruby-hello-world:latest lastTriggeredImage: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e type: ImageChange status: details: causes: - imageTrigger: from: kind: DockerImage name: 172.30.132.215:5000/wiring/ruby-hello-world:latest type: ImageChange latestVersion: 3