atomic-openshift-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-clients-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-master-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-sdn-ovs-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 atomic-openshift-utils-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-filter-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-lookup-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-playbooks-3.0.7-1.git.76.c73ec7b.el7aos.noarch openshift-ansible-roles-3.0.7-1.git.76.c73ec7b.el7aos.noarch tuned-profiles-atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64 [alice@ose3-master ~]$ oc rollback ruby-hello-world-1 #3 rolled back to ruby-hello-world-1 Warning: the following images triggers were disabled: ruby-hello-world You can re-enable them with: oc deploy ruby-hello-world --enable-triggers -n wiring [alice@ose3-master ~]$ oc get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE database-1 mysql registry.access.redhat.com/openshift3/mysql-55-rhel7:latest deployment=database-1,deploymentconfig=database,name =database 1 4m ruby-hello-world-1 ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c66 5e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 app=ruby-hello-world,deployment=ruby-hello-world-1,deploymentconfig=ruby-hello-world 0 4m ruby-hello-world-2 ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e app=ruby-hello-world,deployment=ruby-hello-world-2,d eploymentconfig=ruby-hello-world 1 1m It appears that the DC was changed, and a new version was created... but no new deployment ever happened: apiVersion: v1 kind: DeploymentConfig metadata: annotations: openshift.io/generated-by: OpenShiftNewApp creationTimestamp: 2015-11-06T02:31:52Z labels: app: ruby-hello-world name: ruby-hello-world namespace: wiring resourceVersion: "4058" selfLink: /oapi/v1/namespaces/wiring/deploymentconfigs/ruby-hello-world uid: 8a29a27f-842e-11e5-af5d-525400b33d1d spec: replicas: 1 selector: app: ruby-hello-world deploymentconfig: ruby-hello-world strategy: resources: {} rollingParams: intervalSeconds: 1 maxSurge: 25% maxUnavailable: 25% timeoutSeconds: 600 updatePeriodSeconds: 1 type: Rolling template: metadata: annotations: openshift.io/generated-by: OpenShiftNewApp creationTimestamp: null labels: app: ruby-hello-world deploymentconfig: ruby-hello-world spec: containers: - env: - name: MYSQL_USER value: redhat - name: MYSQL_PASSWORD value: redhat - name: MYSQL_DATABASE value: mydb image: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 imagePullPolicy: Always name: ruby-hello-world ports: - containerPort: 8080 protocol: TCP resources: {} terminationMessagePath: /dev/termination-log dnsPolicy: ClusterFirst restartPolicy: Always securityContext: {} terminationGracePeriodSeconds: 30 triggers: - type: ConfigChange - imageChangeParams: containerNames: - ruby-hello-world from: kind: ImageStreamTag name: ruby-hello-world:latest lastTriggeredImage: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e type: ImageChange status: details: causes: - imageTrigger: from: kind: DockerImage name: 172.30.132.215:5000/wiring/ruby-hello-world:latest type: ImageChange latestVersion: 3
[alice@ose3-master ~]$ oc describe dc ruby-hello-world Name: ruby-hello-world Created: 14 minutes ago Labels: app=ruby-hello-world Annotations: openshift.io/generated-by=OpenShiftNewApp Latest Version: 3 Triggers: Config, Image(ruby-hello-world@latest, auto=false) Strategy: Rolling Template: Selector: app=ruby-hello-world,deploymentconfig=ruby-hello-world Replicas: 1 Containers: NAME IMAGE ENV ruby-hello-world 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52 MYSQL_DATABASE=mydb,MYSQL_PASSWORD=redhat,MYSQL_USER=redhat Latest Deployment: <none> Deployment #2: Created: 10 minutes ago Status: Complete Replicas: 1 current / 1 desired Deployment #1: Created: 13 minutes ago Status: Complete Replicas: 0 current / 0 desired No events.
I use the application-template-stibuild.json, can't reproduce this issue: [root@zhouy testjson]# oc get pods NAME READY STATUS RESTARTS AGE database-1-mq2up 1/1 Running 0 <invalid> frontend-2-e16tg 1/1 Running 0 <invalid> frontend-2-g7arq 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 5s [root@zhouy testjson]# oc rollback frontend --to-version=1 #3 rolled back to frontend-1 Warning: the following images triggers were disabled: origin-ruby-sample You can re-enable them with: oc deploy frontend --enable-triggers -n zhouyt [root@zhouy testjson]# oc get pods NAME READY STATUS RESTARTS AGE database-1-mq2up 1/1 Running 0 1m frontend-3-deploy 1/1 Running 0 <invalid> frontend-3-posthook 0/1 Error 0 <invalid> frontend-3-prehook 0/1 Completed 0 <invalid> frontend-3-vl53j 1/1 Running 0 <invalid> frontend-3-zqdtd 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 1m [root@zhouy testjson]# oc get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE database-1 ruby-helloworld-database openshift/mysql-55-centos7:latest deployment=database-1,deploymentconfig=database,name=database 1 2m frontend-1 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-1,deploymentconfig=frontend,name=frontend 0 1m frontend-2 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-2,deploymentconfig=frontend,name=frontend 0 <invalid> frontend-3 ruby-helloworld 172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39 deployment=frontend-3,deploymentconfig=frontend,name=frontend 2 <invalid>
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.093561 116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2 Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.096765 116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.096805 116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.104967 116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2 Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.113940 116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.114096 116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers So, the "issue" is actually not an issue in anything native to OpenShift, it's actually an issue on quota reporting. It appears that my quota prohibits deploying another RC, but this information (forbidden due to rc limits) doesn't appear to be displayed to the user anywhere. Since the RC is never created, you can't describe it. Describing the DC doesn't propagate the event that the RC creation was prohibited.
https://github.com/openshift/origin/pull/5875 introduces an event when the RC couldn't be created, which will resolve the reporting issue.
Dan, Is this fixed now? The linked PR is merged
Confirmed on ami devenv-rhel7_3076, but the issue still appears. [root@ip-172-18-11-38 amd64]# openshift version openshift v1.1-668-g705a7dc kubernetes v1.1.0-origin-1107-g4c8e6f4 etcd 2.1.2 When the quota prohibits deploying, there was no event describe the RC creation was prohibited. [root@ip-172-18-11-38 amd64]# oc get quota NAME AGE quota 3s [root@ip-172-18-11-38 amd64]# oc get pods NAME READY STATUS RESTARTS AGE hooks-1-13ghh 1/1 Running 0 1m [root@ip-172-18-11-38 amd64]# oc deploy hooks --latest Started deployment #2 [root@ip-172-18-11-38 amd64]# oc get pods NAME READY STATUS RESTARTS AGE hooks-1-13ghh 1/1 Running 0 1m [root@ip-172-18-11-38 amd64]# oc get pods NAME READY STATUS RESTARTS AGE hooks-1-13ghh 1/1 Running 0 1m [root@ip-172-18-11-38 amd64]# oc describe dc hooks Name: hooks Created: 2 minutes ago Labels: name=mysql Latest Version: 2 Triggers: Config Strategy: Recreate Pre-deployment hook (pod type, failure policy: Ignore): Container: mysql-55-rhel7 Command: /bin/bash -c /usr/bin/sleep 6 Env: VAR=pre-deployment Post-deployment hook (pod type, failure policy: Retry): Container: mysql-55-rhel7 Command: /bin/bash -c /usr/bin/sleep 2 Env: VAR=post-deployment Template: Selector: name=mysql Replicas: 1 Containers: NAME IMAGE ENV mysql-55-rhel7 openshift/mysql-55-centos7:latest MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2 Deployment #2 (latest): Name: hooks-2 Created: 13 seconds ago Status: New Replicas: 0 current / 0 desired Selector: deployment=hooks-2,deploymentconfig=hooks,name=mysql Labels: name=mysql,openshift.io/deployment-config.name=hooks Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed Deployment #1: Created: 2 minutes ago Status: Complete Replicas: 1 current / 1 desired Events: FirstSeen LastSeen Count From SubobjectPath Reason Message ───────── ──────── ───── ──── ───────────── ────── ─────── 2m 2m 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-1" for version 1 13s 13s 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-2" for version 2 [root@ip-172-18-11-38 amd64]# oc get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE hooks-1 mysql-55-rhel7 openshift/mysql-55-centos7:latest deployment=hooks-1,deploymentconfig=hooks,name=mysql 1 2m hooks-2 mysql-55-rhel7 openshift/mysql-55-centos7:latest deployment=hooks-2,deploymentconfig=hooks,name=mysql 0 34s [root@ip-172-18-11-38 amd64]# oc describe dc hooks Name: hooks Created: 2 minutes ago Labels: name=mysql Latest Version: 2 Triggers: Config Strategy: Recreate Pre-deployment hook (pod type, failure policy: Ignore): Container: mysql-55-rhel7 Command: /bin/bash -c /usr/bin/sleep 6 Env: VAR=pre-deployment Post-deployment hook (pod type, failure policy: Retry): Container: mysql-55-rhel7 Command: /bin/bash -c /usr/bin/sleep 2 Env: VAR=post-deployment Template: Selector: name=mysql Replicas: 1 Containers: NAME IMAGE ENV mysql-55-rhel7 openshift/mysql-55-centos7:latest MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2 Deployment #2 (latest): Name: hooks-2 Created: 54 seconds ago Status: New Replicas: 0 current / 0 desired Selector: deployment=hooks-2,deploymentconfig=hooks,name=mysql Labels: name=mysql,openshift.io/deployment-config.name=hooks Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed Deployment #1: Created: 2 minutes ago Status: Complete Replicas: 1 current / 1 desired Events: FirstSeen LastSeen Count From SubobjectPath Reason Message ───────── ──────── ───── ──── ───────────── ────── ─────── 2m 2m 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-1" for version 1 54s 54s 1 {deploymentconfig-controller } DeploymentCreated Created new deployment "hooks-2" for version 2
The reported issue was a lack of event for the deployment itself when the RC quota had been reached; there's no event currently produced for an inability to create hook pods as described in https://bugzilla.redhat.com/show_bug.cgi?id=1278630#c7. We could consider trying to produce events for the hook pod creation part, but that would need to be a separate RFE. Let's keep this bug scoped to the original report by testing to ensure an event is created when the deployment itself can't be created due to a quota limit.
Confirmed on devenv-rhel7_3093 could see the event from original reported: [root@ip-172-18-4-210 amd64]# oc get events|grep Failed 14m 14m 1 hooks-1 ReplicationController FailedUpdate {deployer } Error updating deployment zhouy/hooks-1 status to Pending 10m 8m 9 hooks-2 ReplicationController FailedCreate {deployer } Error creating deployer pod for zhouy/hooks-2: Pod "hooks-2-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota. 5m 30s 12 hooks-3 ReplicationController FailedCreate {deployer } Error creating deployer pod for zhouy/hooks-3: Pod "hooks-3-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:0070