Bug 1278630 - oc rollback says it is doing something, but doesn't appear to do it
oc rollback says it is doing something, but doesn't appear to do it
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Deployments (Show other bugs)
3.1.0
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Dan Mace
zhou ying
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-05 21:44 EST by Erik M Jacobs
Modified: 2016-03-08 22:42 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-26 14:16:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Erik M Jacobs 2015-11-05 21:44:49 EST
atomic-openshift-3.1.0.0-0.git.0.0e71938.el7aos.x86_64
atomic-openshift-clients-3.1.0.0-0.git.0.0e71938.el7aos.x86_64
atomic-openshift-master-3.1.0.0-0.git.0.0e71938.el7aos.x86_64
atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64
atomic-openshift-sdn-ovs-3.1.0.0-0.git.0.0e71938.el7aos.x86_64
atomic-openshift-utils-3.0.7-1.git.76.c73ec7b.el7aos.noarch
openshift-ansible-3.0.7-1.git.76.c73ec7b.el7aos.noarch
openshift-ansible-filter-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch
openshift-ansible-lookup-plugins-3.0.7-1.git.76.c73ec7b.el7aos.noarch
openshift-ansible-playbooks-3.0.7-1.git.76.c73ec7b.el7aos.noarch
openshift-ansible-roles-3.0.7-1.git.76.c73ec7b.el7aos.noarch
tuned-profiles-atomic-openshift-node-3.1.0.0-0.git.0.0e71938.el7aos.x86_64

[alice@ose3-master ~]$     oc rollback ruby-hello-world-1
#3 rolled back to ruby-hello-world-1
Warning: the following images triggers were disabled: ruby-hello-world
  You can re-enable them with: oc deploy ruby-hello-world --enable-triggers -n wiring
[alice@ose3-master ~]$ oc get rc
CONTROLLER           CONTAINER(S)       IMAGE(S)                                                         
                                                     SELECTOR                                            
                                   REPLICAS   AGE
database-1           mysql              registry.access.redhat.com/openshift3/mysql-55-rhel7:latest                                                           deployment=database-1,deploymentconfig=database,name
=database                          1          4m
ruby-hello-world-1   ruby-hello-world   172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c66
5e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52   app=ruby-hello-world,deployment=ruby-hello-world-1,deploymentconfig=ruby-hello-world   0          4m
ruby-hello-world-2   ruby-hello-world   172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e   app=ruby-hello-world,deployment=ruby-hello-world-2,d
eploymentconfig=ruby-hello-world   1          1m

It appears that the DC was changed, and a new version was created... but no new deployment ever happened:

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: 2015-11-06T02:31:52Z
  labels:
    app: ruby-hello-world
  name: ruby-hello-world
  namespace: wiring
  resourceVersion: "4058"
  selfLink: /oapi/v1/namespaces/wiring/deploymentconfigs/ruby-hello-world
  uid: 8a29a27f-842e-11e5-af5d-525400b33d1d
spec:
  replicas: 1
  selector:
    app: ruby-hello-world
    deploymentconfig: ruby-hello-world
  strategy:
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        openshift.io/generated-by: OpenShiftNewApp
      creationTimestamp: null
      labels:
        app: ruby-hello-world
        deploymentconfig: ruby-hello-world
    spec:
      containers:
      - env:
        - name: MYSQL_USER
          value: redhat
        - name: MYSQL_PASSWORD
          value: redhat
        - name: MYSQL_DATABASE
          value: mydb
        image: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52
        imagePullPolicy: Always
        name: ruby-hello-world
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
  triggers:
  - type: ConfigChange
  - imageChangeParams:
      containerNames:
      - ruby-hello-world
      from:
        kind: ImageStreamTag
        name: ruby-hello-world:latest
      lastTriggeredImage: 172.30.132.215:5000/wiring/ruby-hello-world@sha256:7cba69f98183b2356af120fc28ce8d21d4134b2ccce85282898d95dfa6d1207e
    type: ImageChange
status:
  details:
    causes:
    - imageTrigger:
        from:
          kind: DockerImage
          name: 172.30.132.215:5000/wiring/ruby-hello-world:latest
      type: ImageChange
  latestVersion: 3
Comment 2 Erik M Jacobs 2015-11-05 21:46:58 EST
[alice@ose3-master ~]$ oc describe dc ruby-hello-world
Name:           ruby-hello-world
Created:        14 minutes ago
Labels:         app=ruby-hello-world
Annotations:    openshift.io/generated-by=OpenShiftNewApp
Latest Version: 3
Triggers:       Config, Image(ruby-hello-world@latest, auto=false)
Strategy:       Rolling
Template:
  Selector:     app=ruby-hello-world,deploymentconfig=ruby-hello-world
  Replicas:     1
  Containers:
  NAME                  IMAGE                                                                                                                   ENV
  ruby-hello-world      172.30.132.215:5000/wiring/ruby-hello-world@sha256:fe97f6d8c90c665e7c0ae3c58a5a421de1818ad2bdb77536486fa2432a75ca52     MYSQL_DATABASE=mydb,MYSQL_PASSWORD=redhat,MYSQL_USER=redhat
Latest Deployment:      <none>
Deployment #2:
        Created:        10 minutes ago
        Status:         Complete
        Replicas:       1 current / 1 desired
Deployment #1:
        Created:        13 minutes ago
        Status:         Complete
        Replicas:       0 current / 0 desired
No events.
Comment 3 zhou ying 2015-11-05 22:12:48 EST
I use the application-template-stibuild.json, can't reproduce this issue:

[root@zhouy testjson]# oc get pods
NAME                        READY     STATUS      RESTARTS   AGE
database-1-mq2up            1/1       Running     0          <invalid>
frontend-2-e16tg            1/1       Running     0          <invalid>
frontend-2-g7arq            1/1       Running     0          <invalid>
ruby-sample-build-1-build   0/1       Completed   0          5s

[root@zhouy testjson]# oc rollback frontend --to-version=1
#3 rolled back to frontend-1
Warning: the following images triggers were disabled: origin-ruby-sample
  You can re-enable them with: oc deploy frontend --enable-triggers -n zhouyt
[root@zhouy testjson]# oc get pods
NAME                        READY     STATUS      RESTARTS   AGE
database-1-mq2up            1/1       Running     0          1m
frontend-3-deploy           1/1       Running     0          <invalid>
frontend-3-posthook         0/1       Error       0          <invalid>
frontend-3-prehook          0/1       Completed   0          <invalid>
frontend-3-vl53j            1/1       Running     0          <invalid>
frontend-3-zqdtd            1/1       Running     0          <invalid>
ruby-sample-build-1-build   0/1       Completed   0          1m
[root@zhouy testjson]# oc get rc
CONTROLLER   CONTAINER(S)               IMAGE(S)                                                                                                               SELECTOR                                                        REPLICAS   AGE
database-1   ruby-helloworld-database   openshift/mysql-55-centos7:latest                                                                                      deployment=database-1,deploymentconfig=database,name=database   1          2m
frontend-1   ruby-helloworld            172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39   deployment=frontend-1,deploymentconfig=frontend,name=frontend   0          1m
frontend-2   ruby-helloworld            172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39   deployment=frontend-2,deploymentconfig=frontend,name=frontend   0          <invalid>
frontend-3   ruby-helloworld            172.30.148.91:5000/zhouyt/origin-ruby-sample@sha256:8b552ac54722fc1f3156aca295a777e652ee0a40c88c1e1ca937406d451bbc39   deployment=frontend-3,deploymentconfig=frontend,name=frontend   2          <invalid>
Comment 4 Erik M Jacobs 2015-11-06 11:55:45 EST
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.093561  116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.096765  116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.096805  116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: I1106 11:52:25.104967  116198 controller.go:131] Desired replicas for wiring/ruby-hello-world:3 set to 1 based on prior completed deployment ruby-hello-world-2
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: W1106 11:52:25.113940  116198 controller.go:150] Cannot create latest deployment for deployment config "wiring/ruby-hello-world:3": ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers
Nov 06 11:52:25 ose3-master.example.com atomic-openshift-master[116198]: E1106 11:52:25.114096  116198 factory.go:91] couldn't create deployment for deployment config wiring/ruby-hello-world:3: ReplicationController "ruby-hello-world-3" is forbidden: limited to 3 replicationcontrollers

So, the "issue" is actually not an issue in anything native to OpenShift, it's actually an issue on quota reporting.

It appears that my quota prohibits deploying another RC, but this information (forbidden due to rc limits) doesn't appear to be displayed to the user anywhere. Since the RC is never created, you can't describe it. Describing the DC doesn't propagate the event that the RC creation was prohibited.
Comment 5 Dan Mace 2015-11-30 09:23:44 EST
https://github.com/openshift/origin/pull/5875 introduces an event when the RC couldn't be created, which will resolve the reporting issue.
Comment 6 Paul Weil 2016-01-04 16:31:13 EST
Dan,

Is this fixed now?  The linked PR is merged
Comment 7 zhou ying 2016-01-06 05:32:35 EST
Confirmed on ami devenv-rhel7_3076, but the issue still appears.
[root@ip-172-18-11-38 amd64]#  openshift version
openshift v1.1-668-g705a7dc
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

When the quota prohibits deploying, there was no event describe the RC creation was prohibited.

[root@ip-172-18-11-38 amd64]# oc get quota
NAME      AGE
quota     3s
[root@ip-172-18-11-38 amd64]# oc get pods
NAME            READY     STATUS    RESTARTS   AGE
hooks-1-13ghh   1/1       Running   0          1m
[root@ip-172-18-11-38 amd64]# oc deploy hooks --latest
Started deployment #2
[root@ip-172-18-11-38 amd64]# oc get pods
NAME            READY     STATUS    RESTARTS   AGE
hooks-1-13ghh   1/1       Running   0          1m
[root@ip-172-18-11-38 amd64]# oc get pods
NAME            READY     STATUS    RESTARTS   AGE
hooks-1-13ghh   1/1       Running   0          1m
[root@ip-172-18-11-38 amd64]# oc describe dc hooks
Name:		hooks
Created:	2 minutes ago
Labels:		name=mysql
Latest Version:	2
Triggers:	Config
Strategy:	Recreate
		  Pre-deployment hook (pod type, failure policy: Ignore):
		    Container:	mysql-55-rhel7
		    Command:	/bin/bash -c /usr/bin/sleep 6
		    Env:	VAR=pre-deployment
		  Post-deployment hook (pod type, failure policy: Retry):
		    Container:	mysql-55-rhel7
		    Command:	/bin/bash -c /usr/bin/sleep 2
		    Env:	VAR=post-deployment
Template:
  Selector:	name=mysql
  Replicas:	1
  Containers:
  NAME			IMAGE					ENV
  mysql-55-rhel7	openshift/mysql-55-centos7:latest	MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2
Deployment #2 (latest):
	Name:		hooks-2
	Created:	13 seconds ago
	Status:		New
	Replicas:	0 current / 0 desired
	Selector:	deployment=hooks-2,deploymentconfig=hooks,name=mysql
	Labels:		name=mysql,openshift.io/deployment-config.name=hooks
	Pods Status:	0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #1:
	Created:	2 minutes ago
	Status:		Complete
	Replicas:	1 current / 1 desired
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Reason			Message
  ─────────	────────	─────	────				─────────────	──────			───────
  2m		2m		1	{deploymentconfig-controller }			DeploymentCreated	Created new deployment "hooks-1" for version 1
  13s		13s		1	{deploymentconfig-controller }			DeploymentCreated	Created new deployment "hooks-2" for version 2


[root@ip-172-18-11-38 amd64]# oc get rc
CONTROLLER   CONTAINER(S)     IMAGE(S)                            SELECTOR                                               REPLICAS   AGE
hooks-1      mysql-55-rhel7   openshift/mysql-55-centos7:latest   deployment=hooks-1,deploymentconfig=hooks,name=mysql   1          2m
hooks-2      mysql-55-rhel7   openshift/mysql-55-centos7:latest   deployment=hooks-2,deploymentconfig=hooks,name=mysql   0          34s
[root@ip-172-18-11-38 amd64]# oc describe dc hooks
Name:		hooks
Created:	2 minutes ago
Labels:		name=mysql
Latest Version:	2
Triggers:	Config
Strategy:	Recreate
		  Pre-deployment hook (pod type, failure policy: Ignore):
		    Container:	mysql-55-rhel7
		    Command:	/bin/bash -c /usr/bin/sleep 6
		    Env:	VAR=pre-deployment
		  Post-deployment hook (pod type, failure policy: Retry):
		    Container:	mysql-55-rhel7
		    Command:	/bin/bash -c /usr/bin/sleep 2
		    Env:	VAR=post-deployment
Template:
  Selector:	name=mysql
  Replicas:	1
  Containers:
  NAME			IMAGE					ENV
  mysql-55-rhel7	openshift/mysql-55-centos7:latest	MYSQL_DATABASE=root,MYSQL_PASSWORD=Plqe5Wev,MYSQL_USER=user8Y2
Deployment #2 (latest):
	Name:		hooks-2
	Created:	54 seconds ago
	Status:		New
	Replicas:	0 current / 0 desired
	Selector:	deployment=hooks-2,deploymentconfig=hooks,name=mysql
	Labels:		name=mysql,openshift.io/deployment-config.name=hooks
	Pods Status:	0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #1:
	Created:	2 minutes ago
	Status:		Complete
	Replicas:	1 current / 1 desired
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Reason			Message
  ─────────	────────	─────	────				─────────────	──────			───────
  2m		2m		1	{deploymentconfig-controller }			DeploymentCreated	Created new deployment "hooks-1" for version 1
  54s		54s		1	{deploymentconfig-controller }			DeploymentCreated	Created new deployment "hooks-2" for version 2
Comment 8 Dan Mace 2016-01-06 08:42:12 EST
The reported issue was a lack of event for the deployment itself when the RC quota had been reached; there's no event currently produced for an inability to create hook pods as described in https://bugzilla.redhat.com/show_bug.cgi?id=1278630#c7.

We could consider trying to produce events for the hook pod creation part, but that would need to be a separate RFE. Let's keep this bug scoped to the original report by testing to ensure an event is created when the deployment itself can't be created due to a quota limit.
Comment 9 zhou ying 2016-01-08 05:01:29 EST
Confirmed on devenv-rhel7_3093
could see the event from original reported:
[root@ip-172-18-4-210 amd64]# oc get events|grep Failed
14m         14m        1         hooks-1            ReplicationController                                       FailedUpdate        {deployer }                              Error updating deployment zhouy/hooks-1 status to Pending
10m         8m         9         hooks-2            ReplicationController                                       FailedCreate        {deployer }                              Error creating deployer pod for zhouy/hooks-2: Pod "hooks-2-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota.
5m          30s        12        hooks-3            ReplicationController                                       FailedCreate        {deployer }                              Error creating deployer pod for zhouy/hooks-3: Pod "hooks-3-deploy" is forbidden: must make a non-zero request for memory since it is tracked by quota.
Comment 11 errata-xmlrpc 2016-01-26 14:16:56 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070

Note You need to log in before you can comment on or make changes to this bug.