Bug 1392780 - Cannot scale deployment using jenkins pipeline step and dsl step
Summary: Cannot scale deployment using jenkins pipeline step and dsl step
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Gabe Montero
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-08 08:57 UTC by Dongbo Yan
Modified: 2017-03-08 18:43 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: pipeline plugin did not use optimal endpoint for scaling Consequence: scaling beyond 1 replica was problematic Fix: pipeline plugin updated to use optimal endpoint Result: user can now scale a DC's RC beyond 1 replica
Clone Of:
Environment:
Last Closed: 2017-01-18 12:50:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0066 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description Dongbo Yan 2016-11-08 08:57:57 UTC
Description of problem:
Cannot scale deployment using jenkins pipeline step "Scale OpenShift Deployment" and dsl step "openshiftScale"

Version-Release number of selected component (if applicable):
brew-pulp.../openshift3/jenkins-2-rhel7         ede34972221e  
brew-pulp.../openshift3/jenkins-1-rhel7         b05627cfafbf 
openshift-login         0.6	
openshift-pipeline      1.0.28	
openshift-sync          0.0.14

openshift v3.4.0.23+24b1a58
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

How reproducible:
Always

Steps to Reproduce:
1. Login openshift via cli and create a new project
2. Setup jenkins server and generate application under project
$oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/jenkins-ephemeral-template.json
$oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/application-template.json
3.Create pipeline job, configure pipeline scripts
node{
    stage 'build'
    openshiftScale( depCfg: 'frontend',replicaCount: '3', verbose: 'true', verifyReplicaCount: 'true')
}

Actual results:
Job failed, cannot scale deployment

Expected results:
scale deployment successfully

Additional info:
The same issue with freestyle job.

jenkins job log:
Starting "Scale OpenShift Deployment" with deployment config "mongodb" from the project "dyan2".
 Scaling to "3" replicas and verifying the replica count is reached ...
Found global job type timeout configuration: 180000 milliseconds
No local timeout configured for this step
Operation will timeout after 180000 milliseconds

OpenShiftScaler setting desired replica count of 3 on mongodb-1

OpenShiftScaler rc returned from update current replica count 1 desired count 3 and state Complete

OpenShiftScaler will wait 10 seconds, then try to scale again

OpenShiftScaler setting desired replica count of 3 on mongodb-1

OpenShiftScaler rc returned from update current replica count 1 desired count 3 and state Complete

Comment 1 Gabe Montero 2016-11-08 16:08:12 UTC
With that particular sample, you *must* build the frontend BC first, otherwise the image stream origin-nodejs-sample does not have the latest tag.

Simply instantiating the template does not achieve that.

I have verified this myself locally.

Add an `openshiftBuild` step prior to the scale step and try again.

Comment 2 Dongbo Yan 2016-11-09 01:53:22 UTC
Actually, I start build before I scale deployment using jenkins, the image stream origin-nodejs-sample has the latest tag.
Sorry for confusing you due to I don't describe reproduce steps in detail.

And an additional info, "oc scale rc" does not work in ocp3.4

Comment 3 Gabe Montero 2016-11-09 16:19:08 UTC
Thanks for the clarification.

First, any perceived issue wrt "oc scale rc" is a separate bug that should be opened against platform management.

I now believe I see the issue in the plugin.  It looks like the API endpoint it is using for scaling no longer matches what `oc scale dc` is doing (it uses the "scale" endpoint).  As a result, there appears to be a problem with scaling beyond 1 replica (the master reports items like this:

I1109 15:48:13.177711    2148 replication_controller.go:323] Observed updated replication controller frontend-1. Desired pod count change: 1->3
I1109 15:48:13.198725    2148 event.go:217] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"myproject", Name:"frontend-1", UID:"d4fd8ecb-a692-11e6-8b95-68f7287398a7", APIVersion:"v1", ResourceVersion:"1721", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: frontend-1-szjln
I1109 15:48:13.198769    2148 event.go:217] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"myproject", Name:"frontend-1", UID:"d4fd8ecb-a692-11e6-8b95-68f7287398a7", APIVersion:"v1", ResourceVersion:"1721", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: frontend-1-fr5c4
I1109 15:48:13.262391    2148 replication_controller.go:323] Observed updated replication controller frontend-1. Desired pod count change: 3->1
I1109 15:48:13.358607    2148 replication_controller.go:323] Observed updated replication controller frontend-1. Desired pod count change: 1->1

That said, the plumbing is there in the restclient to leverage the scale endpoint.  I've started on a fix.

Comment 4 Gabe Montero 2016-11-09 19:08:00 UTC
I have verified the plugin fix locally with commit https://github.com/openshift/jenkins-plugin/commit/7d6de7f52d8ff45e828cc6a96cc4649c7cbad86b

The next steps are:
- cut a new version (v1.0.32) of the plugin at jenkinsci / jenkins download center
- build a new version of the jenkins-1-centos7 image

The problem should exist with both centos and rhel.  If QA can verify on centos, great.  If not, we'll need to wait until I can work with Troy Dawson to cut new rpms for v1.0.32 of the plugin, and then spin up new versions of the jenkins rhel image on brew pulp.

Comment 5 Gabe Montero 2016-11-09 22:10:20 UTC
PR https://github.com/openshift/jenkins/pull/191 is initiating the update of the jenkins centos image with v1.0.32 of the plugin, and Troy Dawson in parallel is updating the RHEL image on brew pulp with the associated RPM.

Barring any unforseen mishap, the images should be available for QE to verify when they arrive Thursday AM China time.

Just to me safe though, go to the manage plugin page after pulling down and starting up the latest image and verify that openshift-pipeline is at v1.0.32.  If it is not post that info here and we'll see what is up.

thanks

Comment 6 Dongbo Yan 2016-11-10 06:44:23 UTC
Test with
brew-pulp.../openshift3/jenkins-2-rhel7           20e80cb29407     brew-pulp.../openshift3/jenkins-1-rhel7           91ba1b4bff1d
openshift-login    0.8
openshift-pipeline 1.0.32	
openshift-sync     0.0.15

openshift v3.4.0.23+24b1a58
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

scale deployment successfully using jenkins

Comment 8 errata-xmlrpc 2017-01-18 12:50:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.