Bug 1550193 - openshift jenkins rhel image release to release migration not working
Summary: openshift jenkins rhel image release to release migration not working
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.0
Assignee: Gabe Montero
QA Contact: Wenjing Zheng
Depends On:
TreeView+ depends on / blocked
Reported: 2018-02-28 18:25 UTC by Gabe Montero
Modified: 2018-06-27 18:02 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The transfer of plugins from the RPM install location to the Jenkins home directory were not occurring properly with the openshfit v2 jenkin rhel image when jenkins was previously deployed on an openshift pod with a persistent volume Consequence: an upgrade of the openshfit v2 jenkin rhel image would not result in the deployment having the most recently plugins associated with the newer image Fix: the openshfit v2 jenkin rhel image run script has been updated to properly transfer the plugins Result: an upgrade of the openshfit v2 jenkin rhel image will result in the deployment having the most recently plugins associated with the newer image
Clone Of:
Last Closed: 2018-06-27 18:01:32 UTC
Target Upstream Version:

Attachments (Terms of Use)
openshift plugin versions after upgrade from 3.7 to 3.9 (33.83 KB, image/png)
2018-03-06 09:52 UTC, Wenjing Zheng
no flags Details
Openshift plugin version before import (51.97 KB, image/png)
2018-03-08 06:15 UTC, Wenjing Zheng
no flags Details
Openshift plugin version after import (38.89 KB, image/png)
2018-03-08 06:16 UTC, Wenjing Zheng
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2013 None None None 2018-06-27 18:02:05 UTC

Description Gabe Montero 2018-02-28 18:25:00 UTC
Per discussions between Ben and I, commit ecea65fc1f3f0c2525043154bea389ce07c9199b for fixing https://github.com/openshift/jenkins/issues/451 ended up removing our /var/lib/jenkins/plugins to /usr/lib/jenkins links on initial startup.

In addition to correcting that situation across the applicable releases (3.7, possibly 3.6), I've realized (and Ben agrees) that we have a situation where

a) user upgrades a plugin our image pulls in to version X when an openshift version Y
b) whoever, in openshift versions Y+1, we happen to upgrade that plugin to versions X+1
c) we we want to override the user's upgrade to X with our X+1

Comment 1 Gabe Montero 2018-02-28 22:52:15 UTC
PR https://github.com/openshift/jenkins/pull/516 has been submitted and is under review

Comment 2 Gabe Montero 2018-03-03 02:37:42 UTC
OK, the referenced PR has merged.

Please pull an openshift jenkins rhel image from the brew-pulp / CD pipeline that has commmit 4b4c6ee00f0ae953e06e691339f900c4ee16a444 to verify.

Reminder on upgrade scenario :

a) have to start a jenkins deployment at 3.6 or 3.7 of the openshift jenkins rhel image on a PV

b) upgrade the image stream with an image container the above commit 

c) confirm that all the openshift pluings versions are 3.9 levels (would all be past v1.0.0)

Comment 3 Wenjing Zheng 2018-03-06 09:51:29 UTC
Hi, Gabe, I have verified this bug with below steps, but found one issue that openshift plugin are updated after image update, but not installed with past v1.0.0 version, is it on purpose?
1. Import image from registry.access.redhat.com/openshift3/jenkins-2-rhel7;
2. Create resources with jenkins-persistent template with above is;
3. Import image from brew registry;
4. Log into jenkins console to check openshift plugin versions.

Please refer to the attachment, thanks!

Comment 4 Wenjing Zheng 2018-03-06 09:52:14 UTC
Created attachment 1404704 [details]
openshift plugin versions after upgrade from 3.7 to 3.9

Comment 5 Gabe Montero 2018-03-06 14:16:07 UTC
Hey Wenjing, 

The image to verify is not at registry.access.redhat.com/openshift3/jenkins-2-rhel7 yet

You need to pull it from brew-pulp, making sure you get the brew-pulp image containing commit 4b4c6ee00f0ae953e06e691339f900c4ee16a444

Comment 6 Wenjing Zheng 2018-03-07 02:10:43 UTC
I am using latest brew-pulp image, isn't this bug about use old image first then import an image with fix to check plugin version? please see my step 3, I import image from brew-pulp.

Comment 7 Gabe Montero 2018-03-07 14:35:35 UTC
Sorry, I should have elaborated .... as part of importing the image, did you explicitly update that "2" tag for the jenkins image stream?  It was unclear to me from your steps.

OK, so aside from answering that question, I'll ask for more debug information in case there is an issue to look into, and to make sure things have lined up like I would expect.

I'll need this information:

1) the precise sha/tag for the brew pulp message you imported - I'll make sure it has the changes I did

2) the output from oc get is jenkins -o yaml .... I can confirm that the jenkins 2 tag was updated

3) oc get events after you import the image ... make sure a new deployment was kicked off after you imported the image 

4) the jenkins pod logs after a new deployment was generated as a result of updating the image stream

Comment 8 Wenjing Zheng 2018-03-08 06:13:29 UTC
Thanks, Gabe! It work now after upgrade, please see the new attachment, I will verify this bug now.


Comment 9 Wenjing Zheng 2018-03-08 06:15:23 UTC
Created attachment 1405703 [details]
Openshift plugin version before import

Comment 10 Wenjing Zheng 2018-03-08 06:16:03 UTC
Created attachment 1405704 [details]
Openshift plugin version after import

Comment 11 Gabe Montero 2018-03-08 14:33:07 UTC
Great thanks Wenjing !

I took a peek and confirmed:

- The "after" plugin screen shot you provided looks good
- I downloaded the image from brewpulp and verified it has the updated run script

Comment 13 errata-xmlrpc 2018-06-27 18:01:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.