Bug 1550193

Summary: openshift jenkins rhel image release to release migration not working
Product: OpenShift Container Platform Reporter: Gabe Montero <gmontero>
Component: ImageAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, bparees, gmontero, jokerman, mmccomas, wzheng, xtian
Target Milestone: ---   
Target Release: 3.9.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The transfer of plugins from the RPM install location to the Jenkins home directory were not occurring properly with the openshfit v2 jenkin rhel image when jenkins was previously deployed on an openshift pod with a persistent volume Consequence: an upgrade of the openshfit v2 jenkin rhel image would not result in the deployment having the most recently plugins associated with the newer image Fix: the openshfit v2 jenkin rhel image run script has been updated to properly transfer the plugins Result: an upgrade of the openshfit v2 jenkin rhel image will result in the deployment having the most recently plugins associated with the newer image
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 18:01:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
openshift plugin versions after upgrade from 3.7 to 3.9
Openshift plugin version before import
Openshift plugin version after import none

Description Gabe Montero 2018-02-28 18:25:00 UTC
Per discussions between Ben and I, commit ecea65fc1f3f0c2525043154bea389ce07c9199b for fixing https://github.com/openshift/jenkins/issues/451 ended up removing our /var/lib/jenkins/plugins to /usr/lib/jenkins links on initial startup.

In addition to correcting that situation across the applicable releases (3.7, possibly 3.6), I've realized (and Ben agrees) that we have a situation where

a) user upgrades a plugin our image pulls in to version X when an openshift version Y
b) whoever, in openshift versions Y+1, we happen to upgrade that plugin to versions X+1
c) we we want to override the user's upgrade to X with our X+1

Comment 1 Gabe Montero 2018-02-28 22:52:15 UTC
PR https://github.com/openshift/jenkins/pull/516 has been submitted and is under review

Comment 2 Gabe Montero 2018-03-03 02:37:42 UTC
OK, the referenced PR has merged.

Please pull an openshift jenkins rhel image from the brew-pulp / CD pipeline that has commmit 4b4c6ee00f0ae953e06e691339f900c4ee16a444 to verify.

Reminder on upgrade scenario :

a) have to start a jenkins deployment at 3.6 or 3.7 of the openshift jenkins rhel image on a PV

b) upgrade the image stream with an image container the above commit 

c) confirm that all the openshift pluings versions are 3.9 levels (would all be past v1.0.0)

Comment 3 Wenjing Zheng 2018-03-06 09:51:29 UTC
Hi, Gabe, I have verified this bug with below steps, but found one issue that openshift plugin are updated after image update, but not installed with past v1.0.0 version, is it on purpose?
1. Import image from registry.access.redhat.com/openshift3/jenkins-2-rhel7;
2. Create resources with jenkins-persistent template with above is;
3. Import image from brew registry;
4. Log into jenkins console to check openshift plugin versions.

Please refer to the attachment, thanks!

Comment 4 Wenjing Zheng 2018-03-06 09:52:14 UTC
Created attachment 1404704 [details]
openshift plugin versions after upgrade from 3.7 to 3.9

Comment 5 Gabe Montero 2018-03-06 14:16:07 UTC
Hey Wenjing, 

The image to verify is not at registry.access.redhat.com/openshift3/jenkins-2-rhel7 yet

You need to pull it from brew-pulp, making sure you get the brew-pulp image containing commit 4b4c6ee00f0ae953e06e691339f900c4ee16a444

Comment 6 Wenjing Zheng 2018-03-07 02:10:43 UTC
I am using latest brew-pulp image, isn't this bug about use old image first then import an image with fix to check plugin version? please see my step 3, I import image from brew-pulp.

Comment 7 Gabe Montero 2018-03-07 14:35:35 UTC
Sorry, I should have elaborated .... as part of importing the image, did you explicitly update that "2" tag for the jenkins image stream?  It was unclear to me from your steps.

OK, so aside from answering that question, I'll ask for more debug information in case there is an issue to look into, and to make sure things have lined up like I would expect.

I'll need this information:

1) the precise sha/tag for the brew pulp message you imported - I'll make sure it has the changes I did

2) the output from oc get is jenkins -o yaml .... I can confirm that the jenkins 2 tag was updated

3) oc get events after you import the image ... make sure a new deployment was kicked off after you imported the image 

4) the jenkins pod logs after a new deployment was generated as a result of updating the image stream

Comment 8 Wenjing Zheng 2018-03-08 06:13:29 UTC
Thanks, Gabe! It work now after upgrade, please see the new attachment, I will verify this bug now.


Comment 9 Wenjing Zheng 2018-03-08 06:15:23 UTC
Created attachment 1405703 [details]
Openshift plugin version before import

Comment 10 Wenjing Zheng 2018-03-08 06:16:03 UTC
Created attachment 1405704 [details]
Openshift plugin version after import

Comment 11 Gabe Montero 2018-03-08 14:33:07 UTC
Great thanks Wenjing !

I took a peek and confirmed:

- The "after" plugin screen shot you provided looks good
- I downloaded the image from brewpulp and verified it has the updated run script

Comment 13 errata-xmlrpc 2018-06-27 18:01:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.