Bug 1995727 - sync plugin no longer catches build deletes that occur between restarts
Summary: sync plugin no longer catches build deletes that occur between restarts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Jenkins
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.9.0
Assignee: Gabe Montero
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-19 17:32 UTC by Gabe Montero
Modified: 2021-10-18 17:47 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the optimizations introduced by v1.0.47 and v1.0.48 of the sync plugin removed explicit listing of watched resources on startup Consequence: any deletes of resources that occurred between the stop and restart of a jenkins instance running with a persistent volume would never be processed Fix: explicit list of resources on start up was added back Result: deletes occurring between the stop of a jenkins instance running on a persistent volume and a restart will again be process once the restart has completed.
Clone Of:
Environment:
Last Closed: 2021-10-18 17:47:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift jenkins-sync-plugin pull 397 0 None None None 2021-08-19 20:28:54 UTC
Github openshift jenkins-sync-plugin pull 400 0 None None None 2021-08-25 21:10:00 UTC
Github openshift jenkins-sync-plugin pull 401 0 None None None 2021-08-26 19:34:40 UTC
Github openshift jenkins pull 1314 0 None None None 2021-08-23 14:57:39 UTC
Github openshift jenkins pull 1316 0 None None None 2021-08-26 19:38:22 UTC
Red Hat Issue Tracker ART-3260 0 None None None 2021-08-23 15:05:49 UTC
Red Hat Issue Tracker ART-3283 0 None None None 2021-08-26 20:44:10 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:47:42 UTC

Description Gabe Montero 2021-08-19 17:32:21 UTC
the recent scaling optimizations removed the list on startup calls for minimally builds (I'll check other resources as well).

however, we need those to capture any deletes of resources that occurred between the stopping of jenkins and the restart of jenkins when the persistent template.

Our current tests had a gap at https://github.com/openshift/origin/blob/master/test/extended/builds/pipeline_jenkins_e2e.go#L260-L272 where the deletes following the scale call could still have been caught by jenkins and sync plugin before jenkins was shut down.

The new in repo e2e's I've been working on caught this error.

I have verified it manually in addition to the e2e's catching it.

Comment 1 Gabe Montero 2021-08-23 15:06:22 UTC
linked ART req for sync plugin RPM:  https://issues.redhat.com/browse/ART-3260

Comment 3 Gabe Montero 2021-08-25 20:24:08 UTC
hold on verification Jitendar ... may have found an issue with the fix for this

Comment 4 Gabe Montero 2021-08-26 15:37:01 UTC
I took a look at the attached customer case https://access.redhat.com/support/cases/#/case/02781060 and I do not believe it is directly related to this bugzilla.

https://bugzilla.redhat.com/show_bug.cgi?id=1925524 is really the most pertinent BZ ... that one is targeted to our current in development release 4.9
Once verification has completed there, I believe it will be backported.

I see from the case that customer is a 4.4.  Any backports will not go that far back, as 4.4 is no longer in service.  Most likely 4.7 will be the oldest release backkports go to.

Certainly getting this BZ on top would not hurt them, but they don't need to wait for it per se.

Comment 5 Gabe Montero 2021-08-26 17:23:07 UTC
still need to attach new jenkins prs after cutting new version of plugin

Comment 8 Jitendar Singh 2021-09-02 06:40:57 UTC
VERIFIED
=======================
 jsingh@fugaku  ~  oc new-project jenkins-test                                                                                        

Now using project "jenkins-test" on server "https://api.dev-svc-4.9-090201.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

 jsingh@fugaku  ~  
 jsingh@fugaku  ~  oc import-image --confirm jenkins-jitsingh --from=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18a5a81edbd705105c821b83a8f27e7fdf387c18600845c7bbb42536ba8fe4df
imagestream.image.openshift.io/jenkins-jitsingh imported

Name:			jenkins-jitsingh
Namespace:		jenkins-test
Created:		Less than a second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2021-09-02T05:07:52Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/jenkins-test/jenkins-jitsingh
Image Lookup:		local=false
Unique Images:		1
Tags:			1

latest
  tagged from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18a5a81edbd705105c821b83a8f27e7fdf387c18600845c7bbb42536ba8fe4df

  * quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18a5a81edbd705105c821b83a8f27e7fdf387c18600845c7bbb42536ba8fe4df
      Less than a second ago

Image Name:	jenkins-jitsingh:latest
Docker Image:	quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18a5a81edbd705105c821b83a8f27e7fdf387c18600845c7bbb42536ba8fe4df
Name:		sha256:18a5a81edbd705105c821b83a8f27e7fdf387c18600845c7bbb42536ba8fe4df
Created:	Less than a second ago
Annotations:	image.openshift.io/dockerLayersOrder=ascending
Image Size:	569.7MB in 6 layers
Layers:		83.37MB	sha256:296e14ee24149e14d573a1fbf5b5a625c7bb0cc22f5b2a8b180e833258187948
		1.792kB	sha256:356f18f3a935b2f226093720b65383048249413ed99da72c87d5be58cc46661c
		2.15MB	sha256:0205529b95f0dcdfb886541f6922cc01b1d968da889a6d51cbfec38e50af0b35
		11.18MB	sha256:1e7e9b5e7bd96e67d552215052ac5172b3f46ec1d44acee24bf19d933aed365c
		47.16MB	sha256:6cb68e7c0df1a8944b21201c66c2c5345124cc732c820de4484b75e2ad3e3697
		425.8MB	sha256:0ee8e7d8fa1968a5f355c56b8bc14e632b7556e62848ea204eb17e27654e9041
Image Created:	2 days ago
Author:		OpenShift Developer Services <openshift-dev-services+jenkins>
Arch:		amd64
Entrypoint:	/usr/bin/go-init -main /usr/libexec/s2i/run
Working Dir:	<none>
User:		1001
Exposes Ports:	50000/tcp, 8080/tcp
Docker Labels:	License=GPLv2+
		architecture=x86_64
		build-date=2021-08-30T17:11:57.229165
		com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com
		com.redhat.component=openshift-jenkins-2-container
		com.redhat.license_terms=https://www.redhat.com/agreements
		description=Jenkins is a continuous integration server
		distribution-scope=public
		io.jenkins.version=2.289.2
		io.k8s.description=Jenkins is a continuous integration server
		io.k8s.display-name=Jenkins 2
		io.openshift.build.commit.id=732eddb3ade5e334472e5d2cf6eeb1449daa304d
		io.openshift.build.commit.url=https://github.com/openshift/jenkins/commit/732eddb3ade5e334472e5d2cf6eeb1449daa304d
		io.openshift.build.source-location=https://github.com/openshift/jenkins
		io.openshift.expose-services=8080:http
		io.openshift.maintainer.component=Jenkins
		io.openshift.maintainer.product=OpenShift Container Platform
		io.openshift.s2i.scripts-url=image:///usr/libexec/s2i
		io.openshift.tags=jenkins,jenkins2,ci
		maintainer=Red Hat, Inc.
		name=openshift/ose-jenkins
		release=202108301627.p0.git.732eddb.assembly.stream
		summary=Provides the latest release of Red Hat Universal Base Image 8.
		url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-jenkins/images/v4.9.0-202108301627.p0.git.732eddb.assembly.stream
		vcs-ref=eaf5169e1808d7e19660d3362610b456f71c894b
		vcs-type=git
		vendor=Red Hat, Inc.
		version=v4.9.0
Environment:	__doozer=merge
		BUILD_RELEASE=202108301627.p0.git.732eddb.assembly.stream
		BUILD_VERSION=v4.9.0
		OS_GIT_MAJOR=4
		OS_GIT_MINOR=9
		OS_GIT_PATCH=0
		OS_GIT_TREE_STATE=clean
		OS_GIT_VERSION=4.9.0-202108301627.p0.git.732eddb.assembly.stream-732eddb
		SOURCE_GIT_TREE_STATE=clean
		OS_GIT_COMMIT=732eddb
		SOURCE_DATE_EPOCH=1630016503
		SOURCE_GIT_COMMIT=732eddb3ade5e334472e5d2cf6eeb1449daa304d
		SOURCE_GIT_TAG=732eddb
		SOURCE_GIT_URL=https://github.com/openshift/jenkins
		JENKINS_VERSION=2
		HOME=/var/lib/jenkins
		JENKINS_HOME=/var/lib/jenkins
		JENKINS_UC=https://updates.jenkins.io
		OPENSHIFT_JENKINS_IMAGE_VERSION=4.9
		LANG=en_US.UTF-8
		LC_ALL=en_US.UTF-8
		INSTALL_JENKINS_VIA_RPMS=true
		KUBE_GIT_COMMIT=14f2b35396ba8c5ce215cec2f91e04af616614c6
		KUBE_GIT_MAJOR=1
		KUBE_GIT_MINOR=13+
		KUBE_GIT_TREE_STATE=clean
		KUBE_GIT_VERSION=v1.13.0+14f2b35
		GODEBUG=x509ignoreCN=0,madvdontneed=1
		PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
		container=oci
Volumes:	/var/lib/jenkins


 jsingh@fugaku  ~  oc new-app jenkins-ephemeral -p NAMESPACE=$(oc project -q) -p JENKINS_IMAGE_STREAM_TAG=jenkins-jitsingh:latest                                                          
--> Deploying template "openshift/jenkins-ephemeral" to project jenkins-test

     Jenkins (Ephemeral)
     ---------
     Jenkins service, without persistent storage.
     
     WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing.

     A Jenkins service has been created in your project.  Log into Jenkins with your OpenShift account.  The tutorial at https://github.com/openshift/origin/blob/master/examples/jenkins/README.md contains more information about using this template.

     * With parameters:
        * Jenkins Service Name=jenkins
        * Jenkins JNLP Service Name=jenkins-jnlp
        * Enable OAuth in Jenkins=true
        * Memory Limit=1Gi
        * Jenkins ImageStream Namespace=jenkins-test
        * Disable memory intensive administrative monitors=false
        * Jenkins ImageStreamTag=jenkins-jitsingh:latest
        * Allows use of Jenkins Update Center repository with invalid SSL certificate=false

--> Creating resources ...
    route.route.openshift.io "jenkins" created
    configmap "jenkins-trusted-ca-bundle" created
    deploymentconfig.apps.openshift.io "jenkins" created
    serviceaccount "jenkins" created
    rolebinding.authorization.openshift.io "jenkins_edit" created
    service "jenkins-jnlp" created
    service "jenkins" created
--> Success
    Access your application via route 'jenkins-jenkins-test.apps.dev-svc-4.9-090201.devcluster.openshift.com' 
    Run 'oc status' to view your app.
 jsingh@fugaku  ~  oc get pods -w
NAME               READY   STATUS              RESTARTS   AGE
jenkins-1-655dx    0/1     ContainerCreating   0          5s
jenkins-1-deploy   1/1     Running             0          8s
jenkins-1-655dx    0/1     Running             0          28s
jenkins-1-655dx    1/1     Running             0          90s
jenkins-1-deploy   0/1     Completed           0          93s
^C%                                                                                                                                                                                            ✘ jsingh@fugaku  ~  oc rsh jenkins-1-655dx
sh-4.4$ cat /var/lib/jenkins/plugins/config-file-provider/META-INF/MANIFEST.MF |grep Implementation-Version
Implementation-Version: 3.7.1
sh-4.4$ cat /var/lib/jenkins/plugins/openshift-sync/META-INF/MANIFEST.MF |grep Implementation-Version
Implementation-Version: 1.0.50
sh-4.4$ exit
========================================
 jsingh@fugaku  /usr/local/go/src/github.com/openshift/verification-tests   master ±  oc get builds -w
NAME                            TYPE              FROM     STATUS      STARTED              DURATION
openshift-jee-sample-3          JenkinsPipeline            Complete    13 minutes ago       
openshift-jee-sample-4          JenkinsPipeline            Complete    10 minutes ago       
openshift-jee-sample-5          JenkinsPipeline            Complete    8 minutes ago        
openshift-jee-sample-6          JenkinsPipeline            Complete    5 minutes ago        
openshift-jee-sample-8          JenkinsPipeline            Cancelled   3 minutes ago        
openshift-jee-sample-docker-3   Docker            Binary   Complete    12 minutes ago       59s
openshift-jee-sample-docker-4   Docker            Binary   Complete    9 minutes ago        1m1s
openshift-jee-sample-docker-5   Docker            Binary   Complete    7 minutes ago        59s
openshift-jee-sample-docker-6   Docker            Binary   Complete    4 minutes ago        1m2s
openshift-jee-sample-9          JenkinsPipeline            Complete    2 minutes ago        
openshift-jee-sample-10         JenkinsPipeline            Cancelled                        
openshift-jee-sample-11         JenkinsPipeline            Running     14 seconds ago       
openshift-jee-sample-12         JenkinsPipeline            New                              
openshift-jee-sample-docker-7   Docker            Binary   Complete    About a minute ago   58s
openshift-jee-sample-11         JenkinsPipeline            Running     23 seconds ago       
openshift-jee-sample-docker-8   Docker            Binary   New                              
openshift-jee-sample-docker-8   Docker            Binary   Pending                          
openshift-jee-sample-11         JenkinsPipeline            Running     38 seconds ago       
openshift-jee-sample-docker-8   Docker            Binary   Running     3 seconds ago        
openshift-jee-sample-docker-8   Docker            Binary   Running     3 seconds ago        
openshift-jee-sample-11         JenkinsPipeline            Running     43 seconds ago       
openshift-jee-sample-11         JenkinsPipeline            Running     About a minute ago   
openshift-jee-sample-docker-8   Docker            Binary   Running     57 seconds ago       
openshift-jee-sample-docker-8   Docker            Binary   Running     58 seconds ago       
openshift-jee-sample-docker-8   Docker            Binary   Complete    58 seconds ago       58s
openshift-jee-sample-docker-3   Docker            Binary   Complete    13 minutes ago       59s
openshift-jee-sample-11         JenkinsPipeline            Running     About a minute ago   
openshift-jee-sample-11         JenkinsPipeline            Running     2 minutes ago        
openshift-jee-sample-11         JenkinsPipeline            Complete    2 minutes ago        
openshift-jee-sample-3          JenkinsPipeline            Complete    15 minutes ago       
openshift-jee-sample-12         JenkinsPipeline            Pending                          
openshift-jee-sample-11         JenkinsPipeline            Complete    2 minutes ago        
openshift-jee-sample-12         JenkinsPipeline            Running     2 seconds ago        
openshift-jee-sample-12         JenkinsPipeline            Running     12 seconds ago       
openshift-jee-sample-12         JenkinsPipeline            Running     22 seconds ago       
openshift-jee-sample-12         JenkinsPipeline            Running     32 seconds ago       
openshift-jee-sample-docker-9   Docker            Binary   New                              
openshift-jee-sample-docker-9   Docker            Binary   Pending                          
openshift-jee-sample-docker-9   Docker            Binary   Running     3 seconds ago        
openshift-jee-sample-docker-9   Docker            Binary   Running     3 seconds ago        
openshift-jee-sample-12         JenkinsPipeline            Running     37 seconds ago       
openshift-jee-sample-12         JenkinsPipeline            Running     42 seconds ago       
openshift-jee-sample-12         JenkinsPipeline            Running     About a minute ago   
openshift-jee-sample-docker-9   Docker            Binary   Running     56 seconds ago       
openshift-jee-sample-docker-9   Docker            Binary   Running     56 seconds ago       
openshift-jee-sample-docker-9   Docker            Binary   Complete    57 seconds ago       57s
openshift-jee-sample-docker-4   Docker            Binary   Complete    13 minutes ago       1m1s
openshift-jee-sample-12         JenkinsPipeline            Running     About a minute ago   
openshift-jee-sample-12         JenkinsPipeline            Running     2 minutes ago        
openshift-jee-sample-12         JenkinsPipeline            Complete    2 minutes ago        
openshift-jee-sample-4          JenkinsPipeline            Complete    15 minutes ago       
openshift-jee-sample-12         JenkinsPipeline            Complete    2 minutes ago

Comment 11 errata-xmlrpc 2021-10-18 17:47:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.