Bug 1566665 - jenkins k8s plugin does not spawn slaves with "jenkins:2" image
Summary: jenkins k8s plugin does not spawn slaves with "jenkins:2" image
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.10.0
Assignee: Ben Parees
QA Contact: Hongkai Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-12 18:05 UTC by Hongkai Liu
Modified: 2018-04-26 17:45 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-26 17:45:19 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Hongkai Liu 2018-04-12 18:05:44 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
# oc new-project ttt
# oc new-app --template=jenkins-persistent -p ENABLE_OAUTH=false -p MEMORY_LIMIT=4096Mi -p VOLUME_CAPACITY=1000Gi

# oc get dc jenkins -o yaml | grep image:
        image: registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:889b6746b0a2041a119d2e65e682f7614d230a5cab14200e20c497ae9f863c83

# yum list installed | grep openshift
atomic-openshift.x86_64         3.9.13-1.git.0.e0acf74.el7

Then create 30 free style jobs and start building them.

Actual results:
No slaves are spawned by k8s plugin.
All builds are performed on jenkins master.


Expected results:
10 slaves are spawned by k8s plugin as configured.


Additional info:
The same test works fine when using "jenkins:1" image
I can see them both on the Jenkins UI and backend via "oc get pod -w"

Comment 1 Ben Parees 2018-04-13 04:00:57 UTC
How did you configure your jobs to use slave pods?

Comment 2 Hongkai Liu 2018-04-13 13:18:35 UTC
I did not configure anything for slaves in the job level.
I think that is the k8s plugin to make it happen which is setup out of box in the Jenkins global configuration. I did not change anything there.
See the attachment.

Jenkins master has 5 executors and k8s plugin can spawn at most 10 slaves according to the default configuration.
With "jenkins:1" image, the Jenkins behaves exactly as expected when I trigger 30 jobs at the same time.

Comment 4 Gabe Montero 2018-04-13 13:38:25 UTC
Please provide a screen shot of the pod template config, not just the cloud config.

Also provide the screen shot of one of the jobs you say spawns a slave with the jenkins 1 image.

Comment 5 Hongkai Liu 2018-04-13 14:49:19 UTC
jenkins:1
# oc new-project ttt
# oc new-app --template=jenkins-persistent -p ENABLE_OAUTH=false -p MEMORY_LIMIT=4096Mi -p VOLUME_CAPACITY=1000Gi -p JENKINS_IMAGE_STREAM_TAG=jenkins:1


# oc get dc -o yaml | grep image:
          image: registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:5a113fb6e551473a399a073e0d69ab878fe7b6a99a9c27cf626194791f60f628


### Those are my job definitions with JJB
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/projects.yaml
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/defaults.yaml
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/job-template.yaml

###After creating 30 jobs with JJB, trigger the jobs with bash

readonly JENKINS_URL=$(oc get route -n ttt --no-headers | awk '{print $2}')

function trigger()
{
  local url
  url=$1
  local job_name
  job_name=$2
  curl -k --user admin:password -X POST "https://${url}/job/${job_name}/build" --data-urlencode json='{"parameter": []}'
}

for i in $(seq 0 29); do trigger "${JENKINS_URL}" "test-${i}_job"; done


###The slaves are created automatically:
# oc get pod -w
NAME                READY     STATUS              RESTARTS   AGE
jenkins-1-bnfhg     1/1       Running             0          7m
jjb-1-sshk2         1/1       Running             0          5m
maven-7faeff1bf3f   0/1       ContainerCreating   0          3s
maven-7fd35891bba   0/1       Pending   0         0s
maven-7fd35891bba   0/1       Pending   0         0s
maven-7fd35891bba   0/1       ContainerCreating   0         0s
maven-7fd36119469   0/1       Pending   0         0s
maven-7fd36119469   0/1       Pending   0         0s
maven-7fd36119469   0/1       ContainerCreating   0         0s
maven-7ff8a63d1a3   0/1       Pending   0         0s
maven-7ff8a63d1a3   0/1       Pending   0         0s
maven-7ff8b66aa92   0/1       Pending   0         0s
maven-7ff8a63d1a3   0/1       ContainerCreating   0         0s
maven-7ff8b66aa92   0/1       Pending   0         0s
maven-7ff8b66aa92   0/1       ContainerCreating   0         0s
maven-801ddd57605   0/1       Pending   0         0s
maven-801ddd57605   0/1       Pending   0         0s
maven-801de85c543   0/1       Pending   0         0s
maven-801ddd57605   0/1       ContainerCreating   0         0s
maven-801de85c543   0/1       Pending   0         0s
maven-801de85c543   0/1       ContainerCreating   0         0s
maven-7faeff1bf3f   1/1       Running   0         34s
maven-7fd36119469   1/1       Running   0         25s
maven-7fd35891bba   1/1       Running   0         25s
maven-7ff8a63d1a3   1/1       Running   0         16s
maven-7ff8b66aa92   1/1       Running   0         16s
maven-801ddd57605   1/1       Running   0         7s
maven-80432256f0e   0/1       Pending   0         0s
maven-80432256f0e   0/1       Pending   0         0s
maven-804330093a8   0/1       Pending   0         0s
maven-80432256f0e   0/1       ContainerCreating   0         0s
maven-804330093a8   0/1       Pending   0         0s
maven-804330093a8   0/1       ContainerCreating   0         0s
maven-80432256f0e   1/1       Running   0         6s
maven-801de85c543   1/1       Running   0         26s
maven-804330093a8   1/1       Running   0         17s
maven-808da7a9bbf   0/1       Pending   0         0s
maven-808da7a9bbf   0/1       Pending   0         0s
maven-808da7a9bbf   0/1       ContainerCreating   0         0s
maven-808dcaa4e91   0/1       Pending   0         0s
maven-808dcaa4e91   0/1       Pending   0         0s
maven-808dcaa4e91   0/1       ContainerCreating   0         0s
maven-808ddecabda   0/1       Pending   0         0s
maven-808ddecabda   0/1       Pending   0         0s
maven-808ddecabda   0/1       ContainerCreating   0         0s
maven-808dcaa4e91   1/1       Running   0         3s
maven-808ddecabda   1/1       Running   0         4s
maven-808da7a9bbf   1/1       Running   0         4s

This can be verified on the Jenkins UI: See the screenshots
http://file.rdu.redhat.com/~hongkliu/test_result/bz1566665/

Other screenshots are there too.


When I did the same test with
# oc new-app --template=jenkins-persistent -p ENABLE_OAUTH=false -p MEMORY_LIMIT=4096Mi -p VOLUME_CAPACITY=1000Gi -p JENKINS_IMAGE_STREAM_TAG=jenkins:2

All builds are done on Jenkins master, no slaves are created.

Comment 6 Ben Parees 2018-04-26 03:42:26 UTC
please provide your jenkins job definitions so we can actually confirm your jobs are configured to use slave labels that are defined in the default jenkins:2 image.

better still, zip up your entire jenkins config directory from your jenkins:2 instance (including config.xml and job definitions) and provide it to us.

Comment 7 Ben Parees 2018-04-26 04:06:06 UTC
also nothing about this screenshot: http://file.rdu.redhat.com/~hongkliu/test_result/bz1566665/job.config.Screenshot%20from%202018-04-13%2010-44-17.png

tells me your job is actually configured to run on a slave.

I would expect you to have configured the "restrict where this job can run" setting, such as seen here:
https://imgur.com/ouSahR0

Comment 8 Hongkai Liu 2018-04-26 13:14:50 UTC
The JJB job definitions are in Comment 5.

We can download the file into `jobs` folder and use JJB to generate the jobs on Jenkins:

jenkins-jobs --flush-cache  update --delete-old jobs

I used the same job definition files for both jenkins:2 and jenkins:1. Are we expecting different behaviors between them? Because jenkins:1 works while jenkins:2 not.

All jenkins config are out of the box. I did not change any.

I will create the Jenkins instance again today and upload the files you want. Will also try the "restrict label" to see if it makes any difference. But I did not set it up for jenkins:1 either.

Comment 9 Ben Parees 2018-04-26 13:30:12 UTC
> I used the same job definition files for both jenkins:2 and jenkins:1. Are we expecting different behaviors between them? Because jenkins:1 works while jenkins:2 not.

I would not expect it to be different, but since jenkins:1 is deprecated, I am only interested in getting your workflow correct on jenkins:2, and I suspect that requires that you explicitly set a restrict label on your job if you do not want the jobs to execute on your master.

You could also try setting the numExecutors to 0 in your jenkins master which *might* be sufficient to force jenkins to instantiate slaves, but you'd have no control over what slave type got instantiated, assuming that is even what would happen.

Comment 10 Hongkai Liu 2018-04-26 13:34:59 UTC
I will try numExecutors as 0 too. Thanks.

Comment 11 Hongkai Liu 2018-04-26 17:40:20 UTC
Tested with same image:
#  oc get dc jenkins -o yaml | grep image:
        image: registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:889b6746b0a2041a119d2e65e682f7614d230a5cab14200e20c497ae9f863c83


both tricks work.

In order to get maven slaves created "automatically" by k8s plugin, we need either

restrict the job with label: (it can run on both master and maven nodes)
master || maven

or

set executor number as 0

Either of them is sufficient.

Those are new for jenkins:2 image.

This maybe a new design of jenkins or a bz.

But this is good enough for my test automation.

Thanks, Ben.

Comment 12 Ben Parees 2018-04-26 17:45:19 UTC
yw, glad we could get you going again.


Note You need to log in before you can comment on or make changes to this bug.