Bug 1566665

How did you configure your jobs to use slave pods?

I did not configure anything for slaves in the job level.
I think that is the k8s plugin to make it happen which is setup out of box in the Jenkins global configuration. I did not change anything there.
See the attachment.

Jenkins master has 5 executors and k8s plugin can spawn at most 10 slaves according to the default configuration.
With "jenkins:1" image, the Jenkins behaves exactly as expected when I trigger 30 jobs at the same time.

Please provide a screen shot of the pod template config, not just the cloud config.

Also provide the screen shot of one of the jobs you say spawns a slave with the jenkins 1 image.

jenkins:1
# oc new-project ttt
# oc new-app --template=jenkins-persistent -p ENABLE_OAUTH=false -p MEMORY_LIMIT=4096Mi -p VOLUME_CAPACITY=1000Gi -p JENKINS_IMAGE_STREAM_TAG=jenkins:1


# oc get dc -o yaml | grep image:
          image: registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:5a113fb6e551473a399a073e0d69ab878fe7b6a99a9c27cf626194791f60f628


### Those are my job definitions with JJB
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/projects.yaml
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/defaults.yaml
curl -L -O https://raw.githubusercontent.com/hongkailiu/svt-case-doc/1b83b892f04f3e77ab9ff4e5564020b04d9f48c0/docker/jjb/svt_jobs/job-template.yaml

###After creating 30 jobs with JJB, trigger the jobs with bash

readonly JENKINS_URL=$(oc get route -n ttt --no-headers | awk '{print $2}')

function trigger()
{
  local url
  url=$1
  local job_name
  job_name=$2
  curl -k --user admin:password -X POST "https://${url}/job/${job_name}/build" --data-urlencode json='{"parameter": []}'
}

for i in $(seq 0 29); do trigger "${JENKINS_URL}" "test-${i}_job"; done


###The slaves are created automatically:
# oc get pod -w
NAME                READY     STATUS              RESTARTS   AGE
jenkins-1-bnfhg     1/1       Running             0          7m
jjb-1-sshk2         1/1       Running             0          5m
maven-7faeff1bf3f   0/1       ContainerCreating   0          3s
maven-7fd35891bba   0/1       Pending   0         0s
maven-7fd35891bba   0/1       Pending   0         0s
maven-7fd35891bba   0/1       ContainerCreating   0         0s
maven-7fd36119469   0/1       Pending   0         0s
maven-7fd36119469   0/1       Pending   0         0s
maven-7fd36119469   0/1       ContainerCreating   0         0s
maven-7ff8a63d1a3   0/1       Pending   0         0s
maven-7ff8a63d1a3   0/1       Pending   0         0s
maven-7ff8b66aa92   0/1       Pending   0         0s
maven-7ff8a63d1a3   0/1       ContainerCreating   0         0s
maven-7ff8b66aa92   0/1       Pending   0         0s
maven-7ff8b66aa92   0/1       ContainerCreating   0         0s
maven-801ddd57605   0/1       Pending   0         0s
maven-801ddd57605   0/1       Pending   0         0s
maven-801de85c543   0/1       Pending   0         0s
maven-801ddd57605   0/1       ContainerCreating   0         0s
maven-801de85c543   0/1       Pending   0         0s
maven-801de85c543   0/1       ContainerCreating   0         0s
maven-7faeff1bf3f   1/1       Running   0         34s
maven-7fd36119469   1/1       Running   0         25s
maven-7fd35891bba   1/1       Running   0         25s
maven-7ff8a63d1a3   1/1       Running   0         16s
maven-7ff8b66aa92   1/1       Running   0         16s
maven-801ddd57605   1/1       Running   0         7s
maven-80432256f0e   0/1       Pending   0         0s
maven-80432256f0e   0/1       Pending   0         0s
maven-804330093a8   0/1       Pending   0         0s
maven-80432256f0e   0/1       ContainerCreating   0         0s
maven-804330093a8   0/1       Pending   0         0s
maven-804330093a8   0/1       ContainerCreating   0         0s
maven-80432256f0e   1/1       Running   0         6s
maven-801de85c543   1/1       Running   0         26s
maven-804330093a8   1/1       Running   0         17s
maven-808da7a9bbf   0/1       Pending   0         0s
maven-808da7a9bbf   0/1       Pending   0         0s
maven-808da7a9bbf   0/1       ContainerCreating   0         0s
maven-808dcaa4e91   0/1       Pending   0         0s
maven-808dcaa4e91   0/1       Pending   0         0s
maven-808dcaa4e91   0/1       ContainerCreating   0         0s
maven-808ddecabda   0/1       Pending   0         0s
maven-808ddecabda   0/1       Pending   0         0s
maven-808ddecabda   0/1       ContainerCreating   0         0s
maven-808dcaa4e91   1/1       Running   0         3s
maven-808ddecabda   1/1       Running   0         4s
maven-808da7a9bbf   1/1       Running   0         4s

This can be verified on the Jenkins UI: See the screenshots
http://file.rdu.redhat.com/~hongkliu/test_result/bz1566665/

Other screenshots are there too.


When I did the same test with
# oc new-app --template=jenkins-persistent -p ENABLE_OAUTH=false -p MEMORY_LIMIT=4096Mi -p VOLUME_CAPACITY=1000Gi -p JENKINS_IMAGE_STREAM_TAG=jenkins:2

All builds are done on Jenkins master, no slaves are created.

please provide your jenkins job definitions so we can actually confirm your jobs are configured to use slave labels that are defined in the default jenkins:2 image.

better still, zip up your entire jenkins config directory from your jenkins:2 instance (including config.xml and job definitions) and provide it to us.

also nothing about this screenshot: http://file.rdu.redhat.com/~hongkliu/test_result/bz1566665/job.config.Screenshot%20from%202018-04-13%2010-44-17.png

tells me your job is actually configured to run on a slave.

I would expect you to have configured the "restrict where this job can run" setting, such as seen here:
https://imgur.com/ouSahR0

The JJB job definitions are in Comment 5.

We can download the file into `jobs` folder and use JJB to generate the jobs on Jenkins:

jenkins-jobs --flush-cache  update --delete-old jobs

I used the same job definition files for both jenkins:2 and jenkins:1. Are we expecting different behaviors between them? Because jenkins:1 works while jenkins:2 not.

All jenkins config are out of the box. I did not change any.

I will create the Jenkins instance again today and upload the files you want. Will also try the "restrict label" to see if it makes any difference. But I did not set it up for jenkins:1 either.

> I used the same job definition files for both jenkins:2 and jenkins:1. Are we expecting different behaviors between them? Because jenkins:1 works while jenkins:2 not.

I would not expect it to be different, but since jenkins:1 is deprecated, I am only interested in getting your workflow correct on jenkins:2, and I suspect that requires that you explicitly set a restrict label on your job if you do not want the jobs to execute on your master.

You could also try setting the numExecutors to 0 in your jenkins master which *might* be sufficient to force jenkins to instantiate slaves, but you'd have no control over what slave type got instantiated, assuming that is even what would happen.

I will try numExecutors as 0 too. Thanks.

Tested with same image:
#  oc get dc jenkins -o yaml | grep image:
        image: registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:889b6746b0a2041a119d2e65e682f7614d230a5cab14200e20c497ae9f863c83


both tricks work.

In order to get maven slaves created "automatically" by k8s plugin, we need either

restrict the job with label: (it can run on both master and maven nodes)
master || maven

or

set executor number as 0

Either of them is sufficient.

Those are new for jenkins:2 image.

This maybe a new design of jenkins or a bz.

But this is good enough for my test automation.

Thanks, Ben.

yw, glad we could get you going again.