Description of problem: $ oc new-app --allow-missing-images registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11.146 * This image will be deployed in deployment config "jenkins-2-rhel7" --> Creating resources ... deploymentconfig.apps.openshift.io "jenkins-2-rhel7" created --> Success Run 'oc status' to view your app. $ oc get po NAME READY STATUS RESTARTS AGE jenkins-2-rhel7-1-deploy 0/1 ContainerCreating 0 4s $ oc get po NAME READY STATUS RESTARTS AGE jenkins-2-rhel7-1-7v96t 1/1 Running 0 5s $ oc log --tail=10 jenkins-2-rhel7-1-7v96t log is DEPRECATED and will be removed in a future version. Use logs instead. Generating kubernetes-plugin credentials (/var/lib/jenkins/credentials.xml.tpl) ... Copying Jenkins configuration to /var/lib/jenkins ... Creating initial Jenkins 'admin' user ... config.xml Administrative monitors that contact the update center will remain active Detected password environment variable change, updating Jenkins configuration ... config.xml Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war /usr/libexec/s2i/run: line 485: exec: java: not found $ oc get po NAME READY STATUS RESTARTS AGE jenkins-2-rhel7-1-7v96t 0/1 CrashLoopBackOff 4 2m $ oc debug jenkins-2-rhel7-1-7v96t Defaulting container name to jenkins-2-rhel7. Use 'oc describe pod/jenkins-2-rhel7-1-7v96t-debug -n jenkins' to see all of the containers in this pod. Debugging with pod/jenkins-2-rhel7-1-7v96t-debug, original command: <image entrypoint> Waiting for pod to start ... If you don't see a command prompt, try pressing enter. sh-4.2$ ls -la /usr/bin/java ls: cannot access /usr/bin/java: No such file or directory sh-4.2$ ls -la /etc/alternatives/java sym ls: cannot access sym: No such file or directory lrwxrwxrwx. 1 root root 73 Sep 18 02:23 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java
*** Bug 1755076 has been marked as a duplicate of this bug. ***
When using jenkins template `jenkins-persistent` and `jenkins-ephemeral` single, jenkind pod could be running well, can't reproduce this issue. When set java var environment 'OPENSHIFT_JENKINS_JVM_ARCH' which enable 'alternatives', could reproduce this error.
Supply cmds for #comment 3 $ oc new-app jenkins-persistent $oc set env dc/jenkins OPENSHIFT_JENKINS_JVM_ARCH=i386 # oc log -f jenkins-2-qbnsx -n xiu4 log is DEPRECATED and will be removed in a future version. Use logs instead. OpenShift client and server versions are oc v3.11.146 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.30.0.1:443 openshift v3.11.146 kubernetes v1.11.0+d4cacc0 alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc. This may be freely redistributed under the terms of the GNU Public License. usage: alternatives --install <link> <name> <path> <priority> [--initscript <service>] [--family <family>] [--slave <link> <name> <path>]* alternatives --remove <name> <path> alternatives --auto <name> alternatives --config <name> alternatives --display <name> alternatives --set <name> <path> alternatives --list common options: --verbose --test --help --usage --version --keep-missing --altdir <directory> --admindir <directory> OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='3072', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac /usr/local/bin/jenkins-common.sh: line 33: java: command not found mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists Administrative monitors that contact the update center will remain active Detected password environment variable change, updating Jenkins configuration ... config.xml Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1536m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war /usr/libexec/s2i/run: line 485: exec: java: not found
I have another customer (02482642) that reported this issue and they are saying that with memory limit set to less than 2 GB, the deployment works fine. I will create a KCS in the interim with that workaround.
@XiuJuan Wang My customer is stating that they have not changed the ARCH varibale. it was there empty '' by default and they are seeing the issue. They did change the memory variable. OpenShift client and server versions are oc v3.11.146 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.30.0.1:443 openshift v3.11.69 kubernetes v1.11.0+d4cacc0 alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc. This may be freely redistributed under the terms of the GNU Public License. usage: alternatives --install <link> <name> <path> <priority> [--initscript <service>] [--family <family>] [--slave <link> <name> <path>]* alternatives --remove <name> <path> alternatives --auto <name> alternatives --config <name> alternatives --display <name> alternatives --set <name> <path> alternatives --list common options: --verbose --test --help --usage --version --keep-missing --altdir <directory> --admindir <directory> OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac /usr/local/bin/jenkins-common.sh: line 33: java: command not found mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists Administrative monitors that contact the update center will remain active Detected password environment variable change, updating Jenkins configuration ... config.xml Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war /usr/libexec/s2i/run: line 485: exec: java: not found
@Akram, @XiuJuan Wang Any update that you can share ? Customer is looking for an advice/ETA etc..
Hi @Anand, the 3.11 image has been reverted. Your customer must re-run an oc import-image to have its image stream updated
Thanks @Akram. Will ask customer to do so and keep you posted.
@Akram Customers IS set to update update and he is still seeing only the image from 13 days ago for Jenkins 2 tag (OC describe of IS below). Does revert mean they have to go back to one revision as opposed to using the latest ? Thanks Anand [root@87879d808cf2 /]# oc describe is jenkins -n openshift Name: jenkins Namespace: openshift Created: 19 months ago Labels: <none> Annotations: openshift.io/display-name=Jenkins openshift.io/image.dockerRepositoryCheck=2019-10-03T12:34:17Z Docker Pull Spec: docker-registry.default.svc:5000/openshift/jenkins Image Lookup: local=false Unique Images: 20 Tags: 3 1 tagged from registry.redhat.io/openshift3/jenkins-1-rhel7:latest prefer registry pullthrough when referencing this tag Provides a Jenkins 1.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: hidden, jenkins * registry.redhat.io/openshift3/jenkins-1-rhel7@sha256:99b16fde353f4b215afbeda2857050010792f3567e21757da55b1a9770084c3f 11 months ago registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:827823c87288e955e493d76d5ad7a39dd800a05d5555c24fad954b268b61b5a4 12 months ago registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:ccb6f09b52e2f8633b65e1bfde18bab62fa7b01f6f0c1e406f6869c461384502 18 months ago registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:f72be1f64dcb10122476c5f1e34567708c13737022be1d6e31aa30b58638fa06 19 months ago registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:ccb6f09b52e2f8633b65e1bfde18bab62fa7b01f6f0c1e406f6869c461384502 19 months ago 2 (latest) updates automatically from registry registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11 prefer registry pullthrough when referencing this tag Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: jenkins * registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474 13 days ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39 4 weeks ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:a33dbdce3ffd886d3aa1d2b0c259a3e55f56dc8feaf4b0f2d96189ccb63c42ee 5 weeks ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:9606a564cce21147bfdb7effbb83a6c7337351ee12633bac44402540c1f6fb98 5 months ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:9aee6b0423712b4bee3094b315dd42186e6776a4484ec17e20a83ab6f35e9859 8 months ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:adcdd0319e87503d6e730923c9f807469ebf2ef97303dd6ed816be8ed7cd6470 9 months ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:6c6a437e1394d3d8b5a7bdfb35c4146190e37eb9008c5d8ec66e77a5a3f40b74 10 months ago registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:bd8ae92065ee1c7a58ce49ab9bf16ce2113920a1684eda88a4cf80cdac96e978 11 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:67279ed65f2d4f1d2fa68406965b974381f25ff0c8ebfa862d0b55c4abaafd96 12 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:45453170e8aedb2648e8ac67d65c435f2c32d7d4c09a1ea4e193e18233d7f3d2 12 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:5b9576eb491e2c21b267d372d934ee2200b4207bbb1c308233626a54c198b28e 12 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:04628e825b33838d2a95c208e3c6e5210945679ec2d44d932a929a08b4f2497a 16 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:d6f64bfc8c2f6a03e16d02627ad6f2859ac741709aa3be80fc2048570f4a4618 16 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:9d9fd30f8cdf87c89b96e01e8b17786cd0643929b0621b719ab6b1353c8a5722 17 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:889b6746b0a2041a119d2e65e682f7614d230a5cab14200e20c497ae9f863c83 18 months ago registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:fc9c7f4b766fe8203a9d792e4a7f99c8962b31da158591f62d3d26ea3904c641 19 months ago [root@87879d808cf2 /]#
Raising severity as another Sev 1 issue is opened by Openshift Dedicated customer. Case 02489116. New build of Jenkins Master using common base image from: registry.access.redhat.com/openshift3/jenkins-2-rhel7:v3.11 or imagestream tag openshift/jenkins:2 in OpenShift Dedicate with versions: - OpenShift Master: v3.11.43 - Kubernetes Master: v1.11.0+d4cacc0 - OpenShift Web Console: v3.11.43 fails at startup with error: ``` Booting Jenkins ... .... .... This may be freely redistributed under the terms of the GNU Public License. usage: alternatives --install <link> <name> <path> <priority> [--initscript <service>] [--family <family>] [--slave <link> <name> <path>]* alternatives --remove <name> <path> alternatives --auto <name> alternatives --config <name> alternatives --display <name> alternatives --set <name> <path> alternatives --list common options: --verbose --test --help --usage --version --keep-missing --altdir <directory> --admindir <directory> OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac /usr/local/bin/jenkins-common.sh: line 33: java: command not found mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists Administrative monitors that contact the update center will remain active /usr/libexec/s2i/run: line 434: [: bhAMzY:a8e040f3e6888d082e84660bab835e211270738404ae7fbb0d318c1184e746b7: unary operator expected Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -Dhudson.tasks.MailSender.SEND_TO_UNKNOWN_USERS=true -Dhudson.tasks.MailSender.SEND_TO_USERS_WITHOUT_READ=true -jar /usr/lib/jenkins/jenkins.war /usr/libexec/s2i/run: line 485: exec: java: not found
Hi Team, One of my customer is hitting the same problem and the demo at https://github.com/siamaksade/openshift-cd-demo would not work because if this issue. Can you help us to point to the Jenkin tag that works please. OpenShift client and server versions are oc v3.11.146 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.30.0.1:443 openshift v3.11.129 kubernetes v1.11.0+d4cacc0 alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc. This may be freely redistributed under the terms of the GNU Public License. usage: alternatives --install <link> <name> <path> <priority> [--initscript <service>] [--family <family>] [--slave <link> <name> <path>]* alternatives --remove <name> <path> alternatives --auto <name> alternatives --config <name> alternatives --display <name> alternatives --set <name> <path> alternatives --list common options: --verbose --test --help --usage --version --keep-missing --altdir <directory> --admindir <directory> OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac /usr/local/bin/jenkins-common.sh: line 33: java: command not found mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists Administrative monitors that contact the update center will remain active Detected password environment variable change, updating Jenkins configuration ... config.xml Migrating slave image configuration to current version tag ... + exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war /usr/libexec/s2i/run: line 485: exec: java: not found
Hi Hieu Nguyen Nhu, Working image: registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11.141
Can't reproduce this issue with latest jenkins-2-rhel7, which points v3.11.152 version. $skopeo inspect docker://registry.reg-aws.openshift.com:443/openshift3/jenkins-2-rhel7:v3.11 | grep version "version": "v3.11.152" The java binary is included in jenkins-2-rhel7:v3.11.152 version $oc new-app openshift/jenkins:2 -n xiu # oc get pods -n xiu NAME READY STATUS RESTARTS AGE jenkins-1-7qhrl 1/1 Running 0 23m [root@qe-xiuwang-311merrn-1 ~]# oc rsh -n xiu jenkins-1-7qhrl sh-4.2$ ls -la /usr/bin/java lrwxrwxrwx. 1 default root 22 Oct 9 02:54 /usr/bin/java -> /etc/alternatives/java $oc new-app jenkins-persistent --env=OPENSHIFT_JENKINS_JVM_ARCH=x86_64 -n xiu1 # oc get pods -n xiu1 NAME READY STATUS RESTARTS AGE jenkins-1-n5cxh 1/1 Running 0 4m # oc rsh -n xiu1 jenkins-1-n5cxh sh-4.2$ ls bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var sh-4.2$ ls -la /usr/bin/java lrwxrwxrwx. 1 default root 22 Oct 9 03:05 /usr/bin/java -> /etc/alternatives/java $oc new-app jenkins-persistent -p MEMORY_LIMIT=2Gi -n xiu2 # oc get pods -n xiu2 NAME READY STATUS RESTARTS AGE jenkins-2-rv8dl 1/1 Running 0 3m oc v3.11.152 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-xiuwang-311merrn-1:8443 openshift v3.11.152 kubernetes v1.11.0+d4cacc0
@ XiuJuan Wang. Do you know if the new version is available to shift hosted customers ? Thanks Anand
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3139