Bug 1754968

Summary: java path is missing in jenkins image version v3.11.146
Product: OpenShift Container Platform Reporter: Abhishek <aabhishe>
Component: JenkinsAssignee: Akram Ben Aissi <abenaiss>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.11.0CC: afleuran, ajohn, apaladug, calfonso, hieumoscow89, ljenkin, vbobade, wsun
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-18 01:34:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1755076    

Description Abhishek 2019-09-24 12:59:35 UTC 
Description of problem:

$ oc new-app --allow-missing-images registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11.146
    * This image will be deployed in deployment config "jenkins-2-rhel7"

--> Creating resources ...
    deploymentconfig.apps.openshift.io "jenkins-2-rhel7" created
--> Success
    Run 'oc status' to view your app.
$ oc get po
NAME                       READY     STATUS              RESTARTS   AGE
jenkins-2-rhel7-1-deploy   0/1       ContainerCreating   0          4s

$ oc get po
NAME                      READY     STATUS    RESTARTS   AGE
jenkins-2-rhel7-1-7v96t   1/1       Running   0          5s

$ oc log --tail=10 jenkins-2-rhel7-1-7v96t
log is DEPRECATED and will be removed in a future version. Use logs instead.
Generating kubernetes-plugin credentials (/var/lib/jenkins/credentials.xml.tpl) ...
Copying Jenkins configuration to /var/lib/jenkins ...
Creating initial Jenkins 'admin' user ...
config.xml
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found

$ oc get po
NAME                      READY     STATUS             RESTARTS   AGE
jenkins-2-rhel7-1-7v96t   0/1       CrashLoopBackOff   4          2m


$ oc debug jenkins-2-rhel7-1-7v96t
Defaulting container name to jenkins-2-rhel7.
Use 'oc describe pod/jenkins-2-rhel7-1-7v96t-debug -n jenkins' to see all of the containers in this pod.

Debugging with pod/jenkins-2-rhel7-1-7v96t-debug, original command: <image entrypoint>
Waiting for pod to start ...
If you don't see a command prompt, try pressing enter.


sh-4.2$ ls -la /usr/bin/java
ls: cannot access /usr/bin/java: No such file or directory
sh-4.2$ ls -la /etc/alternatives/java sym
ls: cannot access sym: No such file or directory
lrwxrwxrwx. 1 root root 73 Sep 18 02:23 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java
Comment 2 Akram Ben Aissi 2019-09-24 17:38:55 UTC 
*** Bug 1755076 has been marked as a duplicate of this bug. ***
Comment 3 XiuJuan Wang 2019-09-26 11:21:55 UTC 
When using jenkins template `jenkins-persistent` and `jenkins-ephemeral` single, jenkind pod could be running well, can't reproduce this issue.
When set java var environment 'OPENSHIFT_JENKINS_JVM_ARCH' which enable 'alternatives', could reproduce this error.
Comment 4 XiuJuan Wang 2019-09-26 11:31:23 UTC 
Supply cmds for #comment 3
$ oc new-app jenkins-persistent
$oc set env dc/jenkins OPENSHIFT_JENKINS_JVM_ARCH=i386

# oc log -f jenkins-2-qbnsx  -n xiu4
log is DEPRECATED and will be removed in a future version. Use logs instead.
OpenShift client and server versions are oc v3.11.146
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.30.0.1:443
openshift v3.11.146
kubernetes v1.11.0+d4cacc0
alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='3072', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1536m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found
Comment 5 Anand Paladugu 2019-09-27 17:05:07 UTC 
I have another customer (02482642) that reported this issue and they are saying that with memory limit set to less than 2 GB, the deployment works fine.

I will create a KCS in the interim with that workaround.
Comment 6 Anand Paladugu 2019-10-01 17:14:46 UTC 
@XiuJuan Wang   

My customer is stating that they have not changed the ARCH varibale. it was there empty '' by default and they are seeing the issue. They did change the memory variable.

OpenShift client and server versions are oc v3.11.146
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.30.0.1:443 
openshift v3.11.69
kubernetes v1.11.0+d4cacc0
alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found
Comment 7 Anand Paladugu 2019-10-02 15:54:13 UTC 
@Akram,  @XiuJuan Wang  

Any update that you can share ? Customer is looking for an advice/ETA etc..
Comment 8 Akram Ben Aissi 2019-10-02 16:00:42 UTC 
Hi @Anand,
the 3.11 image has been reverted. Your customer must re-run an oc import-image to have its image stream updated
Comment 9 Anand Paladugu 2019-10-02 21:39:30 UTC 
Thanks @Akram.  Will ask customer to do so and keep you posted.
Comment 10 Anand Paladugu 2019-10-07 15:28:12 UTC 
@Akram

Customers IS set to update update and he is still seeing only the image from 13 days ago for Jenkins 2 tag (OC describe of IS below).  Does revert mean they have to go back to one revision as opposed to using the latest ?
Thanks
Anand





[root@87879d808cf2 /]# oc describe is jenkins -n openshift
Name:                   jenkins
Namespace:              openshift
Created:                19 months ago
Labels:                 <none>
Annotations:            openshift.io/display-name=Jenkins
                        openshift.io/image.dockerRepositoryCheck=2019-10-03T12:34:17Z
Docker Pull Spec:       docker-registry.default.svc:5000/openshift/jenkins
Image Lookup:           local=false
Unique Images:          20
Tags:                   3

1
  tagged from registry.redhat.io/openshift3/jenkins-1-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 1.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: hidden, jenkins

  * registry.redhat.io/openshift3/jenkins-1-rhel7@sha256:99b16fde353f4b215afbeda2857050010792f3567e21757da55b1a9770084c3f
      11 months ago
    registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:827823c87288e955e493d76d5ad7a39dd800a05d5555c24fad954b268b61b5a4
      12 months ago
    registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:ccb6f09b52e2f8633b65e1bfde18bab62fa7b01f6f0c1e406f6869c461384502
      18 months ago
    registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:f72be1f64dcb10122476c5f1e34567708c13737022be1d6e31aa30b58638fa06
      19 months ago
    registry.access.redhat.com/openshift3/jenkins-1-rhel7@sha256:ccb6f09b52e2f8633b65e1bfde18bab62fa7b01f6f0c1e406f6869c461384502
      19 months ago

2 (latest)
  updates automatically from registry registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  * registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
      13 days ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:386f7a43af90a79eabe521be05fc6c2fad873c14b6dd53bbac38dc5418837f39
      4 weeks ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:a33dbdce3ffd886d3aa1d2b0c259a3e55f56dc8feaf4b0f2d96189ccb63c42ee
      5 weeks ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:9606a564cce21147bfdb7effbb83a6c7337351ee12633bac44402540c1f6fb98
      5 months ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:9aee6b0423712b4bee3094b315dd42186e6776a4484ec17e20a83ab6f35e9859
      8 months ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:adcdd0319e87503d6e730923c9f807469ebf2ef97303dd6ed816be8ed7cd6470
      9 months ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:6c6a437e1394d3d8b5a7bdfb35c4146190e37eb9008c5d8ec66e77a5a3f40b74
      10 months ago
    registry.redhat.io/openshift3/jenkins-2-rhel7@sha256:bd8ae92065ee1c7a58ce49ab9bf16ce2113920a1684eda88a4cf80cdac96e978
      11 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:67279ed65f2d4f1d2fa68406965b974381f25ff0c8ebfa862d0b55c4abaafd96
      12 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:45453170e8aedb2648e8ac67d65c435f2c32d7d4c09a1ea4e193e18233d7f3d2
      12 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:5b9576eb491e2c21b267d372d934ee2200b4207bbb1c308233626a54c198b28e
      12 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:04628e825b33838d2a95c208e3c6e5210945679ec2d44d932a929a08b4f2497a
      16 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:d6f64bfc8c2f6a03e16d02627ad6f2859ac741709aa3be80fc2048570f4a4618
      16 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:9d9fd30f8cdf87c89b96e01e8b17786cd0643929b0621b719ab6b1353c8a5722
      17 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:889b6746b0a2041a119d2e65e682f7614d230a5cab14200e20c497ae9f863c83
      18 months ago
    registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:fc9c7f4b766fe8203a9d792e4a7f99c8962b31da158591f62d3d26ea3904c641
      19 months ago
[root@87879d808cf2 /]#
Comment 11 Anand Paladugu 2019-10-07 21:26:09 UTC 
Raising severity as another Sev 1 issue is opened by Openshift Dedicated customer. Case 02489116.

New build of Jenkins Master using common base image from:
    registry.access.redhat.com/openshift3/jenkins-2-rhel7:v3.11
or     imagestream tag openshift/jenkins:2
in OpenShift Dedicate with versions:
- OpenShift Master: v3.11.43 
- Kubernetes Master: v1.11.0+d4cacc0 
- OpenShift Web Console: v3.11.43 
fails at startup with error:
```
Booting Jenkins ...
....
....
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
/usr/libexec/s2i/run: line 434: [: bhAMzY:a8e040f3e6888d082e84660bab835e211270738404ae7fbb0d318c1184e746b7: unary operator expected
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -Dhudson.tasks.MailSender.SEND_TO_UNKNOWN_USERS=true -Dhudson.tasks.MailSender.SEND_TO_USERS_WITHOUT_READ=true -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found
Comment 12 Hieu Nguyen Nhu 2019-10-08 09:39:57 UTC 
Hi Team,
One of my customer is hitting the same problem and the demo at https://github.com/siamaksade/openshift-cd-demo would not work because if this issue.
Can you help us to point to the Jenkin tag that works please.

OpenShift client and server versions are oc v3.11.146
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.30.0.1:443 
openshift v3.11.129
kubernetes v1.11.0+d4cacc0
alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found
Comment 13 Abhishek 2019-10-08 09:45:38 UTC 
Hi Hieu Nguyen Nhu,

Working image:

registry.redhat.io/openshift3/jenkins-2-rhel7:v3.11.141
Comment 15 XiuJuan Wang 2019-10-09 03:22:52 UTC 
Can't reproduce this issue with latest jenkins-2-rhel7, which points v3.11.152 version.
$skopeo inspect docker://registry.reg-aws.openshift.com:443/openshift3/jenkins-2-rhel7:v3.11   | grep  version  
        "version": "v3.11.152"

The java binary is included in jenkins-2-rhel7:v3.11.152 version

$oc new-app openshift/jenkins:2  -n xiu
# oc get pods  -n xiu 
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-7qhrl   1/1       Running   0          23m
[root@qe-xiuwang-311merrn-1 ~]# oc rsh -n xiu  jenkins-1-7qhrl 
sh-4.2$ ls -la /usr/bin/java
lrwxrwxrwx. 1 default root 22 Oct  9 02:54 /usr/bin/java -> /etc/alternatives/java


$oc new-app jenkins-persistent  --env=OPENSHIFT_JENKINS_JVM_ARCH=x86_64 -n xiu1

# oc get pods -n xiu1
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-n5cxh   1/1       Running   0          4m
# oc rsh -n xiu1 jenkins-1-n5cxh 
sh-4.2$ ls
bin  boot  dev	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var
sh-4.2$ ls -la /usr/bin/java
lrwxrwxrwx. 1 default root 22 Oct  9 03:05 /usr/bin/java -> /etc/alternatives/java

$oc new-app jenkins-persistent  -p MEMORY_LIMIT=2Gi -n xiu2 
# oc get pods -n xiu2
NAME              READY     STATUS    RESTARTS   AGE
jenkins-2-rv8dl   1/1       Running   0          3m

oc v3.11.152
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-xiuwang-311merrn-1:8443
openshift v3.11.152
kubernetes v1.11.0+d4cacc0
Comment 16 Anand Paladugu 2019-10-14 01:30:03 UTC 
@ XiuJuan Wang.

Do you know if the new version is available to shift hosted customers ?

Thanks

Anand
Comment 19 errata-xmlrpc 2019-10-18 01:34:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3139