Bug 1757938

Summary: When launching jenkins-ephemeral or jenkins- persistent on OpenShift v3.11.104 the container fails to start and states that java is not available
Product: OpenShift Container Platform Reporter: Tosin Akinosho <takinosh>
Component: JenkinsAssignee: Akram Ben Aissi <abenaiss>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: calfonso, vbobade, yuxzhu
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-17 09:01:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tosin Akinosho 2019-10-02 18:46:44 UTC 
Description of problem:  when we run oc new-app jenkins-ephemeral --param=MEMORY_LIMIT="2Gi"


Version-Release number of selected component (if applicable):


How reproducible: Consistent in currrent build 


Steps to Reproduce:
1. Start an openshift project on 
2. run oc new-app jenkins-ephemeral --param=MEMORY_LIMIT="2Gi" or oc new-app jenkins- persistent
3.

Actual results: 
]$ oc get pods
NAME                           READY     STATUS             RESTARTS   AGE
catalog-3-6v2w7                1/1       Running            0          55m
catalog-postgresql-1-zdflf     1/1       Running            0          1h
catalog-s2i-1-build            0/1       Completed          0          2h
gateway-3-c9sfh                1/1       Running            0          15m
gateway-s2i-1-build            0/1       Completed          0          2h
gateway-s2i-2-build            0/1       Completed          0          15m
inventory-3-t22dv              1/1       Running            0          55m
inventory-postgresql-1-bdjr8   1/1       Running            0          1h
inventory-s2i-1-build          0/1       Completed          0          3h
jenkins-1-deploy               1/1       Running            0          5m
jenkins-1-kk9d5                0/1       CrashLoopBackOff   5          5m
web-1-build                    0/1       Completed          0          2h
web-2-klsfw                    1/1       Running            0          52m
[user@workspace88ymycuwzpw7ihb8 inventory-thorntail]$ oc logs jenkins-1-kk9d5
OpenShift client and server versions are oc v3.11.146
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.30.0.1:443
openshift v3.11.104
kubernetes v1.11.0+d4cacc0
alternatives version 1.7.4 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <link> <name> <path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list

common options: --verbose --test --help --usage --version --keep-missing
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='2048', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.4.11-1.el7_7.x86_64/bin/javac
/usr/local/bin/jenkins-common.sh: line 33: java: command not found
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Administrative monitors that contact the update center will remain active
Detected password environment variable change, updating Jenkins configuration ...
config.xml
Migrating slave image configuration to current version tag ...
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx1024m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
/usr/libexec/s2i/run: line 485: exec: java: not found


Expected results: Jenkins container should start to run 


Additional info:

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
    template.alpha.openshift.io/wait-for-ready: 'true'
  creationTimestamp: '2019-10-02T18:36:48Z'
  generation: 2
  labels:
    app: jenkins-ephemeral
    template: jenkins-ephemeral-template
  name: jenkins
  namespace: coolstore50
  resourceVersion: '214427'
  selfLink: /apis/apps.openshift.io/v1/namespaces/coolstore50/deploymentconfigs/jenkins
  uid: 97c17dfc-e543-11e9-806f-0a99f7e7ef84
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    name: jenkins
  strategy:
    activeDeadlineSeconds: 21600
    recreateParams:
      timeoutSeconds: 600
    resources: {}
    type: Recreate
  template:
    metadata:
      annotations:
        openshift.io/generated-by: OpenShiftNewApp
      creationTimestamp: null
      labels:
        name: jenkins
    spec:
      containers:
        - env:
            - name: OPENSHIFT_ENABLE_OAUTH
              value: 'true'
            - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT
              value: 'true'
            - name: DISABLE_ADMINISTRATIVE_MONITORS
              value: 'false'
            - name: KUBERNETES_MASTER
              value: 'https://kubernetes.default:443'
            - name: KUBERNETES_TRUST_CERTIFICATES
              value: 'true'
            - name: JENKINS_SERVICE_NAME
              value: jenkins
            - name: JNLP_SERVICE_NAME
              value: jenkins-jnlp
          image: >-
            docker-registry.default.svc:5000/openshift/jenkins@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 2
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 420
            periodSeconds: 360
            successThreshold: 1
            timeoutSeconds: 240
          name: jenkins
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 3
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 240
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 512Mi
          securityContext:
            capabilities: {}
            privileged: false
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/lib/jenkins
              name: jenkins-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: jenkins
      serviceAccountName: jenkins
      terminationGracePeriodSeconds: 30
      volumes:
        - emptyDir: {}
          name: jenkins-data
  test: false
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - jenkins
        from:
          kind: ImageStreamTag
          name: 'jenkins:2'
          namespace: openshift
        lastTriggeredImage: >-
          docker-registry.default.svc:5000/openshift/jenkins@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
      type: ImageChange
    - type: ConfigChange
status:
  availableReplicas: 0
  conditions:
    - lastTransitionTime: '2019-10-02T18:36:48Z'
      lastUpdateTime: '2019-10-02T18:36:48Z'
      message: Deployment config does not have minimum availability.
      status: 'False'
      type: Available
    - lastTransitionTime: '2019-10-02T18:36:51Z'
      lastUpdateTime: '2019-10-02T18:36:51Z'
      message: replication controller "jenkins-1" is progressing
      reason: ReplicationControllerUpdated
      status: 'True'
      type: Progressing
  details:
    causes:
      - type: ConfigChange
    message: config change
  latestVersion: 1
  observedGeneration: 2
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1




apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/deployment-config.latest-version: '1'
    openshift.io/deployment-config.name: jenkins
    openshift.io/deployment.name: jenkins-1
    openshift.io/generated-by: OpenShiftNewApp
    openshift.io/scc: restricted
  creationTimestamp: '2019-10-02T18:36:51Z'
  generateName: jenkins-1-
  labels:
    deployment: jenkins-1
    deploymentconfig: jenkins
    name: jenkins
  name: jenkins-1-kk9d5
  namespace: coolstore50
  ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicationController
      name: jenkins-1
      uid: 97c38978-e543-11e9-806f-0a99f7e7ef84
  resourceVersion: '215987'
  selfLink: /api/v1/namespaces/coolstore50/pods/jenkins-1-kk9d5
  uid: 99b756c9-e543-11e9-806f-0a99f7e7ef84
spec:
  containers:
    - env:
        - name: OPENSHIFT_ENABLE_OAUTH
          value: 'true'
        - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT
          value: 'true'
        - name: DISABLE_ADMINISTRATIVE_MONITORS
          value: 'false'
        - name: KUBERNETES_MASTER
          value: 'https://kubernetes.default:443'
        - name: KUBERNETES_TRUST_CERTIFICATES
          value: 'true'
        - name: JENKINS_SERVICE_NAME
          value: jenkins
        - name: JNLP_SERVICE_NAME
          value: jenkins-jnlp
      image: >-
        docker-registry.default.svc:5000/openshift/jenkins@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 2
        httpGet:
          path: /login
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 420
        periodSeconds: 360
        successThreshold: 1
        timeoutSeconds: 240
      name: jenkins
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /login
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 3
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 240
      resources:
        limits:
          cpu: '2'
          memory: 2Gi
        requests:
          cpu: 200m
          memory: 512Mi
      securityContext:
        capabilities:
          drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        privileged: false
        runAsUser: 1003570000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/lib/jenkins
          name: jenkins-data
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: jenkins-token-6p5hb
          readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
    - name: jenkins-dockercfg-4n98s
  nodeName: node1.delta-b1bf.internal
  nodeSelector:
    node-role.kubernetes.io/compute: 'true'
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1003570000
    seLinuxOptions:
      level: 's0:c60,c15'
  serviceAccount: jenkins
  serviceAccountName: jenkins
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoSchedule
      key: node.kubernetes.io/memory-pressure
      operator: Exists
  volumes:
    - emptyDir: {}
      name: jenkins-data
    - name: jenkins-token-6p5hb
      secret:
        defaultMode: 420
        secretName: jenkins-token-6p5hb
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2019-10-02T18:36:51Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2019-10-02T18:36:51Z'
      message: 'containers with unready status: [jenkins]'
      reason: ContainersNotReady
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: 'containers with unready status: [jenkins]'
      reason: ContainersNotReady
      status: 'False'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2019-10-02T18:36:51Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        docker://d84ff1ef29567231f5ee1dc84f27f34397d7f372eb285e794dbe7aed3ec93537
      image: 'sha256:830c74cd09642cfe1aaedf66623921e15fe7c62b2b9da41cc1a25501b8d86290'
      imageID: >-
        docker-pullable://docker-registry.default.svc:5000/openshift/jenkins@sha256:da3488fb622ca393bcdb635e4d824f57708a7dc11db01ab3171c0d38ccb98474
      lastState:
        terminated:
          containerID: >-
            docker://d84ff1ef29567231f5ee1dc84f27f34397d7f372eb285e794dbe7aed3ec93537
          exitCode: 127
          finishedAt: '2019-10-02T18:42:44Z'
          reason: Error
          startedAt: '2019-10-02T18:42:42Z'
      name: jenkins
      ready: false
      restartCount: 6
      state:
        waiting:
          message: >-
            Back-off 5m0s restarting failed container=jenkins
            pod=jenkins-1-kk9d5_coolstore50(99b756c9-e543-11e9-806f-0a99f7e7ef84)
          reason: CrashLoopBackOff
  hostIP: 192.168.0.136
  phase: Running
  podIP: 10.1.4.100
  qosClass: Burstable
  startTime: '2019-10-02T18:36:51Z'
Comment 2 XiuJuan Wang 2019-12-10 09:07:01 UTC 
Verified with image openshift3/jenkins-2-rhel7:v3.11(v3.11.157-1)

$oc new-app jenkins-ephemeral --param=MEMORY_LIMIT="2Gi"
$oc get pods 
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-x6mb7   1/1       Running   0          47m

# oc rsh jenkins-1-x6mb7
sh-4.2$ ls /usr/bin/java -l 
lrwxrwxrwx. 1 default root 22 Dec 10 08:18 /usr/bin/java -> /etc/alternatives/ja
Comment 4 Yuxiang Zhu 2020-01-06 07:37:10 UTC 
Drop BZ1757938 from https://errata.devel.redhat.com/advisory/49981 because QE reports that bug has been fixed in 3.11.157-1.