Bug 1473538 - Failed to deploy jenkins pod on an Overlay2 openshift cluster
Summary: Failed to deploy jenkins pod on an Overlay2 openshift cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: Lokesh Mandvekar
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On: 1469661
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-21 06:45 UTC by Dongbo Yan
Modified: 2017-11-28 22:05 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-11-28 22:05:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Dongbo Yan 2017-07-21 06:45:26 UTC
Description of problem:
Failed to deploy jenkins pod on openshift cluster behind proxy, it will succeed on a cluster without proxy

Version-Release number of selected component (if applicable):
openshift v3.6.153
kubernetes v1.6.1+5115d708d7

brew-pulp.../openshift3/jenkins-2-rhel7@sha256:81c097633ec45835ffbdc12c27eb6010dca10cffe32d3cca2dfa2b661c820511

How reproducible:
Always

Steps to Reproduce:
1.Deploy a jenkins pod via template
 $ oc new-app jenkins-ephemeral
2.Check pod status
3.

Actual results:
# oc get pod -w
NAME               READY     STATUS    RESTARTS   AGE
jenkins-1-deploy   1/1       Running   0          1m
jenkins-1-w09g1   0/1       CrashLoopBackOff   2         56s

Expected results:
Could deploy jenkins pod successfully

Additional info:
# oc logs -f jenkins-1-w09g1
OPENSHIFT_JENKINS_JVM_ARCH is set to i386 so using 32 bit Java
Error: dl failure on line 893
Error: failed /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/server/libjvm.so, because /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/server/libjvm.so: cannot restore segment prot after reloc: Permission denied
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
Detected password environment variable change, updating Jenkins configuration ...
Error: dl failure on line 893
Error: failed /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/server/libjvm.so, because /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/server/libjvm.so: cannot restore segment prot after reloc: Permission denied

Comment 6 Jan Safranek 2017-07-24 12:46:21 UTC
This is definitely not Kubernetes problem. On openshift-127.lab.sjc.redhat.com (cluster A node) this fails:

$ docker run -ti  -e OPENSHIFT_ENABLE_OAUTH=true -e OPENSHIFT_ENABLE_REDIRECT_PROMPT=true -e OPENSHIFT_JENKINS_JVM_ARCH=i386 -e KUBERNETES_MASTER=https://kubernetes.default:443 -e KUBERNETES_TRUST_CERTIFICATES=true -e JNLP_SERVICE_NAME=jenkins-jnlp brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/jenkins-2-rhel7@sha256:eaa6f51c632f7efe4b21181790126cf8dc6a25db5a0ad2628725ee9970e10908

OPENSHIFT_JENKINS_JVM_ARCH is set to i386 so using 32 bit Java
Error: dl failure on line 893
Error: failed /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.i386/jre/lib/i386/server/libjvm.so, because /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.i386/jre/lib/i386/server/libjvm.so: cannot restore segment prot after reloc: Permission denied


On openshift-120.lab.sjc.redhat.com (cluster B node), it succeeds:

OPENSHIFT_JENKINS_JVM_ARCH is set to i386 so using 32 bit Java
Generating kubernetes-plugin configuration (/opt/openshift/configuration/config.xml.tpl) ...
Copying Jenkins configuration to /var/lib/jenkins ...
Copying 49 Jenkins plugins to /var/lib/jenkins ...
Creating initial Jenkins 'admin' user ...
Running from: /usr/lib/jenkins/jenkins.war

Comment 7 Ben Parees 2017-07-24 12:56:00 UTC
Note to whomever is investigating this that using the x86_64 JVM seems to work ok with that same image (the image contains both JVMs).

Comment 8 Qian Cai 2017-07-24 12:57:50 UTC
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/
> server/libjvm.so, because
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/
> server/libjvm.so: cannot restore segment prot after reloc: Permission denied
> mkdir: cannot create directory ‘/var/lib/jenkins/logs’: File exists
> Detected password environment variable change, updating Jenkins
> configuration ...
> Error: dl failure on line 893
> Error: failed
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/
> server/libjvm.so, because
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.i386/jre/lib/i386/
> server/libjvm.so: cannot restore segment prot after reloc: Permission denied

This looks more like a container-selinux issue that we just fixed recently.

https://bugzilla.redhat.com/show_bug.cgi?id=1469661

You can confirm it by updating the package to container-selinux-2.21-1.el7.noarch
and retry it.

Comment 9 Dongbo Yan 2017-07-25 02:21:17 UTC
After updating to container-selinux-2.21-1.el7.noarch on cluster node, I can deploy jenkins pod successfully

Thanks for caiqian's help

Comment 12 errata-xmlrpc 2017-11-28 22:05:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.