Bug 1293877

Summary: Postgresql pod is CrashLoopBackOff after add nfs volume to dc/rc
Product: OpenShift Container Platform Reporter: Chao Yang <chaoyang>
Component: ImageStreamsAssignee: Ben Parees <bparees>
Status: CLOSED ERRATA QA Contact: Wang Haoran <haowang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, bleanhar, jokerman, mmccomas, pruan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-26 19:20:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chao Yang 2015-12-23 10:43:34 UTC
Description of problem:
postgresql pod could not running after add nfs volume to dc/rc

Version-Release number of selected component (if applicable):
openshift v3.1.0.4
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

How reproducible:
always

Steps to Reproduce:
1.create nfs volume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs3
spec:
  capacity:
    storage: 10Gi
  accessModes:
     - ReadWriteMany
  nfs:
    path: /chaotest
    server: 10.14.6.144

2. oc new-app -i openshift/postgresql -e POSTGRESQL_USER=tester,POSTGRESQL_PASSWORD=xxx,POSTGRESQL_DATABASE=testdb --name=mydb
3.
oc volume dc/mydb --add --type=persistentVolumeClaim --claim-mode='ReadWriteMany' --claim-name='nfsc-chao4' --claim-size=5 --mount-path=/opt
4.check pv and pvc
NAME         LABELS    STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
nfsc-chao4   <none>    Bound     nfs3      10Gi       RWX           50m
5. check the pod status
Actual results:
pod is CrashLoopBackOff

oc get pods mydb-2-xqtfb -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"chao4","name":"mydb-2","uid":"ea4a4060-a925-11e5-a37f-fa163e40ea54","apiVersion":"v1","resourceVersion":"69668"}}
    openshift.io/deployment-config.latest-version: "2"
    openshift.io/deployment-config.name: mydb
    openshift.io/deployment.name: mydb-2
    openshift.io/generated-by: OpenShiftNewApp
    openshift.io/scc: restricted
  creationTimestamp: 2015-12-23T03:33:32Z
  generateName: mydb-2-
  labels:
    app: mydb
    deployment: mydb-2
    deploymentconfig: mydb
  name: mydb-2-xqtfb
  namespace: chao4
  resourceVersion: "69994"
  selfLink: /api/v1/namespaces/chao4/pods/mydb-2-xqtfb
  uid: f0d06bc2-a925-11e5-a37f-fa163e40ea54
spec:
  containers:
  - env:
    - name: POSTGRESQL_DATABASE
      value: testdb
    - name: POSTGRESQL_PASSWORD
      value: xxx
    - name: POSTGRESQL_USER
      value: tester
    image: rcm-img-docker01.build.eng.bos.redhat.com:5001/rhscl/postgresql-94-rhel7:latest
    imagePullPolicy: Always
    name: mydb
    ports:
    - containerPort: 5432
      protocol: TCP
    resources: {}
    securityContext:
      privileged: false
      runAsUser: 1000930000
      seLinuxOptions:
        level: s0:c31,c0
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/lib/pgsql/data
      name: mydb-volume-1
    - mountPath: /opt
      name: volume-obcaj
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-ebvyl
      readOnly: true
  dnsPolicy: ClusterFirst
  host: openshift-111.lab.eng.nay.redhat.com
  imagePullSecrets:
  - name: default-dockercfg-gf35z
  nodeName: openshift-111.lab.eng.nay.redhat.com
  restartPolicy: Always
  securityContext:
    seLinuxOptions:
      level: s0:c31,c0
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - emptyDir: {}
    name: mydb-volume-1
  - name: volume-obcaj
    persistentVolumeClaim:
      claimName: nfsc-chao4
  - name: default-token-ebvyl
    secret:
      secretName: default-token-ebvyl
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2015-12-23T03:38:44Z
    message: 'containers with unready status: [mydb]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  containerStatuses:
  - containerID: docker://4bbaa696b15d081f71916cf16e694121484419efe51b811bb50c489fec4f8cd2
    image: rcm-img-docker01.build.eng.bos.redhat.com:5001/rhscl/postgresql-94-rhel7:latest
    imageID: docker://0d9ea62a74e95b2d3772c53ea257983f822945492f9d364e642b57187a7273f6
    lastState:
      terminated:
        containerID: docker://4bbaa696b15d081f71916cf16e694121484419efe51b811bb50c489fec4f8cd2
        exitCode: 127
        finishedAt: 2015-12-23T03:38:43Z
        reason: Error
        startedAt: 2015-12-23T03:38:42Z
    name: mydb
    ready: false
    restartCount: 6
    state:
      waiting:
        message: Back-off 2m40s restarting failed container=mydb pod=mydb-2-xqtfb_chao4
        reason: CrashLoopBackOff
  hostIP: 10.66.79.111
  phase: Running
  podIP: 10.1.0.250
  startTime: 2015-12-23T03:33:32Z


Expected results:
pod should be running

Additional info:
1.
-bash-4.2# docker ps -a | grep postgresql-94
45daf04e03d3        rcm-img-docker01.build.eng.bos.redhat.com:5001/rhscl/postgresql-94-rhel7:latest                                  "container-entrypoint"   About a minute ago   Exited (127) About a minute ago                       k8s_mydb.b23d22cb_mydb-2-xqtfb_chao4_f0d06bc2-a925-11e5-a37f-fa163e40ea54_c3290e1e

-bash-4.2# docker logs 45daf04e03d3
/usr/bin/scl_source: line 63: /opt/rh/rh-postgresql94/enable: No such file or directory
Can't source enable, skipping.
/usr/share/container-scripts/postgresql/common.sh: line 103: initdb: command not found
2.oc volume --remove dc/mydb --name=volume-obcaj
after above step, pod is running

Comment 1 Ben Parees 2015-12-23 19:27:21 UTC
you mounted the volume under /opt which clobbered necessary files that were in the image.  please choose a different path for mounting the volume that will not overwrite critical image files.

Comment 2 Chao Yang 2015-12-24 07:35:23 UTC
It is OK after chaged the mount-path
oc volume dc/mydb --add --type=persistentVolumeClaim --claim-mode='ReadWriteMany' --claim-name='nfsc-chao3' --claim-size=5 --mount-path=/opt111

Comment 4 errata-xmlrpc 2016-01-26 19:20:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070