Bug 1707072

Summary: no termination message provided by failing openshift-service-catalog-controller-manager-operator pod
Product: OpenShift Container Platform Reporter: Luis Sanchez <sanchezl>
Component: Service CatalogAssignee: Dan Geoffroy <dageoffr>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: calfonso, eparis
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:48:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luis Sanchez 2019-05-06 18:04:09 UTC 
The openshift-service-catalog-apiserver-operator pod does not provide a termination message, hindering debugging efforts when the pods are crash looping.

At minimum, the pod's terminationMessagePolicy should be "FallbackToLogsOnError".

See https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message

Expected Results:
The termination message should appear in a pod container's  .status.lastState.terminated.message field.
Comment 1 Jay Boyd 2019-05-06 19:22:27 UTC 
This is actually set properly on the openshift-service-catalog-apiserver-operator pod:

This was committed April 16 via https://github.com/openshift/cluster-svcat-apiserver-operator/pull/46

Verifying with nightly 05-06-163240:

[jaboyd@jboyd openshift]$ oc get pods -n openshift-service-catalog-apiserver-operator -o yaml | grep Fall -A5 -B30
      uid: 43cb3e57-7032-11e9-a7ea-0a0d33364010
    resourceVersion: "7882"
    selfLink: /api/v1/namespaces/openshift-service-catalog-apiserver-operator/pods/openshift-service-catalog-apiserver-operator-5cfd6f4f5-5k479
    uid: c47e4333-7032-11e9-938c-0ebc19e0b298
  spec:
    containers:
    - args:
      - --config=/var/run/configmaps/config/config.yaml
      - -v=5
      command:
      - cluster-svcat-apiserver-operator
      - operator
      env:
      - name: IMAGE
        value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:203e3a95d47f5e283cfc0822a5f4ffc7f9c038587c093c0d83de611e938671a0
      - name: OPERATOR_IMAGE_VERSION
        value: 4.1.0-0.nightly-2019-05-06-163240
      - name: OPERAND_IMAGE_VERSION
        value: 4.1.0-0.nightly-2019-05-06-163240
      image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1dd3b4443c13b156e686c9eabd5523cfd9a70416d12951815cf094b0f2a8b974
      imagePullPolicy: IfNotPresent
      name: operator
      ports:
      - containerPort: 8443
        name: metrics
        protocol: TCP
      resources:
        requests:
          memory: 50Mi
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError





However there is not one on the cluster-svcat-controller-manager-operator.  

@sanchezl are you ok if I change this bug to track setting this on the controller-manager?
Comment 2 Luis Sanchez 2019-05-07 03:21:24 UTC 
@jaboyd you are correct. I incorrectly flagged the openshift-service-catalog-apiserver-operator pod. Feel free to re-purpose this bug for the cluster-svcat-controller-manager-operator pod.
Comment 4 Jay Boyd 2019-05-07 14:56:59 UTC 
I just submitted the pr (its a simple change).  Should be merged shortly.
Comment 5 Jay Boyd 2019-05-07 14:57:44 UTC 
https://github.com/openshift/cluster-svcat-controller-manager-operator/pull/39
Comment 8 Jian Zhang 2019-05-08 09:08:00 UTC 
LGTM, verify it.
cluster-svcat-controller-manager-operator version info:
io.openshift.build.commit.id=9ac2a14537c4b880f0ac12f8acacb3c29c9823c7
Cluster version is 4.1.0-0.nightly-2019-05-08-012425

mac:~ jianzhang$ oc get pods -o yaml -n openshift-service-catalog-controller-manager-operator|grep terminationMessage -A 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
      - mountPath: /var/run/configmaps/config
        name: config

mac:~ jianzhang$ oc get pods -o yaml -n openshift-service-catalog-apiserver-operator |grep terminationMessage -A 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
      - mountPath: /var/run/configmaps/config
        name: config

mac:~ jianzhang$ oc get pods -o yaml -n openshift-service-catalog-apiserver|grep terminationMessage -A 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
      - mountPath: /var/run/kubernetes-service-catalog
        name: apiserver-ssl
...

mac:~ jianzhang$ oc get pods -o yaml -n openshift-service-catalog-controller-manager|grep terminationMessage -A 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
      - mountPath: /var/run/kubernetes-service-catalog
        name: apiserver-ssl
...
Comment 11 errata-xmlrpc 2019-06-04 10:48:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758