Bug 1288414 - logging-deployment creates logging-es-xxxxx, logging-fluentd and logging-kibana which have wrong image name
logging-deployment creates logging-es-xxxxx, logging-fluentd and logging-kiba...
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.1.0
Unspecified Unspecified
high Severity low
: ---
: ---
Assigned To: Luke Meyer
chunchen
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-04 01:58 EST by Kenjiro Nakayama
Modified: 2016-09-29 22:16 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-18 09:54:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kenjiro Nakayama 2015-12-04 01:58:46 EST
Description of the problem:
- Followed the documentation[1] and after deployed logging-deployment, all of their pod will get their status "CrashLoopBackOff" or "ImagePullBackOff".

To fix this issue:
- Need to update image name in their dc. For example,

(WRONG)
image: logging-fluentd

(TOBE)
image: openshift3/logging-fluentd

[1] https://docs.openshift.com/enterprise/3.1/install_config/aggregate_logging.html
Comment 1 Luke Meyer 2015-12-18 08:07:41 EST
What did you use for your IMAGE_PREFIX and IMAGE_VERSION in the `oc process logging-deployer-template` command?
https://docs.openshift.com/enterprise/3.1/install_config/aggregate_logging.html#deploying-the-efk-stack
It's *supposed* to default to the right thing but maybe it's wrong or you did something non-standard. Definitely the image name shouldn't need manual modification.
Comment 2 Kenjiro Nakayama 2015-12-18 09:10:37 EST
(In reply to Luke Meyer from comment #1)
> What did you use for your IMAGE_PREFIX and IMAGE_VERSION in the `oc process
> logging-deployer-template` command?

I didn't set IMAGE_PREFIX and IMAGE_VERSION in the `oc process logging-deployer-template` command. 

I didn't think it is necessary, since it has been set by default:

~~~
  name: IMAGE_PREFIX
  value: "registry.access.redhat.com/openshift3/"
~~~

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v1.0/infrastructure-templates/enterprise/logging-deployer.yaml#L83-L85
Comment 3 Jeff Cantrill 2016-01-06 15:26:17 EST
Moving this out of 'must fix' as there is a work around if the default is not being set correctly.  The DC can be manually updated.
Comment 4 Luke Meyer 2016-01-08 08:58:45 EST
I think I understand the issue. I don't think it's a bug, but it points to something else that has gone wrong.

How this works is that when the DC is created, the pod spec has a placeholder image setting. This is supposed to be overwritten immediately by the ImageChange trigger. E.g. if you look at logging-fluentd-template it comes out as:

    template:
      metadata:
        labels:
          component: fluentd
          provider: openshift
        name: fluentd-elasticsearch
      spec:
        containers:
        - image: logging-fluentd     <--- placeholder value
          imagePullPolicy: Always
          name: fluentd-elasticsearch
          [...]
    triggers:
    - type: ConfigChange
    - imageChangeParams:             <--- specifies overwrite
        automatic: true
        containerNames:
        - fluentd-elasticsearch
        from:                      
          kind: ImageStreamTag
          name: logging-fluentd:latest
      type: ImageChange

Once the DC is actually created, the image in the template.spec should be changed to the real value, e.g. registry.access.redhat.com/openshift3/logging-fluentd:latest, which comes from the ImageStream "logging-fluentd". If, however, your IS hasn't been populated for some reason, the trigger won't fire and the DC will just try to use the placeholder.

I'm not sure what would cause ImageStream population to fail if you can then just change the image name to the right value and it starts working. ImageStreams do have their own docker client settings (I know this from trying to use an internal registry with an unknown CA, the IS doesn't use the system docker settings for this) however this is working fine with registry.access.redhat.com in all my tests.

It would probably be helpful to get the output of `oc describe is/logging-fluentd` (and/or any others that are misbehaving) as well as master logs if you see this occur.

Since the customer case is closed now (sorry for the delay) shall we close this bug and see if we can get further information if someone reproduces this?
Comment 5 Kenjiro Nakayama 2016-01-17 00:14:11 EST
Thank you for your detail explanation.
I confirmed that the DC has "image: registry.access.redhat.com/openshift3/logging-xxxx" after I ran:

  $ oc process logging-support-template | oc create -f -

As you explained, I also guess that ImageStream population from logging-support-template has been failed.
Please feel free to close this bugzilla ticket.

Note You need to log in before you can comment on or make changes to this bug.