Bug 1338965

Summary: necessary tag is not imported, preventing logging deployment
Product: OpenShift Container Platform Reporter: Luke Meyer <lmeyer>
Component: LoggingAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: chunchen <chunchen>
Severity: high Docs Contact:
Priority: high    
Version: 3.2.0CC: aos-bugs, boris.ruppert, chunchen, dyocum, jkaur, lmeyer, pep, tdawson, wsun, xiazhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: When ImageStreams are created, only a subset of the available tags are imported, and this often excluded the desired tag. Consequence: If the desired tag isn't imported, then the corresponding component never deploys. Workaround (if any): Import each tag manually like so: oc import-image <name>:<version> --from <prefix><name>:<tag> The bug is fixed in 3.3 by not relying on ImageStreams and DC triggers for deployment. Result: Deployment occurs as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 09:33:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1267746    

Description Luke Meyer 2016-05-23 18:31:41 UTC
Description of problem:
Refer to issue https://github.com/openshift/origin/issues/8948

The logging deployment sets up ImageStreams and related DeploymentConfigs. Triggers on each DC result in deployment once the necessary tag is imported into the IS. However, it turns out that only a subset of the available tags are imported automatically (ref https://github.com/openshift/origin/issues/8952), and 3.2.0 (the current default) is frequently not among them. Thus deployment never occurs.


Version-Release number of selected component (if applicable):
Noticed most frequently with 3.2.0, however it will probably affect all versions.

How reproducible:
More often than not. It's not clear why sometimes v3.2 is imported versus 3.2.0.


Steps to Reproduce:
1. Follow deployment instructions at https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html

Actual results:
Deployment occurs in some cases, but not others. In the cases where it doesn't, it's because the 3.2.0 tag was not imported, although "latest" and "v3.2" (which have the same image ID) are.

Expected results:
Consistent deployment.


Additional info:
The quick "fix" will be to update docs to indicate the following workaround:

$ oc import-image logging-auth-proxy:3.2.0 \
     --from registry.access.redhat.com/openshift3/logging-auth-proxy:3.2.0 
$ oc import-image logging-kibana:3.2.0 \
     --from registry.access.redhat.com/openshift3/logging-kibana:3.2.0
$ oc import-image logging-elasticsearch:3.2.0 \
     --from registry.access.redhat.com/openshift3/logging-elasticsearch:3.2.0
$ oc import-image logging-fluentd:3.2.0 \
     --from registry.access.redhat.com/openshift3/logging-fluentd:3.2.0

As a more permanent fix, we will dispense with the ImageStreams and triggers, simply deploying the intended images directly.

Comment 1 openshift-github-bot 2016-05-23 20:42:04 UTC
Commit pushed to master at https://github.com/openshift/openshift-docs

https://github.com/openshift/openshift-docs/commit/0e6244897c81dabf0a6634d69a683d0dd2c48bd7
logging: manually import tags for deployment

Docs fix to bug 1338965
https://bugzilla.redhat.com/show_bug.cgi?id=1338965

This is an ugly workaround, but hopefully short-term.

Comment 2 Luke Meyer 2016-05-24 14:11:58 UTC
*** Bug 1339060 has been marked as a duplicate of this bug. ***

Comment 3 Luke Meyer 2016-05-24 14:16:53 UTC
Docs workaround at https://github.com/openshift/openshift-docs/pull/2141

Comment 5 Josep 'Pep' Turro Mauri 2016-06-23 11:10:42 UTC
While there's a docs workaround that helps in the logging deployment (and possibly others), shouldn't this track the problem of import-image?

I.e. https://github.com/openshift/origin/pull/9163

and if so, we should probably move it from Logging to CLI.

Comment 6 Luke Meyer 2016-06-28 20:36:13 UTC
(In reply to Josep 'Pep' Turro Mauri from comment #5)
> While there's a docs workaround that helps in the logging deployment (and
> possibly others), shouldn't this track the problem of import-image?
> 
> I.e. https://github.com/openshift/origin/pull/9163

I wouldn't expect that to get into Enterprise before 3.3, and by that time I fully expect to have eliminated the problem by removing the usage of ImageStreams in logging deployment.

Agree a note in the upgrade doc would help too until then.

Comment 7 Luke Meyer 2016-06-29 21:25:23 UTC
(In reply to Xia Zhao from comment #4)
> FYI, The change is also needed in upgrade doc
> https://docs.openshift.com/enterprise/3.2/install_config/upgrading/
> manual_upgrades.html#manual-upgrading-efk-logging-stack

Could you validate that https://github.com/openshift/openshift-docs/pull/2428 addresses the problem and the upgrade sequence works with this in place?

I think we'll want to keep this bug open to track the ultimate solution to this; docs are a stopgap.

Comment 8 Xia Zhao 2016-07-01 03:29:22 UTC
@lmeyer The modification on upgrade doc and also the sequence mentioned looks good to me.

Comment 9 Luke Meyer 2016-07-01 20:26:50 UTC
Note published in the upgrade docs https://docs.openshift.com/enterprise/latest/install_config/upgrading/manual_upgrades.html#manual-upgrading-efk-logging-stack

Keeping the bug open for tracking removing imagestreams entirely.

Comment 10 Luke Meyer 2016-08-17 18:57:32 UTC
As of 3.3 imagestreams and DC triggers are not a part of the deployment, so this should not be a problem.

Comment 12 Xia Zhao 2016-08-19 02:59:21 UTC
Yes, we never encountered this issue with the 3.3.0 logging deployer, set to verified.

Comment 14 errata-xmlrpc 2016-09-27 09:33:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933