Bug 1563490 - [3.7] logging annotations are kept in default project after undeployment [NEEDINFO]
Summary: [3.7] logging annotations are kept in default project after undeployment
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.9.z
Assignee: Noriko Hosoi
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-04 02:24 UTC by Junqi Zhao
Modified: 2018-05-14 19:06 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: logging-elasticsearch-ops was missing in the delete configmaps list in the openshift-ansible delete_logging role. Consequence: The logging-elasticsearch-ops configmap still exists after running uninstall ansible playbook for logging as well as the ops logging annotations. Fix: Adding logging-elasticsearch-ops to the delete configmaps list. Result: All the logging configmaps are uninstalled by running uninstall ansible playbook for logging and the following new installation has no more leftover annotations from the previous install.
Clone Of:
Environment:
Last Closed: 2018-05-14 19:06:25 UTC
Target Upstream Version:
nhosoi: needinfo? (tkatarki)


Attachments (Terms of Use)

Description Junqi Zhao 2018-04-04 02:24:12 UTC
Description of problem:
This defect is from https://bugzilla.redhat.com/show_bug.cgi?id=1547348#c11
1. Deploy logging with enabled ops cluster, logging annotations will be added to default project, such as:
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

2. Undeploy logging, logging annotations would be kept for default project
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

3. Deploy logging with non-ops cluster, logging annotations is still
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

If we click "View Archive" link in pod log for default project, it will navigate to kibana-ops UI, but there is not such service, so will throw out error which indicate application is not applicable.

There is not such issue if we deploy logging with non-ops cluster and then deploy logging with enabled ops cluster.
I think there should be one fix in openshift-ansible side, all logging annotations should be deleted after undeployment.

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.7.42
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8


Images
logging-curator/images/v3.7.42-2
logging-elasticsearch/images/v3.7.42-2
logging-kibana/images/v3.7.42-2
logging-fluentd/images/v3.7.42-2
logging-auth-proxy/images/v3.7.42-2

# rpm -qa | grep openshift-ansible
openshift-ansible-lookup-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-callback-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-docs-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-filter-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-roles-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-playbooks-3.7.42-1.git.0.427f18c.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:
logging annotations are kept in default project after undeployment

Expected results:
logging annotations should be removed from default project after undeployment

Additional info:

Comment 1 Noriko Hosoi 2018-04-27 15:13:42 UTC
It might be related to this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1549220

The PR has not been backported to 3.7 yet.  I'm going to do it.
https://bugzilla.redhat.com/show_bug.cgi?id=1549220#c2

Comment 2 Noriko Hosoi 2018-04-27 17:00:49 UTC
https://github.com/openshift/openshift-ansible/pull/8183
[release-3.7] Bug 1549220 - configmap still exist after running uninstall playbook for logging #8183

Comment 4 Junqi Zhao 2018-05-08 02:53:50 UTC
Issue is not fixed, verification steps followed the Description part. Deploy logging with ops enabled and then undeploy it, later, deploy logging without ops enabled. Click "View Archive" link under default project, it still navigated to kibana-ops UI.

The following annotations;
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0507-rhh.qe.rhcloud.com

are still kept for openshift-infra, openshift, kube-system, and default projects.

# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-callback-plugins-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-playbooks-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-docs-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-filter-plugins-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-lookup-plugins-3.7.46-1.git.0.37f607e.el7.noarch

Comment 5 Noriko Hosoi 2018-05-08 19:09:22 UTC
Thanks for testing, @Junqi.

It seems the cause is different...

@Eric, I noticed you fixed this removing annotation in roles/openshift_logging/tasks/delete_logging.yaml in the openshift-ansible master branch.
    Merge pull request #8256 from ewolinetz/logging_fix_remove_annotation
    Updating to remove annotations from namespace rather than project

I don't see the corresponding section "remove annotations added by logging" in the release-3.7 branch.  Do you think it's related to this bz?  Thanks!

Comment 6 ewolinet 2018-05-09 20:37:28 UTC
@Noriko,

I think that would related, yes. However as part of that issue ansible was printing out errors when executing the section to remove the annotation.

@Junqi,

Are there errors in the playbook output when you are running this scenario?

Comment 7 Junqi Zhao 2018-05-10 00:55:30 UTC
(In reply to ewolinet from comment #6)

> @Junqi,
> 
> Are there errors in the playbook output when you are running this scenario?

No, there is not error when running the playbook, this is the scenario:

Deploy logging with ops enabled and then undeploy it, later, deploy logging without ops enabled. Click pod's "View Archive" link  under default project, it will navigated to kibana-ops UI, not kibana UI, actually we don't have kibana-ops service

Comment 8 Noriko Hosoi 2018-05-11 16:33:53 UTC
This bug 1563490 is a dup of Bug 1519808 - View archive link for default and metrics pods redirects to wrong kibana url.
https://bugzilla.redhat.com/show_bug.cgi?id=1519808
Although the bug 1519808 was closed, the fix is not in the release-3.7 branch...

Note: The fix was merged to release-3.9 and newer.
https://github.com/openshift/openshift-ansible/pull/6363

I verified the annotation for logging.ui.hostname was successfully removed by the ansible uninstall on 3.9 and master (3.10).
  metadata:
    annotations:
      openshift.io/logging.ui.hostname: kibana-ops.<DOMAIN>

Since 3.7 is older than (3.10 - 0.2), it needs to be decided by Tushar if the fix should be backported or not...  @Tushar, could you please share your decision with us?  Thanks!


Note You need to log in before you can comment on or make changes to this bug.