Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1563490

Summary: [3.7] logging annotations are kept in default project after undeployment
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.7.0CC: aos-bugs, ewolinet, jcantril, juzhao, nhosoi, rmeggins, tkatarki
Target Milestone: ---Keywords: Regression
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: logging-elasticsearch-ops was missing in the delete configmaps list in the openshift-ansible delete_logging role. Consequence: The logging-elasticsearch-ops configmap still exists after running uninstall ansible playbook for logging as well as the ops logging annotations. Fix: Adding logging-elasticsearch-ops to the delete configmaps list. Result: All the logging configmaps are uninstalled by running uninstall ansible playbook for logging and the following new installation has no more leftover annotations from the previous install.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-14 19:06:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2018-04-04 02:24:12 UTC
Description of problem:
This defect is from https://bugzilla.redhat.com/show_bug.cgi?id=1547348#c11
1. Deploy logging with enabled ops cluster, logging annotations will be added to default project, such as:
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

2. Undeploy logging, logging annotations would be kept for default project
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

3. Deploy logging with non-ops cluster, logging annotations is still
metadata:
  annotations:
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0403-rh2.qe.rhcloud.com

If we click "View Archive" link in pod log for default project, it will navigate to kibana-ops UI, but there is not such service, so will throw out error which indicate application is not applicable.

There is not such issue if we deploy logging with non-ops cluster and then deploy logging with enabled ops cluster.
I think there should be one fix in openshift-ansible side, all logging annotations should be deleted after undeployment.

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.7.42
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8


Images
logging-curator/images/v3.7.42-2
logging-elasticsearch/images/v3.7.42-2
logging-kibana/images/v3.7.42-2
logging-fluentd/images/v3.7.42-2
logging-auth-proxy/images/v3.7.42-2

# rpm -qa | grep openshift-ansible
openshift-ansible-lookup-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-callback-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-docs-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-filter-plugins-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-roles-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-3.7.42-1.git.0.427f18c.el7.noarch
openshift-ansible-playbooks-3.7.42-1.git.0.427f18c.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:
logging annotations are kept in default project after undeployment

Expected results:
logging annotations should be removed from default project after undeployment

Additional info:

Comment 1 Noriko Hosoi 2018-04-27 15:13:42 UTC
It might be related to this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1549220

The PR has not been backported to 3.7 yet.  I'm going to do it.
https://bugzilla.redhat.com/show_bug.cgi?id=1549220#c2

Comment 2 Noriko Hosoi 2018-04-27 17:00:49 UTC
https://github.com/openshift/openshift-ansible/pull/8183
[release-3.7] Bug 1549220 - configmap still exist after running uninstall playbook for logging #8183

Comment 4 Junqi Zhao 2018-05-08 02:53:50 UTC
Issue is not fixed, verification steps followed the Description part. Deploy logging with ops enabled and then undeploy it, later, deploy logging without ops enabled. Click "View Archive" link under default project, it still navigated to kibana-ops UI.

The following annotations;
    openshift.io/logging.data.prefix: .operations
    openshift.io/logging.ui.hostname: kibana-ops.apps.0507-rhh.qe.rhcloud.com

are still kept for openshift-infra, openshift, kube-system, and default projects.

# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-callback-plugins-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-playbooks-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-docs-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-filter-plugins-3.7.46-1.git.0.37f607e.el7.noarch
openshift-ansible-lookup-plugins-3.7.46-1.git.0.37f607e.el7.noarch

Comment 5 Noriko Hosoi 2018-05-08 19:09:22 UTC
Thanks for testing, @Junqi.

It seems the cause is different...

@Eric, I noticed you fixed this removing annotation in roles/openshift_logging/tasks/delete_logging.yaml in the openshift-ansible master branch.
    Merge pull request #8256 from ewolinetz/logging_fix_remove_annotation
    Updating to remove annotations from namespace rather than project

I don't see the corresponding section "remove annotations added by logging" in the release-3.7 branch.  Do you think it's related to this bz?  Thanks!

Comment 6 ewolinet 2018-05-09 20:37:28 UTC
@Noriko,

I think that would related, yes. However as part of that issue ansible was printing out errors when executing the section to remove the annotation.

@Junqi,

Are there errors in the playbook output when you are running this scenario?

Comment 7 Junqi Zhao 2018-05-10 00:55:30 UTC
(In reply to ewolinet from comment #6)

> @Junqi,
> 
> Are there errors in the playbook output when you are running this scenario?

No, there is not error when running the playbook, this is the scenario:

Deploy logging with ops enabled and then undeploy it, later, deploy logging without ops enabled. Click pod's "View Archive" link  under default project, it will navigated to kibana-ops UI, not kibana UI, actually we don't have kibana-ops service

Comment 8 Noriko Hosoi 2018-05-11 16:33:53 UTC
This bug 1563490 is a dup of Bug 1519808 - View archive link for default and metrics pods redirects to wrong kibana url.
https://bugzilla.redhat.com/show_bug.cgi?id=1519808
Although the bug 1519808 was closed, the fix is not in the release-3.7 branch...

Note: The fix was merged to release-3.9 and newer.
https://github.com/openshift/openshift-ansible/pull/6363

I verified the annotation for logging.ui.hostname was successfully removed by the ansible uninstall on 3.9 and master (3.10).
  metadata:
    annotations:
      openshift.io/logging.ui.hostname: kibana-ops.<DOMAIN>

Since 3.7 is older than (3.10 - 0.2), it needs to be decided by Tushar if the fix should be backported or not...  @Tushar, could you please share your decision with us?  Thanks!

Comment 11 Red Hat Bugzilla 2023-09-14 04:26:20 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days