This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2227787 - SSPCommonTemplatesModificationReverted alert in firing state during cnv upgrade (4.12.4->4.12.5)
Summary: SSPCommonTemplatesModificationReverted alert in firing state during cnv upgra...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Infrastructure
Version: 4.12.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.14.1
Assignee: João Vilaça
QA Contact: Geetika Kapoor
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-31 13:23 UTC by Ahmad
Modified: 2024-04-13 04:25 UTC (History)
7 users (show)

Fixed In Version: cnv v4.14.1.rhel9-22 / ssp v4.14.1-3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-12-14 16:16:01 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt ssp-operator pull 721 0 None Merged [release-v0.16] Fix total_restored_common_templates metric update 2023-11-15 10:28:34 UTC
Red Hat Issue Tracker   CNV-31564 0 None None None 2023-12-14 16:16:00 UTC

Description Ahmad 2023-07-31 13:23:18 UTC 
Description of problem:
During cnv upgrade v4.12.4 to 4.12.5-50 (OCP: 4.12.26), noticed SSPCommonTemplatesModificationReverted (SSP operator) alert in firing state.


Version-Release number of selected component (if applicable):
cnv: 4.12.4 to 4.12.5-50  (OCP: 4.12.26)

How reproducible:
1 out of 1 attempt 

Steps to Reproduce:
1. Check alerts fired during cnv upgrade
2.
3.

Actual results:
[{'labels': {'alertname': 'SSPCommonTemplatesModificationReverted', 'kubernetes_operator_component': 'ssp-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}, 'annotations': {'runbook_url': 'https://kubevirt.io/monitoring/runbooks/SSPCommonTemplatesModificationReverted', 'summary': 'Common Templates manual modifications were reverted by the operator'}, 'state': 'firing', 'activeAt': '2023-07-26T20:49:22.832759694Z', 'value': '1.1266666666666666e+02'}

Expected results:
we are trying to capture all the alerts that are fired during upgrades and reduce the noise generated.


Additional info:

http://pastebin.test.redhat.com/1106300

must-gather.log
http://pastebin.test.redhat.com/1106301
Comment 1 Krzysztof Majcher 2023-08-22 12:46:00 UTC 
Please try to reproduce
Comment 2 Dharmit Shah 2023-09-06 09:00:18 UTC 
Can't reproduce this. I tried this with OCP 4.12.26 and upgraded OCP-V Operator from 4.12.4 to 4.12.5. To install OCP-V 4.12.4 I used below manifest:

```
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/kubevirt-hyperconverged.openshift-cnv: ""
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  channel: stable
  installPlanApproval: Manual
  name: kubevirt-hyperconverged
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: kubevirt-hyperconverged-operator.v4.12.4
```

And then upgraded to 4.12.5 through the Operator Hub UI.

Ahmad, do we need to try anything else to reproduce this?
Comment 3 Debarati Basu-Nag 2023-09-07 22:19:21 UTC 
@dshah We see this during our upgrade automation test run. We can help with how to run this test. But if you can let us know what data we should collect when such alerts are fired, we can do that and that would save time on reproducing effort. Please let us know what you think.
Comment 4 Dharmit Shah 2023-09-08 05:57:33 UTC 
(In reply to Debarati Basu-Nag from comment #3)
> @dshah We see this during our upgrade automation test run. We can
> help with how to run this test. 

Is anything wrong/inaccurate with how I have tried to reproduce this? It was my first time trying to upgrade, so I'm curious to know.

> But if you can let us know what data we
> should collect when such alerts are fired, we can do that and that would
> save time on reproducing effort. Please let us know what you think.

I'm not sure what data needs to be collected when such an alert is fired. @
Comment 5 Dharmit Shah 2023-09-08 05:59:50 UTC 
> But if you can let us know what data we
> should collect when such alerts are fired, we can do that and that would
> save time on reproducing effort. Please let us know what you think.

I'm not sure what data needs to be collected when such an alert is fired. Joao, can you help answer this query from Debarati?
Comment 8 Krzysztof Majcher 2023-09-26 12:50:09 UTC 
Dominik, could you look at this alert?
Shirly and our team can assist if needed.
Comment 9 Geetika Kapoor 2023-10-03 10:01:32 UTC 
Change the target version on discussion with HCO team.
Comment 10 João Vilaça 2023-10-16 10:33:30 UTC 
@kmajcher 

The target release is set to 4.14.1
Ahmad had this issue in v4.12.4 to 4.12.5

I already fixed this issue in https://github.com/kubevirt/ssp-operator/pull/559,
which became available from v4.14.0-67

Should we backport this to older versions, or should we consider this as done?
Comment 11 Krzysztof Majcher 2023-10-17 09:08:56 UTC 
Hi,
for questions like this what we need to do is to look at the telemeter report here:
https://telemeter-lts-dashboards.datahub.redhat.com/d/7tLVLsNMz/cnv-cluster-overview?orgId=1

and see how many of our customers are on this version.If you look at the "Number of OLM cvs per version" you'll see that roughly half of our customers is on 4.12.x.

Please close this bug, open one with Target version for 4.12.z and backport the fix.
Comment 12 Ahmad 2023-10-29 13:43:07 UTC 
Hi @kmajcher @jvilaca 
we still can reproduce this in 4.12 & 4.13 can please see if this can be backported?
Comment 14 Krzysztof Majcher 2023-11-15 07:52:48 UTC 
I believe the latest PR from Joao will do the backport.
Comment 15 Geetika Kapoor 2023-12-06 09:42:07 UTC 
looks like issue exist based on JIRA comments : https://issues.redhat.com/browse/CNV-31564. Could you please check.
Comment 16 Red Hat Bugzilla 2024-04-13 04:25:16 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.