Bug 1890476 - [Kuryr] KuryrSDNPodNotReady alert is missing the node name in the message
Summary: [Kuryr] KuryrSDNPodNotReady alert is missing the node name in the message
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.6.z
Assignee: Robin Cernin
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1990728
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-22 10:48 UTC by Jon Uriarte
Modified: 2021-08-12 09:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1990725 1990726 1990728 (view as bug list)
Environment:
Last Closed: 2021-08-12 09:28:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 1176 0 None None None 2021-08-06 02:03:23 UTC

Description Jon Uriarte 2020-10-22 10:48:24 UTC
Description of problem:

KuryrSDNPodNotReady alert doesn't print the node name in the message, i.e:

"message": "SDN pod kuryr-controller-d5c669d95-jzx89 on node  is not ready."
"message": "SDN pod kuryr-cni-vmx6q on node  is not ready."

The message format as defined should be:
message: SDN pod {{"{{"}} $labels.pod {{"}}"}} on node {{"{{"}} $labels.node {{"}}"}} is not ready.


Version-Release number of selected component (if applicable):
OCP 4.6.0-0.nightly-2020-10-20-101225
OSP13 2020-10-06.2

How reproducible: always

Steps to Reproduce:
1. rsh to the kuryr controller pod and remove /tmp/pools_loaded file (to make it not ready)
   $ oc -n openshift-kuryr rsh <kuryr-controller-pod>
   $ rm /tmp/pools_loaded
2. Check the KuryrSDNPodNotReady alert is raised (the apps fip needs to be asigned to the ingress port and the entry for prometheus added in /etc/hosts file - <apps fip> prometheus-k8s-openshift-monitoring.apps.ostest.shiftstack.com)

token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
curl -sk -H "Authorization: Bearer $token" 'https://prometheus-k8s-openshift-monitoring.apps.ostest.shiftstack.com/api/v1/alerts' | jq '.data.alerts[] | select(.labels.alertname == "KuryrSDNPodNotReady")'

Check the message field.

Actual results: "message": "SDN pod kuryr-controller-d5c669d95-jzx89 on node  is not ready."

Expected results: it should reflect the node in where the pod is running


Additional info:
The NodeWithoutKuryrCNIPodRunning alert which also prints the node is correctly printed, i.e:
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-worker-0-rrjqm is not.\n"
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-worker-0-wsqjr is not.\n"
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-master-0 is not.\n"
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-master-1 is not.\n"
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-master-2 is not.\n"
"message": "All nodes should be running a kuryr-cni pod, ostest-ftv4z-worker-0-mpbwv is not.\n"

Comment 6 Michał Dulko 2021-08-12 09:28:12 UTC
For now we've decided not to backport a low priority fix unless requested. It'll be fixed in 4.9.


Note You need to log in before you can comment on or make changes to this bug.