Bug 1770387
| Summary: | network_attachment_definition_enabled_instance_up{networks="any"} should not be 1 if pod is not running | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Weibin Liang <weliang> |
| Component: | Networking | Assignee: | Feng Pan <fpan> |
| Networking sub component: | multus | QA Contact: | Weibin Liang <weliang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | alegrand, anpicker, aos-bugs, aputtur, bbennett, erooth, kakkoyun, lcosic, mloibl, tohayash |
| Version: | 4.3.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-23 11:11:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Weibin Liang
2019-11-08 20:48:21 UTC
By looking at metrics I think this might be a problem with some network component - reassigning. Yes, Checking for Pod status periodically is expensive, So not implemented to check the pod status. The event is captured when the pod is created and metrics are incremented. It doesn't consider the state of the metrics. (Creating forever, error ). The metrics decrease count when delete event is fired. Verification failed on 4.3.0-0.nightly-2019-12-04-054458. 1. Create a pod oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/multus-cni/Pods/1interface-macvlan-bridge.yaml 2. check pod: pod is in ContainerCreating state 3. Metrics show: network_attachment_definition_enabled_instance_up{networks="any"} 0 network_attachment_definition_instances{networks="any"} 0 4. create net-attach-def curl -s https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/multus-cni/NetworkAttachmentDefinitions/macvlan-bridge.yaml | sed s/eth0/ens5/g | oc create -f- 5. check pod: pod is in Running state 6. Metrics show: network_attachment_definition_enabled_instance_up{networks="any"} 0 network_attachment_definition_instances{networks="any"} 0 Expect: network_attachment_definition_instances{networks="macvlan"} 1 Tried with AWS cluster and could not recreate. 1. oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/multus-cni/Pods/1interface-macvlan-bridge.yaml 2. oc get pods NAME READY STATUS RESTARTS AGE macvlan-bridge-pod-9swhw 0/1 ContainerCreating 0 4s 3. curl -s https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/multus-cni/NetworkAttachmentDefinitions/macvlan-bridge.yaml | sed s/eth0/ens3/g | oc create -f- 4. oc get pods NAME READY STATUS RESTARTS AGE macvlan-bridge-pod-9swhw 1/1 Running 0 3m20s 5. oc logs -f multus-admission-controller-nzl2p -n openshift-multus .... I1204 17:54:36.245547 1 webhook.go:142] AdmissionReview request allowed: Network Attachment Definition '{"cniVersion":"0.3.0","ipam":{"gateway":"10.1.1.1","rangeEnd":"10.1.1.200","rangeStart":"10.1.1.100","routes":[{"dst":"0.0.0.0/0"}],"subnet":"10.1.1.0/24","type":"host-local"},"master":"ens3","mode":"bridge","type":"macvlan"}' is valid I1204 17:54:49.247354 1 localmetrics.go:50] UPdating net-attach-def metrics for macvlan with value 1 I1204 17:54:49.247388 1 localmetrics.go:50] UPdating net-attach-def metrics for any with value 1 6. oc rsh -n openshift-multus multus-admission-controller-nzl2p curl localhost:9091/metrics ... ..... # HELP network_attachment_definition_enabled_instance_up Metric to identify clusters with network attachment definition enabled instances. # TYPE network_attachment_definition_enabled_instance_up gauge network_attachment_definition_enabled_instance_up{networks="any"} 1 network_attachment_definition_enabled_instance_up{networks="sriov"} 0 # HELP network_attachment_definition_instances Metric to get number of instance using network attachment definition in the cluster. # TYPE network_attachment_definition_instances gauge network_attachment_definition_instances{networks="any"} 1 network_attachment_definition_instances{networks="macvlan"} 1 network_attachment_definition_instances{networks="sriov"} 0 (In reply to Aneesh Puttur from comment #7) > Tried with AWS cluster and could not recreate. > 1. oc create -f > https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/ > networking/multus-cni/Pods/1interface-macvlan-bridge.yaml > > 2. oc get pods > NAME READY STATUS RESTARTS AGE > macvlan-bridge-pod-9swhw 0/1 ContainerCreating 0 4s > > 3. curl -s > https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/ > networking/multus-cni/NetworkAttachmentDefinitions/macvlan-bridge.yaml | sed > s/eth0/ens3/g | oc create -f- > > 4. oc get pods > NAME READY STATUS RESTARTS AGE > macvlan-bridge-pod-9swhw 1/1 Running 0 3m20s > > 5. oc logs -f multus-admission-controller-nzl2p -n openshift-multus > .... > I1204 17:54:36.245547 1 webhook.go:142] AdmissionReview request > allowed: Network Attachment Definition > '{"cniVersion":"0.3.0","ipam":{"gateway":"10.1.1.1","rangeEnd":"10.1.1.200", > "rangeStart":"10.1.1.100","routes":[{"dst":"0.0.0.0/0"}],"subnet":"10.1.1.0/ > 24","type":"host-local"},"master":"ens3","mode":"bridge","type":"macvlan"}' > is valid > I1204 17:54:49.247354 1 localmetrics.go:50] UPdating net-attach-def > metrics for macvlan with value 1 > I1204 17:54:49.247388 1 localmetrics.go:50] UPdating net-attach-def > metrics for any with value 1 > > 6. oc rsh -n openshift-multus multus-admission-controller-nzl2p curl > localhost:9091/metrics > ... > ..... > # HELP network_attachment_definition_enabled_instance_up Metric to identify > clusters with network attachment definition enabled instances. > # TYPE network_attachment_definition_enabled_instance_up gauge > network_attachment_definition_enabled_instance_up{networks="any"} 1 > network_attachment_definition_enabled_instance_up{networks="sriov"} 0 > # HELP network_attachment_definition_instances Metric to get number of > instance using network attachment definition in the cluster. > # TYPE network_attachment_definition_instances gauge > network_attachment_definition_instances{networks="any"} 1 > network_attachment_definition_instances{networks="macvlan"} 1 > network_attachment_definition_instances{networks="sriov"} 0 Worked with Aneesh, when creating the pod and NAD under a new project, the problem will be shown up Tested and verified on 4.3.0-0.nightly-2019-12-10-120829 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |