Bug 1937575
| Summary: | nfd-operator generates over 15,000 log entries per minute, appears to be logging in "info" mode | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | Node Feature Discovery Operator | Assignee: | Carlos Eduardo Arango Gutierrez <carangog> |
| Status: | CLOSED ERRATA | QA Contact: | Walid A. <wabouham> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.7 | CC: | carangog, dahernan, msweiker, scott.worthington, sejug, wabouham |
| Target Milestone: | --- | ||
| Target Release: | 4.6.z | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-03-25 04:36:58 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1936464 | ||
| Bug Blocks: | 1940205 | ||
Verified on OCP 4.6.22 with the release-4.6 image of nfd-operator. The nfd-operator logs are stable and not increasingly growing over time.
After deploying NFD operator, waiting 10 minutes:
# oc logs -n openshift-nfd nfd-operator-7dc55fc567-hrhhk | wc -l
506
# date; oc logs -n openshift-nfd nfd-operator-7dc55fc567-hrhhk | wc -l
Thu Mar 18 21:51:14 UTC 2021
506
After about 30 minutes after deployment of NFD operator:
# date; oc logs -n openshift-nfd nfd-operator-7dc55fc567-hrhhk | wc -l
Thu Mar 18 22:18:06 UTC 2021
506
# oc describe pod -n openshift-nfd nfd-operator-7dc55fc567-hrhhk
Name: nfd-operator-7dc55fc567-hrhhk
Namespace: openshift-nfd
Priority: 0
Node: ip-10-0-199-45.us-east-2.compute.internal/10.0.199.45
Start Time: Thu, 18 Mar 2021 21:45:59 +0000
Labels: name=nfd-operator
pod-template-hash=7dc55fc567
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.129.0.8"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.129.0.8"
],
"default": true,
"dns": {}
}]
openshift.io/scc: anyuid
Status: Running
IP: 10.129.0.8
IPs:
IP: 10.129.0.8
Controlled By: ReplicaSet/nfd-operator-7dc55fc567
Containers:
nfd-operator:
Container ID: cri-o://98409af96c20edf75778849c2356b70a74550ac8c4d6112215dc7755bd96811d
Image: quay.io/wabouham/cluster-nfd-operator:release-4.6
Image ID: quay.io/wabouham/cluster-nfd-operator@sha256:e8f4a7d7b7b551fc2ab1dcda65677ffc44f73b588280e514019690672036c561
Port: 60000/TCP
Host Port: 0/TCP
Command:
cluster-nfd-operator
State: Running
Started: Thu, 18 Mar 2021 21:46:09 +0000
Ready: True
Restart Count: 0
Environment:
WATCH_NAMESPACE: openshift-nfd (v1:metadata.namespace)
POD_NAME: nfd-operator-7dc55fc567-hrhhk (v1:metadata.name)
OPERATOR_NAME: cluster-nfd-operator
NODE_FEATURE_DISCOVERY_IMAGE: quay.io/openshift/origin-node-feature-discovery:4.6
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from nfd-operator-token-5cn4h (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
nfd-operator-token-5cn4h:
Type: Secret (a volume populated by a Secret)
SecretName: nfd-operator-token-5cn4h
Optional: false
QoS Class: BestEffort
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32m default-scheduler Successfully assigned openshift-nfd/nfd-operator-7dc55fc567-hrhhk to ip-10-0-199-45.us-east-2.compute.internal
Normal AddedInterface 32m multus Add eth0 [10.129.0.8/23]
Normal Pulling 32m kubelet Pulling image "quay.io/wabouham/cluster-nfd-operator:release-4.6"
Normal Pulled 32m kubelet Successfully pulled image "quay.io/wabouham/cluster-nfd-operator:release-4.6" in 8.216914759s
Normal Created 32m kubelet Created container nfd-operator
Normal Started 32m kubelet Started container nfd-operator
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.22 extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0828 |
Verified on OCP 4.6.21 with the release-4.7 image of nfd-operator. The nfd-operator logs are not increasingly growing over time. After deploying NFD operator, waiting 10 minutes: # oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l 599 # date; oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l Mon Mar 15 20:19:32 UTC 2021 599 # date; oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l Mon Mar 15 20:25:52 UTC 2021 599 # date; oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l Mon Mar 15 20:26:10 UTC 2021 599 # date; oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l Mon Mar 15 20:29:33 UTC 2021 599 # oc get pods -n openshift-nfd NAME READY STATUS RESTARTS AGE nfd-master-pzxlj 1/1 Running 0 20m nfd-master-tj7pv 1/1 Running 0 20m nfd-master-whkwl 1/1 Running 0 20m nfd-operator-6444df8d77-qqj64 1/1 Running 0 20m nfd-worker-9jnxf 1/1 Running 0 20m nfd-worker-hk786 1/1 Running 0 20m nfd-worker-k4474 1/1 Running 0 20m # oc describe pod -n openshift-nfd nfd-operator-6444df8d77-qqj64 Name: nfd-operator-6444df8d77-qqj64 Namespace: openshift-nfd Priority: 0 Node: ip-10-0-179-12.us-east-2.compute.internal/10.0.179.12 Start Time: Mon, 15 Mar 2021 20:09:07 +0000 Labels: name=nfd-operator pod-template-hash=6444df8d77 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.130.0.10" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.130.0.10" ], "default": true, "dns": {} }] openshift.io/scc: anyuid Status: Running IP: 10.130.0.10 IPs: IP: 10.130.0.10 Controlled By: ReplicaSet/nfd-operator-6444df8d77 Containers: nfd-operator: Container ID: cri-o://9d9114eaf9d3db7a4dd60b7e187c480f0891eae20e9a4579ddac8eb8ea98b32e Image: quay.io/wabouham/cluster-nfd-operator:release-4.7 Image ID: quay.io/wabouham/cluster-nfd-operator@sha256:94176df3b81d9fc77d5ca3064fca5f8436f31e5b50036ffdea604f9de46baa7d Port: 60000/TCP Host Port: 0/TCP Command: cluster-nfd-operator State: Running Started: Mon, 15 Mar 2021 20:09:18 +0000 Ready: True Restart Count: 0 Environment: WATCH_NAMESPACE: openshift-nfd (v1:metadata.namespace) POD_NAME: nfd-operator-6444df8d77-qqj64 (v1:metadata.name) OPERATOR_NAME: cluster-nfd-operator NODE_FEATURE_DISCOVERY_IMAGE: quay.io/openshift/origin-node-feature-discovery:4.7 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from nfd-operator-token-fwvm5 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: nfd-operator-token-fwvm5: Type: Secret (a volume populated by a Secret) SecretName: nfd-operator-token-fwvm5 Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 20m default-scheduler Successfully assigned openshift-nfd/nfd-operator-6444df8d77-qqj64 to ip-10-0-179-12.us-east-2.compute.internal Normal AddedInterface 20m multus Add eth0 [10.130.0.10/23] Normal Pulling 20m kubelet Pulling image "quay.io/wabouham/cluster-nfd-operator:release-4.7" Normal Pulled 20m kubelet Successfully pulled image "quay.io/wabouham/cluster-nfd-operator:release-4.7" in 8.279171975s Normal Created 20m kubelet Created container nfd-operator Normal Started 20m kubelet Started container nfd-operator === About 50 minutes later: logs volume remains stable # date; oc logs -n openshift-nfd nfd-operator-6444df8d77-qqj64 | wc -l Mon Mar 15 20:58:55 UTC 2021 599