Bug 2060032

First, can you capture a must-gather so we have logs to see what is going on to cause this?

Also, does reapplying the NNCP after the outage workaround the problem (but please get the must-gather first if the policy is still in error)?

Attempting to reapply the NNCP results in:

$ oc apply -f br-bond0.yaml 
Error from server (failed to admit NodeNetworkConfigurationPolicy br-bond0-policy: message: policy br-bond0-policy is still in progress. ): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"nmstate.io/v1beta1\",\"kind\":\"NodeNetworkConfigurationPolicy\",\"metadata\":{\"annotations\":{},\"name\":\"br-bond0-policy\"},\"spec\":{\"desiredState\":{\"interfaces\":[{\"bridge\":{\"options\":{\"stp\":{\"enabled\":false}},\"port\":[{\"name\":\"bond0\"}]},\"description\":\"linux-bridge with bond0\",\"ipv4\":{\"dhcp\":true,\"enabled\":true},\"name\":\"br-bond0\",\"state\":\"up\",\"type\":\"linux-bridge\"}]},\"nodeSelector\":{\"node-role.kubernetes.io/worker\":\"\"}}}\n"}},"spec":{"desiredState":{"interfaces":[{"bridge":{"options":{"stp":{"enabled":false}},"port":[{"name":"bond0"}]},"description":"linux-bridge with bond0","ipv4":{"dhcp":true,"enabled":true},"name":"br-bond0","state":"up","type":"linux-bridge"}]}}}
to:
Resource: "nmstate.io/v1beta1, Resource=nodenetworkconfigurationpolicies", GroupVersionKind: "nmstate.io/v1beta1, Kind=NodeNetworkConfigurationPolicy"
Name: "br-bond0-policy", Namespace: ""
for: "br-bond0.yaml": admission webhook "nodenetworkconfigurationpolicies-progress-validate.nmstate.io" denied the request: failed to admit NodeNetworkConfigurationPolicy br-bond0-policy: message: policy br-bond0-policy is still in progress.

Will upload must-gather when complete.

Moving to CNV since this is related to that version of the operator.

Thanks for reporting this. We will triage it on our bug scrub next week.

Hello Tom,

Thanks for uploading the must-gather. While analyzing the logs, could you try to delete the policy and create it again?
(Deleting the policy doesn't remove the configuration on nodes, where it was already applied)

That will get you pass the validation error, as progressing policies cannot be updated.

Backported fix in https://github.com/nmstate/kubernetes-nmstate/pull/1013 
Now if enactments are blocked due to network outage, progress will unblock after a timeout of 10 minutes.

All the upstream work is merged. We are now waiting for 4.8.6 errata to get created

Errata was created. Pushing new M/S patch

Correction to my previous comment:
> Now if enactments are blocked due to network outage, progress will unblock after a timeout of 10 minutes.

The timeout is 18 minutes

Verified on a bare-metal cluster with
OCP 4.8.35
CNV 4.8.6 
nmstate-handler v4.8.6-6

Verified using the following scenario:
1. Applied the following policy:
apiVersion: nmstate.io/v1beta1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br1-eno3-nncp
spec:
  desiredState:
    interfaces:
    - bridge:
        options:
          stp:
            enabled: false
        port:
        - name: eno3
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        enabled: false
      name: br1-eno3
      state: up
      type: linux-bridge
  nodeSelector:
    node-role.kubernetes.io/worker: ""

2. Waited for the policy to be in Progressing status
cnv-qe-jenkins@cnv-qe-infra-01:~/yossi$ oc get nncp -w
NAME            STATUS
br1-eno3-nncp   
br1-eno3-nncp   ConfigurationProgressing
br1-eno3-nncp   ConfigurationProgressing

3. ssh'd to each of the worker nodes.

4. Simulated network outage, by shutting down the primary interface on each worker node, by running this shell script on each worker:
ip link show dev eno1
echo
sudo ip link set dev eno1 down
ip link show dev eno1
echo
sleep 120
sudo ip link set dev eno1 up

5. Verified that the ssh connectivity to all workers was lost.

6. After the outage timeout (120 seconds) was over, I verified that the connectivity was again possible to the nodes.

7. Followed the policy status, and saw that it continued from Progressing, until it completely succeeded.
br1-eno3-nncp   ConfigurationProgressing
br1-eno3-nncp   ConfigurationProgressing
br1-eno3-nncp   ConfigurationProgressing
br1-eno3-nncp   ConfigurationProgressing
br1-eno3-nncp   SuccessfullyConfigured
br1-eno3-nncp   SuccessfullyConfigured

8. Verified, on each worker, that the bridge (br1-eno3) was created successfully.
[core@cnv-qe-infra-33 yossi]$ nmcli c show
NAME              UUID                                  TYPE      DEVICE   
Wired Connection  e3018248-d4e3-4e0e-8946-7cf74e526b60  ethernet  eno1     
br1-eno3          73272c12-fd5a-47f3-ba1c-85eb41655380  bridge    br1-eno3 
eno3              19d51264-7d7d-4284-82e3-48e142f31ed9  ethernet  eno3     
Wired Connection  fcbd3574-c06a-4b25-b465-72e26c9836d9  ethernet  --       
[core@cnv-qe-infra-33 yossi]$ 
[core@cnv-qe-infra-33 yossi]$ 
[core@cnv-qe-infra-33 yossi]$ ip link show dev br1-eno3
151: br1-eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 78:ac:44:4b:3c:e2 brd ff:ff:ff:ff:ff:ff

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.8.6 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:4843

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days