Description of problem: Right now, the only way to see what incremental processing nodes' handlers return in ovn-controller during a DB change processing is to enable DBG logs. It might be useful for debugging to have (coverage) counters instead that show how many times an I-P node could process changes incrementally/had to recompute/had to abort.
test with following script: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 systemctl restart ovn-controller sleep 5 ovn-appctl -t ovn-controller inc-engine/show-stats ovn-appctl -t ovn-controller inc-engine/clear-stats ovn-appctl -t ovn-controller inc-engine/show-stats ovn-appctl -t ovn-controller inc-engine/show-stats result on ovn-2021-21.03.0-21.el8fdp.x86_64 [root@wsfd-advnetlab18 bz1890902]# rpm -qa | grep -E "ovn-2021|openvswitch2.15" ovn-2021-central-21.03.0-21.el8fdp.x86_64 ovn-2021-21.03.0-21.el8fdp.x86_64 ovn-2021-host-21.03.0-21.el8fdp.x86_64 openvswitch2.15-2.15.0-15.el8fdp.x86_64 + sleep 5 + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 9 - compute: 0 - abort: 0 Node: addr_sets - recompute: 1 - compute: 0 - abort: 1 Node: SB_port_group - recompute: 8 - compute: 0 - abort: 0 Node: port_groups - recompute: 1 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 8 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 7 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 7 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 7 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 7 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 7 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 7 - compute: 0 - abort: 0 Node: OVS_port - recompute: 7 - compute: 0 - abort: 0 Node: OVS_interface - recompute: 7 - compute: 0 - abort: 0 Node: runtime_data - recompute: 1 - compute: 0 - abort: 0 Node: mff_ovn_geneve - recompute: 7 - compute: 0 - abort: 0 Node: ct_zones - recompute: 1 - compute: 0 - abort: 0 Node: physical_flow_changes - recompute: 1 - compute: 0 - abort: 0 Node: SB_encap - recompute: 6 - compute: 0 - abort: 0 Node: SB_multicast_group - recompute: 6 - compute: 0 - abort: 0 Node: SB_mac_binding - recompute: 6 - compute: 0 - abort: 0 Node: SB_logical_flow - recompute: 6 - compute: 0 - abort: 0 Node: SB_logical_dp_group - recompute: 6 - compute: 0 - abort: 0 Node: SB_dhcp_options - recompute: 6 - compute: 0 - abort: 0 Node: SB_dhcpv6_options - recompute: 6 - compute: 0 - abort: 0 Node: SB_dns - recompute: 6 - compute: 0 - abort: 0 Node: SB_load_balancer - recompute: 6 - compute: 0 - abort: 0 Node: SB_fdb - recompute: 6 - compute: 0 - abort: 0 Node: flow_output - recompute: 1 - compute: 0 - abort: 0 + ovn-appctl -t ovn-controller inc-engine/clear-stats + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 2 - compute: 0 - abort: 0 Node: addr_sets - recompute: 0 - compute: 0 - abort: 0 Node: SB_port_group - recompute: 2 - compute: 0 - abort: 0 Node: port_groups - recompute: 0 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 2 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 2 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 2 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 2 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 2 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 2 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 2 - compute: 0 - abort: 0 ...... Hi lorenzo, what does every item mean in the output? so that we can check if the result is right.
(In reply to Jianlin Shi from comment #3) > test with following script: > > systemctl start openvswitch > systemctl start ovn-northd > ovn-nbctl set-connection ptcp:6641 > ovn-sbctl set-connection ptcp:6642 > ovs-vsctl set open . external_ids:system-id=hv1 > external_ids:ovn-remote=tcp:20.0.175.25:6642 > external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 > systemctl restart ovn-controller > > sleep 5 > ovn-appctl -t ovn-controller inc-engine/show-stats > ovn-appctl -t ovn-controller inc-engine/clear-stats > ovn-appctl -t ovn-controller inc-engine/show-stats > ovn-appctl -t ovn-controller inc-engine/show-stats > > result on ovn-2021-21.03.0-21.el8fdp.x86_64 > > [root@wsfd-advnetlab18 bz1890902]# rpm -qa | grep -E > "ovn-2021|openvswitch2.15" > ovn-2021-central-21.03.0-21.el8fdp.x86_64 > ovn-2021-21.03.0-21.el8fdp.x86_64 > ovn-2021-host-21.03.0-21.el8fdp.x86_64 > openvswitch2.15-2.15.0-15.el8fdp.x86_64 > > + sleep 5 > + ovn-appctl -t ovn-controller inc-engine/show-stats > Node: SB_address_set > - recompute: 9 > - compute: 0 > - abort: 0 > Node: addr_sets > - recompute: 1 > - compute: 0 > - abort: 1 > Node: SB_port_group > - recompute: 8 > - compute: 0 > - abort: 0 > Node: port_groups > - recompute: 1 > - compute: 0 > - abort: 0 > Node: ofctrl_is_connected > - recompute: 8 > - compute: 0 > > - abort: 0 > Node: OVS_open_vswitch > - recompute: 7 > - compute: 0 > - abort: 0 > Node: OVS_bridge > - recompute: 7 > - compute: 0 > > - abort: 0 > Node: OVS_qos > - recompute: 7 > - compute: 0 > - abort: 0 > Node: SB_chassis > - recompute: 7 > - compute: 0 > - abort: 0 > Node: SB_datapath_binding > - recompute: 7 > - compute: 0 > - abort: 0 > Node: SB_port_binding > - recompute: 7 > - compute: 0 > - abort: 0 > Node: OVS_port > - recompute: 7 > - compute: 0 > - abort: 0 > Node: OVS_interface > - recompute: 7 > - compute: 0 > - abort: 0 > Node: runtime_data > - recompute: 1 > - compute: 0 > - abort: 0 > Node: mff_ovn_geneve > - recompute: 7 > - compute: 0 > - abort: 0 > Node: ct_zones > - recompute: 1 > - compute: 0 > - abort: 0 > Node: physical_flow_changes > - recompute: 1 > - compute: 0 > - abort: 0 > Node: SB_encap > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_multicast_group > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_mac_binding > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_logical_flow > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_logical_dp_group > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_dhcp_options > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_dhcpv6_options > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_dns > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_load_balancer > - recompute: 6 > - compute: 0 > - abort: 0 > Node: SB_fdb > - recompute: 6 > - compute: 0 > - abort: 0 > Node: flow_output > - recompute: 1 > - compute: 0 > - abort: 0 > + ovn-appctl -t ovn-controller inc-engine/clear-stats > + ovn-appctl -t ovn-controller inc-engine/show-stats > Node: SB_address_set > - recompute: 2 > - compute: 0 > - abort: 0 > Node: addr_sets > - recompute: 0 > - compute: 0 > - abort: 0 > Node: SB_port_group > - recompute: 2 > - compute: 0 > - abort: 0 > Node: port_groups > - recompute: 0 > - compute: 0 > - abort: 0 > Node: ofctrl_is_connected > - recompute: 2 > - compute: 0 > - abort: 0 > Node: OVS_open_vswitch > - recompute: 2 > - compute: 0 > - abort: 0 > Node: OVS_bridge > - recompute: 2 > - compute: 0 > - abort: 0 > Node: OVS_qos > - recompute: 2 > - compute: 0 > - abort: 0 > Node: SB_chassis > - recompute: 2 > - compute: 0 > - abort: 0 > Node: SB_datapath_binding > - recompute: 2 > - compute: 0 > - abort: 0 > Node: SB_port_binding > - recompute: 2 > - compute: 0 > - abort: 0 > ...... > > Hi lorenzo, > what does every item mean in the output? so that we can check if the result > is right. Hi Jianlin Shi, can you please check the new commands (inc-engine/show-stats, inc-engine/clear-stats) work fine? the output of inc-engine/show-stats command are the new counters added in Incremental Processing engine running in ovn-controller
Verified on ovn2.13-20.12.0-104: [root@wsfd-advnetlab18 bz1890902]# rpm -qa | grep -E "ovn2.13|openvswitch2.13" ovn2.13-20.12.0-104.el8fdp.x86_64 python3-openvswitch2.13-2.13.0-99.el8fdp.x86_64 ovn2.13-central-20.12.0-104.el8fdp.x86_64 openvswitch2.13-2.13.0-99.el8fdp.x86_64 ovn2.13-host-20.12.0-104.el8fdp.x86_64 + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 14 - compute: 0 - abort: 0 Node: addr_sets - recompute: 1 - compute: 0 - abort: 1 Node: SB_port_group - recompute: 13 - compute: 0 - abort: 0 Node: port_groups - recompute: 1 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 13 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 12 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 12 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 12 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 12 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 11 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 11 - compute: 0 - abort: 0 Node: OVS_port - recompute: 11 - compute: 0 - abort: 0 Node: OVS_interface - recompute: 10 - compute: 0 - abort: 0 Node: runtime_data - recompute: 1 - compute: 0 - abort: 0 Node: mff_ovn_geneve - recompute: 10 - compute: 0 - abort: 0 Node: ct_zones - recompute: 1 - compute: 0 - abort: 0 Node: physical_flow_changes - recompute: 1 - compute: 0 - abort: 0 Node: SB_encap - recompute: 10 - compute: 0 - abort: 0 Node: SB_multicast_group - recompute: 10 - compute: 0 - abort: 0 Node: SB_mac_binding - recompute: 10 - compute: 0 - abort: 0 Node: SB_logical_flow - recompute: 10 - compute: 0 - abort: 0 Node: SB_logical_dp_group - recompute: 10 - compute: 0 - abort: 0 Node: SB_dhcp_options - recompute: 10 - compute: 0 - abort: 0 Node: SB_dhcpv6_options - recompute: 10 - compute: 0 - abort: 0 Node: SB_dns - recompute: 10 - compute: 0 - abort: 0 Node: SB_load_balancer - recompute: 10 - compute: 0 - abort: 0 Node: SB_fdb - recompute: 10 - compute: 0 - abort: 0 Node: flow_output - recompute: 1 - compute: 0 - abort: 0 + ovn-appctl -t ovn-controller inc-engine/clear-stats + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 2 - compute: 0 - abort: 0 Node: addr_sets - recompute: 0 - compute: 0 - abort: 0 Node: SB_port_group - recompute: 2 - compute: 0 - abort: 0 Node: port_groups - recompute: 0 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 2 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 2 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 2 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 2 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 2 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 2 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 2 - compute: 0 - abort: 0 Node: OVS_port - recompute: 2 - compute: 0 - abort: 0 Node: OVS_interface - recompute: 2 - compute: 0 - abort: 0 Node: runtime_data - recompute: 0 - compute: 0 - abort: 0 Node: mff_ovn_geneve - recompute: 2 - compute: 0 - abort: 0 Node: ct_zones - recompute: 0 - compute: 0 - abort: 0 Node: physical_flow_changes - recompute: 0 - compute: 0 - abort: 0 Node: SB_encap - recompute: 2 - compute: 0 - abort: 0 Node: SB_multicast_group - recompute: 2 - compute: 0 - abort: 0 Node: SB_mac_binding - recompute: 2 - compute: 0 - abort: 0 Node: SB_logical_flow - recompute: 2 - compute: 0 - abort: 0 Node: SB_logical_dp_group - recompute: 2 - compute: 0 - abort: 0 Node: SB_dhcp_options - recompute: 2 - compute: 0 - abort: 0 Node: SB_dhcpv6_options - recompute: 2 - compute: 0 - abort: 0 Node: SB_dns - recompute: 2 - compute: 0 - abort: 0 Node: SB_load_balancer - recompute: 2 - compute: 0 - abort: 0 Node: SB_fdb - recompute: 2 - compute: 0 - abort: 0 Node: flow_output - recompute: 0 - compute: 0 - abort: 0
Verified on ovn2.13-20.12.0-104.el7: + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 13 - compute: 0 - abort: 0 Node: addr_sets - recompute: 1 - compute: 0 - abort: 1 Node: SB_port_group - recompute: 12 - compute: 0 - abort: 0 Node: port_groups - recompute: 1 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 12 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 11 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 11 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 11 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 11 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 10 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 10 - compute: 0 - abort: 0 Node: OVS_port - recompute: 10 - compute: 0 - abort: 0 Node: OVS_interface - recompute: 9 - compute: 0 - abort: 0 Node: runtime_data - recompute: 1 - compute: 0 - abort: 0 Node: mff_ovn_geneve - recompute: 9 - compute: 0 - abort: 0 Node: ct_zones - recompute: 1 - compute: 0 - abort: 0 Node: physical_flow_changes - recompute: 1 - compute: 0 - abort: 0 Node: SB_encap - recompute: 9 - compute: 0 - abort: 0 Node: SB_multicast_group - recompute: 9 - compute: 0 - abort: 0 Node: SB_mac_binding - recompute: 9 - compute: 0 - abort: 0 Node: SB_logical_flow - recompute: 9 - compute: 0 - abort: 0 Node: SB_logical_dp_group - recompute: 9 - compute: 0 - abort: 0 Node: SB_dhcp_options - recompute: 9 - compute: 0 - abort: 0 Node: SB_dhcpv6_options - recompute: 9 - compute: 0 - abort: 0 Node: SB_dns - recompute: 9 - compute: 0 - abort: 0 Node: SB_load_balancer - recompute: 9 - compute: 0 - abort: 0 Node: SB_fdb - recompute: 9 - compute: 0 - abort: 0 Node: flow_output - recompute: 1 - compute: 0 - abort: 0 + ovn-appctl -t ovn-controller inc-engine/clear-stats + ovn-appctl -t ovn-controller inc-engine/show-stats Node: SB_address_set - recompute: 2 - compute: 0 - abort: 0 Node: addr_sets - recompute: 0 - compute: 0 - abort: 0 Node: SB_port_group - recompute: 2 - compute: 0 - abort: 0 Node: port_groups - recompute: 0 - compute: 0 - abort: 0 Node: ofctrl_is_connected - recompute: 2 - compute: 0 - abort: 0 Node: OVS_open_vswitch - recompute: 2 - compute: 0 - abort: 0 Node: OVS_bridge - recompute: 2 - compute: 0 - abort: 0 Node: OVS_qos - recompute: 2 - compute: 0 - abort: 0 Node: SB_chassis - recompute: 2 - compute: 0 - abort: 0 Node: SB_datapath_binding - recompute: 2 - compute: 0 - abort: 0 Node: SB_port_binding - recompute: 2 - compute: 0 - abort: 0 Node: OVS_port - recompute: 2 - compute: 0 - abort: 0 Node: OVS_interface - recompute: 2 - compute: 0 - abort: 0 Node: runtime_data - recompute: 0 - compute: 0 - abort: 0 Node: mff_ovn_geneve - recompute: 2 - compute: 0 - abort: 0 Node: ct_zones - recompute: 0 - compute: 0 - abort: 0 Node: physical_flow_changes - recompute: 0 - compute: 0 - abort: 0 Node: SB_encap - recompute: 2 - compute: 0 - abort: 0 Node: SB_multicast_group - recompute: 2 - compute: 0 - abort: 0 Node: SB_mac_binding - recompute: 2 - compute: 0 - abort: 0 Node: SB_logical_flow - recompute: 2 - compute: 0 - abort: 0 Node: SB_logical_dp_group - recompute: 2 - compute: 0 - abort: 0 Node: SB_dhcp_options - recompute: 2 - compute: 0 - abort: 0 Node: SB_dhcpv6_options - recompute: 2 - compute: 0 - abort: 0 Node: SB_dns - recompute: 2 - compute: 0 - abort: 0 Node: SB_load_balancer - recompute: 2 - compute: 0 - abort: 0 Node: SB_fdb - recompute: 2 - compute: 0 - abort: 0 Node: flow_output - recompute: 0 - compute: 0 - abort: 0 [root@wsfd-advnetlab19 bz1890902]# rpm -qa | grep -E "openvswitch2.13|ovn2.13" openvswitch2.13-2.13.0-90.el7fdp.x86_64 ovn2.13-20.12.0-104.el7fdp.x86_64 ovn2.13-central-20.12.0-104.el7fdp.x86_64 ovn2.13-host-20.12.0-104.el7fdp.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2080