Seems like the Hybird Overlay flag is enabled by default: https://github.com/openshift/cluster-network-operator/blob/6b85589493bb5ef25e9a75c4d19513009f1ce9e6/bindata/network/ovn-kubernetes/004-config.yaml#L33 but no CIDRs need to be set necessarily? I still don't understand what this feature does.

(In reply to Surya Seetharaman from comment #3)
> Seems like the Hybird Overlay flag is enabled by default:
> https://github.com/openshift/cluster-network-operator/blob/
> 6b85589493bb5ef25e9a75c4d19513009f1ce9e6/bindata/network/ovn-kubernetes/004-
> config.yaml#L33 but no CIDRs need to be set necessarily? I still don't
> understand what this feature does.

Okay its not enabled by default: https://github.com/openshift/cluster-network-operator/blob/1808251ababd4b60002d424bbccad2203609567d/pkg/network/ovn_kubernetes.go#L104-L120

On a default config cluster:

I0706 06:03:26.897124    3686 config.go:1455] Parsed config: {Default:{MTU:1360 ConntrackZone:64000 EncapType:geneve EncapIP: EncapPort:6081 InactivityProbe:100000 OpenFlowProbe:180 RawClusterSubnets:10.128.0.0/16/23 ClusterSubnets:[]} Logging:{File: CNIFile: Level:4 LogFileMaxSize:100 LogFileMaxBackups:5 LogFileMaxAge:5 ACLLoggingRateLimit:20} Monitoring:{RawNetFlowTargets: RawSFlowTargets: RawIPFIXTargets: NetFlowTargets:[] SFlowTargets:[] IPFIXTargets:[]} CNI:{ConfDir:/etc/cni/net.d Plugin:ovn-k8s-cni-overlay} OVNKubernetesFeature:{EnableEgressIP:true EnableEgressFirewall:true} Kubernetes:{Kubeconfig: CACert: APIServer:https://api-int.ci-ln-7zfsdb2-f76d1.origin-ci-int-gce.dev.openshift.com:6443 Token: CompatServiceCIDR: RawServiceCIDRs:172.30.0.0/16 ServiceCIDRs:[] OVNConfigNamespace:openshift-ovn-kubernetes MetricsBindAddress: OVNMetricsBindAddress: MetricsEnablePprof:false OVNEmptyLbEvents:false PodIP: RawNoHostSubnetNodes: NoHostSubnetNodes:nil HostNetworkNamespace:openshift-host-network} OvnNorth:{Address: PrivKey: Cert: CACert: CertCommonName: Scheme: northbound:false exec:<nil>} OvnSouth:{Address: PrivKey: Cert: CACert: CertCommonName: Scheme: northbound:false exec:<nil>} Gateway:{Mode:local Interface: NextHop: VLANID:0 NodeportEnable:true DisableSNATMultipleGWs:false V4JoinSubnet:100.64.0.0/16 V6JoinSubnet:fd98::/64 DisablePacketMTUCheck:false} MasterHA:{ElectionLeaseDuration:60 ElectionRenewDeadline:30 ElectionRetryPeriod:20} HybridOverlay:{Enabled:false RawClusterSubnets: ClusterSubnets:[] VXLANPort:4789} OvnKubeNode:{Mode:full}}

Looks like the upgrade test cluster had this enabled.

> 
> Looks like the upgrade test cluster had this enabled.

hybridOverlayConfig: {}

This in itself makes it enabled.

https://github.com/ovn-org/ovn-kubernetes/pull/2314

upstream fix posted.

Upstream fixed is merged, waiting for the cherry-pick process to downstream.

Downstream cherry-pick merged. Moving to modified.

verified this bug on 4.9.0-0.nightly-2021-08-03-200806

steps: 

prepare install-config.yaml with OVNKubernetes 
1. Create manifests with below:
 openshift-install create manifests

2. Create the following file

cat <<EOF > manifests/cluster-network-03-config.yml
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  defaultNetwork:
    ovnKubernetesConfig:
      hybridOverlayConfig: {}
       
EOF

3. create the cluster 

 /openshift-install create cluster --log-level debug

4. After the cluster is running and check no error logs in ovn-controller logs

for i in $(oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers | awk '{ print $1}') ; do oc logs $i -n openshift-ovn-kubernetes -c ovn-controller | grep -i error ; done

5.  and also check no related lr-policy
oc rsh -n openshift-ovn-kubernetes ovnkube-master-tg7kt
Defaulting container name to northd.
Use 'oc describe pod/ovnkube-master-tg7kt -n openshift-ovn-kubernetes' to see all of the containers in this pod.
sh-4.4# ovn-nbctl lr-policy-list ovn_cluster_router | grep 1002
sh-4.4# exit

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759