Roy, This is most likely a case where the required network ACLs aren't in place. We fixed this in GCP UPI here https://github.com/openshift/installer/pull/2984 udp ports 9000-9999 and 30000-32767 need to be open between control plane and nodes in both directions
We don't block any of these at all, we have layer 2 networking and no libvirt network filters set by default. Gal please share you're findings so far on the bug.
I tried to debug the issue and I found that it is a SDN bug, I opened: https://bugzilla.redhat.com/show_bug.cgi?id=1808856 I also found the the problem doesn't occur when we use Network Type: OVNKubernetes. After talking with network engineer I understood that we are going towards OVNKubernetes as the default network in the future. So I have created this PR to switch to OVNKubernetes on our e2e Jobs and it will fix the NodePort test fails: https://github.com/openshift/release/pull/7392
Scott have a look at this search - a lot of other platforms are failing on this and other related network tests, constantly https://search.svc.ci.openshift.org/?search=failed%3A+.*+%22%5C%5Bsig-network%5C%5D+.*Networking+Granular+Checks%3A+&maxAge=168h&context=0&type=all
Scott please notice that there is a open bug on the problem: https://bugzilla.redhat.com/show_bug.cgi?id=1794714
Closing as a dupe given your confirmation that there are no network ACLs restricting connectivity between masters and workers. *** This bug has been marked as a duplicate of bug 1794714 ***