Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1829080

Summary: 4.5 promote job failing on "[sig-network] Services should be rejected when no endpoints exist [Skipped:Network/OVNKubernetes]"
Product: OpenShift Container Platform Reporter: Micah Abbott <miabbott>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: unspecified CC: mkumatag, walters, wking
Version: 4.5   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-01 17:02:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Micah Abbott 2020-04-28 20:38:17 UTC 
The `promote-release-openshift-machine-os-content-e2e-aws-4.5` job has recently started failing and it appears to coincide with the change to RHCOS to start using RHEL 8.2 content.

One of the failing tests that jumps out is "[sig-network] Services should be rejected when no endpoints exist [Skipped:Network/OVNKubernetes]"

https://storage.googleapis.com/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.5/10242/build-log.txt

The logs contain errors like this:

```
[It] should be rejected when no endpoints exist [Skipped:Network/OVNKubernetes] [Skipped:ibmcloud] [Suite:openshift/conformance/parallel] [Suite:k8s]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/network/service.go:2582
STEP: creating a service with no endpoints
STEP: creating execpod-noendpoints on node ip-10-0-134-8.ec2.internal
Apr 24 21:25:06.568: INFO: Creating new exec pod
Apr 24 21:25:12.778: INFO: waiting up to 30s to connect to no-pods:80
STEP: hitting service no-pods:80 from pod execpod-noendpoints on node ip-10-0-134-8.ec2.internal
Apr 24 21:25:12.778: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-d982qc1l-8c7ff.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-services-7069 execpod-noendpoints982dr -- /bin/sh -x -c /agnhost connect --timeout=3s no-pods:80'
Apr 24 21:25:16.197: INFO: rc: 1
Apr 24 21:25:16.197: INFO: error didn't contain 'REFUSED', keep trying: error running /usr/bin/kubectl --server=https://api.ci-op-d982qc1l-8c7ff.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-services-7069 execpod-noendpoints982dr -- /bin/sh -x -c /agnhost connect --timeout=3s no-pods:80:
Command stdout:

stderr:
+ /agnhost connect --timeout=3s no-pods:80
TIMEOUT
command terminated with exit code 1
```


Since RHCOS uses the same RHEL content across all the versions, the 4.3 and 4.4 promote jobs are showing similar failures:

https://search.svc.ci.openshift.org/?search=failed%3A.*Services+should+be+rejected+when+no+endpoints+exist&maxAge=48h&context=1&type=build-log&name=promote&maxMatches=5&maxBytes=20971520&groupBy=job



The RPM diff between RHEL 8.1 and RHEL 8.2 for RHCOS 4.5 looks like:

```
ostree diff commit from: 7fd358579a296b3b9e5a94e79a7ba91d7fe1970c7a07dbddf659b712d5e42fdf
ostree diff commit to:   d81d5af73b9ae67a1c6e20d1bee13f9e412b29b1e283dac01270b1e8a3bfe0a7
Upgraded:
  NetworkManager 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8
  NetworkManager-libnm 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8
  NetworkManager-ovs 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8
  NetworkManager-team 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8
  adcli 0.8.2-3.el8 -> 0.8.2-5.el8
  audit 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8
  audit-libs 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8
  bind-export-libs 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  bind-libs 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  bind-libs-lite 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  bind-license 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  bind-utils 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  bubblewrap 0.3.0-1.el8 -> 0.4.0-1.el8
  cifs-utils 6.8-2.el8 -> 6.8-3.el8
  clevis 11-2.el8 -> 11-9.el8
  clevis-dracut 11-2.el8 -> 11-9.el8
  clevis-luks 11-2.el8 -> 11-9.el8
  clevis-systemd 11-2.el8 -> 11-9.el8
  container-selinux 2:2.124.0-1.module+el8.1.1+5259+bcdd613a -> 2:2.124.0-1.module+el8.2.0+5183+0d77e161
  containernetworking-plugins 0.8.3-4.module+el8.1.1+5259+bcdd613a -> 0.8.3-5.module+el8.2.0+5201+6b31f0d9
  containers-common 1:0.1.40-8.module+el8.1.1+5351+506397b0 -> 1:0.1.40-10.module+el8.2.0+5955+6cd70ceb
  cri-o 1.18.0-4.dev.rhaos4.5.git7d79f42.el8 -> 1.18.0-5.dev.rhaos4.5.git7da4850.el8
  criu 3.12-9.el8 -> 3.12-9.module+el8.2.0+5030+403b061b
  crypto-policies 20190807-1.git9b1477b.el8 -> 20191128-2.git23e1bf1.el8
  cryptsetup 2.2.0-2.el8 -> 2.2.2-1.el8
  cryptsetup-libs 2.2.0-2.el8 -> 2.2.2-1.el8
  cryptsetup-reencrypt 2.2.0-2.el8 -> 2.2.2-1.el8
  cups-libs 1:2.2.6-28.el8 -> 1:2.2.6-33.el8
  curl 7.61.1-11.el8 -> 7.61.1-12.el8
  device-mapper 8:1.02.163-5.el8 -> 8:1.02.169-3.el8
  device-mapper-event 8:1.02.163-5.el8 -> 8:1.02.169-3.el8
  device-mapper-event-libs 8:1.02.163-5.el8 -> 8:1.02.169-3.el8
  device-mapper-libs 8:1.02.163-5.el8 -> 8:1.02.169-3.el8
  device-mapper-multipath 0.8.0-5.el8 -> 0.8.3-3.el8
  device-mapper-multipath-libs 0.8.0-5.el8 -> 0.8.3-3.el8
  device-mapper-persistent-data 0.8.5-2.el8 -> 0.8.5-3.el8
  dhcp-client 12:4.3.6-34.el8 -> 12:4.3.6-40.el8
  dhcp-common 12:4.3.6-34.el8 -> 12:4.3.6-40.el8
  dhcp-libs 12:4.3.6-34.el8 -> 12:4.3.6-40.el8
  diffutils 3.6-5.el8 -> 3.6-6.el8
  dnsmasq 2.79-6.el8 -> 2.79-11.el8
  e2fsprogs 1.44.6-3.el8 -> 1.45.4-3.el8
  e2fsprogs-libs 1.44.6-3.el8 -> 1.45.4-3.el8
  elfutils-default-yama-scope 0.176-5.el8 -> 0.178-7.el8
  elfutils-libelf 0.176-5.el8 -> 0.178-7.el8
  elfutils-libs 0.176-5.el8 -> 0.178-7.el8
  file 5.33-8.el8 -> 5.33-13.el8
  file-libs 5.33-8.el8 -> 5.33-13.el8
  firewalld-filesystem 0.7.0-5.el8_1.1 -> 0.8.0-4.el8
  fuse-overlayfs 0.7.2-5.module+el8.1.1+6114+953c5a57 -> 0.7.2-5.module+el8.2.0+6060+9dbc027d
  glib2 2.56.4-7.el8 -> 2.56.4-8.el8
  glibc 2.28-72.el8_1.1 -> 2.28-101.el8
  glibc-all-langpacks 2.28-72.el8_1.1 -> 2.28-101.el8
  glibc-common 2.28-72.el8_1.1 -> 2.28-101.el8
  gnutls 3.6.8-8.el8 -> 3.6.8-9.el8
  grub2-common 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-efi-x64 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-pc 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-pc-modules 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-tools 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-tools-extra 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grub2-tools-minimal 1:2.02-78.el8_1.1 -> 1:2.02-81.el8
  grubby 8.40-37.el8 -> 8.40-38.el8
  gssproxy 0.8.0-14.el8 -> 0.8.0-15.el8
  hwdata 0.314-8.2.el8_1 -> 0.314-8.4.el8
  info 6.5-4.el8 -> 6.5-6.el8
  initscripts 10.00.4-1.el8 -> 10.00.6-1.el8
  ipcalc 0.2.4-3.el8 -> 0.2.4-4.el8
  iproute 4.18.0-15.el8 -> 5.3.0-1.el8
  iscsi-initiator-utils 6.2.0.877-1.gitf71581b.el8 -> 6.2.0.878-4.gitd791ce0.el8
  iscsi-initiator-utils-iscsiuio 6.2.0.877-1.gitf71581b.el8 -> 6.2.0.878-4.gitd791ce0.el8
  kernel 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kernel-core 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kernel-devel 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kernel-headers 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kernel-modules 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kernel-modules-extra 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8
  kmod 25-13.el8 -> 25-16.el8
  kmod-libs 25-13.el8 -> 25-16.el8
  kpartx 0.8.0-5.el8 -> 0.8.3-3.el8
  krb5-libs 1.17-9.el8 -> 1.17-18.el8
  libblkid 2.32.1-17.el8 -> 2.32.1-22.el8
  libcap 2.26-1.el8 -> 2.26-3.el8
  libcap-ng 0.7.9-4.el8 -> 0.7.9-5.el8
  libcom_err 1.44.6-3.el8 -> 1.45.4-3.el8
  libcurl 7.61.1-11.el8 -> 7.61.1-12.el8
  libdrm 2.4.98-2.el8 -> 2.4.100-1.el8
  libfdisk 2.32.1-17.el8 -> 2.32.1-22.el8
  libgcc 8.3.1-4.5.el8 -> 8.3.1-5.el8
  libgomp 8.3.1-4.5.el8 -> 8.3.1-5.el8
  libibverbs 22.3-1.el8 -> 26.0-8.el8
  libipa_hbac 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libldb 1.5.4-2.el8 -> 2.0.7-3.el8
  libluksmeta 9-3.el8_1.1 -> 9-4.el8
  libmaxminddb 1.2.0-6.el8 -> 1.2.0-7.el8
  libmodulemd1 1.8.11-4.el8_1 -> 1.8.16-0.2.8.2.1
  libmount 2.32.1-17.el8 -> 2.32.1-22.el8
  libmspack 0.7-0.1.alpha.el8.3 -> 0.7-0.3.alpha.el8.4
  libndp 1.7-1.el8 -> 1.7-3.el8
  libnfsidmap 1:2.3.3-26.el8 -> 1:2.3.3-31.el8
  libnl3 3.4.0-5.el8 -> 3.5.0-1.el8
  libnl3-cli 3.4.0-5.el8 -> 3.5.0-1.el8
  librepo 1.10.3-3.el8 -> 1.11.0-2.el8
  libreport-filesystem 2.9.5-9.el8 -> 2.9.5-10.el8
  libselinux 2.9-2.1.el8 -> 2.9-3.el8
  libselinux-utils 2.9-2.1.el8 -> 2.9-3.el8
  libsemanage 2.9-1.el8 -> 2.9-2.el8
  libsmartcols 2.32.1-17.el8 -> 2.32.1-22.el8
  libsmbclient 4.10.4-101.el8_1 -> 4.11.2-13.el8
  libsolv 0.7.4-3.el8 -> 0.7.7-1.el8
  libss 1.44.6-3.el8 -> 1.45.4-3.el8
  libsss_autofs 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libsss_certmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libsss_idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libsss_nss_idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libsss_sudo 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  libstdc++ 8.3.1-4.5.el8 -> 8.3.1-5.el8
  libtalloc 2.1.16-3.el8 -> 2.2.0-7.el8
  libtdb 1.3.18-2.el8 -> 1.4.2-2.el8
  libteam 1.28-4.el8 -> 1.29-1.el8
  libtevent 0.9.39-2.el8 -> 0.10.0-2.el8
  libuuid 2.32.1-17.el8 -> 2.32.1-22.el8
  libwbclient 4.10.4-101.el8_1 -> 4.11.2-13.el8
  libxml2 2.9.7-5.el8 -> 2.9.7-7.el8
  libxslt 1.1.32-3.el8 -> 1.1.32-4.el8
  linux-firmware 20190516-94.git711d3297.el8 -> 20191202-97.gite8a0f4c9.el8
  luksmeta 9-3.el8_1.1 -> 9-4.el8
  lvm2 8:2.03.05-5.el8 -> 8:2.03.08-3.el8
  lvm2-libs 8:2.03.05-5.el8 -> 8:2.03.08-3.el8
  machine-config-daemon 4.5.0-202004241417.git.1.e84f244.el8 -> 4.5.0-202004241817.git.1.09877e1.el8
  make 1:4.2.1-9.el8 -> 1:4.2.1-10.el8
  mdadm 4.1-9.el8 -> 4.1-13.el8
  microcode_ctl 4:20190618-1.20191115.3.el8_1 -> 4:20191115-4.el8
  mozjs60 60.9.0-3.el8 -> 60.9.0-4.el8
  nfs-utils 1:2.3.3-26.el8 -> 1:2.3.3-31.el8
  nftables 1:0.9.0-14.el8_1.1 -> 1:0.9.3-12.el8
  numactl-libs 2.0.12-7.el8 -> 2.0.12-9.el8
  open-vm-tools 10.3.10-3.el8_1.1 -> 11.0.0-4.el8
  openssl 1:1.1.1c-2.el8_1.1 -> 1:1.1.1c-15.el8
  openssl-libs 1:1.1.1c-2.el8_1.1 -> 1:1.1.1c-15.el8
  openssl-pkcs11 0.4.8-2.el8 -> 0.4.10-2.el8
  pam 1.3.1-4.el8 -> 1.3.1-8.el8
  passwd 0.80-2.el8 -> 0.80-3.el8
  platform-python 3.6.8-15.1.el8 -> 3.6.8-23.el8
  platform-python-pip 9.0.3-15.el8 -> 9.0.3-16.el8
  policycoreutils 2.9-3.el8_1.1 -> 2.9-9.el8
  policycoreutils-python-utils 2.9-3.el8_1.1 -> 2.9-9.el8
  polkit 0.115-9.el8 -> 0.115-11.el8
  polkit-libs 0.115-9.el8 -> 0.115-11.el8
  psmisc 23.1-3.el8 -> 23.1-4.el8
  python3-audit 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8
  python3-bind 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8
  python3-libs 3.6.8-15.1.el8 -> 3.6.8-23.el8
  python3-libselinux 2.9-2.1.el8 -> 2.9-3.el8
  python3-libsemanage 2.9-1.el8 -> 2.9-2.el8
  python3-pip-wheel 9.0.3-15.el8 -> 9.0.3-16.el8
  python3-ply 3.9-7.el8 -> 3.9-8.el8
  python3-policycoreutils 2.9-3.el8_1.1 -> 2.9-9.el8
  python3-setools 4.2.2-1.el8 -> 4.2.2-2.el8
  python3-sssdconfig 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  rdma-core 22.3-1.el8 -> 26.0-8.el8
  rpcbind 1.2.5-4.el8 -> 1.2.5-7.el8
  rpm 4.14.2-26.el8_1 -> 4.14.2-37.el8
  rpm-libs 4.14.2-26.el8_1 -> 4.14.2-37.el8
  rpm-plugin-selinux 4.14.2-26.el8_1 -> 4.14.2-37.el8
  rsync 3.1.3-6.el8 -> 3.1.3-7.el8
  samba-client-libs 4.10.4-101.el8_1 -> 4.11.2-13.el8
  samba-common 4.10.4-101.el8_1 -> 4.11.2-13.el8
  samba-common-libs 4.10.4-101.el8_1 -> 4.11.2-13.el8
  selinux-policy 3.14.3-20.el8 -> 3.14.3-41.el8
  selinux-policy-targeted 3.14.3-20.el8 -> 3.14.3-41.el8
  setup 2.12.2-2.el8_1.1 -> 2.12.2-5.el8
  sg3_utils 1.44-3.el8 -> 1.44-5.el8
  sg3_utils-libs 1.44-3.el8 -> 1.44-5.el8
  skopeo 1:0.1.40-8.module+el8.1.1+5351+506397b0 -> 1:0.1.40-10.module+el8.2.0+5955+6cd70ceb
  slirp4netns 0.4.2-3.git21fdece.module+el8.1.1+5657+524a77d7 -> 0.4.2-3.git21fdece.module+el8.2.0+5658+9a15711d
  socat 1.7.3.2-6.el8 -> 1.7.3.3-2.el8
  sqlite-libs 3.26.0-4.el8_1 -> 3.26.0-6.el8
  sssd 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-ad 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-client 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-common 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-common-pac 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-ipa 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-krb5 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-krb5-common 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-ldap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-nfs-idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  sssd-proxy 2.2.0-19.el8_1.1 -> 2.2.3-20.el8
  strace 4.24-5.el8 -> 4.24-9.el8
  subscription-manager-rhsm-certificates 1.25.17-1.el8 -> 1.26.16-1.el8
  sudo 1.8.25p1-8.el8_1.1 -> 1.8.29-5.el8
  systemd 239-27.el8 -> 239-29.el8
  systemd-journal-remote 239-27.el8 -> 239-29.el8
  systemd-libs 239-27.el8 -> 239-29.el8
  systemd-pam 239-27.el8 -> 239-29.el8
  systemd-udev 239-27.el8 -> 239-29.el8
  teamd 1.28-4.el8 -> 1.29-1.el8
  tpm2-tools 3.1.4-5.el8 -> 3.2.1-1.el8
  unbound-libs 1.7.3-8.el8 -> 1.7.3-10.el8
  util-linux 2.32.1-17.el8 -> 2.32.1-22.el8
  which 2.21-10.el8 -> 2.21-12.el8
  xfsprogs 5.0.0-1.el8 -> 5.0.0-2.el8
  zlib 1.2.11-10.el8 -> 1.2.11-13.el8
Removed:
  libdnet-1.12-26.el8.x86_64
  libicu-60.3-2.el8_1.x86_64
Added:
  elfutils-debuginfod-client-0.178-7.el8.x86_64
  kbd-2.0.4-8.el8.x86_64
  kbd-legacy-2.0.4-8.el8.noarch
  kbd-misc-2.0.4-8.el8.noarch
  libxkbcommon-0.9.1-1.el8.x86_64
  libzstd-1.4.2-2.el8.x86_64
  xkeyboard-config-2.28-1.el8.noarch
```
Comment 1 Micah Abbott 2020-04-28 20:39:12 UTC 
I found a similar BZ but it appears to be for FCOS/RHEL7 nodes - https://bugzilla.redhat.com/show_bug.cgi?id=1781575
Comment 2 Colin Walters 2020-05-01 17:02:07 UTC 
For now marking as a duplicate because evidence points that way.

*** This bug has been marked as a duplicate of bug 1781575 ***
Comment 3 mkumatag 2020-05-02 14:32:03 UTC 
(In reply to Colin Walters from comment #2)
> For now marking as a duplicate because evidence points that way.
> 
> *** This bug has been marked as a duplicate of bug 1781575 ***

I don't really have access to see what is happening in bug 1781575, can someone cc me into that issue?
Comment 4 W. Trevor King 2020-05-04 23:55:15 UTC 
> I don't really have access...

It's public now :).