The `promote-release-openshift-machine-os-content-e2e-aws-4.5` job has recently started failing and it appears to coincide with the change to RHCOS to start using RHEL 8.2 content. One of the failing tests that jumps out is "[sig-network] Services should be rejected when no endpoints exist [Skipped:Network/OVNKubernetes]" https://storage.googleapis.com/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.5/10242/build-log.txt The logs contain errors like this: ``` [It] should be rejected when no endpoints exist [Skipped:Network/OVNKubernetes] [Skipped:ibmcloud] [Suite:openshift/conformance/parallel] [Suite:k8s] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/network/service.go:2582 STEP: creating a service with no endpoints STEP: creating execpod-noendpoints on node ip-10-0-134-8.ec2.internal Apr 24 21:25:06.568: INFO: Creating new exec pod Apr 24 21:25:12.778: INFO: waiting up to 30s to connect to no-pods:80 STEP: hitting service no-pods:80 from pod execpod-noendpoints on node ip-10-0-134-8.ec2.internal Apr 24 21:25:12.778: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-d982qc1l-8c7ff.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-services-7069 execpod-noendpoints982dr -- /bin/sh -x -c /agnhost connect --timeout=3s no-pods:80' Apr 24 21:25:16.197: INFO: rc: 1 Apr 24 21:25:16.197: INFO: error didn't contain 'REFUSED', keep trying: error running /usr/bin/kubectl --server=https://api.ci-op-d982qc1l-8c7ff.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-services-7069 execpod-noendpoints982dr -- /bin/sh -x -c /agnhost connect --timeout=3s no-pods:80: Command stdout: stderr: + /agnhost connect --timeout=3s no-pods:80 TIMEOUT command terminated with exit code 1 ``` Since RHCOS uses the same RHEL content across all the versions, the 4.3 and 4.4 promote jobs are showing similar failures: https://search.svc.ci.openshift.org/?search=failed%3A.*Services+should+be+rejected+when+no+endpoints+exist&maxAge=48h&context=1&type=build-log&name=promote&maxMatches=5&maxBytes=20971520&groupBy=job The RPM diff between RHEL 8.1 and RHEL 8.2 for RHCOS 4.5 looks like: ``` ostree diff commit from: 7fd358579a296b3b9e5a94e79a7ba91d7fe1970c7a07dbddf659b712d5e42fdf ostree diff commit to: d81d5af73b9ae67a1c6e20d1bee13f9e412b29b1e283dac01270b1e8a3bfe0a7 Upgraded: NetworkManager 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8 NetworkManager-libnm 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8 NetworkManager-ovs 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8 NetworkManager-team 1:1.20.0-5.el8_1 -> 1:1.22.8-4.el8 adcli 0.8.2-3.el8 -> 0.8.2-5.el8 audit 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8 audit-libs 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8 bind-export-libs 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 bind-libs 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 bind-libs-lite 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 bind-license 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 bind-utils 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 bubblewrap 0.3.0-1.el8 -> 0.4.0-1.el8 cifs-utils 6.8-2.el8 -> 6.8-3.el8 clevis 11-2.el8 -> 11-9.el8 clevis-dracut 11-2.el8 -> 11-9.el8 clevis-luks 11-2.el8 -> 11-9.el8 clevis-systemd 11-2.el8 -> 11-9.el8 container-selinux 2:2.124.0-1.module+el8.1.1+5259+bcdd613a -> 2:2.124.0-1.module+el8.2.0+5183+0d77e161 containernetworking-plugins 0.8.3-4.module+el8.1.1+5259+bcdd613a -> 0.8.3-5.module+el8.2.0+5201+6b31f0d9 containers-common 1:0.1.40-8.module+el8.1.1+5351+506397b0 -> 1:0.1.40-10.module+el8.2.0+5955+6cd70ceb cri-o 1.18.0-4.dev.rhaos4.5.git7d79f42.el8 -> 1.18.0-5.dev.rhaos4.5.git7da4850.el8 criu 3.12-9.el8 -> 3.12-9.module+el8.2.0+5030+403b061b crypto-policies 20190807-1.git9b1477b.el8 -> 20191128-2.git23e1bf1.el8 cryptsetup 2.2.0-2.el8 -> 2.2.2-1.el8 cryptsetup-libs 2.2.0-2.el8 -> 2.2.2-1.el8 cryptsetup-reencrypt 2.2.0-2.el8 -> 2.2.2-1.el8 cups-libs 1:2.2.6-28.el8 -> 1:2.2.6-33.el8 curl 7.61.1-11.el8 -> 7.61.1-12.el8 device-mapper 8:1.02.163-5.el8 -> 8:1.02.169-3.el8 device-mapper-event 8:1.02.163-5.el8 -> 8:1.02.169-3.el8 device-mapper-event-libs 8:1.02.163-5.el8 -> 8:1.02.169-3.el8 device-mapper-libs 8:1.02.163-5.el8 -> 8:1.02.169-3.el8 device-mapper-multipath 0.8.0-5.el8 -> 0.8.3-3.el8 device-mapper-multipath-libs 0.8.0-5.el8 -> 0.8.3-3.el8 device-mapper-persistent-data 0.8.5-2.el8 -> 0.8.5-3.el8 dhcp-client 12:4.3.6-34.el8 -> 12:4.3.6-40.el8 dhcp-common 12:4.3.6-34.el8 -> 12:4.3.6-40.el8 dhcp-libs 12:4.3.6-34.el8 -> 12:4.3.6-40.el8 diffutils 3.6-5.el8 -> 3.6-6.el8 dnsmasq 2.79-6.el8 -> 2.79-11.el8 e2fsprogs 1.44.6-3.el8 -> 1.45.4-3.el8 e2fsprogs-libs 1.44.6-3.el8 -> 1.45.4-3.el8 elfutils-default-yama-scope 0.176-5.el8 -> 0.178-7.el8 elfutils-libelf 0.176-5.el8 -> 0.178-7.el8 elfutils-libs 0.176-5.el8 -> 0.178-7.el8 file 5.33-8.el8 -> 5.33-13.el8 file-libs 5.33-8.el8 -> 5.33-13.el8 firewalld-filesystem 0.7.0-5.el8_1.1 -> 0.8.0-4.el8 fuse-overlayfs 0.7.2-5.module+el8.1.1+6114+953c5a57 -> 0.7.2-5.module+el8.2.0+6060+9dbc027d glib2 2.56.4-7.el8 -> 2.56.4-8.el8 glibc 2.28-72.el8_1.1 -> 2.28-101.el8 glibc-all-langpacks 2.28-72.el8_1.1 -> 2.28-101.el8 glibc-common 2.28-72.el8_1.1 -> 2.28-101.el8 gnutls 3.6.8-8.el8 -> 3.6.8-9.el8 grub2-common 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-efi-x64 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-pc 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-pc-modules 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-tools 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-tools-extra 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grub2-tools-minimal 1:2.02-78.el8_1.1 -> 1:2.02-81.el8 grubby 8.40-37.el8 -> 8.40-38.el8 gssproxy 0.8.0-14.el8 -> 0.8.0-15.el8 hwdata 0.314-8.2.el8_1 -> 0.314-8.4.el8 info 6.5-4.el8 -> 6.5-6.el8 initscripts 10.00.4-1.el8 -> 10.00.6-1.el8 ipcalc 0.2.4-3.el8 -> 0.2.4-4.el8 iproute 4.18.0-15.el8 -> 5.3.0-1.el8 iscsi-initiator-utils 6.2.0.877-1.gitf71581b.el8 -> 6.2.0.878-4.gitd791ce0.el8 iscsi-initiator-utils-iscsiuio 6.2.0.877-1.gitf71581b.el8 -> 6.2.0.878-4.gitd791ce0.el8 kernel 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kernel-core 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kernel-devel 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kernel-headers 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kernel-modules 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kernel-modules-extra 4.18.0-147.8.1.el8_1 -> 4.18.0-193.el8 kmod 25-13.el8 -> 25-16.el8 kmod-libs 25-13.el8 -> 25-16.el8 kpartx 0.8.0-5.el8 -> 0.8.3-3.el8 krb5-libs 1.17-9.el8 -> 1.17-18.el8 libblkid 2.32.1-17.el8 -> 2.32.1-22.el8 libcap 2.26-1.el8 -> 2.26-3.el8 libcap-ng 0.7.9-4.el8 -> 0.7.9-5.el8 libcom_err 1.44.6-3.el8 -> 1.45.4-3.el8 libcurl 7.61.1-11.el8 -> 7.61.1-12.el8 libdrm 2.4.98-2.el8 -> 2.4.100-1.el8 libfdisk 2.32.1-17.el8 -> 2.32.1-22.el8 libgcc 8.3.1-4.5.el8 -> 8.3.1-5.el8 libgomp 8.3.1-4.5.el8 -> 8.3.1-5.el8 libibverbs 22.3-1.el8 -> 26.0-8.el8 libipa_hbac 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libldb 1.5.4-2.el8 -> 2.0.7-3.el8 libluksmeta 9-3.el8_1.1 -> 9-4.el8 libmaxminddb 1.2.0-6.el8 -> 1.2.0-7.el8 libmodulemd1 1.8.11-4.el8_1 -> 1.8.16-0.2.8.2.1 libmount 2.32.1-17.el8 -> 2.32.1-22.el8 libmspack 0.7-0.1.alpha.el8.3 -> 0.7-0.3.alpha.el8.4 libndp 1.7-1.el8 -> 1.7-3.el8 libnfsidmap 1:2.3.3-26.el8 -> 1:2.3.3-31.el8 libnl3 3.4.0-5.el8 -> 3.5.0-1.el8 libnl3-cli 3.4.0-5.el8 -> 3.5.0-1.el8 librepo 1.10.3-3.el8 -> 1.11.0-2.el8 libreport-filesystem 2.9.5-9.el8 -> 2.9.5-10.el8 libselinux 2.9-2.1.el8 -> 2.9-3.el8 libselinux-utils 2.9-2.1.el8 -> 2.9-3.el8 libsemanage 2.9-1.el8 -> 2.9-2.el8 libsmartcols 2.32.1-17.el8 -> 2.32.1-22.el8 libsmbclient 4.10.4-101.el8_1 -> 4.11.2-13.el8 libsolv 0.7.4-3.el8 -> 0.7.7-1.el8 libss 1.44.6-3.el8 -> 1.45.4-3.el8 libsss_autofs 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libsss_certmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libsss_idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libsss_nss_idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libsss_sudo 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 libstdc++ 8.3.1-4.5.el8 -> 8.3.1-5.el8 libtalloc 2.1.16-3.el8 -> 2.2.0-7.el8 libtdb 1.3.18-2.el8 -> 1.4.2-2.el8 libteam 1.28-4.el8 -> 1.29-1.el8 libtevent 0.9.39-2.el8 -> 0.10.0-2.el8 libuuid 2.32.1-17.el8 -> 2.32.1-22.el8 libwbclient 4.10.4-101.el8_1 -> 4.11.2-13.el8 libxml2 2.9.7-5.el8 -> 2.9.7-7.el8 libxslt 1.1.32-3.el8 -> 1.1.32-4.el8 linux-firmware 20190516-94.git711d3297.el8 -> 20191202-97.gite8a0f4c9.el8 luksmeta 9-3.el8_1.1 -> 9-4.el8 lvm2 8:2.03.05-5.el8 -> 8:2.03.08-3.el8 lvm2-libs 8:2.03.05-5.el8 -> 8:2.03.08-3.el8 machine-config-daemon 4.5.0-202004241417.git.1.e84f244.el8 -> 4.5.0-202004241817.git.1.09877e1.el8 make 1:4.2.1-9.el8 -> 1:4.2.1-10.el8 mdadm 4.1-9.el8 -> 4.1-13.el8 microcode_ctl 4:20190618-1.20191115.3.el8_1 -> 4:20191115-4.el8 mozjs60 60.9.0-3.el8 -> 60.9.0-4.el8 nfs-utils 1:2.3.3-26.el8 -> 1:2.3.3-31.el8 nftables 1:0.9.0-14.el8_1.1 -> 1:0.9.3-12.el8 numactl-libs 2.0.12-7.el8 -> 2.0.12-9.el8 open-vm-tools 10.3.10-3.el8_1.1 -> 11.0.0-4.el8 openssl 1:1.1.1c-2.el8_1.1 -> 1:1.1.1c-15.el8 openssl-libs 1:1.1.1c-2.el8_1.1 -> 1:1.1.1c-15.el8 openssl-pkcs11 0.4.8-2.el8 -> 0.4.10-2.el8 pam 1.3.1-4.el8 -> 1.3.1-8.el8 passwd 0.80-2.el8 -> 0.80-3.el8 platform-python 3.6.8-15.1.el8 -> 3.6.8-23.el8 platform-python-pip 9.0.3-15.el8 -> 9.0.3-16.el8 policycoreutils 2.9-3.el8_1.1 -> 2.9-9.el8 policycoreutils-python-utils 2.9-3.el8_1.1 -> 2.9-9.el8 polkit 0.115-9.el8 -> 0.115-11.el8 polkit-libs 0.115-9.el8 -> 0.115-11.el8 psmisc 23.1-3.el8 -> 23.1-4.el8 python3-audit 3.0-0.13.20190507gitf58ec40.el8 -> 3.0-0.17.20191104git1c2f876.el8 python3-bind 32:9.11.4-26.P2.el8 -> 32:9.11.13-3.el8 python3-libs 3.6.8-15.1.el8 -> 3.6.8-23.el8 python3-libselinux 2.9-2.1.el8 -> 2.9-3.el8 python3-libsemanage 2.9-1.el8 -> 2.9-2.el8 python3-pip-wheel 9.0.3-15.el8 -> 9.0.3-16.el8 python3-ply 3.9-7.el8 -> 3.9-8.el8 python3-policycoreutils 2.9-3.el8_1.1 -> 2.9-9.el8 python3-setools 4.2.2-1.el8 -> 4.2.2-2.el8 python3-sssdconfig 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 rdma-core 22.3-1.el8 -> 26.0-8.el8 rpcbind 1.2.5-4.el8 -> 1.2.5-7.el8 rpm 4.14.2-26.el8_1 -> 4.14.2-37.el8 rpm-libs 4.14.2-26.el8_1 -> 4.14.2-37.el8 rpm-plugin-selinux 4.14.2-26.el8_1 -> 4.14.2-37.el8 rsync 3.1.3-6.el8 -> 3.1.3-7.el8 samba-client-libs 4.10.4-101.el8_1 -> 4.11.2-13.el8 samba-common 4.10.4-101.el8_1 -> 4.11.2-13.el8 samba-common-libs 4.10.4-101.el8_1 -> 4.11.2-13.el8 selinux-policy 3.14.3-20.el8 -> 3.14.3-41.el8 selinux-policy-targeted 3.14.3-20.el8 -> 3.14.3-41.el8 setup 2.12.2-2.el8_1.1 -> 2.12.2-5.el8 sg3_utils 1.44-3.el8 -> 1.44-5.el8 sg3_utils-libs 1.44-3.el8 -> 1.44-5.el8 skopeo 1:0.1.40-8.module+el8.1.1+5351+506397b0 -> 1:0.1.40-10.module+el8.2.0+5955+6cd70ceb slirp4netns 0.4.2-3.git21fdece.module+el8.1.1+5657+524a77d7 -> 0.4.2-3.git21fdece.module+el8.2.0+5658+9a15711d socat 1.7.3.2-6.el8 -> 1.7.3.3-2.el8 sqlite-libs 3.26.0-4.el8_1 -> 3.26.0-6.el8 sssd 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-ad 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-client 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-common 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-common-pac 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-ipa 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-krb5 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-krb5-common 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-ldap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-nfs-idmap 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 sssd-proxy 2.2.0-19.el8_1.1 -> 2.2.3-20.el8 strace 4.24-5.el8 -> 4.24-9.el8 subscription-manager-rhsm-certificates 1.25.17-1.el8 -> 1.26.16-1.el8 sudo 1.8.25p1-8.el8_1.1 -> 1.8.29-5.el8 systemd 239-27.el8 -> 239-29.el8 systemd-journal-remote 239-27.el8 -> 239-29.el8 systemd-libs 239-27.el8 -> 239-29.el8 systemd-pam 239-27.el8 -> 239-29.el8 systemd-udev 239-27.el8 -> 239-29.el8 teamd 1.28-4.el8 -> 1.29-1.el8 tpm2-tools 3.1.4-5.el8 -> 3.2.1-1.el8 unbound-libs 1.7.3-8.el8 -> 1.7.3-10.el8 util-linux 2.32.1-17.el8 -> 2.32.1-22.el8 which 2.21-10.el8 -> 2.21-12.el8 xfsprogs 5.0.0-1.el8 -> 5.0.0-2.el8 zlib 1.2.11-10.el8 -> 1.2.11-13.el8 Removed: libdnet-1.12-26.el8.x86_64 libicu-60.3-2.el8_1.x86_64 Added: elfutils-debuginfod-client-0.178-7.el8.x86_64 kbd-2.0.4-8.el8.x86_64 kbd-legacy-2.0.4-8.el8.noarch kbd-misc-2.0.4-8.el8.noarch libxkbcommon-0.9.1-1.el8.x86_64 libzstd-1.4.2-2.el8.x86_64 xkeyboard-config-2.28-1.el8.noarch ```
I found a similar BZ but it appears to be for FCOS/RHEL7 nodes - https://bugzilla.redhat.com/show_bug.cgi?id=1781575
For now marking as a duplicate because evidence points that way. *** This bug has been marked as a duplicate of bug 1781575 ***
(In reply to Colin Walters from comment #2) > For now marking as a duplicate because evidence points that way. > > *** This bug has been marked as a duplicate of bug 1781575 *** I don't really have access to see what is happening in bug 1781575, can someone cc me into that issue?
> I don't really have access... It's public now :).