looking at recent failures of https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-aws-fips-4.6 , the artifacts for all the last six failing runs indicate that we have four or five nodes, not six. We need to figure out what happened to our sixth node. Some tests require six nodes and I would expect the topologies to match. 1. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295648148575752192 2. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295618541667487744 3. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295601073251160064 4. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295582596859695104 5. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295540587214147584 6. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295524370566877184 Compared to a passing one with six nodes: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295473462776696832
In all cases, the instances for the failing nodes are failing to get past ignition during boot ========== [0;32m OK [0m] Started Populate OSTree /var. [ 13.644709] systemd[1]: Starting Ignition (files)... Starting Ignition (files)... [ 13.674347] ignition[908]: INFO : Ignition 2.6.0 [ 13.678770] ignition[908]: INFO : Stage: files [ 13.683065] ignition[908]: INFO : reading system config file "/usr/lib/ignition/base.ign" [ 13.690774] ignition[908]: DEBUG : parsing config with SHA512: ff6a5153be363997e4d5d3ea8cc4048373a457c48c4a5b134a08a30aacd167c1e0f099f0bdf1e24c99ad180628cd02b767b863b5fe3a8fce3fe1886847eb8e2e [ 13.703081] ignition[908]: INFO : Adding "root-ca" to list of CAs [ 13.950008] ignition[908]: INFO : files: ensureUsers: op(1): [started] creating or modifying user "core" [ 13.957283] ignition[908]: DEBUG : files: ensureUsers: op(1): executing: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--comment" "CoreOS Admin" "--groups" "adm,sudo,systemd-journal,wheel" "core" [ 16.017682] ignition[908]: INFO : files: ensureUsers: op(1): [finished] creating or modifying user "core" [ 16.025785] ignition[908]: INFO : files: ensureUsers: op(2): [started] adding ssh keys to user "core" [ 16.033335] ignition[908]: INFO : files: ensureUsers: op(2): [finished] adding ssh keys to user "core" [ 16.039808] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(3): [started] writing file "/sysroot/etc/mcs-machine-config-content.json" [ 16.049718] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(3): [finished] writing file "/sysroot/etc/mcs-machine-config-content.json" [ 16.059417] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(4): [started] writing file "/sysroot/etc/ignition-machine-config-encapsulated.json" [ 16.069779] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(4): [finished] writing file "/sysroot/etc/ignition-machine-config-encapsulated.json" [ 16.080406] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(5): [started] writing file "/sysroot/etc/containers/registries.conf" [ 16.089361] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(5): [finished] writing file "/sysroot/etc/containers/registries.conf" [ 16.098199] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(6): [started] writing file "/sysroot/etc/kubernetes/kubeconfig" [ 16.106929] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(6): [finished] writing file "/sysroot/etc/kubernetes/kubeconfig" [ 16.116197] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(7): [started] writing file "/sysroot/etc/containers/storage.conf" [ 16.125159] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(7): [finished] writing file "/sysroot/etc/containers/storage.conf" [ 16.134274] ignition[908]: INFO : files: createFilesystemsFiles: createFiles: op(8): [started] writing file "/sysroot/etc/machine-config-daemon/node-annotations.json" [ 16.144864] ignition[908]: INFO 2020-08-18T07:38:00.000Z === END OF CONSOLE LOG === Oddly enough, the console log for a successful node is 65581 bytes and a failed one is 64046 bytes
In https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295648148575752192/artifacts/e2e-aws-fips/nodes/i-0e84a0c7dd7538352/console I see: > [ 8.904136] rhcos-fips[736]: No Ignition config provided. That's a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1862957. *** This bug has been marked as a duplicate of bug 1862957 ***