Bug 1869654

Summary:	fips tests are missing nodes
Product:	OpenShift Container Platform	Reporter:	David Eads <deads>
Component:	RHCOS	Assignee:	Micah Abbott <miabbott>
Status:	CLOSED DUPLICATE	QA Contact:	Michael Nguyen <mnguyen>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.6	CC:	aos-bugs, bbreard, imcleod, jlebon, jligon, jokerman, nstielau, sjenning
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-08-18 15:17:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Eads 2020-08-18 13:12:51 UTC

looking at recent failures of https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-aws-fips-4.6 , the artifacts for all the last six failing runs indicate that we have four or five nodes, not six.

We need to figure out what happened to our sixth node.  Some tests require six nodes and I would expect the topologies to match.


1. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295648148575752192
2. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295618541667487744
3. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295601073251160064
4. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295582596859695104
5. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295540587214147584
6. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295524370566877184

Compared to a passing one with six nodes: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295473462776696832

Comment 1 Seth Jennings 2020-08-18 14:36:42 UTC

In all cases, the instances for the failing nodes are failing to get past ignition during boot

==========
[0;32m  OK  [0m] Started Populate OSTree /var.
         [   13.644709] systemd[1]: Starting Ignition (files)...
Starting Ignition (files)...
[   13.674347] ignition[908]: INFO     : Ignition 2.6.0
[   13.678770] ignition[908]: INFO     : Stage: files
[   13.683065] ignition[908]: INFO     : reading system config file "/usr/lib/ignition/base.ign"
[   13.690774] ignition[908]: DEBUG    : parsing config with SHA512: ff6a5153be363997e4d5d3ea8cc4048373a457c48c4a5b134a08a30aacd167c1e0f099f0bdf1e24c99ad180628cd02b767b863b5fe3a8fce3fe1886847eb8e2e
[   13.703081] ignition[908]: INFO     : Adding "root-ca" to list of CAs
[   13.950008] ignition[908]: INFO     : files: ensureUsers: op(1): [started]  creating or modifying user "core"
[   13.957283] ignition[908]: DEBUG    : files: ensureUsers: op(1): executing: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--comment" "CoreOS Admin" "--groups" "adm,sudo,systemd-journal,wheel" "core"
[   16.017682] ignition[908]: INFO     : files: ensureUsers: op(1): [finished] creating or modifying user "core"
[   16.025785] ignition[908]: INFO     : files: ensureUsers: op(2): [started]  adding ssh keys to user "core"
[   16.033335] ignition[908]: INFO     : files: ensureUsers: op(2): [finished] adding ssh keys to user "core"
[   16.039808] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [started]  writing file "/sysroot/etc/mcs-machine-config-content.json"
[   16.049718] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [finished] writing file "/sysroot/etc/mcs-machine-config-content.json"
[   16.059417] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [started]  writing file "/sysroot/etc/ignition-machine-config-encapsulated.json"
[   16.069779] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [finished] writing file "/sysroot/etc/ignition-machine-config-encapsulated.json"
[   16.080406] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [started]  writing file "/sysroot/etc/containers/registries.conf"
[   16.089361] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [finished] writing file "/sysroot/etc/containers/registries.conf"
[   16.098199] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(6): [started]  writing file "/sysroot/etc/kubernetes/kubeconfig"
[   16.106929] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(6): [finished] writing file "/sysroot/etc/kubernetes/kubeconfig"
[   16.116197] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(7): [started]  writing file "/sysroot/etc/containers/storage.conf"
[   16.125159] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(7): [finished] writing file "/sysroot/etc/containers/storage.conf"
[   16.134274] ignition[908]: INFO     : files: createFilesystemsFiles: createFiles: op(8): [started]  writing file "/sysroot/etc/machine-config-daemon/node-annotations.json"
[   16.144864] ignition[908]: INFO   	2020-08-18T07:38:00.000Z
=== END OF CONSOLE LOG ===

Oddly enough, the console log for a successful node is 65581 bytes and a failed one is 64046 bytes

Comment 2 Jonathan Lebon 2020-08-18 15:17:12 UTC

In https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.6/1295648148575752192/artifacts/e2e-aws-fips/nodes/i-0e84a0c7dd7538352/console I see:

> [    8.904136] rhcos-fips[736]: No Ignition config provided.

That's a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1862957.

*** This bug has been marked as a duplicate of bug 1862957 ***