Created attachment 1782458 [details] ironic-conductor.log Description of problem: In install-config.yaml bootMode: UEFISecureBoot is set for master-0 node, but actually secure boot is enabled on another master. From ironic-conductor log: 2021-05-12 14:27:46.488 1 DEBUG ironic_lib.json_rpc.server [req-b2060778-9a7f-4c62-b9ac-f8a92fac49bd bootstrap-user - - - -] RPC create_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'uuid': '2b743e50-db1c-4441-8da8-43b51e441821', 'name': 'openshift-master-0', 'chassis_id': None, 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true', 'cpu_arch': 'x86_64', 'local_gb': '50', 'root_device': {'name': 's== /dev/sda'}}, 'conductor_group': '', 'provision_state': 'enroll', 'resource_class': 'baremetal', 'boot_interface': 'redfish-virtual-media', 'raid_interface': 'no-raid'}, 'ironic_object.changes': ['name', 'provision_state', 'resource_class', 'boot_interface', 'driver_info', 'uuid', 'raid_interface', 'conductor_group', 'chassis_id', 'driver', 'properties']}, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-b2060778-9a7f-4c62-b9ac-f8a92fac49bd', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279 ....... 2021-05-12 14:37:11.171 1 DEBUG ironic_lib.json_rpc.server [req-9eaf5172-1381-4440-8a31-c18982906533 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '4e21c154-9e20-4010-9d5c-63e3d51ce4b7', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '556bf644-ca5c-47f8-90ee-fbdd9e01b913', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-12T14:28:07.719666', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-12T14:37:03.421856', 'agent_verify_ca': '/var/lib/ironic/certificates/4e21c154-9e20-4010-9d5c-63e3d51ce4b7.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-12 14:36:41.836198'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-12T14:37:07Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-12T14:27:51Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 3, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-12T14:27:46Z', 'updated_at': '2021-05-12T14:37:11Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-9eaf5172-1381-4440-8a31-c18982906533', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279 Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-05-12-002851 How reproducible: Run it twice, the first time secure boot was enabled on master-1, the second on master-2 Steps to Reproduce: 1. Ensure secure boot disabled on all machines 2. In install-config set bootMode: UEFISecureBoot for master-0. Rebuild manifest 3. Deploy the cluster Actual results: secure boot is enabled on other machine Expected results: secure boot is enabled on configured machine Additional info:
Created attachment 1782459 [details] install-config.yaml
The only thing I can think about that could lead to this situation is that the list with maps with the instance_info that is used doesn't keep the order
must-gather http://rhos-compute-node-10.lab.eng.rdu2.redhat.com/logs/BZ1959920-must-gather.tar.gz
Created attachment 1782768 [details] manifests
In the manifests directory, the terraform.tfstate correctly shows openshift-master-0 is the only one with the secure boot settings. The first log in comment #0 is for master #0, and it shows it has the correct settings. The second log in comment #0 from conductor shows that master-2 doesn't have secureboot in the capabilities section: 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', The only reference to secure boot in that second log is a separate `capabilities': {'secure_boot': 'true'}`. I am not sure where it's coming from but it isn't terraform AFAICT.
Looking at terraform.tfstate I don't see the "instance_info" with "capabilities": "secure_boot:true" (Lines 299 325 351) I would expect Line 299 (since is "index_key": 0 to have something like: "instance_info": { "image_checksum": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum", "image_source": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2", "capabilities": "secure_boot:true" },
terraform.baremetal.auto.tfvars.json seems to have things in the right order (master-0 would end up with secure_boot: true in instance_info/capabilities (but it didn't, only in properties the value is present)
Please keep in mind these files have two different purposes. terraform.baremetal.auto.tfvars.json is what the installer sends as the values of variables to terraform. terraform.tfstate is the state of terraform after it's run. > terraform.baremetal.auto.tfvars.json seems to have things in the right order (master-0 would end up with secure_boot: true in instance_info/capabilities (but it didn't, only in properties the value is present) I don't see this. In tfvars capabitilies set in instance_infos[0] (i.e. master-0): "instance_infos": [ { "capabilities": "secure_boot:true", "image_checksum": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum", "image_source": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2" }, Then in tfstate, I see properties has capabilities set: "properties": { "capabilities": "boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true"} Instance info is not present in tfstate as you remove it: https://github.com/openshift-metal3/terraform-provider-ironic/commit/edfa8be1c9ede51b205230123f15ef0afaff780f#diff-1eb399fd0f2c2f90b6c4d903f9ac21375437e09a73e010fb72341542e2384a1eR110 I don't know what Ironic is expecting, but everything on the installer/terraform side seems to be transforming and sending the data it was asked to.
Ironic would expect that master-0 has the "properties" like you mentioned and the instance_info field with "capabilities"" {"secure_boot":"true"} (This information is missing, but it's present in the master-2 instance_info). From the ironic-conductor in the boostrap: master-0 receives an update with 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'} 2021-05-13 12:45:01.851 1 DEBUG ironic_lib.json_rpc.server [req-ad771de0-1685-428e-93c2-751a742b9b65 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 3, 'uuid': '8cded29d-2a34-460c-a61e-f733b9b06650', 'name': 'openshift-master-0', 'chassis_id': None, 'instance_uuid': '4cd20209-9a31-4068-84ff-9bd19f3dfa62', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:02.937411', 'agent_url': 'https://10.46.29.129:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:54.788726', 'agent_verify_ca': '/var/lib/ironic/certificates/8cded29d-2a34-460c-a61e-f733b9b06650.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:44:32.962377'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:44:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 1, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:44:59Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-ad771de0-1685-428e-93c2-751a742b9b65', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279 And conductor confirms the update 2021-05-13 12:45:01.909 1 DEBUG ironic_lib.json_rpc.server [req-ad771de0-1685-428e-93c2-751a742b9b65 bootstrap-user - - - -] RPC update_node returned {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 3, 'uuid': '8cded29d-2a34-460c-a61e-f733b9b06650', 'name': 'openshift-master-0', 'chassis_id': None, 'instance_uuid': '4cd20209-9a31-4068-84ff-9bd19f3dfa62', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:02.937411', 'agent_url': 'https://10.46.29.129:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:54.788726', 'agent_verify_ca': '/var/lib/ironic/certificates/8cded29d-2a34-460c-a61e-f733b9b06650.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:44:32.962377'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': '10.46.29.179', 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:44:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 1, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:294 master-2 receives the update with 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}} 2021-05-13 12:45:01.949 1 DEBUG ironic_lib.json_rpc.server [req-ffe82a2f-a8e4-4559-945b-a8688e7a9963 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '56610aa1-a6e9-4b34-8964-4d6d3bad3438', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '7b4f32ae-0ad5-4e7f-b517-398c3241a922', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:00.919689', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:57.467642', 'agent_verify_ca': '/var/lib/ironic/certificates/56610aa1-a6e9-4b34-8964-4d6d3bad3438.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:43:33.879009'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:43:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 2, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-ffe82a2f-a8e4-4559-945b-a8688e7a9963', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279 master-2 is updated with the wrong information. 2021-05-13 12:45:02.016 1 DEBUG ironic_lib.json_rpc.server [req-ffe82a2f-a8e4-4559-945b-a8688e7a9963 bootstrap-user - - - -] RPC update_node returned {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '56610aa1-a6e9-4b34-8964-4d6d3bad3438', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '7b4f32ae-0ad5-4e7f-b517-398c3241a922', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:00.919689', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:57.467642', 'agent_verify_ca': '/var/lib/ironic/certificates/56610aa1-a6e9-4b34-8964-4d6d3bad3438.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:43:33.879009'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': '10.46.29.179', 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:43:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 2, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:294 Both have the correct properties but not instance_info.
Ok, I have a guess about what's happening -- we use an Ironic allocation: https://github.com/openshift/installer/blob/master/data/data/baremetal/masters/main.tf#L35 My guess is Ironic is allocating openshift-master-2 for our first allocation, but the allocation and deployment object get called master-0 and get master-0's information. Can you find in the Ironic logs the allocation request and if the first one we get back is indeed master-2? We probably need to set candidate notes here: https://github.com/openshift/installer/blob/master/data/data/baremetal/masters/main.tf#L35 to something like "candidate_nodes = [ironic_node_v1.openshift-master-host[count].id]" instead, so the only candidate for master-2 is master-2.
Oh! That would explain (I totally forgot to check allocation, will do that and update the bz)
Setting blocker+, since this would affect customers trying to deploy in a scenario where we have only one node with secure boot and the information will be set in a different node.
Moving to Post, since can be related to the allocation
Changed the Component to Installer (since the fix is in the installer repo)
Since this is a non-standard scenario (only 1 master with secure boot) and it's a new feature in 4.8) we decided to lower the priority and not consider this a blocker.
Verified on 4.8.0-0.nightly-2021-05-21-233425 - run twice: for both attempts the right master was set to SecureBoot
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438