Bug 1959920

Summary: UEFISecureBoot set not on the right master node
Product: OpenShift Container Platform Reporter: Lubov <lshilin>
Component: InstallerAssignee: Iury Gregory Melo Ferreira <imelofer>
Installer sub component: openshift-installer QA Contact: Lubov <lshilin>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: imelofer, stbenjam
Version: 4.8Keywords: Triaged
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:08:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ironic-conductor.log
none
install-config.yaml
none
manifests none

Description Lubov 2021-05-12 15:57:15 UTC
Created attachment 1782458 [details]
ironic-conductor.log

Description of problem:
In install-config.yaml bootMode: UEFISecureBoot is set for master-0 node, but actually secure boot is enabled on another master.

From ironic-conductor log:

2021-05-12 14:27:46.488 1 DEBUG ironic_lib.json_rpc.server [req-b2060778-9a7f-4c62-b9ac-f8a92fac49bd bootstrap-user - - - -] RPC create_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'uuid': '2b743e50-db1c-4441-8da8-43b51e441821', 'name': 'openshift-master-0', 'chassis_id': None, 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true', 'cpu_arch': 'x86_64', 'local_gb': '50', 'root_device': {'name': 's== /dev/sda'}}, 'conductor_group': '', 'provision_state': 'enroll', 'resource_class': 'baremetal', 'boot_interface': 'redfish-virtual-media', 'raid_interface': 'no-raid'}, 'ironic_object.changes': ['name', 'provision_state', 'resource_class', 'boot_interface', 'driver_info', 'uuid', 'raid_interface', 'conductor_group', 'chassis_id', 'driver', 'properties']}, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-b2060778-9a7f-4c62-b9ac-f8a92fac49bd', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279

.......

2021-05-12 14:37:11.171 1 DEBUG ironic_lib.json_rpc.server [req-9eaf5172-1381-4440-8a31-c18982906533 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '4e21c154-9e20-4010-9d5c-63e3d51ce4b7', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '556bf644-ca5c-47f8-90ee-fbdd9e01b913', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-12T14:28:07.719666', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-12T14:37:03.421856', 'agent_verify_ca': '/var/lib/ironic/certificates/4e21c154-9e20-4010-9d5c-63e3d51ce4b7.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-12 14:36:41.836198'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-12T14:37:07Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-12T14:27:51Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 3, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-12T14:27:46Z', 'updated_at': '2021-05-12T14:37:11Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-9eaf5172-1381-4440-8a31-c18982906533', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-05-12-002851

How reproducible:
Run it twice, the first time secure boot was enabled on master-1, the second on master-2

Steps to Reproduce:
1. Ensure secure boot disabled on all machines
2. In install-config set bootMode: UEFISecureBoot for master-0. Rebuild manifest
3. Deploy the cluster

Actual results:
secure boot is enabled on other machine

Expected results:
secure boot is enabled on configured machine

Additional info:

Comment 1 Lubov 2021-05-12 15:58:26 UTC
Created attachment 1782459 [details]
install-config.yaml

Comment 2 Iury Gregory Melo Ferreira 2021-05-12 17:26:16 UTC
The only thing I can think about that could lead to this situation is that the list with maps with the instance_info that is used doesn't keep the order

Comment 4 Lubov 2021-05-13 13:28:01 UTC
Created attachment 1782768 [details]
manifests

Comment 5 Stephen Benjamin 2021-05-13 14:21:46 UTC
In the manifests directory, the terraform.tfstate correctly shows openshift-master-0 is the only one with the secure boot settings.


The first log in comment #0 is for master #0, and it shows it has the correct settings.  The second log in comment #0 from conductor shows that master-2 doesn't have secureboot in the capabilities section:


     'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true',

The only reference to secure boot in that second log is a separate `capabilities': {'secure_boot': 'true'}`. I am not sure where it's coming from but it isn't terraform AFAICT.

Comment 6 Iury Gregory Melo Ferreira 2021-05-13 14:41:50 UTC
Looking at terraform.tfstate I don't see the "instance_info" with "capabilities": "secure_boot:true" (Lines 299 325 351)
I would expect Line 299 (since is "index_key": 0 to have something like:

            "instance_info": {
              "image_checksum": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum",
              "image_source": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2",
              "capabilities": "secure_boot:true"
            },

Comment 7 Iury Gregory Melo Ferreira 2021-05-13 14:44:33 UTC
terraform.baremetal.auto.tfvars.json seems to have things in the right order (master-0 would end up with secure_boot: true in instance_info/capabilities (but it didn't, only in properties the value is present)

Comment 8 Stephen Benjamin 2021-05-13 18:12:03 UTC
Please keep in mind these files have two different purposes. terraform.baremetal.auto.tfvars.json is what the installer sends as the values of variables to terraform. terraform.tfstate is the state of terraform after it's run.


> terraform.baremetal.auto.tfvars.json seems to have things in the right order (master-0 would end up with secure_boot: true in instance_info/capabilities (but it didn't, only in properties the value is present)

I don't see this. In tfvars capabitilies set in instance_infos[0] (i.e. master-0):

  "instance_infos": [
    {
      "capabilities": "secure_boot:true",
      "image_checksum": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum",
      "image_source": "http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2"
    },


Then in tfstate, I see properties has capabilities set:

    "properties": { "capabilities": "boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true"}

Instance info is not present in tfstate as you remove it: https://github.com/openshift-metal3/terraform-provider-ironic/commit/edfa8be1c9ede51b205230123f15ef0afaff780f#diff-1eb399fd0f2c2f90b6c4d903f9ac21375437e09a73e010fb72341542e2384a1eR110


I don't know what Ironic is expecting, but everything on the installer/terraform side seems to be transforming and sending the data it was asked to.

Comment 9 Iury Gregory Melo Ferreira 2021-05-13 18:56:48 UTC
Ironic would expect that master-0 has the "properties" like you mentioned and the instance_info field with "capabilities"" {"secure_boot":"true"}  (This information is missing, but it's present in the master-2 instance_info).

From the ironic-conductor in the boostrap:
master-0 receives an update with 

'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'}

2021-05-13 12:45:01.851 1 DEBUG ironic_lib.json_rpc.server [req-ad771de0-1685-428e-93c2-751a742b9b65 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 3, 'uuid': '8cded29d-2a34-460c-a61e-f733b9b06650', 'name': 'openshift-master-0', 'chassis_id': None, 'instance_uuid': '4cd20209-9a31-4068-84ff-9bd19f3dfa62', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:02.937411', 'agent_url': 'https://10.46.29.129:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:54.788726', 'agent_verify_ca': '/var/lib/ironic/certificates/8cded29d-2a34-460c-a61e-f733b9b06650.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:44:32.962377'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:44:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 1, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:44:59Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-ad771de0-1685-428e-93c2-751a742b9b65', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279

And conductor confirms the update 
2021-05-13 12:45:01.909 1 DEBUG ironic_lib.json_rpc.server [req-ad771de0-1685-428e-93c2-751a742b9b65 bootstrap-user - - - -] RPC update_node returned {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 3, 'uuid': '8cded29d-2a34-460c-a61e-f733b9b06650', 'name': 'openshift-master-0', 'chassis_id': None, 'instance_uuid': '4cd20209-9a31-4068-84ff-9bd19f3dfa62', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.16', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:02.937411', 'agent_url': 'https://10.46.29.129:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:54.788726', 'agent_verify_ca': '/var/lib/ironic/certificates/8cded29d-2a34-460c-a61e-f733b9b06650.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:44:32.962377'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2'}, 'properties': {'capabilities': 'boot_mode:uefi,secure_boot:true,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': '10.46.29.179', 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:44:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 1, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:294


master-2 receives the update with 
'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}

2021-05-13 12:45:01.949 1 DEBUG ironic_lib.json_rpc.server [req-ffe82a2f-a8e4-4559-945b-a8688e7a9963 bootstrap-user - - - -] RPC update_node with {'node_obj': {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '56610aa1-a6e9-4b34-8964-4d6d3bad3438', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '7b4f32ae-0ad5-4e7f-b517-398c3241a922', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:00.919689', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:57.467642', 'agent_verify_ca': '/var/lib/ironic/certificates/56610aa1-a6e9-4b34-8964-4d6d3bad3438.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:43:33.879009'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': None, 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:43:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 2, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}, 'ironic_object.changes': ['instance_info']}, 'reset_interfaces': None, 'context': {'user': 'bootstrap-user', 'tenant': None, 'system_scope': None, 'project': None, 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': False, 'read_only': False, 'show_deleted': False, 'auth_token': '***', 'request_id': 'req-ffe82a2f-a8e4-4559-945b-a8688e7a9963', 'global_request_id': None, 'resource_uuid': None, 'roles': [], 'user_identity': 'bootstrap-user - - - -', 'is_admin_project': True}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:279

master-2 is updated with the wrong information.

2021-05-13 12:45:02.016 1 DEBUG ironic_lib.json_rpc.server [req-ffe82a2f-a8e4-4559-945b-a8688e7a9963 bootstrap-user - - - -] RPC update_node returned {'ironic_object.name': 'Node', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.35', 'ironic_object.data': {'id': 2, 'uuid': '56610aa1-a6e9-4b34-8964-4d6d3bad3438', 'name': 'openshift-master-2', 'chassis_id': None, 'instance_uuid': '7b4f32ae-0ad5-4e7f-b517-398c3241a922', 'driver': 'redfish', 'driver_info': {'deploy_kernel': 'http://10.46.29.199:80/images/ironic-python-agent.kernel', 'deploy_ramdisk': 'http://10.46.29.199:80/images/ironic-python-agent.initramfs', 'redfish_address': 'https://10.46.61.18', 'redfish_password': '***', 'redfish_system_id': '/redfish/v1/Systems/1', 'redfish_username': 'admin', 'redfish_verify_ca': 'false'}, 'driver_internal_info': {'agent_secret_token': '***', 'agent_secret_token_pregenerated': '***', 'last_power_state_change': '2021-05-13T12:36:00.919689', 'agent_url': 'https://10.46.29.131:9999', 'agent_version': '7.0.1.dev5', 'agent_last_heartbeat': '2021-05-13T12:44:57.467642', 'agent_verify_ca': '/var/lib/ironic/certificates/56610aa1-a6e9-4b34-8964-4d6d3bad3438.crt', 'clean_steps': None, 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True, 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False, 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 1, 'agent_erase_skip_read_only': False, 'hardware_manager_version': {'IntelCnaHardwareManager': '1.0', 'generic_hardware_manager': '1.1'}, 'agent_cached_clean_steps_refreshed': '2021-05-13 12:43:33.879009'}, 'clean_step': {}, 'deploy_step': {}, 'raid_config': {}, 'target_raid_config': {}, 'instance_info': {'image_checksum': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2.md5sum', 'image_source': 'http://10.46.29.199:80/images/rhcos-48.84.202104271417-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202104271417-0-openstack.x86_64.qcow2', 'capabilities': {'secure_boot': 'true'}}, 'properties': {'capabilities': 'boot_mode:uefi,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'cpu_arch': 'x86_64', 'local_gb': '893', 'root_device': {'name': 's== /dev/sda'}, 'vendor': 'HPE', 'cpus': '64', 'memory_mb': '262144'}, 'reservation': '10.46.29.179', 'conductor_affinity': None, 'conductor_group': '', 'power_state': 'power on', 'target_power_state': None, 'provision_state': 'available', 'provision_updated_at': '2021-05-13T12:43:59Z', 'target_provision_state': None, 'maintenance': False, 'maintenance_reason': None, 'fault': None, 'console_enabled': False, 'last_error': None, 'resource_class': 'baremetal', 'inspection_finished_at': None, 'inspection_started_at': '2021-05-13T12:35:43Z', 'extra': {}, 'automated_clean': None, 'protected': False, 'protected_reason': None, 'allocation_id': 2, 'bios_interface': 'redfish', 'boot_interface': 'redfish-virtual-media', 'console_interface': 'no-console', 'deploy_interface': 'direct', 'inspect_interface': 'inspector', 'management_interface': 'redfish', 'network_interface': 'noop', 'power_interface': 'redfish', 'raid_interface': 'no-raid', 'rescue_interface': 'no-rescue', 'storage_interface': 'noop', 'vendor_interface': 'no-vendor', 'traits': {'ironic_object.name': 'TraitList', 'ironic_object.namespace': 'ironic', 'ironic_object.version': '1.0', 'ironic_object.data': {'objects': []}}, 'owner': None, 'lessee': None, 'description': None, 'retired': False, 'retired_reason': None, 'network_data': {}, 'created_at': '2021-05-13T12:35:38Z', 'updated_at': '2021-05-13T12:45:01Z'}} _handle_requests /usr/lib/python3.6/site-packages/ironic_lib/json_rpc/server.py:294


Both have the correct properties but not instance_info.

Comment 10 Stephen Benjamin 2021-05-14 00:53:58 UTC
Ok, I have a guess about what's happening -- we use an Ironic allocation: https://github.com/openshift/installer/blob/master/data/data/baremetal/masters/main.tf#L35

My guess is Ironic is allocating openshift-master-2 for our first allocation, but the allocation and deployment object get called master-0 and get master-0's information. Can you find in the Ironic logs the allocation request and if the first one we get back is indeed master-2?

We probably need to set candidate notes here: https://github.com/openshift/installer/blob/master/data/data/baremetal/masters/main.tf#L35

to something like "candidate_nodes = [ironic_node_v1.openshift-master-host[count].id]" instead, so the only candidate for master-2 is master-2.

Comment 11 Iury Gregory Melo Ferreira 2021-05-14 07:03:53 UTC
Oh! That would explain (I totally forgot to check allocation, will do that and update the bz)

Comment 12 Iury Gregory Melo Ferreira 2021-05-17 14:27:01 UTC
Setting blocker+, since this would affect customers trying to deploy in a scenario where we have only one node with secure boot and the information will be set in a different node.

Comment 14 Iury Gregory Melo Ferreira 2021-05-18 16:03:18 UTC
Moving to Post, since can be related to the allocation

Comment 15 Iury Gregory Melo Ferreira 2021-05-18 16:05:32 UTC
Changed the Component to Installer (since the fix is in the installer repo)

Comment 16 Iury Gregory Melo Ferreira 2021-05-19 10:13:07 UTC
Since this is a non-standard scenario (only 1 master with secure boot) and it's a new feature in 4.8) we decided to lower the priority and not consider this a blocker.

Comment 18 Lubov 2021-05-23 12:12:35 UTC
Verified on 4.8.0-0.nightly-2021-05-21-233425 - run twice: for both attempts the right master was set to SecureBoot

Comment 21 errata-xmlrpc 2021-07-27 23:08:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438