This looks like another environment where compute-2 has hostname compute-1, and compute-1 has hostname compute-2. Bug #2079935 is tracking a fix for choosing the correct node in this case. Can you reproduce this issue with tripleo-ansible-3.3.1-0.20220519231828.5e0485e.el8ost?

It would be useful to know how this hostname swap happened, in bug #2079935 it started as a 16.2 environment and the nodes were selected from the profile pool, so the names didn't match up. If this started as a 17.0 environment then it would be good to look into a fix to avoid this state entirely.

I will attempt to recreate when a puddle with tripleo-ansible-3.3.1-0.20220519231828.5e0485e.el8ost is available. It looks like it is not available yet.

This was a standard OSP17 deploy and was not the result of an upgrade. Have seen many OSP17 deploys where compute-2 had hostname compute-1.

Can still be recreated with updated tripleo-ansible. Using RHOS-17.0-RHEL-9-20220610.n.0:

yum list installed | grep tripleo-ansible
tripleo-ansible.noarch                           3.3.1-0.20220607162207.ae139c3.el9ost    @rhelosp-17.0   

After running scale down automation:

(undercloud) [stack@undercloud-0 ~]$ metalsmith list
[2022-06-14 17:00:20,686] No Port found for 327207da-5c0a-49c2-96c4-0a8e5c26fb52: Client Error for url: https://192.168.24.2:13696/v2.0/ports/327207da-5c0a-49c2-96c4-0a8e5c26fb52, Port 327207da-5c0a-49c2-96c4-0a8e5c26fb52 could not be found.

Could you provide the output of this:

metalsmith --debug --verbose list

Also it may be useful to have access to the environment, looking at the logs during the scale-down may reveal something.

metalsmith: error: argument -v/--verbose: not allowed with argument --debug
(undercloud) [stack@undercloud-0 ~]$ metalsmith --debug list
2022-06-15 16:50:45,060 DEBUG stevedore.extension: found extension EntryPoint(name='admin_token', value='keystoneauth1.loading._plugins.admin_token:AdminToken', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='http_basic', value='keystoneauth1.loading._plugins.http_basic:HTTPBasicAuth', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='none', value='keystoneauth1.loading._plugins.noauth:NoAuth', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='password', value='keystoneauth1.loading._plugins.identity.generic:Password', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='token', value='keystoneauth1.loading._plugins.identity.generic:Token', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='v2password', value='keystoneauth1.loading._plugins.identity.v2:Password', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='v2token', value='keystoneauth1.loading._plugins.identity.v2:Token', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='v3adfspassword', value='keystoneauth1.extras._saml2._loading:ADFSPassword', group='keystoneauth1.plugin')
2022-06-15 16:50:45,061 DEBUG stevedore.extension: found extension EntryPoint(name='v3applicationcredential', value='keystoneauth1.loading._plugins.identity.v3:ApplicationCredential', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3fedkerb', value='keystoneauth1.extras.kerberos._loading:MappedKerberos', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3kerberos', value='keystoneauth1.extras.kerberos._loading:Kerberos', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3multifactor', value='keystoneauth1.loading._plugins.identity.v3:MultiFactor', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3oauth1', value='keystoneauth1.extras.oauth1._loading:V3OAuth1', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3oidcaccesstoken', value='keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3oidcauthcode', value='keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3oidcclientcredentials', value='keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3oidcpassword', value='keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3password', value='keystoneauth1.loading._plugins.identity.v3:Password', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3samlpassword', value='keystoneauth1.extras._saml2._loading:Saml2Password', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3token', value='keystoneauth1.loading._plugins.identity.v3:Token', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3tokenlessauth', value='keystoneauth1.loading._plugins.identity.v3:TokenlessAuth', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v3totp', value='keystoneauth1.loading._plugins.identity.v3:TOTP', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='v1password', value='swiftclient.authv1:PasswordLoader', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='noauth', value='cinderclient.contrib.noauth:CinderNoAuthLoader', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='aodh-noauth', value='aodhclient.noauth:AodhNoAuthLoader', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='gnocchi-basic', value='gnocchiclient.auth:GnocchiBasicLoader', group='keystoneauth1.plugin')
2022-06-15 16:50:45,062 DEBUG stevedore.extension: found extension EntryPoint(name='gnocchi-noauth', value='gnocchiclient.auth:GnocchiNoAuthLoader', group='keystoneauth1.plugin')
2022-06-15 16:50:45,068 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10"
2022-06-15 16:50:45,198 DEBUG keystoneauth.session: RESP: [300] content-length: 269 content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT location: https://192.168.24.2:13000/v3/ server: Apache vary: X-Auth-Token x-openstack-request-id: req-8e7f0454-f4ea-4666-a556-e95e0bd63074
2022-06-15 16:50:45,198 DEBUG keystoneauth.session: RESP BODY: {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://192.168.24.2:13000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
2022-06-15 16:50:45,198 DEBUG keystoneauth.session: GET call to https://192.168.24.2:13000/ used request id req-8e7f0454-f4ea-4666-a556-e95e0bd63074
2022-06-15 16:50:45,199 DEBUG keystoneauth.identity.v3.base: Making authentication request to https://192.168.24.2:13000/v3/auth/tokens
2022-06-15 16:50:45,598 DEBUG keystoneauth.identity.v3.base: {"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "791ae6de856b413a97e943fb86942ac5", "name": "admin", "password_expires_at": null}, "audit_ids": ["5yZsqlhUQTyvEwEmHof5Sg"], "expires_at": "2022-06-15T20:50:45.000000Z", "issued_at": "2022-06-15T16:50:45.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "216558ff141544f18f91340c9b144c8b", "name": "admin"}, "is_domain": false, "roles": [{"id": "3a9fd0a06c8a475c9494b86b95996b86", "name": "admin"}, {"id": "03ed47ac8abc465293524982e0beed91", "name": "_member_"}, {"id": "71074bb9513549878e9734d5b7a085ca", "name": "reader"}, {"id": "33aea27e2aa64acaa7f91fa91bc100d2", "name": "member"}], "catalog": [{"endpoints": [{"id": "656563e07c434d7bbc1a37b49973ee6c", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:35357", "region": "regionOne"}, {"id": "92ace60c19b84fafb890bd82ee2aa07b", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5000", "region": "regionOne"}, {"id": "ebb5a909ed484c91ada485aa31876ad3", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13000", "region": "regionOne"}], "id": "53dbd0acbbd040739ace729edfb2ae57", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "a6a170a88f474a1e93f7f19e816fd3b9", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13050", "region": "regionOne"}, {"id": "b0dc7fe59a2d4ed7a2a3a2a809abb0fd", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}, {"id": "dfae05ced239403185402151b8b74752", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}], "id": "760ea145f6ed42a0b5fcce427da973a2", "type": "baremetal-introspection", "name": "ironic-inspector"}, {"endpoints": [{"id": "0a378cc6f4d84586b19bd86c153e2a8c", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}, {"id": "39cc485595ef47efac7421257ae265bc", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13696", "region": "regionOne"}, {"id": "5636022482954794b11d56d907a33fbc", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}], "id": "a23d54a2342c4f0d9abd972591dbc716", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "213378c181ae42f4807861641bec9735", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}, {"id": "3f0f323099d349e6ae12aff941c73238", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}, {"id": "a95f72d43a96481aa6dfe8f5fe6235b3", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13385", "region": "regionOne"}], "id": "e61cb794489642fc9ba3b59c73dfc5d5", "type": "baremetal", "name": "ironic"}]}}
2022-06-15 16:50:45,599 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10"
2022-06-15 16:50:45,706 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-9c4563a9-f21c-4c50-af7c-e7709abf7e30 server: Apache transfer-encoding: chunked vary: Accept-Encoding
2022-06-15 16:50:45,706 DEBUG keystoneauth.session: RESP BODY: {"name": "OpenStack Ironic API", "description": "Ironic is an OpenStack project which aims to provision baremetal machines.", "default_version": {"id": "v1", "links": [{"href": "https://192.168.24.2:13385/v1/", "rel": "self"}], "status": "CURRENT", "min_version": "1.1", "version": "1.72"}, "versions": [{"id": "v1", "links": [{"href": "https://192.168.24.2:13385/v1/", "rel": "self"}], "status": "CURRENT", "min_version": "1.1", "version": "1.72"}]}
2022-06-15 16:50:45,711 DEBUG keystoneauth.session: REQ: curl -g -i -X GET "https://192.168.24.2:13385/v1/nodes/detail?associated=True" -H "Accept: application/json" -H "OpenStack-API-Version: baremetal 1.61" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.61"
2022-06-15 16:50:45,820 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-4b2beeb0-43bb-4e2d-bf16-f26db68e023f server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.61
2022-06-15 16:50:45,820 DEBUG keystoneauth.session: RESP BODY: {"nodes": [{"uuid": "d9478cce-1172-42c9-b7d0-060701f9f553", "created_at": "2022-06-15T13:45:10+00:00", "updated_at": "2022-06-15T14:08:49+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6230"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:37.204366", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:08:05.877526", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:26.179635", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["7c48bcbc-701c-44bf-9ff6-99e853a42c30"], "metalsmith_created_ports": ["7c48bcbc-701c-44bf-9ff6-99e853a42c30"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "ceph-storage"}, "display_name": "ceph-0", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 29, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "0a5d075d-f20c-4771-83d3-0c8b885b32c7", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "ceph-0", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "2", "memory_mb": "4096", "local_gb": "30", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:ceph-storage,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown", "root_device": {"name": "/dev/vda"}}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:08:49+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/d9478cce-1172-42c9-b7d0-060701f9f553", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "0a5d075d-f20c-4771-83d3-0c8b885b32c7", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/volume", "rel": "bookmark"}]}, {"uuid": "8819df63-e4c5-4ced-bfe7-fd5d1c978277", "created_at": "2022-06-15T13:45:11+00:00", "updated_at": "2022-06-15T14:09:00+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6231"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:48.999524", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:08:16.895860", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:32.341355", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["f27b5656-55d9-4b2c-92e0-af1ba9c23d73"], "metalsmith_created_ports": ["f27b5656-55d9-4b2c-92e0-af1ba9c23d73"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "compute"}, "display_name": "compute-1", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 48, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "3a0a30ed-ea7a-463b-ade5-afb2517fbaec", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "compute-0", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "4", "memory_mb": "8192", "local_gb": "49", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:compute,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown"}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:09:00+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "3a0a30ed-ea7a-463b-ade5-afb2517fbaec", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/volume", "rel": "bookmark"}]}, {"uuid": "45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa", "created_at": "2022-06-15T13:45:12+00:00", "updated_at": "2022-06-15T14:08:16+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6233"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:04.902328", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:07:41.364002", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:01.596007", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["dc5dbc80-0b8c-441c-a6c5-f3a1a8557a04"], "metalsmith_created_ports": ["dc5dbc80-0b8c-441c-a6c5-f3a1a8557a04"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "compute"}, "display_name": "compute-2", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 48, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "67089ba3-e1a0-4a34-9b80-766a570a0080", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "compute-2", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "4", "memory_mb": "8192", "local_gb": "49", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:compute,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown"}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:08:16+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "67089ba3-e1a0-4a34-9b80-766a570a0080", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa/volume", "rel": "bookmark"}]}, {"uuid": "69ded144-f212-4d40-9069-a6e28d16039c", "created_at": "2022-06-15T13:45:13+00:00", "updated_at": "2022-06-15T14:08:54+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6234"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:42.823168", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:08:16.135820", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:31.475544", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["4335ef34-0b27-4f34-820f-9666a2e34238"], "metalsmith_created_ports": ["4335ef34-0b27-4f34-820f-9666a2e34238"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "control"}, "display_name": "controller-2", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 38, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "2dfa7c45-b35d-427e-8e72-4e1d61f87145", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "controller-0", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "8", "memory_mb": "32768", "local_gb": "39", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:control,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown"}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:08:54+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/69ded144-f212-4d40-9069-a6e28d16039c", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/69ded144-f212-4d40-9069-a6e28d16039c", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "2dfa7c45-b35d-427e-8e72-4e1d61f87145", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/69ded144-f212-4d40-9069-a6e28d16039c/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/69ded144-f212-4d40-9069-a6e28d16039c/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/69ded144-f212-4d40-9069-a6e28d16039c/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/69ded144-f212-4d40-9069-a6e28d16039c/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/69ded144-f212-4d40-9069-a6e28d16039c/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/69ded144-f212-4d40-9069-a6e28d16039c/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/69ded144-f212-4d40-9069-a6e28d16039c/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/69ded144-f212-4d40-9069-a6e28d16039c/volume", "rel": "bookmark"}]}, {"uuid": "06581073-1f30-4715-ae53-df539f2e954f", "created_at": "2022-06-15T13:45:14+00:00", "updated_at": "2022-06-15T14:08:49+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6235"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:37.475593", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:08:09.556365", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:24.818093", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["cbc7e670-e258-47fa-8a84-c76ae68a719b"], "metalsmith_created_ports": ["cbc7e670-e258-47fa-8a84-c76ae68a719b"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "control"}, "display_name": "controller-0", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 38, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "controller-1", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "8", "memory_mb": "32768", "local_gb": "39", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:control,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown"}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:08:49+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/06581073-1f30-4715-ae53-df539f2e954f", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/06581073-1f30-4715-ae53-df539f2e954f", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/06581073-1f30-4715-ae53-df539f2e954f/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/06581073-1f30-4715-ae53-df539f2e954f/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/06581073-1f30-4715-ae53-df539f2e954f/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/06581073-1f30-4715-ae53-df539f2e954f/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/06581073-1f30-4715-ae53-df539f2e954f/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/06581073-1f30-4715-ae53-df539f2e954f/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/06581073-1f30-4715-ae53-df539f2e954f/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/06581073-1f30-4715-ae53-df539f2e954f/volume", "rel": "bookmark"}]}, {"uuid": "6cfdc765-220f-4116-9ddf-3ddfc73d40ac", "created_at": "2022-06-15T13:45:14+00:00", "updated_at": "2022-06-15T14:08:54+00:00", "automated_clean": null, "bios_interface": "no-bios", "boot_interface": "ipxe", "clean_step": {}, "conductor_group": "", "console_enabled": false, "console_interface": "ipmitool-socat", "deploy_interface": "direct", "deploy_step": {}, "description": null, "driver": "ipmi", "driver_info": {"deploy_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "rescue_kernel": "file:///var/lib/ironic/httpboot/agent.kernel", "deploy_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "rescue_ramdisk": "file:///var/lib/ironic/httpboot/agent.ramdisk", "ipmi_address": "172.16.0.27", "ipmi_username": "admin", "ipmi_password": "******", "ipmi_port": "6236"}, "driver_internal_info": {"last_power_state_change": "2022-06-15T14:08:42.878729", "is_whole_disk_image": true, "deploy_steps": null, "deploy_boot_mode": "uefi", "agent_version": "7.0.3.dev15", "agent_last_heartbeat": "2022-06-15T14:08:11.889449", "hardware_manager_version": {"generic_hardware_manager": "1.1"}, "agent_cached_deploy_steps_refreshed": "2022-06-15 14:07:32.279346", "root_uuid_or_disk_id": "0x00000000"}, "extra": {"metalsmith_attached_ports": ["2b4bdd24-5931-4a94-80e9-c6821ad7695d"], "metalsmith_created_ports": ["2b4bdd24-5931-4a94-80e9-c6821ad7695d"]}, "fault": null, "inspection_finished_at": null, "inspection_started_at": "2022-06-15T13:45:34+00:00", "inspect_interface": "inspector", "instance_info": {"traits": [], "capabilities": {"profile": "control"}, "display_name": "controller-1", "image_source": "file:///var/lib/ironic/images/overcloud-hardened-uefi-full.raw", "root_gb": 38, "configdrive": "******", "image_disk_format": "raw", "image_checksum": null, "image_os_hash_algo": "sha256", "image_os_hash_value": "7958da6dca08cdde1d49a71aac1eff98b3e17d062b866aa28f19af675d9e81e6", "image_url": "******", "image_type": "whole-disk-image"}, "instance_uuid": "1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8", "last_error": null, "maintenance": false, "maintenance_reason": null, "management_interface": "ipmitool", "name": "controller-2", "network_interface": "flat", "owner": null, "power_interface": "ipmitool", "power_state": "power on", "properties": {"cpus": "8", "memory_mb": "32768", "local_gb": "39", "cpu_arch": "x86_64", "capabilities": "boot_option:local,profile:control,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true", "vendor": "unknown"}, "protected": false, "protected_reason": null, "provision_state": "active", "provision_updated_at": "2022-06-15T14:08:54+00:00", "raid_config": {}, "raid_interface": "no-raid", "rescue_interface": "agent", "reservation": null, "resource_class": "baremetal", "retired": false, "retired_reason": null, "storage_interface": "noop", "target_power_state": null, "target_provision_state": null, "target_raid_config": {}, "traits": [], "vendor_interface": "ipmitool", "links": [{"href": "https://192.168.24.2:13385/v1/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac", "rel": "bookmark"}], "conductor": "undercloud-0.redhat.local", "allocation_uuid": "1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8", "chassis_uuid": null, "ports": [{"href": "https://192.168.24.2:13385/v1/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/ports", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/ports", "rel": "bookmark"}], "states": [{"href": "https://192.168.24.2:13385/v1/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/states", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/states", "rel": "bookmark"}], "portgroups": [{"href": "https://192.168.24.2:13385/v1/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/portgroups", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/portgroups", "rel": "bookmark"}], "volume": [{"href": "https://192.168.24.2:13385/v1/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/volume", "rel": "self"}, {"href": "https://192.168.24.2:13385/nodes/6cfdc765-220f-4116-9ddf-3ddfc73d40ac/volume", "rel": "bookmark"}]}]}
2022-06-15 16:50:45,826 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/0a5d075d-f20c-4771-83d3-0c8b885b32c7 -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:45,851 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-bdf03865-1e5e-4f65-88be-2b85e959bfbb server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:45,851 DEBUG keystoneauth.session: RESP BODY: {"uuid": "0a5d075d-f20c-4771-83d3-0c8b885b32c7", "created_at": "2022-06-15T14:03:20+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["d9478cce-1172-42c9-b7d0-060701f9f553"], "extra": {}, "last_error": null, "name": "ceph-0", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/0a5d075d-f20c-4771-83d3-0c8b885b32c7", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/0a5d075d-f20c-4771-83d3-0c8b885b32c7", "rel": "bookmark"}], "node_uuid": "d9478cce-1172-42c9-b7d0-060701f9f553"}
2022-06-15 16:50:45,856 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/3a0a30ed-ea7a-463b-ade5-afb2517fbaec -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:45,882 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-ea5c6f11-9e08-409c-ba11-8788aba033cf server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:45,883 DEBUG keystoneauth.session: RESP BODY: {"uuid": "3a0a30ed-ea7a-463b-ade5-afb2517fbaec", "created_at": "2022-06-15T14:03:16+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["8819df63-e4c5-4ced-bfe7-fd5d1c978277", "45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa"], "extra": {}, "last_error": null, "name": "compute-1", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/3a0a30ed-ea7a-463b-ade5-afb2517fbaec", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/3a0a30ed-ea7a-463b-ade5-afb2517fbaec", "rel": "bookmark"}], "node_uuid": "8819df63-e4c5-4ced-bfe7-fd5d1c978277"}
2022-06-15 16:50:45,888 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/67089ba3-e1a0-4a34-9b80-766a570a0080 -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:45,914 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-adcf0222-687b-4aaa-9076-c250bec68f81 server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:45,915 DEBUG keystoneauth.session: RESP BODY: {"uuid": "67089ba3-e1a0-4a34-9b80-766a570a0080", "created_at": "2022-06-15T14:03:18+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa"], "extra": {}, "last_error": null, "name": "compute-2", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/67089ba3-e1a0-4a34-9b80-766a570a0080", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/67089ba3-e1a0-4a34-9b80-766a570a0080", "rel": "bookmark"}], "node_uuid": "45ab66ae-4ef6-4863-b8f1-6a1a8adc7dfa"}
2022-06-15 16:50:45,919 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/2dfa7c45-b35d-427e-8e72-4e1d61f87145 -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:45,951 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-7d680145-8f7d-4771-8bf0-ed3619e3998e server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:45,951 DEBUG keystoneauth.session: RESP BODY: {"uuid": "2dfa7c45-b35d-427e-8e72-4e1d61f87145", "created_at": "2022-06-15T14:03:11+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["69ded144-f212-4d40-9069-a6e28d16039c"], "extra": {}, "last_error": null, "name": "controller-2", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/2dfa7c45-b35d-427e-8e72-4e1d61f87145", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/2dfa7c45-b35d-427e-8e72-4e1d61f87145", "rel": "bookmark"}], "node_uuid": "69ded144-f212-4d40-9069-a6e28d16039c"}
2022-06-15 16:50:45,958 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:45,998 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:45 GMT openstack-request-id: req-5b63fa24-5cbf-4a65-8ed7-8304eb1fb7a2 server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:45,998 DEBUG keystoneauth.session: RESP BODY: {"uuid": "f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c", "created_at": "2022-06-15T14:03:07+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["69ded144-f212-4d40-9069-a6e28d16039c", "06581073-1f30-4715-ae53-df539f2e954f", "6cfdc765-220f-4116-9ddf-3ddfc73d40ac"], "extra": {}, "last_error": null, "name": "controller-0", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/f9aab9d2-0b26-4f0d-9c69-2e875c7ab00c", "rel": "bookmark"}], "node_uuid": "06581073-1f30-4715-ae53-df539f2e954f"}
2022-06-15 16:50:46,003 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/allocations/1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8 -H "OpenStack-API-Version: baremetal 1.58" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.58"
2022-06-15 16:50:46,030 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:46 GMT openstack-request-id: req-76c6f2dd-3505-4087-a4ea-60a261961110 server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.58
2022-06-15 16:50:46,030 DEBUG keystoneauth.session: RESP BODY: {"uuid": "1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8", "created_at": "2022-06-15T14:03:09+00:00", "updated_at": "2022-06-15T14:03:25+00:00", "candidate_nodes": ["69ded144-f212-4d40-9069-a6e28d16039c", "6cfdc765-220f-4116-9ddf-3ddfc73d40ac"], "extra": {}, "last_error": null, "name": "controller-1", "resource_class": "baremetal", "state": "active", "traits": [], "links": [{"href": "https://192.168.24.2:13385/v1/allocations/1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8", "rel": "self"}, {"href": "https://192.168.24.2:13385/allocations/1cc482c3-9d20-4de7-ad9b-5c1eefb6c0c8", "rel": "bookmark"}], "node_uuid": "6cfdc765-220f-4116-9ddf-3ddfc73d40ac"}
2022-06-15 16:50:46,035 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/nodes/d9478cce-1172-42c9-b7d0-060701f9f553/vifs -H "OpenStack-API-Version: baremetal 1.61" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.61"
2022-06-15 16:50:46,719 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:46 GMT openstack-request-id: req-3fe90977-944d-4371-8c4e-4f96c4f16ba3 server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.61
2022-06-15 16:50:46,719 DEBUG keystoneauth.session: RESP BODY: {"vifs": [{"id": "7c48bcbc-701c-44bf-9ff6-99e853a42c30"}]}
2022-06-15 16:50:46,727 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13696/v2.0/ports/7c48bcbc-701c-44bf-9ff6-99e853a42c30 -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356"
2022-06-15 16:50:46,878 DEBUG keystoneauth.session: RESP: [200] content-length: 1565 content-type: application/json date: Wed, 15 Jun 2022 16:50:46 GMT x-openstack-request-id: req-3271ea13-694b-4efd-ab38-eadf06e59927
2022-06-15 16:50:46,878 DEBUG keystoneauth.session: RESP BODY: {"port":{"id":"7c48bcbc-701c-44bf-9ff6-99e853a42c30","name":"ceph-0-ctlplane","network_id":"5d544069-306d-4c11-9f4a-c8601e39beb3","tenant_id":"216558ff141544f18f91340c9b144c8b","mac_address":"52:54:00:f0:b9:29","admin_state_up":true,"status":"ACTIVE","device_id":"","device_owner":"","fixed_ips":[{"subnet_id":"ac4231a6-2f13-43a1-a12c-b4743eba3d73","ip_address":"192.168.24.33"}],"allowed_address_pairs":[],"extra_dhcp_opts":[{"opt_name":"150","opt_value":"192.168.24.1","ip_version":4},{"opt_name":"66","opt_value":"192.168.24.1","ip_version":4},{"opt_name":"server-ip-address","opt_value":"192.168.24.1","ip_version":4},{"opt_name":"tag:!ipxe,67","opt_value":"snponly.efi","ip_version":4},{"opt_name":"tag:ipxe,67","opt_value":"http://192.168.24.1:8088/boot.ipxe","ip_version":4}],"security_groups":["567f84d4-61f1-4e05-afbb-9a2429098fff"],"description":"","binding:vnic_type":"baremetal","binding:profile":{},"binding:host_id":"d9478cce-1172-42c9-b7d0-060701f9f553","binding:vif_type":"other","binding:vif_details":{"connectivity":"legacy"},"port_security_enabled":true,"dns_name":"ceph-0","dns_assignment":[{"ip_address":"192.168.24.33","hostname":"ceph-0","fqdn":"ceph-0.redhat.local."}],"dns_domain":"","ip_allocation":"immediate","tags":["tripleo_default_route=true","tripleo_ironic_uuid=d9478cce-1172-42c9-b7d0-060701f9f553","tripleo_ironic_vif_port=true","tripleo_role=CephStorage","tripleo_stack_name=overcloud"],"created_at":"2022-06-15T14:03:26Z","updated_at":"2022-06-15T14:09:09Z","revision_number":17,"project_id":"216558ff141544f18f91340c9b144c8b"}}
2022-06-15 16:50:46,878 DEBUG keystoneauth.session: GET call to network for https://192.168.24.2:13696/v2.0/ports/7c48bcbc-701c-44bf-9ff6-99e853a42c30 used request id req-3271ea13-694b-4efd-ab38-eadf06e59927
2022-06-15 16:50:46,880 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13696/v2.0/networks/5d544069-306d-4c11-9f4a-c8601e39beb3 -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356"
2022-06-15 16:50:46,975 DEBUG keystoneauth.session: RESP: [200] content-length: 747 content-type: application/json date: Wed, 15 Jun 2022 16:50:46 GMT x-openstack-request-id: req-158c6021-85cc-4b25-b5dd-d85dd5370fca
2022-06-15 16:50:46,975 DEBUG keystoneauth.session: RESP BODY: {"network":{"id":"5d544069-306d-4c11-9f4a-c8601e39beb3","name":"ctlplane","tenant_id":"216558ff141544f18f91340c9b144c8b","admin_state_up":true,"mtu":1500,"status":"ACTIVE","subnets":["ac4231a6-2f13-43a1-a12c-b4743eba3d73"],"shared":false,"availability_zone_hints":[],"availability_zones":["nova"],"ipv4_address_scope":null,"ipv6_address_scope":null,"router:external":false,"description":"","port_security_enabled":true,"dns_domain":"ctlplane.redhat.local.","l2_adjacency":true,"tags":["192.168.24.0/24"],"created_at":"2022-06-15T13:38:49Z","updated_at":"2022-06-15T13:38:50Z","revision_number":3,"project_id":"216558ff141544f18f91340c9b144c8b","provider:network_type":"flat","provider:physical_network":"ctlplane","provider:segmentation_id":null}}
2022-06-15 16:50:46,975 DEBUG keystoneauth.session: GET call to network for https://192.168.24.2:13696/v2.0/networks/5d544069-306d-4c11-9f4a-c8601e39beb3 used request id req-158c6021-85cc-4b25-b5dd-d85dd5370fca
2022-06-15 16:50:46,978 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13385/v1/nodes/8819df63-e4c5-4ced-bfe7-fd5d1c978277/vifs -H "OpenStack-API-Version: baremetal 1.61" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356" -H "X-OpenStack-Ironic-API-Version: 1.61"
2022-06-15 16:50:47,609 DEBUG keystoneauth.session: RESP: [200] content-encoding: gzip content-type: application/json date: Wed, 15 Jun 2022 16:50:46 GMT openstack-request-id: req-5046543b-f4a6-4b19-ba22-f1bf6c20372b server: Apache transfer-encoding: chunked vary: Accept-Encoding x-openstack-ironic-api-maximum-version: 1.72 x-openstack-ironic-api-minimum-version: 1.1 x-openstack-ironic-api-version: 1.61
2022-06-15 16:50:47,609 DEBUG keystoneauth.session: RESP BODY: {"vifs": [{"id": "f27b5656-55d9-4b2c-92e0-af1ba9c23d73"}]}
2022-06-15 16:50:47,610 DEBUG keystoneauth.session: REQ: curl -g -i -X GET https://192.168.24.2:13696/v2.0/ports/f27b5656-55d9-4b2c-92e0-af1ba9c23d73 -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.9.10" -H "X-Auth-Token: {SHA256}756378142f6f32652946f719ccdf1be09526edbd27d0bf4979ddd0fba1d60356"
2022-06-15 16:50:47,644 DEBUG keystoneauth.session: RESP: [404] content-length: 132 content-type: application/json date: Wed, 15 Jun 2022 16:50:47 GMT x-openstack-request-id: req-60609cb1-99be-4fef-b3b7-5e8bdf4a15b7
2022-06-15 16:50:47,644 DEBUG keystoneauth.session: RESP BODY: {"NeutronError": {"type": "PortNotFound", "message": "Port f27b5656-55d9-4b2c-92e0-af1ba9c23d73 could not be found.", "detail": ""}}
2022-06-15 16:50:47,644 DEBUG keystoneauth.session: GET call to network for https://192.168.24.2:13696/v2.0/ports/f27b5656-55d9-4b2c-92e0-af1ba9c23d73 used request id req-60609cb1-99be-4fef-b3b7-5e8bdf4a15b7
2022-06-15 16:50:47,644 CRITICAL metalsmith._cmd: No Port found for f27b5656-55d9-4b2c-92e0-af1ba9c23d73: Client Error for url: https://192.168.24.2:13696/v2.0/ports/f27b5656-55d9-4b2c-92e0-af1ba9c23d73, Port f27b5656-55d9-4b2c-92e0-af1ba9c23d73 could not be found.
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/metalsmith/_cmd.py", line 261, in main
    args.func(api, args, formatter)
  File "/usr/lib/python3.9/site-packages/metalsmith/_cmd.py", line 108, in _do_list
    formatter.show(instances)
  File "/usr/lib/python3.9/site-packages/metalsmith/_format.py", line 104, in show
    for row in self._iter_rows(instances):
  File "/usr/lib/python3.9/site-packages/metalsmith/_format.py", line 71, in _iter_rows
    instance.ip_addresses().items())
  File "/usr/lib/python3.9/site-packages/metalsmith/_instance.py", line 93, in ip_addresses
    for nic in self.nics():
  File "/usr/lib/python3.9/site-packages/metalsmith/_instance.py", line 120, in nics
    port = self._connection.network.get_port(vif)
  File "/usr/lib/python3.9/site-packages/openstack/network/v2/_proxy.py", line 1890, in get_port
    return self._get(_port.Port, port)
  File "/usr/lib/python3.9/site-packages/openstack/proxy.py", line 47, in check
    return method(self, expected, actual, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/openstack/proxy.py", line 506, in _get
    return res.fetch(
  File "/usr/lib/python3.9/site-packages/openstack/resource.py", line 1461, in fetch
    self._translate_response(response, **kwargs)
  File "/usr/lib/python3.9/site-packages/openstack/resource.py", line 1158, in _translate_response
    exceptions.raise_from_response(response, error_message=error_message)
  File "/usr/lib/python3.9/site-packages/openstack/exceptions.py", line 236, in raise_from_response
    raise cls(
openstack.exceptions.ResourceNotFound: No Port found for f27b5656-55d9-4b2c-92e0-af1ba9c23d73: Client Error for url: https://192.168.24.2:13696/v2.0/ports/f27b5656-55d9-4b2c-92e0-af1ba9c23d73, Port f27b5656-55d9-4b2c-92e0-af1ba9c23d73 could not be found.

OK, I've attached a fix for missing ports in the metalsmith list call.

I'm still concerned that the scale-down process is deleting the wrong port, maybe related to the node-name/hostname swap.

Having access to the environment or attaching the ironic-conductor logs would help.

These are Jenkins logs. Maybe ironic-conductor logs were archived:

https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/df/view/rfe/job/DFG-df-rfe-17.0-virsh-3cont_3comp_1ceph-blacklist-1compute-scaledown/39/

I'm making progress on diagnosing the root cause, according to the logs on the failing job[1] name:compute-0 hostname:compute-1 should be scaled down, but the vif_detach call is for the vif of node name:compute-1.

This will be why the job is occasionally green, when name/hostname match.

I'll continue next week.

[1] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-df-rfe-17.0-virsh-3cont_3comp_1ceph-blacklist-1compute-scaledown/39/undercloud-0/

I believe I've found the root cause, unprovision was being referenced by node name, but the metalsmith node lookup first attempts the allocation name. The fix is attached.

Downstream backports are now proposed

*** Bug 2104675 has been marked as a duplicate of this bug. ***

Moving back to on_dev. See https://bugzilla.redhat.com/show_bug.cgi?id=2104675 for a description of the problems still being seen with scale up/scale down.

Fix proposed

Follow-up fix from Rabi has been proposed to rhos-17.0-trunk-patches:

https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/421639

Latest patch[1] still open, so it's not yet in.

Since it's not a blocker, it will probably be for potential z1. Moving back to POST.

The compute replacement job that was failing when this BZ was written has been passing for a long time now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577