Description of problem: When running in a split stack environment, nodes have hostnames defined outside of rhosp. Its possible that a node leaves the cloud and rejoins the cloud with the same name later on. In this case, since the node scale down doesn't remove information from the default nova cell or the resource proviers table of nova api, the node will not work. It gets conflicts when trying to self register as a resource provider, and the cell discover is not able to place the node into the cell. Version-Release number of selected component (if applicable): RHOSP12 How reproducible: Add a node as compute to a working nova environment remove the node add it with the same hostname again try to place a VM on it (will fail) Steps to Reproduce: 1. remove compute resource 2. add it back to the cloud 3. boom Actual results: The node is not able to take workloads as its not fully registered as resource provider nor is it placed in the default cell. Expected results: Node successfully rejoins the cloud and can take workloads. Additional info: The removal of a node doesn't clean up the node from the cell or the "resource_provider" (database, nova_api). This prevents the re-added node from "joining" the resource_provider list again, because it requires nodes to be uniq. In a normal deployment this is not an issue, as the node name usually contains a counting number. In pre-deployed environments, this might (will) happen. So the artifacts of the node stays inside the database, preventing it from rejoining the cloud. So the resource_provider issue looks like this: Conflicting resource provider <hostname>.<domain> already exists. The old entry for the node still exists in the database, a new one with the same name can't be added. The cell issue looks like this: ()[root@<hostname> /]# nova-manage cell_v2 discover_hosts --verbose Found 2 cell mappings. Skipping cell0 since it does not contain hosts. Getting compute nodes from cell 'default': e165ac8a-ec98-4b53-bda9-1c3ee7efcd5d Found 1 unmapped computes in cell: e165ac8a-ec98-4b53-bda9-1c3ee7efcd5d Checking host mapping for compute host '<hostname>.<domain>': b42d76a5-5c22-46d5-a37b-80f69f91c127 ()[root@<hostname> /]# nova-manage cell_v2 discover_hosts --verbose Found 2 cell mappings. Skipping cell0 since it does not contain hosts. Getting compute nodes from cell 'default': e165ac8a-ec98-4b53-bda9-1c3ee7efcd5d Found 1 unmapped computes in cell: e165ac8a-ec98-4b53-bda9-1c3ee7efcd5d Checking host mapping for compute host '<hostname>.<domain>': b42d76a5-5c22-46d5-a37b-80f69f91c127 You can call this command over and over again, the host will never be mapped. And why? because its already mapped (the old one with the same name). So to solve the "host not found" and "unable to update my resource provider record" issue, two tasks need to be done: 1. remove the existing host entry from mysql database "nova_api" table "resource_provider" 2. delete host from default cell "nova-manage cell_v2 delete_host --cell_uuid e165ac8a-ec98-4b53-bda9-1c3ee7efcd5d $hostname" (this command is from my memory, as i don't have access right now, sorry). After those two commands, the node will self register again as a resource provider and you can execute the cell_v2 discover_hosts with a successful mapping.
This was fixed in upstream pike: https://review.openstack.org/#/c/553829/
(In reply to Dan Smith from comment #1) > This was fixed in upstream pike: > > https://review.openstack.org/#/c/553829/ Hey Dan, but this was only for cell deletion, the issue we see is when a node is deleted, right? We would need to add a node delete from cell to the "service delete" of compute or to the templates when we scale down a node. The first one would probably the best solution, as this could also happen without director. Cheers, Sven
Sorry, this is the upstream fix for delete_host that I was thinking, which unmaps the node when deleting the host mapping: https://review.openstack.org/#/c/527560/ Maybe you could elaborate more on what you mean by "remove compute resource" so I can know what you're deleting and re-adding? The thing is, if the compute node and service come back in the same configuration (i.e. the same hostname) the old host mapping should still be sufficient to find it again (i.e. you shouldn't need to discover again).
Hey Dan, sorry for the delay, missed that one :( To clearify: The issue was triggered first, when we started to do some scale in and out tests in our test environment. So what we basically did: - install RHOSP12 with 2 Compute Nodes - scale to 3 nodes (adding node c) - scale to 2 nodes (removing a, b or c) - reinstall the node - scale to 3 nodes (adding the removed node) This is in a split stack environment, so the nodes are not installed by ironic, but externally by the customer. For that reason the node name won't change. So if you remove compute-b and reinstall it, it will be compute-b again. In this scenario, the node is only removed from the environment within heat. Since there is no delete_host executed or done, the whole mapping stays as is. So if you try to bring a node "back" (or lets simplify it: if you add a node which has the exact same hostname as one node had before), it doesn't work. The existing, orphaned entry prevents the node from being re-added to the cell. So the commit you're refering to might fix the need of manually fiddleing around in the database, as a delete_host would be enough. But then we need to add the delete_host as a needed step to our documentation. Or we add a task into our templates executing exactly the delete_host command. But as we ask the customer to disable and remove the service manually already, the first one might be easier (except that this needs to be executed inside a container as there is no external command for it, right?). If you still miss something, please let me know. Cheers and thanks, Sven
This is not an issue only with in the deployed server scenario. It can also happen when we use hostname mappings to keep the same hostname even if the internal index goes up. What we'd need is: 1) https://review.openstack.org/#/c/527560/ as part of next OSP12 maintenance release (16.1.x). 2) a doc bug to enhance our scale down procedure to remove the compute from the cell [1]. existing instructions: ~~~ ... Finally, remove the node’s Compute service: (undercloud) $ source ~/stack/overcloudrc (overcloud) $ openstack compute service list (overcloud) $ openstack compute service delete [service-id] ~~~ Here we need to add: ~~~ Login to one of the overcloud controllers and delete the removed host $ ssh heat-admin@overcloud-controller-X $ nova-manage --config-dir /var/lib/config-data/puppet-generated/nova/etc/nova cell_v2 list_hosts $ nova-manage --config-dir /var/lib/config-data/puppet-generated/nova/etc/nova cell_v2 delete_host --cell_uuid <Cell UUID> --host <Hostname> ~~~ [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/director_installation_and_usage/#sect-Removing_Compute_Nodes
This bugzilla has been removed from the release since it has not been triaged, and needs to be reviewed for targeting another release.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2332