Bug 1728335

Description of problem:
When starting multiple instances after a number of previous build failures, all instances are scheduled on one or a small subset of compute nodes instead of being spread across all compute nodes.

Version-Release number of selected component (if applicable):
OSP 14 (Rocky), and I suspect also OSP 13 (Queens)

How reproducible:
Create some build failures on OSP14, then start an instance. The instance will not start on node(s) where a previous failure had occurred, assuming there are other node(s) where failures had not previously occurred.

Steps to Reproduce:
1. Validate that the failure count on the all compute nodes is 0:
[heat-admin@controller0 ]$ sudo docker exec -ti -u root galera-bundle-docker-0 mysql nova -e "select hypervisor_hostname,stats from compute_nodes"

2. Build an instance that will fail. For example, use an image that has no "min-disk" set (or set too small) and then build an instance that uses a disk size that is smaller than the actual image size (this will cause a failure).

3. Validate that the failure count on one of the compute hosts has increased by 1. (See step 1.)

4. Start a new instance and look at the nova-scheduler.log for the weight of the compute nodes. The weight for the compute host with a failure will be N minus 500,000 (e.g. -499997.412955).

5. So long as there are no failures on all other nodes, no instances will be scheduled to the node with the positive failure count until the nova_compute container is restarted on that node.

Actual results:
The node with a positive failure count will have an impossibly low weight, so it will never be selected for scheduling unless all other nodes have also had failures.

Expected results:
Nodes should be scheduled, even if a previous failure had occurred, especially when that failure is caused by a user error (see step 2., for example).

Additional info:
Please see https://access.redhat.com/solutions/4274472 where I have give outputs and workaround instructions.