Description of problem:

We are running an a scaling operation via our VNFM that performs a stack update to update the number of resources in one of our resource groups in order to scale up the number of instances.  We were trying a failure scenario be capping the number instances by setting the instance quota to the current number of VMs.  The expectation is that the VNFM will attempt the scale out operation several times and then eventually give up due to the multiple stack update failures.  Instead what we see is that the first stack update fails, the second update also fails, but the third and any subsequent update completes successfully despite the new server never being created.  The first stack update fails with the following error:

Resource UPDATE failed: resources.payload_group: Resource CREATE failed: Forbidden: resources[3].resources.rg_wrapper.resources[0].resources.server: Quota exceeded for instances: Requested 1, but already used 32 of 32 instances (HTTP 403) (Request-ID: req

The second with:

Resource UPDATE failed: resources[0]: resources.payload_group.resources[3].resources.rg_wrapper.Resource CREATE failed: Forbidden: resources.server: Quota exceeded for instances: Requested 1, but already used 32 of 32 instances (HTTP 403) (Request-ID: req

But for the third attempt the status for the resource group that is missing the addional server simply says:

state changed

After this third update it seems Heat is no longer recognizing that the server instance is missing.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

I created a simpler heat template that doesn't require our VNFM to be brought up and uses a standard RHEL image.  It is attached as teststack.tar.gz.  With this the issue can be reproduced using just the OpenStack CLI.

1. Create a project with a instance limit of 3, and grab its OpenStack RC file.
2. Source theRC file and bring up the stack in that project with :  openstack stack create -t main_oam_only.yaml -e env.yaml teststack
3 Once the stack is up and stable, attempt to scale it out to 4 instances via a stack update: openstack stack update -t main_oam_only.yaml -e env_scale_up.yaml teststack.
4. Wait for this stack update to fail.  It will fail with the expected failure reason: "Quota exceeded for instances: Requested 1, but already used 3 of 3 instances (HTTP 403)"
5. Do another stack update to again attempt to scale it to 4 instances: openstack stack create -t main_oam_only.yaml -e env.yaml teststack, this will again fail with the expected error "Quota exceeded for instances: Requested 1, but already used 3 of 3 instances (HTTP 403)"
6 Do a third stack update to again attempt to scale it to 4 instances: openstack stack create -t main_oam_only.yaml -e env.yaml teststack, this will succeed as will any subsequent stack updates that reattempt to scale it to 4 instances.


Actual results:

Example from our system:

bash-4.2$ openstack stack create -t main_oam_only.yaml -e env.yaml teststack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | 2aeb02c8-46f0-4fe8-9409-60e755e83480 |
| stack_name          | teststack                            |
| description         | No description                       |
| creation_time       | 2021-06-15T18:38:32Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
bash-4.2$ openstack stack show -c stack_status -c stack_status_reason teststack
+---------------------+-------------------------------------+
| Field               | Value                               |
+---------------------+-------------------------------------+
| stack_status        | CREATE_COMPLETE                     |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+-------------------------------------+
bash-4.2$ openstack stack update -t main_oam_only.yaml -e env_scale_up.yaml teststack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | 2aeb02c8-46f0-4fe8-9409-60e755e83480 |
| stack_name          | teststack                            |
| description         | No description                       |
| creation_time       | 2021-06-15T18:38:32Z                 |
| updated_time        | 2021-06-15T18:40:54Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
bash-4.2$ openstack stack show -c stack_status -c stack_status_reason teststack
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                                                                                                              |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| stack_status        | UPDATE_FAILED                                                                                                                                                                                                                                                      |
| stack_status_reason | Resource UPDATE failed: resources.backend_server: Resource CREATE failed: Forbidden: resources[2].resources.server: Quota exceeded for instances: Requested 1, but already used 3 of 3 instances (HTTP 403) (Request-ID: req-3869a405-d3c1-40c8-a5a6-4d55d459afa7) |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
bash-4.2$ openstack stack update -t main_oam_only.yaml -e env_scale_up.yaml teststack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | 2aeb02c8-46f0-4fe8-9409-60e755e83480 |
| stack_name          | teststack                            |
| description         | No description                       |
| creation_time       | 2021-06-15T18:38:32Z                 |
| updated_time        | 2021-06-15T18:42:42Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
bash-4.2$ openstack stack show -c stack_status -c stack_status_reason teststack
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                                                                                                              |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| stack_status        | UPDATE_FAILED                                                                                                                                                                                                                                                      |
| stack_status_reason | Resource UPDATE failed: resources[2]: resources.backend_server.Resource CREATE failed: Forbidden: resources.server: Quota exceeded for instances: Requested 1, but already used 3 of 3 instances (HTTP 403) (Request-ID: req-69c7052e-44d9-4d5d-8797-da73e8a40e38) |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
bash-4.2$ openstack stack update -t main_oam_only.yaml -e env_scale_up.yaml teststack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | 2aeb02c8-46f0-4fe8-9409-60e755e83480 |
| stack_name          | teststack                            |
| description         | No description                       |
| creation_time       | 2021-06-15T18:38:32Z                 |
| updated_time        | 2021-06-15T18:46:09Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
bash-4.2$ openstack stack show -c stack_status -c stack_status_reason teststack
+---------------------+-------------------------------------+
| Field               | Value                               |
+---------------------+-------------------------------------+
| stack_status        | UPDATE_COMPLETE                     |
| stack_status_reason | Stack UPDATE completed successfully |
+---------------------+-------------------------------------+
bash-4.2$


Expected results:


Additional info: