Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1749077

Summary: Instance <uuid> has allocations against this compute host but is not found in the database. in OSP 13
Product: Red Hat OpenStack Reporter: Andreas Karis <akaris>
Component: openstack-novaAssignee: Sylvain Bauza <sbauza>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, vcojot, vromanso
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-13 14:50:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2019-09-04 19:57:30 UTC 
Description of problem:
Instance <uuid> has allocations against this compute host but is not found in the database. in OSP 13

Version-Release number of selected component (if applicable):
]$ grep nova ./sos_commands/docker/docker_ps
afe9ccc4ac7e        10.8.68.1:8787/rhosp13/openstack-nova-compute:latest                "kolla_start"       26 hours ago        Up 3 minutes (healthy)                       nova_compute
8f621b3da7ea        10.8.68.1:8787/rhosp13/openstack-nova-compute:latest                "kolla_start"       26 hours ago        Up 3 minutes (healthy)                       nova_migration_target
420a84f35193        10.8.68.1:8787/rhosp13/openstack-nova-libvirt:latest                "kolla_start"       26 hours ago        Up 3 minutes                                 nova_libvirt
e52f6b89a10d        10.8.68.1:8787/rhosp13/openstack-nova-libvirt:latest                "kolla_start"       26 hours ago        Up 3 minutes                                 nova_virtlogd
[supportshell.prod.useraccess-us-west-2.redhat.com] [19:48:11+0000]
]$ grep nova installed-rpms 
openstack-nova-api-17.0.9-9.el7ost.noarch                   Thu Apr 18 15:03:32 2019
openstack-nova-common-17.0.9-9.el7ost.noarch                Thu Apr 18 15:02:48 2019
openstack-nova-compute-17.0.9-9.el7ost.noarch               Thu Apr 18 15:03:00 2019
openstack-nova-conductor-17.0.9-9.el7ost.noarch             Thu Apr 18 15:03:32 2019
openstack-nova-console-17.0.9-9.el7ost.noarch               Thu Apr 18 15:03:32 2019
openstack-nova-migration-17.0.9-9.el7ost.noarch             Thu Apr 18 15:03:28 2019
openstack-nova-novncproxy-17.0.9-9.el7ost.noarch            Thu Apr 18 15:03:32 2019
openstack-nova-placement-api-17.0.9-9.el7ost.noarch         Thu Apr 18 15:03:31 2019
openstack-nova-scheduler-17.0.9-9.el7ost.noarch             Thu Apr 18 15:03:31 2019
puppet-nova-12.4.0-17.el7ost.noarch                         Thu Apr 18 15:06:49 2019
python2-novaclient-10.1.0-1.el7ost.noarch                   Thu Apr 18 15:02:09 2019
python-nova-17.0.9-9.el7ost.noarch                          Thu Apr 18 15:02:47 2019
[supportshell.prod.useraccess-us-west-2.redhat.com] [19:48:14+0000]
[]$ find . -name '*docker*images*'
./sos_commands/docker/docker_images
[supportshell.prod.useraccess-us-west-2.redhat.com] [19:48:36+0000]
]$ grep nova ./sos_commands/docker/docker_images
10.8.68.1:8787/rhosp13/openstack-nova-compute                latest              e8dcdb866cd4        3 months ago        1.5 GB
10.8.68.1:8787/rhosp13/openstack-nova-libvirt                latest              debd9e959854        3 months ago        1.42 GB
[supportshell.prod.useraccess-us-west-2.redhat.com] [19:48:40+0000]


How reproducible:
Instance was deleted at some point with command:
#openstack server delete <ID1> <ID2> <ID3> 

Instance is gone, but holds 3x its allocation size. Issue could be found for 2 instances and was worked around with https://access.redhat.com/solutions/3537351

Instance 1:
2019-08-29 12:13:25.125 1 INFO nova.compute.resource_tracker [req-ddf21868-8ebc-4a38-9af6-9ad2929c2a44 - - - - -] Instance fb8bb08f-0f66-4659-a805-c784e7c17cfd has allocations against this compute host but is not found in the database.

Instance 2:
2019-09-03 12:18:12.777 1 INFO nova.compute.resource_tracker [req-e1bda850-6fa4-4875-811b-2efa143be243 - - - - -] Instance c2de752a-73bf-4cb9-9cea-89746f0c15ad has allocations against this compute host but is not found in the database.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Quick summary:

We used https://access.redhat.com/solutions/3537351 to clean up a rogue allocation that caused issues.
~~~
[root@controller-0 nova]# curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OS_TOKEN" "$PLACEMENT_ENDPOINT/resource_providers/$rp/allocations" | python -m json.tool
{
    "allocations": {
        "d732b089-2ff4-4526-8604-cf7904e63acb": {
            "resources": {
                "DISK_GB": 550,
                "MEMORY_MB": 32768,
                "VCPU": 16
            }
        },
        "fb8bb08f-0f66-4659-a805-c784e7c17cfd": {
            "resources": {
                "DISK_GB": 1650,
                "MEMORY_MB": 98304,
                "VCPU": 48
            }
        }
    },
    "resource_provider_generation": 6
}
[root@controller-0 nova]# curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OS_TOKEN" "$PLACEMENT_ENDPOINT/allocations/fb8bb08f-0f66-4659-a805-c784e7c17cfd" | python -m json.tool
{
    "allocations": {
        "b7dd3fc1-d705-4991-a976-10e687952626": {
            "generation": 6,
            "resources": {
                "DISK_GB": 1650,
                "MEMORY_MB": 98304,
                "VCPU": 48
            }
        }
    },
    "project_id": "bc3721cc88a3494cbb8f350d01ebfa09",
    "user_id": "c807b1b765544eff9473c68aaf036949"
}
[root@controller-0 nova]# curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OS_TOKEN" "$PLACEMENT_ENDPOINT/allocations/fb8bb08f-0f66-4659-a805-c784e7c17cfd" -X DELETE | python -m json.tool
No JSON object could be decoded
[root@rt-331-control-01-controller-0 nova]# curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OS_TOKEN" "$PLACEMENT_ENDPOINT/allocations/fb8bb08f-0f66-4659-a805-c784e7c17cfd" | python -m json.tool
{
    "allocations": {}
}
[root@controller-0 nova]# curl -s -H "accept: application/json" -H "User-Agent: nova-scheduler keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "OpenStack-API-Version: placement 1.17" -H "X-Auth-Token: $OS_TOKEN" "$PLACEMENT_ENDPOINT/resource_providers/$rp/allocations" | python -m json.tool
{
    "allocations": {
        "d732b089-2ff4-4526-8604-cf7904e63acb": {
            "resources": {
                "DISK_GB": 550,
                "MEMORY_MB": 32768,
                "VCPU": 16
            }
        }
    },
    "resource_provider_generation": 6
}
~~~