Bug 1496145 - Invalid instance state after hypervisor has been forcefully shut down
Summary: Invalid instance state after hypervisor has been forcefully shut down
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-26 14:36 UTC by Gabriel Szasz
Modified: 2023-03-21 18:45 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-01 14:23:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1238592 0 high CLOSED [RFE] Nova to get valid server/instance state 2021-02-22 00:41:40 UTC
Red Hat Issue Tracker OSP-23321 0 None None None 2023-03-21 18:45:13 UTC

Description Gabriel Szasz 2017-09-26 14:36:48 UTC
Description of problem:
Nova does not update its state when the hypervisor has been forcefully shut down.  Note that the issue was filed as a RFE for OSP 9 (Bug 1238592) where it was implemented on API level.


Version-Release number of selected component (if applicable):
openstack-nova-api-16.0.1-0.20170830161812.cdf08b7.el7ost


How reproducible:
100%


Steps to Reproduce:

1. Deploy RHOSP using OSP-d (containerized deployment) in non-HA setup (1 controller, 1 compute node)

2. Perform post-deployment steps for manual testing.

https://wiki.test.redhat.com/RHOSP/Compute/ManualTesting

3. Run an instance using nova boot command.

$ nova boot --poll --image cirros --flavor m1.tiny test --nic net-id=$SID

4. Get auth token and instance ID

$ openstack token issue
+------------+--------------------------+
| Field      | Value                    |
+------------+--------------------------+
| expires    | 2017-09-26T17:50:02+0000 |
| id         | <TOKEN_ID>               |
| project_id | <PROJECT_ID>             |
| user_id    | <USER_ID>                |
+------------+--------------------------+

$ nova list
+-----------+------+--------+------------+-------------+-------------------+
| ID        | Name | Status | Task State | Power State | Networks          |
+-----------+------+--------+------------+-------------+-------------------+
| <INST_ID> | test | ACTIVE | -          | Running     | private=<INST_IP> |
+-----------+------+--------+------------+-------------+-------------------+

5. Get instance status via API call

$ curl -i -X GET -H 'X-Auth-Token: <TOKEN_ID>' -H 'X-OpenStack-Nova-API-Version: 2.16' http://X.X.X.X:8774/v2.1/<PROJECT_ID>/servers/<INST_ID>

..."host_status": "UP",...

6. Shutdown the compute service on the compute node

[root@compute-0 ~] docker stop nova_compute

7. Wait a couple of minutes and convince yourself that the service is down

$ nova service-list
+--------+------------------+--------------+----------+---------+-------+
| Id     | Binary           | Host         | Zone     | Status  | State |
+--------+------------------+--------------+----------+---------+-------+
| <UUID> | nova-conductor   | controller-0 | internal | enabled | up    |
| <UUID> | nova-compute     | compute-0    | nova     | enabled | down  |
| <UUID> | nova-consoleauth | controller-0 | internal | enabled | up    |
| <UUID> | nova-scheduler   | controller-0 | internal | enabled | up    |
+--------+------------------+--------------+----------+---------+-------+

8. Get instance status via API call

$ curl -i -X GET -H 'X-Auth-Token: <TOKEN_ID>' -H 'X-OpenStack-Nova-API-Version: 2.16' http://X.X.X.X:8774/v2.1/<PROJECT_ID>/servers/<INST_ID>


Actual results:

..."host_status": "UP",...


Expected results:

..."host_status": "DOWN",...


Additional info:

Note that this issue was fixed in OSP 9.

Comment 3 melanie witt 2017-10-06 15:01:42 UTC
(In reply to Gabriel Szasz from comment #0)
> Actual results:
> 
> ..."host_status": "UP",...
> 
> 
> Expected results:
> 
> ..."host_status": "DOWN",...

As discussed on the triage call, the host_status will only be "DOWN" if nova-compute has been forced_down and the host_status is expected to be "UNKNOWN" if nova-compute is simply stopped and hasn't been seen in the last configured interval.

Comment 4 Gabriel Szasz 2017-10-24 15:23:29 UTC
(Correcting description according comment 3)

...

Actual results:

..."host_status": "UP",...
 
Expected results:
 
..."host_status": "UNKNOWN",...

Additional info:

Note that this issue was fixed in OSP 9.

Comment 7 Lee Yarwood 2018-08-01 14:23:43 UTC
Closing this out as WORKSFORME with an OSP 12 env I see the instance host_status move to UNKNOWN once the nova-compute service is marked as down.


Note You need to log in before you can comment on or make changes to this bug.