Bug 1262986

Summary:	unable to delete heat stack or nodes from ironic
Product:	Red Hat OpenStack	Reporter:	bigswitch <rhosp-bugs-internal>
Component:	rhosp-director	Assignee:	chris alfonso <calfonso>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	yeylon <yeylon>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	hbrock, jslagle, mburns, rhel-osp-director-maint, rhosp-bugs-internal, srevivo
Target Milestone:	---
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-09-30 16:48:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-14 20:01:21 UTC

Description of problem:
Not sure how I hit into this problem, notice mariadb wasnt running on undercloud (1262971), restarted the mariadb service, and attempt to delete heat stack failed. Also attempt to delete node from ironic and set its maintenance mode to off failed with the below message:

[stack@dell-undercloud ~]$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| acffaf46-1518-45fb-883e-88f7ec1726ae | None | d6efae4d-d76c-441f-93e8-48411f6e23c2 | None        | active          | True        |
| 0cfff7c9-229e-48e5-afbc-eac4fe15130a | None | e02db87b-8b83-4f3f-abef-67597eaed760 | None        | active          | True        |
| 8b9dbac2-b2f7-4e37-a612-a63de3f1c80f | None | 72d56c64-96d0-432f-bdd6-012d76c27659 | None        | active          | True        |
| b4aeb7fb-f4a1-455b-935c-168517586fb8 | None | e1f24a55-0481-4a9d-844d-24bb8f792cae | None        | active          | True        |
| d59b9930-aea8-4087-aa6d-e9ce22ce0f84 | None | e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | None        | active          | True        |
| d1d215fc-ec7e-4271-a08b-04ae7e7f198f | None | 1222300f-4977-4d88-8348-c59df3e9c6f7 | None        | active          | True        |
| d7f5b58a-61e1-4c24-ab69-7870811e5fbf | None | None                                 | None        | available       | True        |
| db2b7860-4a2a-4528-9898-0fec76fa9613 | None | 59b17eb7-e119-4c69-8378-80ade7d3aeeb | None        | active          | True        |
| a68bcc55-798d-41f0-8c1d-9ecc2951c80a | None | 058f4064-b149-4160-bf94-813f1d755480 | None        | active          | True        |
| 4f10ad68-b880-426d-92ce-5d6069148634 | None | 946a09a4-b094-405a-9794-63755ea49648 | None        | active          | True        |
| 132545ab-45a2-4343-9932-b99bcddd9ed2 | None | 21851e51-c329-43ad-bdc2-403ce9c38eb2 | None        | active          | True        |
| 53df5f89-bafc-4b03-b814-732cd6ce75f6 | None | 08c5e611-4885-412f-8641-a4452a8bdcd5 | None        | active          | True        |
| 9f64113c-6b6e-40a0-ae8d-bc87c88d037e | None | a19bb1d0-8b0c-4276-9323-68b2b07938a2 | None        | active          | True        |
| b40ddec2-9d4f-42a9-b68f-4887a217ec26 | None | 7e81cd61-81cf-4b6e-8d77-8607bd16e276 | None        | active          | True        |
| e005e38c-026d-4438-a9da-255efce35978 | None | None                                 | None        | available       | True        |
| b73ffb6f-b0b2-4a51-8731-4b80ad512907 | None | db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | None        | active          | True        |
| fb2b2300-cbe5-40c8-a8d1-1626b23a23e2 | None | 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | None        | active          | True        |
| 6ded664f-da01-45a1-9adb-990af6dd8ba4 | None | 040f55f4-28a4-448f-b2ff-864d26812bb1 | None        | active          | True        |
| 0c7394e6-b7af-4ceb-a2a3-712571b19a8b | None | None                                 | None        | available       | True        |
| 3086b196-eaca-46c2-94db-764e1c750cb4 | None | bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | None        | active          | True        |
| 4883759d-fa07-4d19-bf01-e6b08e6e0cba | None | 1ff336ae-8d73-4893-b970-d2dce66176b0 | None        | active          | True        |
| f8fc2900-2e3b-4f7c-ab90-9c910f6b22a5 | None | c87353f1-d60e-4cee-9e0a-706902030992 | None        | active          | True        |
| c9fddb2f-65d8-4433-924b-72dac00dde57 | None | None                                 | None        | available       | True        |
| ea319814-f4e9-4586-afd5-b37550904d4f | None | d6f057ba-9509-453a-b08f-05d58dbe7a4c | None        | active          | True        |
| 8d52d5c5-2515-4404-bfb3-c2f77edaf801 | None | 6bc8560b-4b43-4587-8727-8af4b1ede22a | None        | active          | True        |
| 56aecd60-ca8c-446d-b523-5481adbd0c0f | None | 458c6e45-c0dc-44ed-8be8-f082034e2427 | None        | active          | True        |
| 82a9724b-46ce-4ef4-86c4-689c568fd855 | None | None                                 | None        | available       | True        |
| d72e23aa-380a-4197-91c6-9b83196da903 | None | a1f1327a-fbf1-4c5c-9507-6801aa24be87 | None        | active          | True        |
| edcbb1c9-6be3-4887-8319-38932685a424 | None | c9fd0800-864f-40ea-b3a2-4e47590405e6 | None        | active          | True        |
| 41bb9457-cd03-4163-b8b3-76998f05a210 | None | None                                 | None        | available       | True        |
| be65163e-0422-4df8-8c26-22b07b9cd3fd | None | f0c24cbc-d018-42d4-a18e-fc320f8fdaba | None        | active          | True        |
| 73f4698a-deba-4ba5-a0ea-23079e75b9c4 | None | 9d504f22-0d70-40bc-89d7-ca4fa0b56c1f | None        | active          | True        |
| b1f4805f-5d5a-4ee2-b3f5-f6ba48c44113 | None | dd090ff9-4c75-4872-abe9-3f40c529cb67 | None        | active          | True        |
| 771b52d5-9582-4de4-99af-0b23bc8f8135 | None | 79f875c6-f155-4343-9122-62b31b3f11d9 | None        | active          | True        |
| d58ad309-1af5-4c3b-8acb-75940457baaf | None | 94b059ff-a1b1-4c74-b387-de8d6d343e1f | None        | active          | True        |
| e9da7e88-c18b-4a9a-a9ad-5fea15d3a1af | None | 6dd2c32a-3dfe-49b3-a17a-faebb16b208c | None        | active          | True        |
| 2f0247c4-ee0c-414f-8bef-039e16b5c2d0 | None | None                                 | None        | available       | True        |
| ab00ff13-ab47-4379-88b1-f2862f25bf9f | None | None                                 | None        | available       | True        |
| 57926d21-671a-4308-a165-edbcc0e2a5e9 | None | None                                 | None        | available       | True        |
| a1f7c05a-29e9-49ed-8c69-86cfd2d2109b | None | None                                 | None        | available       | True        |
| b88c0c7e-e3de-4931-8ce0-3805fb303fac | None | None                                 | None        | available       | True        |
| 6b13fd48-d41a-485e-a1eb-2e512db4ffbc | None | None                                 | None        | available       | True        |
| 7b990e34-c0e9-4109-9b8b-427df31eec6a | None | None                                 | None        | available       | True        |
| 52d47561-11c7-457e-bad5-c922fc1728b8 | None | None                                 | None        | available       | True        |
| 5524144c-82f7-4188-ba1f-b97cee3512dd | None | None                                 | None        | available       | True        |
| 3af4d971-003f-48dd-bd8b-415ec885a1d8 | None | None                                 | None        | available       | True        |
| 76b44a5d-ab90-42d9-a40d-6260b0b819f5 | None | None                                 | None        | available       | True        |
| 5253aeb9-fdcb-4ff6-9c6e-3c04a3d3164f | None | 4e9494e2-6ff6-4500-8bfb-51fa74130c74 | None        | error           | True        |
| b3cc8bc1-7fdb-4bef-aef4-517b4bdbca23 | None | None                                 | None        | available       | True        |
| 89ffc4ae-4c9c-42d7-bd0f-924ff57e357d | None | None                                 | None        | available       | True        |
| ba4b4e5a-e6fa-4f79-b046-bae8af17d7d3 | None | None                                 | None        | available       | True        |
| fabc9c5a-a1e4-4cab-8c8f-faca64e24947 | None | None                                 | None        | available       | True        |
| 1df44c3c-0b55-478e-a520-a464b1e3076a | None | None                                 | None        | available       | True        |
| c395a170-1d2e-43d1-8a8d-e35650658336 | None | None                                 | None        | available       | True        |
| 20c7c779-e731-4d8b-a8ab-0f7523d6e77a | None | None                                 | None        | available       | True        |
| 21584880-24e6-4d10-aced-2a7d9d4f4b2d | None | None                                 | None        | available       | True        |
| 6eae36f0-147f-43b6-8322-55c2d2bbac69 | None | None                                 | None        | available       | True        |
| b95fcce9-4362-4428-890c-739beb60d131 | None | None                                 | None        | available       | True        |
| e69dcbe3-40ac-4654-bc2d-c2b5d1a61b2a | None | None                                 | None        | available       | True        |
| 3f91fad5-c5ca-4102-a785-ef0d7488b40d | None | None                                 | None        | available       | True        |
| 06355c42-f6b0-4a1f-850b-712b8032aab9 | None | None                                 | None        | available       | True        |
| 514dfd11-bdd0-48c8-a7d5-504105c4162f | None | None                                 | None        | available       | True        |
| b9870479-327f-440b-9edd-aae3fbd7ce36 | None | None                                 | None        | available       | True        |
| 7503ab04-166f-4ee9-81fe-22f3ce240722 | None | None                                 | None        | available       | True        |
| 4b142996-a0f8-435e-b1ac-2d3e655372e3 | None | None                                 | None        | available       | True        |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
[stack@dell-undercloud ~]$ ironic node-delete 4b142996-a0f8-435e-b1ac-2d3e655372e3
No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400)
[stack@dell-undercloud ~]$ ironic node-set-power-state 4b142996-a0f8-435e-b1ac-2d3e655372e3 off
No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 404)
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | DELETE_FAILED | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+---------------+----------------------+


Version-Release number of selected component (if applicable):
7.0

How reproducible:
not sure if its reproducible, this is seen on 3 controller, 30 compute nodes deployment

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 chris alfonso 2015-09-14 20:12:57 UTC

Is there a way you can try to reproduce it? It'll be fairly difficult for us to fix if we can't reproduce it.

Comment 4 bigswitch 2015-09-16 23:07:27 UTC

Chris,
  We will try to reproduce and bump up the severity if it is seen again. Do you need anything other than sosreport when we have reproduction?

Comment 5 chris alfonso 2015-09-17 15:27:28 UTC

You don't need to bump up the severity for us to see the bug. The sos report will be a good start. The steps to reproduce the issue are the most important thing you can provide.

Comment 6 chris alfonso 2015-09-17 16:51:56 UTC

Also, can you provide the ironic-conductor log, if you do reproduce the issue?

Comment 7 James Slagle 2015-09-17 16:52:28 UTC

use journalctl -u openstack-ironic-conductor to get the log

Comment 8 James Slagle 2015-09-22 16:32:51 UTC

had the OOM killer killed these services and mariadb? can you check in the dmesg output? try restarting the services that are down.

Comment 9 bigswitch 2015-09-22 21:24:06 UTC

havent hit into this issue so far. I am redeploying everything from scratch at this moment, and will provide the output if I hit into this issue again.

Comment 10 chris alfonso 2015-09-30 16:48:55 UTC

Closing this since it hasn't been observed again. If it happens again, please reopen.