Bug 1262986 - unable to delete heat stack or nodes from ironic
unable to delete heat stack or nodes from ironic
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity medium
: ---
: 7.0 (Kilo)
Assigned To: chris alfonso
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-14 16:01 EDT by bigswitch
Modified: 2016-04-18 02:54 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-30 12:48:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2015-09-14 16:01:21 EDT
Description of problem:
Not sure how I hit into this problem, notice mariadb wasnt running on undercloud (1262971), restarted the mariadb service, and attempt to delete heat stack failed. Also attempt to delete node from ironic and set its maintenance mode to off failed with the below message:

[stack@dell-undercloud ~]$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| acffaf46-1518-45fb-883e-88f7ec1726ae | None | d6efae4d-d76c-441f-93e8-48411f6e23c2 | None        | active          | True        |
| 0cfff7c9-229e-48e5-afbc-eac4fe15130a | None | e02db87b-8b83-4f3f-abef-67597eaed760 | None        | active          | True        |
| 8b9dbac2-b2f7-4e37-a612-a63de3f1c80f | None | 72d56c64-96d0-432f-bdd6-012d76c27659 | None        | active          | True        |
| b4aeb7fb-f4a1-455b-935c-168517586fb8 | None | e1f24a55-0481-4a9d-844d-24bb8f792cae | None        | active          | True        |
| d59b9930-aea8-4087-aa6d-e9ce22ce0f84 | None | e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | None        | active          | True        |
| d1d215fc-ec7e-4271-a08b-04ae7e7f198f | None | 1222300f-4977-4d88-8348-c59df3e9c6f7 | None        | active          | True        |
| d7f5b58a-61e1-4c24-ab69-7870811e5fbf | None | None                                 | None        | available       | True        |
| db2b7860-4a2a-4528-9898-0fec76fa9613 | None | 59b17eb7-e119-4c69-8378-80ade7d3aeeb | None        | active          | True        |
| a68bcc55-798d-41f0-8c1d-9ecc2951c80a | None | 058f4064-b149-4160-bf94-813f1d755480 | None        | active          | True        |
| 4f10ad68-b880-426d-92ce-5d6069148634 | None | 946a09a4-b094-405a-9794-63755ea49648 | None        | active          | True        |
| 132545ab-45a2-4343-9932-b99bcddd9ed2 | None | 21851e51-c329-43ad-bdc2-403ce9c38eb2 | None        | active          | True        |
| 53df5f89-bafc-4b03-b814-732cd6ce75f6 | None | 08c5e611-4885-412f-8641-a4452a8bdcd5 | None        | active          | True        |
| 9f64113c-6b6e-40a0-ae8d-bc87c88d037e | None | a19bb1d0-8b0c-4276-9323-68b2b07938a2 | None        | active          | True        |
| b40ddec2-9d4f-42a9-b68f-4887a217ec26 | None | 7e81cd61-81cf-4b6e-8d77-8607bd16e276 | None        | active          | True        |
| e005e38c-026d-4438-a9da-255efce35978 | None | None                                 | None        | available       | True        |
| b73ffb6f-b0b2-4a51-8731-4b80ad512907 | None | db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | None        | active          | True        |
| fb2b2300-cbe5-40c8-a8d1-1626b23a23e2 | None | 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | None        | active          | True        |
| 6ded664f-da01-45a1-9adb-990af6dd8ba4 | None | 040f55f4-28a4-448f-b2ff-864d26812bb1 | None        | active          | True        |
| 0c7394e6-b7af-4ceb-a2a3-712571b19a8b | None | None                                 | None        | available       | True        |
| 3086b196-eaca-46c2-94db-764e1c750cb4 | None | bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | None        | active          | True        |
| 4883759d-fa07-4d19-bf01-e6b08e6e0cba | None | 1ff336ae-8d73-4893-b970-d2dce66176b0 | None        | active          | True        |
| f8fc2900-2e3b-4f7c-ab90-9c910f6b22a5 | None | c87353f1-d60e-4cee-9e0a-706902030992 | None        | active          | True        |
| c9fddb2f-65d8-4433-924b-72dac00dde57 | None | None                                 | None        | available       | True        |
| ea319814-f4e9-4586-afd5-b37550904d4f | None | d6f057ba-9509-453a-b08f-05d58dbe7a4c | None        | active          | True        |
| 8d52d5c5-2515-4404-bfb3-c2f77edaf801 | None | 6bc8560b-4b43-4587-8727-8af4b1ede22a | None        | active          | True        |
| 56aecd60-ca8c-446d-b523-5481adbd0c0f | None | 458c6e45-c0dc-44ed-8be8-f082034e2427 | None        | active          | True        |
| 82a9724b-46ce-4ef4-86c4-689c568fd855 | None | None                                 | None        | available       | True        |
| d72e23aa-380a-4197-91c6-9b83196da903 | None | a1f1327a-fbf1-4c5c-9507-6801aa24be87 | None        | active          | True        |
| edcbb1c9-6be3-4887-8319-38932685a424 | None | c9fd0800-864f-40ea-b3a2-4e47590405e6 | None        | active          | True        |
| 41bb9457-cd03-4163-b8b3-76998f05a210 | None | None                                 | None        | available       | True        |
| be65163e-0422-4df8-8c26-22b07b9cd3fd | None | f0c24cbc-d018-42d4-a18e-fc320f8fdaba | None        | active          | True        |
| 73f4698a-deba-4ba5-a0ea-23079e75b9c4 | None | 9d504f22-0d70-40bc-89d7-ca4fa0b56c1f | None        | active          | True        |
| b1f4805f-5d5a-4ee2-b3f5-f6ba48c44113 | None | dd090ff9-4c75-4872-abe9-3f40c529cb67 | None        | active          | True        |
| 771b52d5-9582-4de4-99af-0b23bc8f8135 | None | 79f875c6-f155-4343-9122-62b31b3f11d9 | None        | active          | True        |
| d58ad309-1af5-4c3b-8acb-75940457baaf | None | 94b059ff-a1b1-4c74-b387-de8d6d343e1f | None        | active          | True        |
| e9da7e88-c18b-4a9a-a9ad-5fea15d3a1af | None | 6dd2c32a-3dfe-49b3-a17a-faebb16b208c | None        | active          | True        |
| 2f0247c4-ee0c-414f-8bef-039e16b5c2d0 | None | None                                 | None        | available       | True        |
| ab00ff13-ab47-4379-88b1-f2862f25bf9f | None | None                                 | None        | available       | True        |
| 57926d21-671a-4308-a165-edbcc0e2a5e9 | None | None                                 | None        | available       | True        |
| a1f7c05a-29e9-49ed-8c69-86cfd2d2109b | None | None                                 | None        | available       | True        |
| b88c0c7e-e3de-4931-8ce0-3805fb303fac | None | None                                 | None        | available       | True        |
| 6b13fd48-d41a-485e-a1eb-2e512db4ffbc | None | None                                 | None        | available       | True        |
| 7b990e34-c0e9-4109-9b8b-427df31eec6a | None | None                                 | None        | available       | True        |
| 52d47561-11c7-457e-bad5-c922fc1728b8 | None | None                                 | None        | available       | True        |
| 5524144c-82f7-4188-ba1f-b97cee3512dd | None | None                                 | None        | available       | True        |
| 3af4d971-003f-48dd-bd8b-415ec885a1d8 | None | None                                 | None        | available       | True        |
| 76b44a5d-ab90-42d9-a40d-6260b0b819f5 | None | None                                 | None        | available       | True        |
| 5253aeb9-fdcb-4ff6-9c6e-3c04a3d3164f | None | 4e9494e2-6ff6-4500-8bfb-51fa74130c74 | None        | error           | True        |
| b3cc8bc1-7fdb-4bef-aef4-517b4bdbca23 | None | None                                 | None        | available       | True        |
| 89ffc4ae-4c9c-42d7-bd0f-924ff57e357d | None | None                                 | None        | available       | True        |
| ba4b4e5a-e6fa-4f79-b046-bae8af17d7d3 | None | None                                 | None        | available       | True        |
| fabc9c5a-a1e4-4cab-8c8f-faca64e24947 | None | None                                 | None        | available       | True        |
| 1df44c3c-0b55-478e-a520-a464b1e3076a | None | None                                 | None        | available       | True        |
| c395a170-1d2e-43d1-8a8d-e35650658336 | None | None                                 | None        | available       | True        |
| 20c7c779-e731-4d8b-a8ab-0f7523d6e77a | None | None                                 | None        | available       | True        |
| 21584880-24e6-4d10-aced-2a7d9d4f4b2d | None | None                                 | None        | available       | True        |
| 6eae36f0-147f-43b6-8322-55c2d2bbac69 | None | None                                 | None        | available       | True        |
| b95fcce9-4362-4428-890c-739beb60d131 | None | None                                 | None        | available       | True        |
| e69dcbe3-40ac-4654-bc2d-c2b5d1a61b2a | None | None                                 | None        | available       | True        |
| 3f91fad5-c5ca-4102-a785-ef0d7488b40d | None | None                                 | None        | available       | True        |
| 06355c42-f6b0-4a1f-850b-712b8032aab9 | None | None                                 | None        | available       | True        |
| 514dfd11-bdd0-48c8-a7d5-504105c4162f | None | None                                 | None        | available       | True        |
| b9870479-327f-440b-9edd-aae3fbd7ce36 | None | None                                 | None        | available       | True        |
| 7503ab04-166f-4ee9-81fe-22f3ce240722 | None | None                                 | None        | available       | True        |
| 4b142996-a0f8-435e-b1ac-2d3e655372e3 | None | None                                 | None        | available       | True        |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
[stack@dell-undercloud ~]$ ironic node-delete 4b142996-a0f8-435e-b1ac-2d3e655372e3
No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400)
[stack@dell-undercloud ~]$ ironic node-set-power-state 4b142996-a0f8-435e-b1ac-2d3e655372e3 off
No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 404)
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | DELETE_FAILED | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+---------------+----------------------+


Version-Release number of selected component (if applicable):
7.0

How reproducible:
not sure if its reproducible, this is seen on 3 controller, 30 compute nodes deployment

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 3 chris alfonso 2015-09-14 16:12:57 EDT
Is there a way you can try to reproduce it? It'll be fairly difficult for us to fix if we can't reproduce it.
Comment 4 bigswitch 2015-09-16 19:07:27 EDT
Chris,
  We will try to reproduce and bump up the severity if it is seen again. Do you need anything other than sosreport when we have reproduction?
Comment 5 chris alfonso 2015-09-17 11:27:28 EDT
You don't need to bump up the severity for us to see the bug. The sos report will be a good start. The steps to reproduce the issue are the most important thing you can provide.
Comment 6 chris alfonso 2015-09-17 12:51:56 EDT
Also, can you provide the ironic-conductor log, if you do reproduce the issue?
Comment 7 James Slagle 2015-09-17 12:52:28 EDT
use journalctl -u openstack-ironic-conductor to get the log
Comment 8 James Slagle 2015-09-22 12:32:51 EDT
had the OOM killer killed these services and mariadb? can you check in the dmesg output? try restarting the services that are down.
Comment 9 bigswitch 2015-09-22 17:24:06 EDT
havent hit into this issue so far. I am redeploying everything from scratch at this moment, and will provide the output if I hit into this issue again.
Comment 10 chris alfonso 2015-09-30 12:48:55 EDT
Closing this since it hasn't been observed again. If it happens again, please reopen.

Note You need to log in before you can comment on or make changes to this bug.