Description of problem: Not sure how I hit into this problem, notice mariadb wasnt running on undercloud (1262971), restarted the mariadb service, and attempt to delete heat stack failed. Also attempt to delete node from ironic and set its maintenance mode to off failed with the below message: [stack@dell-undercloud ~]$ ironic node-list +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ | UUID | Name | Instance UUID | Power State | Provision State | Maintenance | +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ | acffaf46-1518-45fb-883e-88f7ec1726ae | None | d6efae4d-d76c-441f-93e8-48411f6e23c2 | None | active | True | | 0cfff7c9-229e-48e5-afbc-eac4fe15130a | None | e02db87b-8b83-4f3f-abef-67597eaed760 | None | active | True | | 8b9dbac2-b2f7-4e37-a612-a63de3f1c80f | None | 72d56c64-96d0-432f-bdd6-012d76c27659 | None | active | True | | b4aeb7fb-f4a1-455b-935c-168517586fb8 | None | e1f24a55-0481-4a9d-844d-24bb8f792cae | None | active | True | | d59b9930-aea8-4087-aa6d-e9ce22ce0f84 | None | e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | None | active | True | | d1d215fc-ec7e-4271-a08b-04ae7e7f198f | None | 1222300f-4977-4d88-8348-c59df3e9c6f7 | None | active | True | | d7f5b58a-61e1-4c24-ab69-7870811e5fbf | None | None | None | available | True | | db2b7860-4a2a-4528-9898-0fec76fa9613 | None | 59b17eb7-e119-4c69-8378-80ade7d3aeeb | None | active | True | | a68bcc55-798d-41f0-8c1d-9ecc2951c80a | None | 058f4064-b149-4160-bf94-813f1d755480 | None | active | True | | 4f10ad68-b880-426d-92ce-5d6069148634 | None | 946a09a4-b094-405a-9794-63755ea49648 | None | active | True | | 132545ab-45a2-4343-9932-b99bcddd9ed2 | None | 21851e51-c329-43ad-bdc2-403ce9c38eb2 | None | active | True | | 53df5f89-bafc-4b03-b814-732cd6ce75f6 | None | 08c5e611-4885-412f-8641-a4452a8bdcd5 | None | active | True | | 9f64113c-6b6e-40a0-ae8d-bc87c88d037e | None | a19bb1d0-8b0c-4276-9323-68b2b07938a2 | None | active | True | | b40ddec2-9d4f-42a9-b68f-4887a217ec26 | None | 7e81cd61-81cf-4b6e-8d77-8607bd16e276 | None | active | True | | e005e38c-026d-4438-a9da-255efce35978 | None | None | None | available | True | | b73ffb6f-b0b2-4a51-8731-4b80ad512907 | None | db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | None | active | True | | fb2b2300-cbe5-40c8-a8d1-1626b23a23e2 | None | 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | None | active | True | | 6ded664f-da01-45a1-9adb-990af6dd8ba4 | None | 040f55f4-28a4-448f-b2ff-864d26812bb1 | None | active | True | | 0c7394e6-b7af-4ceb-a2a3-712571b19a8b | None | None | None | available | True | | 3086b196-eaca-46c2-94db-764e1c750cb4 | None | bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | None | active | True | | 4883759d-fa07-4d19-bf01-e6b08e6e0cba | None | 1ff336ae-8d73-4893-b970-d2dce66176b0 | None | active | True | | f8fc2900-2e3b-4f7c-ab90-9c910f6b22a5 | None | c87353f1-d60e-4cee-9e0a-706902030992 | None | active | True | | c9fddb2f-65d8-4433-924b-72dac00dde57 | None | None | None | available | True | | ea319814-f4e9-4586-afd5-b37550904d4f | None | d6f057ba-9509-453a-b08f-05d58dbe7a4c | None | active | True | | 8d52d5c5-2515-4404-bfb3-c2f77edaf801 | None | 6bc8560b-4b43-4587-8727-8af4b1ede22a | None | active | True | | 56aecd60-ca8c-446d-b523-5481adbd0c0f | None | 458c6e45-c0dc-44ed-8be8-f082034e2427 | None | active | True | | 82a9724b-46ce-4ef4-86c4-689c568fd855 | None | None | None | available | True | | d72e23aa-380a-4197-91c6-9b83196da903 | None | a1f1327a-fbf1-4c5c-9507-6801aa24be87 | None | active | True | | edcbb1c9-6be3-4887-8319-38932685a424 | None | c9fd0800-864f-40ea-b3a2-4e47590405e6 | None | active | True | | 41bb9457-cd03-4163-b8b3-76998f05a210 | None | None | None | available | True | | be65163e-0422-4df8-8c26-22b07b9cd3fd | None | f0c24cbc-d018-42d4-a18e-fc320f8fdaba | None | active | True | | 73f4698a-deba-4ba5-a0ea-23079e75b9c4 | None | 9d504f22-0d70-40bc-89d7-ca4fa0b56c1f | None | active | True | | b1f4805f-5d5a-4ee2-b3f5-f6ba48c44113 | None | dd090ff9-4c75-4872-abe9-3f40c529cb67 | None | active | True | | 771b52d5-9582-4de4-99af-0b23bc8f8135 | None | 79f875c6-f155-4343-9122-62b31b3f11d9 | None | active | True | | d58ad309-1af5-4c3b-8acb-75940457baaf | None | 94b059ff-a1b1-4c74-b387-de8d6d343e1f | None | active | True | | e9da7e88-c18b-4a9a-a9ad-5fea15d3a1af | None | 6dd2c32a-3dfe-49b3-a17a-faebb16b208c | None | active | True | | 2f0247c4-ee0c-414f-8bef-039e16b5c2d0 | None | None | None | available | True | | ab00ff13-ab47-4379-88b1-f2862f25bf9f | None | None | None | available | True | | 57926d21-671a-4308-a165-edbcc0e2a5e9 | None | None | None | available | True | | a1f7c05a-29e9-49ed-8c69-86cfd2d2109b | None | None | None | available | True | | b88c0c7e-e3de-4931-8ce0-3805fb303fac | None | None | None | available | True | | 6b13fd48-d41a-485e-a1eb-2e512db4ffbc | None | None | None | available | True | | 7b990e34-c0e9-4109-9b8b-427df31eec6a | None | None | None | available | True | | 52d47561-11c7-457e-bad5-c922fc1728b8 | None | None | None | available | True | | 5524144c-82f7-4188-ba1f-b97cee3512dd | None | None | None | available | True | | 3af4d971-003f-48dd-bd8b-415ec885a1d8 | None | None | None | available | True | | 76b44a5d-ab90-42d9-a40d-6260b0b819f5 | None | None | None | available | True | | 5253aeb9-fdcb-4ff6-9c6e-3c04a3d3164f | None | 4e9494e2-6ff6-4500-8bfb-51fa74130c74 | None | error | True | | b3cc8bc1-7fdb-4bef-aef4-517b4bdbca23 | None | None | None | available | True | | 89ffc4ae-4c9c-42d7-bd0f-924ff57e357d | None | None | None | available | True | | ba4b4e5a-e6fa-4f79-b046-bae8af17d7d3 | None | None | None | available | True | | fabc9c5a-a1e4-4cab-8c8f-faca64e24947 | None | None | None | available | True | | 1df44c3c-0b55-478e-a520-a464b1e3076a | None | None | None | available | True | | c395a170-1d2e-43d1-8a8d-e35650658336 | None | None | None | available | True | | 20c7c779-e731-4d8b-a8ab-0f7523d6e77a | None | None | None | available | True | | 21584880-24e6-4d10-aced-2a7d9d4f4b2d | None | None | None | available | True | | 6eae36f0-147f-43b6-8322-55c2d2bbac69 | None | None | None | available | True | | b95fcce9-4362-4428-890c-739beb60d131 | None | None | None | available | True | | e69dcbe3-40ac-4654-bc2d-c2b5d1a61b2a | None | None | None | available | True | | 3f91fad5-c5ca-4102-a785-ef0d7488b40d | None | None | None | available | True | | 06355c42-f6b0-4a1f-850b-712b8032aab9 | None | None | None | available | True | | 514dfd11-bdd0-48c8-a7d5-504105c4162f | None | None | None | available | True | | b9870479-327f-440b-9edd-aae3fbd7ce36 | None | None | None | available | True | | 7503ab04-166f-4ee9-81fe-22f3ce240722 | None | None | None | available | True | | 4b142996-a0f8-435e-b1ac-2d3e655372e3 | None | None | None | available | True | +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ [stack@dell-undercloud ~]$ ironic node-delete 4b142996-a0f8-435e-b1ac-2d3e655372e3 No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400) [stack@dell-undercloud ~]$ ironic node-set-power-state 4b142996-a0f8-435e-b1ac-2d3e655372e3 off No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 404) [stack@dell-undercloud ~]$ heat stack-list +--------------------------------------+------------+---------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+---------------+----------------------+ | 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud | DELETE_FAILED | 2015-09-10T02:27:07Z | +--------------------------------------+------------+---------------+----------------------+ Version-Release number of selected component (if applicable): 7.0 How reproducible: not sure if its reproducible, this is seen on 3 controller, 30 compute nodes deployment Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Is there a way you can try to reproduce it? It'll be fairly difficult for us to fix if we can't reproduce it.
Chris, We will try to reproduce and bump up the severity if it is seen again. Do you need anything other than sosreport when we have reproduction?
You don't need to bump up the severity for us to see the bug. The sos report will be a good start. The steps to reproduce the issue are the most important thing you can provide.
Also, can you provide the ironic-conductor log, if you do reproduce the issue?
use journalctl -u openstack-ironic-conductor to get the log
had the OOM killer killed these services and mariadb? can you check in the dmesg output? try restarting the services that are down.
havent hit into this issue so far. I am redeploying everything from scratch at this moment, and will provide the output if I hit into this issue again.
Closing this since it hasn't been observed again. If it happens again, please reopen.