Bug 1263861

Summary: Not able to introspect all nodes when doing a bulk introspection on 70+ nodes
Product: Red Hat OpenStack Reporter: bigswitch <rhosp-bugs-internal>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED DUPLICATE QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: hbrock, mburns, rhel-osp-director-maint, srevivo
Target Milestone: ---   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-16 22:04:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-16 22:03:39 UTC
Description of problem:
import 70+ nodes into openstack baremetal, and do a bulk introspection. After about 3 hours, not all nodes are discovered. some nodes are timing out
subsequently doing a node by node introspection of the failed nodes work


[stack@dell-undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| c7fdba2d-8177-4729-bdd7-5b2aedb58044 | None | None          | power off   | available       | False       |
| 8e3eeef0-5bf8-48fa-86ae-89706bcc3cd1 | None | None          | power off   | available       | False       |
| 73a75b9f-f4f1-4dd0-af62-9237cc2779d6 | None | None          | power off   | available       | False       |
| c7526e61-fe94-4857-bb39-33fb604de908 | None | None          | power off   | available       | False       |
| 389f6cab-f712-4a9a-86f3-c3c7e1d14076 | None | None          | power off   | available       | False       |
| 98ae6b68-cb01-4bf2-b8df-8ce1dc0b2de5 | None | None          | power off   | available       | False       |
| 26554c4a-8318-4371-a11d-a18c5f7f9bf7 | None | None          | power off   | available       | False       |
| beec77e1-bbd5-4c13-b5cf-5014779a3f6a | None | None          | power off   | available       | False       |
| f50d37a6-ad5f-418f-8f9c-a2e480f2d442 | None | None          | power off   | available       | False       |
| 9ed44703-8f39-4057-ace1-950415cde3eb | None | None          | power off   | available       | False       |
| 42105127-7526-4cd6-a5dd-ca41a4268e6a | None | None          | power off   | available       | False       |
| ed0b0f12-710c-4c3d-96b1-fcc1a5fc36a8 | None | None          | power off   | available       | False       |
| d259b087-0bab-455f-86ee-036fb1f23e5f | None | None          | power off   | available       | False       |
| 4e866cef-fb3c-4935-8f30-d0993e2a139c | None | None          | power off   | available       | False       |
| 9fc5350f-df33-4b1b-bfbf-735c11f202ab | None | None          | power off   | available       | False       |
| 3859a0eb-4b10-483f-978b-d73f3b8c162a | None | None          | power off   | available       | False       |
| 3d057044-1939-4ef8-ae61-c985bbb919f7 | None | None          | power off   | available       | False       |
| daf9009b-1a64-41ab-9a7f-a302407efdb9 | None | None          | power off   | available       | False       |
| b8abced9-eb33-4313-b6c6-79e3b4f4f1b3 | None | None          | power off   | available       | False       |
| 712ec08c-67d6-4d42-baa2-39d883b226c6 | None | None          | power off   | available       | False       |
| 769786b0-fef1-4859-b784-0ca25969b75a | None | None          | power off   | available       | False       |
| 804d63e8-d99e-40cf-8ea2-872df80ad28a | None | None          | power off   | available       | False       |
| 20a2fe3a-630d-41c4-bba5-e24bb22a5fb0 | None | None          | power off   | available       | False       |
| 790f1f0f-6efe-4c08-b710-03ce18ae912d | None | None          | power off   | available       | False       |
| ef625302-86f5-4af4-ad38-6414d4b268fc | None | None          | power off   | available       | False       |
| 009d0e77-c355-4904-952d-35f068725c20 | None | None          | power off   | available       | False       |
| a7bf099a-208f-4746-a19a-631cccaee46f | None | None          | power off   | available       | False       |
| fe4256fe-398b-46c6-964f-a0e1bd06fbd3 | None | None          | power off   | available       | False       |
| 7ac0d4ec-a4a8-438b-8b2f-5f817a1a70cd | None | None          | power off   | available       | False       |
| 1554cfe9-28c5-405f-9a28-7cbc71c8053f | None | None          | power off   | available       | False       |
| 4795e1b0-0708-4be2-99de-412bd5401065 | None | None          | power off   | available       | False       |
| 429aacc5-3476-40ff-86fd-9c7858c1b1d0 | None | None          | power off   | available       | False       |
| 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa | None | None          | power off   | available       | False       |
| 76e0f972-2a8b-4a74-916b-edcdca9b28d7 | None | None          | power off   | available       | False       |
| 98140410-0311-407a-88c9-ce6d3bb9c32f | None | None          | power off   | available       | False       |
| 8885598e-b9b8-4a16-9a19-a1dee15ddd44 | None | None          | power off   | available       | False       |
| 0e26c0eb-92fa-4ab8-a13f-449653a6e19b | None | None          | power off   | available       | False       |
| 1f0fd18e-2de1-4d12-8fd3-117c6b1968d1 | None | None          | power off   | available       | False       |
| 560c181d-8da6-4997-9f5a-d899a6860cf5 | None | None          | power off   | available       | False       |
| 372f7b66-9cf8-44c4-9136-b6554ed60e29 | None | None          | power off   | available       | False       |
| 217323a6-614a-47db-90d0-7f00069d7038 | None | None          | power off   | available       | False       |
| 71c8b4c6-7843-414a-8d3d-3f94f0d45089 | None | None          | power off   | available       | False       |
| a736c564-a8bc-451f-996e-dd709551f45a | None | None          | power off   | available       | False       |
| a0ef8d26-d61e-44cc-a058-764192663500 | None | None          | power off   | available       | False       |
| 8dadaec4-2bc6-4f88-b205-5cdb016d0365 | None | None          | power off   | available       | False       |
| 85fa62b6-10cd-4249-aaa3-cabfa3e85751 | None | None          | power off   | available       | False       |
| eb9a4e05-6fc9-434b-9f33-6a31208dd888 | None | None          | power off   | available       | False       |
| 4b819e73-4b3c-4746-a360-d4e2d6cb2c97 | None | None          | power off   | available       | False       |
| 82e7ecfa-7a73-40e9-8ae3-55ba3b467ee8 | None | None          | power off   | available       | False       |
| 391e5097-8cd7-40ac-8f2d-9f80adaf76ec | None | None          | power off   | available       | False       |
| 29773e1b-05d6-4f2b-98e1-981f84b217ec | None | None          | power off   | available       | False       |
| 40b034bc-0c99-4881-be8f-4e4ff34bce69 | None | None          | power off   | available       | False       |
| 9658eb40-69be-4d79-b10d-15eee12bfd7a | None | None          | power off   | available       | False       |
| 30b7480a-ff06-4510-8282-6d7c37073a49 | None | None          | power off   | available       | False       |
| 216632a7-5012-4eb8-8117-2a4ac5245673 | None | None          | power off   | available       | False       |
| 523d8a18-5f10-4c3e-82ff-c4ada7e06dba | None | None          | power off   | available       | False       |
| aa8bc63c-0a3d-4924-9cc6-a81813126e07 | None | None          | power off   | available       | False       |
| c274afdb-efc7-423b-abbd-2bb836ff1f1c | None | None          | power off   | available       | False       |
| 9452b455-e67e-4fc8-b81b-0bc1ee7c97e8 | None | None          | power off   | available       | False       |
| 503ae852-1dff-4bec-8ae6-0f9bcbd0d322 | None | None          | power off   | available       | False       |
| 9d4be85a-40eb-41c1-a008-5922e39e865d | None | None          | power off   | available       | False       |
| be7506a3-3211-4606-b1dd-20d822dfb4c0 | None | None          | power off   | available       | False       |
| 011f6b22-059c-42c5-9e84-15b3c074d455 | None | None          | power off   | available       | False       |
| cfb3ea0d-2f49-4bd0-aa7b-ab72a46438c0 | None | None          | power off   | available       | False       |
| ff28c010-2d5b-4cc7-b111-270908a18e1f | None | None          | power off   | available       | False       |
| 1da69901-7a14-4393-a61c-be02347039ec | None | None          | power off   | available       | False       |
| 90b14609-c8ef-45bb-b173-0af540e3183f | None | None          | power off   | available       | False       |
| 18c6a449-68f3-4b2d-93f5-cdfaa12ceeeb | None | None          | power off   | available       | False       |
| ebbd548f-091c-410a-9e8f-7e783e947734 | None | None          | power off   | available       | False       |
| 64dfd30c-50ec-41f1-9aa8-498ac3b8026a | None | None          | power off   | available       | False       |
| 3c2abe43-42c4-4689-b0df-3d02d116bbe1 | None | None          | power off   | available       | False       |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+


[stack@dell-undercloud ~]$ openstack baremetal introspection bulk start
Setting available nodes to manageable...
Starting introspection of node: c7fdba2d-8177-4729-bdd7-5b2aedb58044
Starting introspection of node: 8e3eeef0-5bf8-48fa-86ae-89706bcc3cd1
Starting introspection of node: 73a75b9f-f4f1-4dd0-af62-9237cc2779d6
Starting introspection of node: c7526e61-fe94-4857-bb39-33fb604de908
Starting introspection of node: 389f6cab-f712-4a9a-86f3-c3c7e1d14076
Starting introspection of node: 98ae6b68-cb01-4bf2-b8df-8ce1dc0b2de5
Starting introspection of node: 26554c4a-8318-4371-a11d-a18c5f7f9bf7
Starting introspection of node: beec77e1-bbd5-4c13-b5cf-5014779a3f6a
Starting introspection of node: f50d37a6-ad5f-418f-8f9c-a2e480f2d442
Starting introspection of node: 9ed44703-8f39-4057-ace1-950415cde3eb
Starting introspection of node: 42105127-7526-4cd6-a5dd-ca41a4268e6a
Starting introspection of node: ed0b0f12-710c-4c3d-96b1-fcc1a5fc36a8
Starting introspection of node: d259b087-0bab-455f-86ee-036fb1f23e5f
Starting introspection of node: 4e866cef-fb3c-4935-8f30-d0993e2a139c
Starting introspection of node: 9fc5350f-df33-4b1b-bfbf-735c11f202ab
Starting introspection of node: 3859a0eb-4b10-483f-978b-d73f3b8c162a
Starting introspection of node: 3d057044-1939-4ef8-ae61-c985bbb919f7
Starting introspection of node: daf9009b-1a64-41ab-9a7f-a302407efdb9
Starting introspection of node: b8abced9-eb33-4313-b6c6-79e3b4f4f1b3
Starting introspection of node: 712ec08c-67d6-4d42-baa2-39d883b226c6
Starting introspection of node: 769786b0-fef1-4859-b784-0ca25969b75a
Starting introspection of node: 804d63e8-d99e-40cf-8ea2-872df80ad28a
Starting introspection of node: 20a2fe3a-630d-41c4-bba5-e24bb22a5fb0
Starting introspection of node: 790f1f0f-6efe-4c08-b710-03ce18ae912d
Starting introspection of node: ef625302-86f5-4af4-ad38-6414d4b268fc
Starting introspection of node: 009d0e77-c355-4904-952d-35f068725c20
Starting introspection of node: a7bf099a-208f-4746-a19a-631cccaee46f
Starting introspection of node: fe4256fe-398b-46c6-964f-a0e1bd06fbd3
Starting introspection of node: 7ac0d4ec-a4a8-438b-8b2f-5f817a1a70cd
Starting introspection of node: 1554cfe9-28c5-405f-9a28-7cbc71c8053f
Starting introspection of node: 4795e1b0-0708-4be2-99de-412bd5401065
Starting introspection of node: 429aacc5-3476-40ff-86fd-9c7858c1b1d0
Starting introspection of node: 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa
Starting introspection of node: 76e0f972-2a8b-4a74-916b-edcdca9b28d7
Starting introspection of node: 98140410-0311-407a-88c9-ce6d3bb9c32f
Starting introspection of node: 8885598e-b9b8-4a16-9a19-a1dee15ddd44
Starting introspection of node: 0e26c0eb-92fa-4ab8-a13f-449653a6e19b
Starting introspection of node: 1f0fd18e-2de1-4d12-8fd3-117c6b1968d1
Starting introspection of node: 560c181d-8da6-4997-9f5a-d899a6860cf5
Starting introspection of node: 372f7b66-9cf8-44c4-9136-b6554ed60e29
Starting introspection of node: 217323a6-614a-47db-90d0-7f00069d7038
Starting introspection of node: 71c8b4c6-7843-414a-8d3d-3f94f0d45089
Starting introspection of node: a736c564-a8bc-451f-996e-dd709551f45a
Starting introspection of node: a0ef8d26-d61e-44cc-a058-764192663500
Starting introspection of node: 8dadaec4-2bc6-4f88-b205-5cdb016d0365
Starting introspection of node: 85fa62b6-10cd-4249-aaa3-cabfa3e85751
Starting introspection of node: eb9a4e05-6fc9-434b-9f33-6a31208dd888
Starting introspection of node: 4b819e73-4b3c-4746-a360-d4e2d6cb2c97
Starting introspection of node: 82e7ecfa-7a73-40e9-8ae3-55ba3b467ee8
Starting introspection of node: 391e5097-8cd7-40ac-8f2d-9f80adaf76ec
Starting introspection of node: 29773e1b-05d6-4f2b-98e1-981f84b217ec
Starting introspection of node: 40b034bc-0c99-4881-be8f-4e4ff34bce69
Starting introspection of node: 9658eb40-69be-4d79-b10d-15eee12bfd7a
Starting introspection of node: 30b7480a-ff06-4510-8282-6d7c37073a49
Starting introspection of node: 216632a7-5012-4eb8-8117-2a4ac5245673
Starting introspection of node: 523d8a18-5f10-4c3e-82ff-c4ada7e06dba
Starting introspection of node: aa8bc63c-0a3d-4924-9cc6-a81813126e07
Starting introspection of node: c274afdb-efc7-423b-abbd-2bb836ff1f1c
Starting introspection of node: 9452b455-e67e-4fc8-b81b-0bc1ee7c97e8
Starting introspection of node: 503ae852-1dff-4bec-8ae6-0f9bcbd0d322
Starting introspection of node: 9d4be85a-40eb-41c1-a008-5922e39e865d
Starting introspection of node: be7506a3-3211-4606-b1dd-20d822dfb4c0
Starting introspection of node: 011f6b22-059c-42c5-9e84-15b3c074d455
Starting introspection of node: cfb3ea0d-2f49-4bd0-aa7b-ab72a46438c0
Starting introspection of node: ff28c010-2d5b-4cc7-b111-270908a18e1f
Starting introspection of node: 1da69901-7a14-4393-a61c-be02347039ec
Starting introspection of node: 90b14609-c8ef-45bb-b173-0af540e3183f
Starting introspection of node: 18c6a449-68f3-4b2d-93f5-cdfaa12ceeeb
Starting introspection of node: ebbd548f-091c-410a-9e8f-7e783e947734
Starting introspection of node: 64dfd30c-50ec-41f1-9aa8-498ac3b8026a
Starting introspection of node: 3c2abe43-42c4-4689-b0df-3d02d116bbe1
Waiting for discovery to finish...
Discovery for UUID c7fdba2d-8177-4729-bdd7-5b2aedb58044 finished successfully.
Discovery for UUID 73a75b9f-f4f1-4dd0-af62-9237cc2779d6 finished successfully.
Discovery for UUID 389f6cab-f712-4a9a-86f3-c3c7e1d14076 finished successfully.
Discovery for UUID 26554c4a-8318-4371-a11d-a18c5f7f9bf7 finished successfully.
Discovery for UUID f50d37a6-ad5f-418f-8f9c-a2e480f2d442 finished successfully.
Discovery for UUID 42105127-7526-4cd6-a5dd-ca41a4268e6a finished successfully.
Discovery for UUID d259b087-0bab-455f-86ee-036fb1f23e5f finished successfully.
Discovery for UUID 9fc5350f-df33-4b1b-bfbf-735c11f202ab finished successfully.
Discovery for UUID 3d057044-1939-4ef8-ae61-c985bbb919f7 finished successfully.
Discovery for UUID b8abced9-eb33-4313-b6c6-79e3b4f4f1b3 finished successfully.
Discovery for UUID 769786b0-fef1-4859-b784-0ca25969b75a finished successfully.
Discovery for UUID 20a2fe3a-630d-41c4-bba5-e24bb22a5fb0 finished successfully.
Discovery for UUID ef625302-86f5-4af4-ad38-6414d4b268fc finished successfully.
Discovery for UUID a7bf099a-208f-4746-a19a-631cccaee46f finished successfully.
Discovery for UUID 7ac0d4ec-a4a8-438b-8b2f-5f817a1a70cd finished successfully.
Discovery for UUID 4795e1b0-0708-4be2-99de-412bd5401065 finished with error: Introspection timeout
Discovery for UUID 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa finished with error: Introspection timeout
Discovery for UUID 98140410-0311-407a-88c9-ce6d3bb9c32f finished with error: Introspection timeout
Discovery for UUID 0e26c0eb-92fa-4ab8-a13f-449653a6e19b finished with error: Introspection timeout
Discovery for UUID 560c181d-8da6-4997-9f5a-d899a6860cf5 finished with error: Introspection timeout
Discovery for UUID 217323a6-614a-47db-90d0-7f00069d7038 finished with error: Introspection timeout
Discovery for UUID a736c564-a8bc-451f-996e-dd709551f45a finished successfully.
Discovery for UUID 8dadaec4-2bc6-4f88-b205-5cdb016d0365 finished successfully.
Discovery for UUID eb9a4e05-6fc9-434b-9f33-6a31208dd888 finished successfully.
Discovery for UUID 82e7ecfa-7a73-40e9-8ae3-55ba3b467ee8 finished successfully.
Discovery for UUID 29773e1b-05d6-4f2b-98e1-981f84b217ec finished successfully.
Discovery for UUID 9658eb40-69be-4d79-b10d-15eee12bfd7a finished successfully.
Discovery for UUID 216632a7-5012-4eb8-8117-2a4ac5245673 finished successfully.
Discovery for UUID aa8bc63c-0a3d-4924-9cc6-a81813126e07 finished successfully.
Discovery for UUID 9452b455-e67e-4fc8-b81b-0bc1ee7c97e8 finished successfully.
Discovery for UUID 9d4be85a-40eb-41c1-a008-5922e39e865d finished successfully.
Discovery for UUID 011f6b22-059c-42c5-9e84-15b3c074d455 finished successfully.
Discovery for UUID ff28c010-2d5b-4cc7-b111-270908a18e1f finished successfully.
Discovery for UUID 8e3eeef0-5bf8-48fa-86ae-89706bcc3cd1 finished successfully.
Discovery for UUID 98ae6b68-cb01-4bf2-b8df-8ce1dc0b2de5 finished successfully.
Discovery for UUID 9ed44703-8f39-4057-ace1-950415cde3eb finished successfully.
Discovery for UUID 4e866cef-fb3c-4935-8f30-d0993e2a139c finished successfully.
Discovery for UUID daf9009b-1a64-41ab-9a7f-a302407efdb9 finished successfully.
Discovery for UUID 804d63e8-d99e-40cf-8ea2-872df80ad28a finished successfully.
Discovery for UUID 009d0e77-c355-4904-952d-35f068725c20 finished successfully.
Discovery for UUID 1554cfe9-28c5-405f-9a28-7cbc71c8053f finished with error: Introspection timeout
Discovery for UUID 76e0f972-2a8b-4a74-916b-edcdca9b28d7 finished with error: Introspection timeout
Discovery for UUID 1f0fd18e-2de1-4d12-8fd3-117c6b1968d1 finished with error: Introspection timeout
Discovery for UUID 71c8b4c6-7843-414a-8d3d-3f94f0d45089 finished with error: Introspection timeout
Discovery for UUID 85fa62b6-10cd-4249-aaa3-cabfa3e85751 finished successfully.
Discovery for UUID 391e5097-8cd7-40ac-8f2d-9f80adaf76ec finished successfully.
Discovery for UUID 30b7480a-ff06-4510-8282-6d7c37073a49 finished successfully.
Discovery for UUID c274afdb-efc7-423b-abbd-2bb836ff1f1c finished successfully.
Discovery for UUID be7506a3-3211-4606-b1dd-20d822dfb4c0 finished successfully.
Discovery for UUID c7526e61-fe94-4857-bb39-33fb604de908 finished successfully.
Discovery for UUID ed0b0f12-710c-4c3d-96b1-fcc1a5fc36a8 finished successfully.
Discovery for UUID 712ec08c-67d6-4d42-baa2-39d883b226c6 finished successfully.
Discovery for UUID fe4256fe-398b-46c6-964f-a0e1bd06fbd3 finished successfully.
Discovery for UUID 8885598e-b9b8-4a16-9a19-a1dee15ddd44 finished with error: Introspection timeout
Discovery for UUID a0ef8d26-d61e-44cc-a058-764192663500 finished successfully.
Discovery for UUID 40b034bc-0c99-4881-be8f-4e4ff34bce69 finished successfully.
Discovery for UUID 503ae852-1dff-4bec-8ae6-0f9bcbd0d322 finished successfully.
Discovery for UUID beec77e1-bbd5-4c13-b5cf-5014779a3f6a finished successfully.
Discovery for UUID 790f1f0f-6efe-4c08-b710-03ce18ae912d finished successfully.
Discovery for UUID 372f7b66-9cf8-44c4-9136-b6554ed60e29 finished with error: Introspection timeout
Discovery for UUID 523d8a18-5f10-4c3e-82ff-c4ada7e06dba finished successfully.
Discovery for UUID 3859a0eb-4b10-483f-978b-d73f3b8c162a finished successfully.
Discovery for UUID 4b819e73-4b3c-4746-a360-d4e2d6cb2c97 finished successfully.
Discovery for UUID 429aacc5-3476-40ff-86fd-9c7858c1b1d0 finished with error: Introspection timeout
Discovery for UUID cfb3ea0d-2f49-4bd0-aa7b-ab72a46438c0 finished successfully.
Discovery for UUID 1da69901-7a14-4393-a61c-be02347039ec finished successfully.
Discovery for UUID 18c6a449-68f3-4b2d-93f5-cdfaa12ceeeb finished successfully.
Discovery for UUID 64dfd30c-50ec-41f1-9aa8-498ac3b8026a finished successfully.
Discovery for UUID 90b14609-c8ef-45bb-b173-0af540e3183f finished successfully.
Discovery for UUID 3c2abe43-42c4-4689-b0df-3d02d116bbe1 finished successfully.
Discovery for UUID ebbd548f-091c-410a-9e8f-7e783e947734 finished successfully.
Setting manageable nodes to available...
Node c7fdba2d-8177-4729-bdd7-5b2aedb58044 has been set to available.
Node 8e3eeef0-5bf8-48fa-86ae-89706bcc3cd1 has been set to available.
Node 73a75b9f-f4f1-4dd0-af62-9237cc2779d6 has been set to available.
Node c7526e61-fe94-4857-bb39-33fb604de908 has been set to available.
Node 389f6cab-f712-4a9a-86f3-c3c7e1d14076 has been set to available.
Node 98ae6b68-cb01-4bf2-b8df-8ce1dc0b2de5 has been set to available.
Node 26554c4a-8318-4371-a11d-a18c5f7f9bf7 has been set to available.
Node beec77e1-bbd5-4c13-b5cf-5014779a3f6a has been set to available.
Node f50d37a6-ad5f-418f-8f9c-a2e480f2d442 has been set to available.
Node 9ed44703-8f39-4057-ace1-950415cde3eb has been set to available.
Node 42105127-7526-4cd6-a5dd-ca41a4268e6a has been set to available.
Node ed0b0f12-710c-4c3d-96b1-fcc1a5fc36a8 has been set to available.
Node d259b087-0bab-455f-86ee-036fb1f23e5f has been set to available.
Node 4e866cef-fb3c-4935-8f30-d0993e2a139c has been set to available.
Node 9fc5350f-df33-4b1b-bfbf-735c11f202ab has been set to available.
Node 3859a0eb-4b10-483f-978b-d73f3b8c162a has been set to available.
Node 3d057044-1939-4ef8-ae61-c985bbb919f7 has been set to available.
Node daf9009b-1a64-41ab-9a7f-a302407efdb9 has been set to available.
Node b8abced9-eb33-4313-b6c6-79e3b4f4f1b3 has been set to available.
Node 712ec08c-67d6-4d42-baa2-39d883b226c6 has been set to available.
Node 769786b0-fef1-4859-b784-0ca25969b75a has been set to available.
Node 804d63e8-d99e-40cf-8ea2-872df80ad28a has been set to available.
Node 20a2fe3a-630d-41c4-bba5-e24bb22a5fb0 has been set to available.
Node 790f1f0f-6efe-4c08-b710-03ce18ae912d has been set to available.
Node ef625302-86f5-4af4-ad38-6414d4b268fc has been set to available.
Node 009d0e77-c355-4904-952d-35f068725c20 has been set to available.
Node a7bf099a-208f-4746-a19a-631cccaee46f has been set to available.
Node fe4256fe-398b-46c6-964f-a0e1bd06fbd3 has been set to available.
Node 7ac0d4ec-a4a8-438b-8b2f-5f817a1a70cd has been set to available.
Node 1554cfe9-28c5-405f-9a28-7cbc71c8053f has been set to available.
Node 4795e1b0-0708-4be2-99de-412bd5401065 has been set to available.
Node 429aacc5-3476-40ff-86fd-9c7858c1b1d0 has been set to available.
Node 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa has been set to available.
Node 76e0f972-2a8b-4a74-916b-edcdca9b28d7 has been set to available.
Node 98140410-0311-407a-88c9-ce6d3bb9c32f has been set to available.
Node 8885598e-b9b8-4a16-9a19-a1dee15ddd44 has been set to available.
Node 0e26c0eb-92fa-4ab8-a13f-449653a6e19b has been set to available.
Node 1f0fd18e-2de1-4d12-8fd3-117c6b1968d1 has been set to available.
Node 560c181d-8da6-4997-9f5a-d899a6860cf5 has been set to available.
Node 372f7b66-9cf8-44c4-9136-b6554ed60e29 has been set to available.
Node 217323a6-614a-47db-90d0-7f00069d7038 has been set to available.
Node 71c8b4c6-7843-414a-8d3d-3f94f0d45089 has been set to available.
Node a736c564-a8bc-451f-996e-dd709551f45a has been set to available.
Node a0ef8d26-d61e-44cc-a058-764192663500 has been set to available.
Node 8dadaec4-2bc6-4f88-b205-5cdb016d0365 has been set to available.
Node 85fa62b6-10cd-4249-aaa3-cabfa3e85751 has been set to available.
Node eb9a4e05-6fc9-434b-9f33-6a31208dd888 has been set to available.
Node 4b819e73-4b3c-4746-a360-d4e2d6cb2c97 has been set to available.
Node 82e7ecfa-7a73-40e9-8ae3-55ba3b467ee8 has been set to available.
Node 391e5097-8cd7-40ac-8f2d-9f80adaf76ec has been set to available.
Node 29773e1b-05d6-4f2b-98e1-981f84b217ec has been set to available.
Node 40b034bc-0c99-4881-be8f-4e4ff34bce69 has been set to available.
Node 9658eb40-69be-4d79-b10d-15eee12bfd7a has been set to available.
Node 30b7480a-ff06-4510-8282-6d7c37073a49 has been set to available.
Node 216632a7-5012-4eb8-8117-2a4ac5245673 has been set to available.
Node 523d8a18-5f10-4c3e-82ff-c4ada7e06dba has been set to available.
Node aa8bc63c-0a3d-4924-9cc6-a81813126e07 has been set to available.
Node c274afdb-efc7-423b-abbd-2bb836ff1f1c has been set to available.
Node 9452b455-e67e-4fc8-b81b-0bc1ee7c97e8 has been set to available.
Node 503ae852-1dff-4bec-8ae6-0f9bcbd0d322 has been set to available.
Node 9d4be85a-40eb-41c1-a008-5922e39e865d has been set to available.
Node be7506a3-3211-4606-b1dd-20d822dfb4c0 has been set to available.
Node 011f6b22-059c-42c5-9e84-15b3c074d455 has been set to available.
Node cfb3ea0d-2f49-4bd0-aa7b-ab72a46438c0 has been set to available.
Node ff28c010-2d5b-4cc7-b111-270908a18e1f has been set to available.
Node 1da69901-7a14-4393-a61c-be02347039ec has been set to available.
Node 90b14609-c8ef-45bb-b173-0af540e3183f has been set to available.
Node 18c6a449-68f3-4b2d-93f5-cdfaa12ceeeb has been set to available.
Node ebbd548f-091c-410a-9e8f-7e783e947734 has been set to available.
Node 64dfd30c-50ec-41f1-9aa8-498ac3b8026a has been set to available.
Node 3c2abe43-42c4-4689-b0df-3d02d116bbe1 has been set to available.
Discovery completed.



[stack@dell-undercloud ~]$ ironic node-show 1554cfe9-28c5-405f-9a28-7cbc71c8053f
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'on_discovery': u'true'}                                            |
| last_error             | None                                                                  |
| updated_at             | 2015-09-16T21:45:27+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | 1554cfe9-28c5-405f-9a28-7cbc71c8053f                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | True                                                                  |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power on                                                              |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'16384', u'cpu_arch': u'x86_64', u'local_gb': u'500', |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.17.64.109',       |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'bf8c3775-bb1d-466e-    |
|                        | bee1-057472379440', u'deploy_ramdisk': u'8ea8b4b4-004a-               |
|                        | 491d-a900-f62b82080fdf'}                                              |
| created_at             | 2015-09-16T19:31:09+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+
[stack@dell-undercloud ~]$ ironic node-show 1554cfe9-28c5-405f-9a28-7cbc71c8053f
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':         |
|                        | [u'35H5XVPLS']}, u'hardware_swift_object': u'extra_hardware-          |
|                        | 1554cfe9-28c5-405f-9a28-7cbc71c8053f'}                                |
| last_error             | None                                                                  |
| updated_at             | 2015-09-16T21:50:49+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | 1554cfe9-28c5-405f-9a28-7cbc71c8053f                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | True                                                                  |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power off                                                             |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'16384', u'cpu_arch': u'x86_64', u'local_gb': u'464', |
|                        | u'cpus': u'8', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.17.64.109',       |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'bf8c3775-bb1d-466e-    |
|                        | bee1-057472379440', u'deploy_ramdisk': u'8ea8b4b4-004a-               |
|                        | 491d-a900-f62b82080fdf'}                                              |
| created_at             | 2015-09-16T19:31:09+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+
[stack@dell-undercloud ~]$
[stack@dell-undercloud ~]$
[stack@dell-undercloud ~]$
[stack@dell-undercloud ~]$ ironic node-show 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'on_discovery': u'true'}                                            |
| last_error             | None                                                                  |
| updated_at             | 2015-09-16T21:47:24+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | True                                                                  |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power on                                                              |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'16384', u'cpu_arch': u'x86_64', u'local_gb': u'500', |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.17.64.100',       |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'bf8c3775-bb1d-466e-    |
|                        | bee1-057472379440', u'deploy_ramdisk': u'8ea8b4b4-004a-               |
|                        | 491d-a900-f62b82080fdf'}                                              |
| created_at             | 2015-09-16T19:31:10+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+
[stack@dell-undercloud ~]$ ironic node-show 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':         |
|                        | [u'35H1625LS']}, u'hardware_swift_object': u'extra_hardware-5d32d61f- |
|                        | 0e62-4f38-b11a-5fdd7e9265fa'}                                         |
| last_error             | None                                                                  |
| updated_at             | 2015-09-16T21:57:09+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | 5d32d61f-0e62-4f38-b11a-5fdd7e9265fa                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | True                                                                  |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power off                                                             |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'16384', u'cpu_arch': u'x86_64', u'local_gb': u'464', |
|                        | u'cpus': u'8', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.17.64.100',       |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'bf8c3775-bb1d-466e-    |
|                        | bee1-057472379440', u'deploy_ramdisk': u'8ea8b4b4-004a-               |
|                        | 491d-a900-f62b82080fdf'}                                              |
| created_at             | 2015-09-16T19:31:10+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+


Version-Release number of selected component (if applicable):
7.0

How reproducible:


Steps to Reproduce:
1. import 72 nodes into openstack baremetal
2. openstack baremetal configure boot
3. openstack baremetal introspection bulk start

after a few hours, some nodes will fail introspection with timeout message

Actual results:


Expected results:
should be able to inspect all nodes

Additional info:

Comment 3 bigswitch 2015-09-16 22:04:46 UTC

*** This bug has been marked as a duplicate of bug 1263860 ***