Created attachment 1242028 [details] Logs Description of problem: Inspection fails with RHOSP9 for all node with the error message "Introspection didn't finish for nodes ... " However, it gets pass with RHOSP10 I have attached, all required logs and screenshot. Please find it. Environment: - Undercloud(VM on ESXi6.5) : RHEL7.2 - Bare metal node using Overcloud: DELL R230 (BIOS v1.4.5, Firmware v2.41.40.40, Lifecycle Controller Firmware v2.41.40.40) * 6EA Note: As per below reference, DELL R230 server is certified with both OSP9 & OSP10 >> https://access.redhat.com/ecosystem/hardware/2064063#note3 Detailed Analysis: After analyzing the logs, i gets below findings, Overcloud node gets ip: X.X.X.10/24, and a set of mac addresses[u'X:X:X:a4:a7:d7', u'X:X:X:a4:a5:eb', u'X:X:X:a4:a8:af', u'X:X:X:a4:a8:e7', u'X:X:X:a4:a4:97'] available for the overcloud node. Overcloud node gets bmc_address': [u'X.X.X.11'], u'mac' for bmc: [u'X:X:X:a4:a7:bf'] ***Inspection of overcloud started at >> Jan 11 20:49:23 director.naimnetworks.com ironic-inspector[9080]: 2017-01-11 20:49:23.891 9080 INFO ironic_inspector.introspect [-] [node: 684bb2e8-3fb6-4e30-9380-1493bd360afd] Introspection started successfully ***However, just after starting of introspection, ironic-conductor logged a strong warning for adding the certificate for HTTPS connection as shown below >> Jan 11 20:49:24 director.naimnetworks.com ironic-conductor[9401]: /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html ***After analyzing, tcp dump i find all the communication happens between undercloud and overcloud node over HTTP except the request that goes to [https://X.X.X.11:443/wsman] as shown below. >> Jan 11 20:49:24 director.naimnetworks.com ironic-conductor[9401]: 2017-01-11 20:49:24.836 9401 DEBUG dracclient.wsman [req-18a19a38-4a45-4842-b047-f996c6412bbd - - - - -] Sending request to https://X.X.X.11:443/wsman: <s:Envelope xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:wsman="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd"><s:Header><wsa:To s:mustUnderstand="true">https://X.X.X.11:443/wsman</wsa:To><wsman:ResourceURI s:mustUnderstand="true">http://schemas.dell.com/wbem/wscim/1/cim-schema/2/DCIM_LifecycleJob</wsman:ResourceURI><wsa:MessageID s:mustUnderstand="true">uuid:3d576955-1de0-4c6d-92f7-d8d88413d14e</wsa:MessageID><wsa:ReplyTo><wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address></wsa:ReplyTo><wsa:Action s:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/enumeration/Enumerate</wsa:Action></s:Header><s:Body><wsen:Enumerate xmlns:wsen="http://schemas.xmlsoap.org/ws/2004/09/enumeration"><wsman:Filter Dialect="http://schemas.dmtf.org/wbem/cql/1/dsp0202.pdf">select * from DCIM_LifecycleJob where Name != "CLEARALL" and JobStatus != "Reboot Completed" and JobStatus != "Completed" and JobStatus != "Completed with Errors" and JobStatus != "Failed"</wsman:Filter><wsman:OptimizeEnumeration/><wsman:MaxElements>100</wsman:MaxElements></wsen:Enumerate></s:Body></s:Envelope> _do_request /usr/lib/python2.7/site-packages/dracclient/wsman.py:60 ***Ironic driver module and draclient modules throw below errors >>2017-01-11 15:11:16.074 2253 ERROR dracclient.wsman SSLError: EOF occurred in violation of protocol (_ssl.c:579) >> 2017-01-11 15:11:16.083 2253 ERROR ironic.drivers.modules.drac.power [req-014548b4-3451-40a1-b65f-bc0dd0f77db6 admin - - - -] DRAC driver failed to get power state for node 79ad4497-b759-4d06-9340-c5bf59a45145. Reason: WSMan request failed. 2017-01-11 15:11:16.084 2253 WARNING ironic.conductor.manager [req-014548b4-3451-40a1-b65f-bc0dd0f77db6 admin - - - -] During sync_power_state, could not get power state for node 79ad4497-b759-4d06-9340-c5bf59a45145, attempt 1 of 3. Error: DRAC operation failed. Reason: WSMan request failed. >> dracclient.wsman SSLError: EOF occurred in violation of protocol (_ssl.c:579) *** Ironic-inspector logged authentication error as mentioned below >> Jan 11 14:53:29 director.naimnetworks.com ironic-inspector[28360]: 2017-01-11 14:53:29.813 28360 ERROR ironic_inspector.main AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token, error was: The request you have made requires authentication. (HTTP 401) (Request-ID: req-de692231-3474-4595-af47-ae05030d24f3) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Introspection didn't finish for nodes Expected results: Introspection should successfully completed for all nodes. Additional info: The same issue occurs while try to introspect a single server also.
Customer is experiencing similar problem when running introspection with mistral workflow tool in OSP10. Probably he's hitting the same bug in his environment: $ mistral execution-list | grep "ERROR"; | 2a7c2499-03e7-4c04-a1a4-4a23f5d4bb52 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect | sub-workflow execution | b5f64f1d-f79b-4a0a-b6cb-dbdf57c1e55a | ERROR | Failure caused by error i... | 2017-03-21 11:08:08 | 2017-03-21 11:19:32 | | 5da548ed-ba25-4e6b-96e6-8dc0ca7dcacc | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect | sub-workflow execution | 4b513c7c-0566-4ae8-879a-2e2ff2e51f1f | ERROR | Failure caused by error i... | 2017-03-21 13:59:02 | 2017-03-21 14:08:56 | | cc2b7e9e-7286-425d-bedd-aa77ddc54185 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect | sub-workflow execution | b8204bfa-6d27-4700-93c4-d05296a61654 | ERROR | Failure caused by error i... | 2017-03-21 15:17:54 | 2017-03-21 15:27:25 | | 9962345f-97d2-47f5-ab1b-0c857e8ba305 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect | sub-workflow execution | 82726201-9cf1-4ce2-9f6e-2fd39104d2f8 | ERROR | Failure caused by error i... | 2017-03-21 16:10:15 | 2017-03-21 16:19:51 | $ mistral execution-get 9962345f-97d2-47f5-ab1b-0c857e8ba305 +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | 9962345f-97d2-47f5-ab1b-0c857e8ba305 | | Workflow ID | 70db247b-1c8d-414a-ae98-595545183399 | | Workflow name | tripleo.baremetal.v1.introspect | | Description | sub-workflow execution | | Task Execution ID | 82726201-9cf1-4ce2-9f6e-2fd39104d2f8 | | State | ERROR | | State info | Failure caused by error in tasks: wait_for_introspection_to_finish | | | | | | wait_for_introspection_to_finish [task_ex_id=6415c7f2-451a-483e-8b4c-74e4bcd2b277] -> Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class | | | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': | | | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca- | | | 9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}'] | | | BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried. | | | [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, idx=0]: Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class | | | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': | | | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca- | | | 9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}'] | | | BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried. | | | | | Created at | 2017-03-21 16:10:15 | | Updated at | 2017-03-21 16:19:51 | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ $ mistral execution-get-output 9962345f-97d2-47f5-ab1b-0c857e8ba305 { "result": "Failure caused by error in tasks: wait_for_introspection_to_finish\n\n wait_for_introspection_to_finish [task_ex_id=6415c7f2-451a-483e-8b4c-74e4bcd2b277] -> Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, idx=0]: Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n" } $ mistral execution-get cc2b7e9e-7286-425d-bedd-aa77ddc54185 +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ID | cc2b7e9e-7286-425d-bedd-aa77ddc54185 | | Workflow ID | 70db247b-1c8d-414a-ae98-595545183399 | | Workflow name | tripleo.baremetal.v1.introspect | | Description | sub-workflow execution | | Task Execution ID | b8204bfa-6d27-4700-93c4-d05296a61654 | | State | ERROR | | State info | Failure caused by error in tasks: wait_for_introspection_to_finish | | | | | | wait_for_introspection_to_finish [task_ex_id=a43444e1-ff02-4c61-bfec-12daabdc5517] -> Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class | | | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': | | | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c- | | | 6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}'] | | | BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried. | | | [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, idx=0]: Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class | | | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': | | | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c- | | | 6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}'] | | | BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried. | | | | | Created at | 2017-03-21 15:17:54 | | Updated at | 2017-03-21 15:27:25 | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ $ mistral execution-get-output cc2b7e9e-7286-425d-bedd-aa77ddc54185 { "result": "Failure caused by error in tasks: wait_for_introspection_to_finish\n\n wait_for_introspection_to_finish [task_ex_id=a43444e1-ff02-4c61-bfec-12daabdc5517] -> Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, idx=0]: Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n" } The following can be observed in the log: --- 17750:Mar 21 16:17:19 ospd ironic-inspector: 2017-03-21 16:17:19.576 923 DEBUG ironic_inspector.process [-] [node: MAC a0:36:9f:d3:f4:b0 BMC 192.0.2.129] Running pre-processing hook ramdisk_error _run_pre_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:119 17790:Mar 21 16:17:20 ospd ironic-inspector: 2017-03-21 16:17:20.006 923 DEBUG ironic_inspector.process [-] [node: ac893774-9a1a-4d7c-be8e-8971cd9a5c28 MAC a0:36:9f:d3:f4:b0 BMC 192.0.2.129] Running post-process ing hook ramdisk_error _run_post_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:262 17861:Mar 21 16:17:22 ospd ironic-inspector: [{u'rotational': False, u'vendor': u'ATA', u'name': u'/dev/sda', u'wwn_vendor_extension': None, u'wwn_with_extension': u'0x55cd2e414d4d43d0', u'model': u'INTEL SSDSC2 BB12', u'wwn': u'0x55cd2e414d4d43d0', u'serial': u'PHWA640601HW120CGN', u'size': 120034123776}], u'boot': {u'current_boot_mode': u'bios', u'pxe_interface': u'a0:36:9f:d3:f1:f4'}, u'system_vendor': {u'serial_numb er': u'7SRCZG2', u'product_name': u'PowerEdge C6320', u'manufacturer': u'Dell Inc.'}, u'memory': {u'physical_mb': 262144, u'total': 270375514112}, u'cpu': {u'count': 64, u'frequency': u'1202.414', u'flags': [u'f pu', u'vme', u'de', u'pse', u'tsc', u'msr', u'pae', u'mce', u'cx8', u'apic', u'sep', u'mtrr', u'pge', u'mca', u'cmov', u'pat', u'pse36', u'clflush', u'dts', u'acpi', u'mmx', u'fxsr', u'sse', u'sse2', u'ss', u'ht ', u'tm', u'pbe', u'syscall', u'nx', u'pdpe1gb', u'rdtscp', u'lm', u'constant_tsc', u'arch_perfmon', u'pebs', u'bts', u'rep_good', u'nopl', u'xtopology', u'nonstop_tsc', u'aperfmperf', u'eagerfpu', u'pni', u'pcl mulqdq', u'dtes64', u'monitor', u'ds_cpl', u'vmx', u'smx', u'est', u'tm2', u'ssse3', u'fma', u'cx16', u'xtpr', u'pdcm', u'pcid', u'dca', u'sse4_1', u'sse4_2', u'x2apic', u'movbe', u'popcnt', u'tsc_deadline_timer ', u'aes', u'xsave', u'avx', u'f16c', u'rdrand', u'lahf_lm', u'abm', u'3dnowprefetch', u'ida', u'arat', u'epb', u'pln', u'pts', u'dtherm', u'intel_pt', u'tpr_shadow', u'vnmi', u'flexpriority', u'ept', u'vpid', u 'fsgsbase', u'tsc_adjust', u'bmi1', u'hle', u'avx2', u'smep', u'bmi2', u'erms', u'invpcid', u'rtm', u'cqm', u'rdseed', u'adx', u'smap', u'xsaveopt', u'cqm_llc', u'cqm_occup_llc', u'cqm_mbm_total', u'cqm_mbm_loca l'], u'model_name': u'Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz', u'architecture': u'x86_64'}}, u'error': None, u'local_gb': 110, u'logs': '<hidden>'} api_continue /usr/lib/python2.7/site-packages/ironic_inspect or/main.py:180 17862:Mar 21 16:17:22 ospd ironic-inspector: 2017-03-21 16:17:22.090 923 DEBUG ironic_inspector.process [-] [node: MAC a0:36:9f:d3:f1:f4 BMC 192.0.2.132] Running pre-processing hook ramdisk_error _run_pre_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:119 17891:Mar 21 16:17:22 ospd ironic-inspector: 2017-03-21 16:17:22.217 923 DEBUG ironic_inspector.process [-] [node: 4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5 MAC a0:36:9f:d3:f1:f4 BMC 192.0.2.132] Running post-process ing hook ramdisk_error _run_post_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:262 18031:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman [req-a9c58c37-1e59-4395-8879-45942b004894 - - - - -] Request failed 18032:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman Traceback (most recent call last): 18033:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/dracclient/wsman.py", line 67, in _do_request 18034:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman verify=False) 18035:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/requests/api.py", line 111, in post 18036:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman return request('post', url, data=data, json=json, **kwargs) 18037:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/requests/api.py", line 57, in request 18038:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman return session.request(method=method, url=url, **kwargs) 18039:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request 18040:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman resp = self.send(prep, **send_kwargs) 18041:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send 18042:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman r = adapter.send(request, **kwargs) 18043:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 453, in send 18044:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman raise ConnectionError(err, request=request) 18045:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman ConnectionError: ('Connection aborted.', BadStatusLine("''",)) 18046:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman 18047:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.202 9729 ERROR ironic.drivers.modules.drac.power [req-a9c58c37-1e59-4395-8879-45942b004894 - - - - -] DRAC driver failed to get power state for no de ea88388f-8c83-498e-9299-e23dfe7bcd56. Reason: WSMan request failed. ---
Hi! In the later releases we've moved away from openwsman, this is why it works for you with OSP 10. I recommend trying the pxe_ipmitool driver instead in your release. Note that https://github.com/OpenNMS/wsman is a Java library, it's unlikely to be related. Sergii, your problem looks different and at first glance is related to either Keystone or ironic-inspector services being down. Please check their logs.
There are known issues using dracclient in OSP-9, see upstream bug related to get/set power - https://bugs.launchpad.net/python-dracclient/+bug/1667088, and the BZ referencing it https://bugzilla.redhat.com/show_bug.cgi?id=1463224. Its currently recommended to use pxe_ipmitool in OSP-9.
Can someone having these problems please try: 1. updating the firmware of their Drac's, 2. installing python-dracclient from a more recent release (OSP 10, for example), 3. try pxe_ipmitool driver instead of pxe_ilo? OSP 9 has python-dracclient 0.0.5, OSP 10 has 1.3.0 (!). This is a huge difference, we can probably risk rebasing if it solves problems.
In OSP-9 it is recommended to use pxe_ipmitool per comments 7 and 8, or switch to the python-dracclient in OSP-10. As there is has been response to the request for info in 4 months, closing this out.